Automated flight data ingestion and deal scoring engine
Flight deal platforms depend on large volumes of external API data that change frequently and contain a lot of noise. Raw Skyscanner responses include thousands of price combinations, but most of them are not meaningful deals. The challenge was to continuously fetch fresh data, process it efficiently and identify only the most valuable offers based on multiple factors such as price, seasonality, route quality and airline.
We built a background processing pipeline using Celery that fetches flight data from the Skyscanner API at regular intervals. The system analyzes incoming data against internally defined thresholds for destinations and departure regions, and evaluates each connection using a custom scoring algorithm. This algorithm takes into account price levels, travel season, number of transfers, destination attractiveness and airline quality. The result is a continuously updated stream of curated flight deals, filtered and ranked automatically.