AllTheRooms

AllTheRooms is a portal for booking rooms, an extensive search engine for travelers that combines the largest sites (Expedia, Airbnb and Jetsetter) with other popular travel resources. Room prices are compared on all these sites, so you can book a room or a house anywhere in the world at the best possible price. Our team was hired to develop a supporting project for AllTheRooms. It was necessary to create an application that independently collected information from dozens of providers through the API, and used custom scrappers for booking sites that did not have an API (or had access restrictions). The complexity of the task was due to the requirement that each subprogram be created at speed (no more than 2–3 hours per provider), taking into account the fact that for each, it was necessary to select the most optimal set of tools.

What was done

For this service, we implemented:

• the balanced use of a distributed network of proxy servers (to prevent the provider from blocking requests by IP);
• the selection of timing and frequency of requests, as well as the logic of repeated requests using throttling;
• the сalibration of prices depending on tax conditions and various formulas for calculating/displaying commissions by providers that affect the final actual price of a reservation;
• the main method for parsing fields of interest (API, PhantomJS/CasperJS, http request & regexp parser logic, specificity of ajax & pagination & crawling handling);
• the quick display of available booking options at the user's request via the logic of extracting information from the cache with background updating, which makes the architecture able to perform many parallel requests (high load) and also allows users to see the current availability and prices of booking options.