Case: All the Rooms

AllTheRooms allows users to find the perfect room for the next trip with one search and results from more than 400 travel sites. Room prices are compared on all these sites, so you can book a room or a house anywhere in the world at the best possible price. It was necessary to create an application that independently collected information from dozens of providers through the API, and used custom scrappers for booking sites that did not have an API (or had access restrictions). The complexity of the task was due to the requirement that each subprogram be created at speed (no more than 2–3 hours per provider), taking into account the fact that for each, it was necessary to select the most optimal set of tools.

Task on the project

To develop a supporting project for AllTheRooms

For this service, we implemented

the balanced use of a distributed network of proxy servers (to prevent the provider from blocking requests by IP);
the selection of timing and frequency of requests, as well as the logic of repeated requests using throttling;
the сalibration of prices depending on tax conditions and various formulas for calculating/displaying commissions by providers that affect the final actual price of a reservation;
the main method for parsing fields of interest (API, PhantomJS/CasperJS, http request & regexp parser logic, specificity of ajax & pagination & crawling handling);
the quick display of available booking options at the user's request via the logic of extracting information from the cache with background updating, which makes the architecture able to perform many parallel requests (high load) and also allows users to see the current availability and prices of booking options.