AllTheRooms

AllTheRooms - portal for booking rooms. This is an extensive search engine for travelers, combining the largest sites (Expedia, Airbnb and Jetsetter) and other popular travel resources. All the rooms compares prices on all these sites so that you can book a room or house anywhere in the world at the best price. Our team was hired to develop a supporting project for AllTheRooms. It was necessary to create an application that independently collects information from dozens of providers through the API, as well as using custom scrappers for booking sites that do not have an API (or have access restrictions). The complexity of the task was determined by the requirement for the speed of creating each subprogram (no more than 2-3 hours per provider), taking into account the fact that for each of them it was necessary to select the most optimal set of tools.

What was done

For the service we have implemented:

• balanced use of a distributed network of proxy servers (to prevent the provider from blocking requests by IP)
• selection of timings and frequency of requests, as well as the logic of repeated requests using throttling
• сalibration of prices depending on tax conditions and various formulas for calculating/displaying commissions by providers that affect the final actual price of a reservation
• selection and implementation of the main method for parsing fields of interest (API, PhantomJS/CasperJS, http request & regexp parser logic, specificity of ajax & pagination & crawling handling)
• the logic of extracting information from the cache with background updating is implemented for quick display of available booking options upon the user's request. That makes the architecture able to perform many parallel requests (highload), and also allows users to see the current availability and prices of booking options.