Using the Apify platform to scrape prices, we ran an extensive analysis of the three leading Czech e-shops for the month leading up to Black Friday. We found that only 2% of products were discounted by an average of 12%, while the original price was artificially inflated so that “discounts of up to 80%” could be advertised online.
We used three crawlers that scraped the basic product parameters (mainly URL, name, current and original price). Our crawlers tracked daily price fluctuations of all products for the month before Black Friday. During the Black Friday sale (which ran for almost the full week up to Friday, November 24), we tracked all prices four times every day. In total, we processed approx. 30,000 pages per day and collected approx. 733,000 entries.
After collecting the data, we prepared a MySQL DB on AWS and wrote a simple Apify actor that went through all our crawler runs and added the data into the DB. Finally, we deduplicated the records, cleaned them up and uploaded the resulting data to Google Data Studio. From there we were able to search, calculate and visualize the results.