3 days trial then $25.00/month - No credit card required now
BBB Data Crawler
3 days trial then $25.00/month - No credit card required now
Scrape data from the Better Business Bureau as known as BBB. Crawl and extract company information, insights, financial projections, social links, accreditations and much more. Scrape the huge BBB Cloud, retrieve charities and businesses in the United States, Canada, and Mexico by their categories.
How do I maximize the number of records scraped?
Closed
Using build 0.0.892, I did a scrape for "Plumbers in New York, NY" using the URL: https://www.bbb.org/search?find_country=USA&find_entity=10113-000&find_id=10113-000&find_loc=New%20York%2c%20NY&find_text=Plumber&find_type=Category&page=1&sort=Distance.
This scrape produced approximately 200 records. However, when I manually visit the BBB site, using the same URL, I can see that there are over 7K results for the same query?
Any reason why the BBB Data Crawler is producing less results than what is actually available online? Should I optimize any other settings before running? I have ran multiple scrapes using multiple build #s. I have found that build 0.0.892 seems to produce the most consistent # of results with each scrape. Any feedback would be helpful.
Hey there,
Thank you very much for reaching out and letting us know about your problem. BBB website has a limitation that we call the "Soft Limitation". This is a technique that the website tells you that it found a high amount of items but only shows a limited amount. In your search query, you can only reach up to the 15th page maximum. There are no more than 15th pages on the website no matter what is your search query.
To maximize your search scope, and capture more items you have to split your search query into multiple chunks. For example; if you are using "California", you might want to use a list of California's cities or counties.
Best
epctex
