Business Directory Scraper — Yellow Pages, BBB, Gelbe Seiten, PagineGialle, Europages, Hotfrog

Seven business directories, one run, one table, one schema. Type a trade and a city, pick the directories you care about, and get company records with phone numbers, websites, e-mail addresses, postal addresses, coordinates, star ratings, BBB letter grades, accreditation status, staff counts and founding years — all in the same 38 columns, no matter which directory a row came from.

Everything below is measured. Every number in this file comes from a run you can repeat, and tools/verify_readme.py fails the build if a number here stops matching the code or the measurement file.

What it covers

Directory	Countries	Rows per page	What it is good for
Yellow Pages US (yellowpages.com)	United States	30	websites, years in business, star ratings
Yellow Pages Canada (yellowpages.ca)	Canada	35	websites, star ratings, parsed postal addresses
Better Business Bureau (bbb.org)	United States, Canada	15	A+ letter grades, accreditation, coordinates
Gelbe Seiten (gelbeseiten.de)	Germany	50 then 10	e-mail addresses, websites
PagineGialle (paginegialle.it)	Italy	25	ratings, opening hours, multiple phone numbers
Europages (europages.co.uk)	28 European countries	30	B2B suppliers, staff count, founding year
Hotfrog (27 country sites)	US, GB, CA, AU, DE, FR, IT, ES, NL and 18 more	~12	latitude and longitude, long descriptions

All seven answer plain HTTP from the Apify cloud. No proxy, no browser, no API key. The actor is pure Python standard library.

The headline: breaking the Better Business Bureau result cap

BBB is the most valuable directory in this set — it is the one with letter grades, accreditation and complaint history behind it — and it is also the one that hides most of what it finds.

Ask it for plumbers in Austin, Texas and it answers, in its own JSON: totalResults: 2537. Then it sets totalPages: 15 and serves 15 rows a page. Fifteen times fifteen is 225, and the pages repeat each other, so crawling every page a human could ever click gives 175 unique businesses. That is 6.9% of what the directory just told you it has. No amount of paging changes it: page 16 returns record-not-found.

The way past it is a split the directory itself hands you. Next to every result list BBB returns its own category list under filters.byId.filter_category, it understands a distance ring in filter_distance, and it understands a sort order in sort. Each of those is a separate query with its own cap of 225. Running the same search once per bucket and merging on the business id gives, on the same city and the same word:

What was crawled	Unique businesses
Plain search, all 15 pages	175
+ 10 category buckets BBB itself lists	366
+ distance rings of 5, 10 and 15 miles	517
+ sorted by distance	524

524 against 175 — three times as many, for 21 extra queries. Both numbers come from cloud runs of this actor with the same input and expandBuckets flipped: run kt2dccRUXf2Itm2BL returned 175 rows, run rKU71p8Dxq62dCgQ0 returned 524.

Two honest notes about that. First, 524 is not 2537: the buckets overlap and some businesses sit in no narrow category at all, so this lifts the reachable ceiling threefold, it does not empty the directory. Second, the buckets that earn nothing are not requested at all — rings of 25, 50, 100 and 200 miles and the sort orders TOB, Alphabetical and HighestRating each added exactly zero new businesses in the measurement, so asking for them would be 60 wasted requests against the source and 60 wasted seconds of your run.

Why seven directories in one actor matters

The Apify store sells these one at a time. To cover what this single actor covers you would currently run trudax/yellow-pages-us-scraper (United States, $5.00 a month flat), a separate Yellow Pages Canada actor, easyapi/bbb-business-directory-scraper ($19.99 a month flat), plowdata/gelbe-seiten-ppr for Germany, an Italian one, and easyapi/europages-business-directory-scraper ($19.99 a month flat) — five or six subscriptions, five or six different output shapes, and a merge job of your own before any of it reaches a CRM.

Here it is one run, one dataset and one set of column names. source and source_name on every row say where it came from; everything else means the same thing in every row.

Fields, and how often they are actually filled

Thirty-eight columns. The leading Yellow Pages actor on the store documents ten (isAd, url, name, address, phone, rating, ratingCount, infoSnippet, image, categories). The extra columns here are not padding — they are parsed address components, coordinates, contact channels and company facts that the search pages really carry.

An empty cell means the directory did not publish that fact. Nothing is inferred, guessed or filled in from another source. Measured on the cloud runs listed in tools/measurements.json:

Directory	rows	phone	website	e-mail	address	city	lat/lng	categories	rating
Yellow Pages US	150	100%	79%	—	99%	99%	—	100%	13%
Yellow Pages Canada	150	100%	49%	—	53%	51%	—	99%	17%
BBB	524	99%	—	—	100%	100%	100%	100%	—
Gelbe Seiten	132	88%	56%	35%	100%	—	—	100%	—
PagineGialle	150	67%	—	—	100%	99%	—	100%	26%
Europages	150	—	—	—	100%	100%	—	48%	—
Hotfrog	120	98%	—	—	71%	71%	63%	62%	—

Directory-specific columns on top of that: years_in_business (Yellow Pages US, 25% of rows), bbb_rating and bbb_score (BBB, 82%), accredited (BBB, 62%), employee_count (Europages, 38%), founded_year (Europages, 51%), service_areas (BBB service counties and Europages supplier types), description (Europages 89%, Hotfrog 100%), opening_hours (Yellow Pages US 67%, PagineGialle 29%, Gelbe Seiten 27%).

E-mail addresses, which the others do not return

Gelbe Seiten is the only directory in this set that puts a real e-mail address on the search page — and it does not put it anywhere a naive scraper would look. It sits inside the JSON configuration of the page's chat widget, alongside a cleanly split street, postcode, city and phone list. Measured per page: 16 e-mail addresses on 50 cards for klempner in Berlin and 32 on 50 for zahnarzt in Berlin; across a 132-row run, 35%.

The same directory hides two more things in plain sight. The website address is base64 in a data-webseiteLink attribute, and the phone number of a listing without the chat widget is base64 in data-prg. Both are decoded here, which is why phone coverage is 88% rather than the 35% you get from the widget alone.

Getting past the Gelbe Seiten fifty-row wall

Gelbe Seiten shows 50 businesses and then a "Mehr Anzeigen" button. The button is not a link — it posts a form to /ajaxsuche, and the answer is not HTML but JSON shaped {"anzahlTreffer":10,"html":"<escaped markup>"}. A scraper that parses the raw body finds zero cards in it, stops at the first page and reports success. That is exactly what this actor did before the fix: 48 rows while the source itself printed 210 Treffer at the top of the page. Unwrapping the JSON first takes the same query to 132 rows. The answer always carries ten cards no matter what anzahl asks for — measured at 50, 10 and 50 again.

Directories that are not here, and why

This is the part most listings leave out. Nine directories were tested from the Apify cloud and rejected, each for a named reason:

• yell.com (United Kingdom) — HTTP 403.
• pagesjaunes.fr (France) — HTTP 403.
• thomasnet.com (US industrial) — HTTP 403.
• manta.com — HTTP 403, Cloudflare interstitial.
• cylex.us.com — HTTP 403.
• goldenpages.ie (Ireland) — HTTP 202 with an AWS WAF challenge.
• yellowpages.com.sg (Singapore) — redirected to a Radware captcha.
• BBB business profile pages — HTTP 403. The BBB search API is open, the per-company profile page is not, which is why this actor reads everything from search results and never promises profile-only fields.
• yellowpages.com.au (Australia) — and this one is worth spelling out. It answers HTTP 200 and returns thirty perfectly formed listings. It also returns the same thirty for plumber in Sydney and for dentist in Melbourne: a list of data-processing firms that has nothing to do with either query. A scraper that trusts the status code would sell you that as Australian plumbers. It is excluded on purpose.

Two more silent traps, handled

Hotfrog ignores the city. /search/us/plumber/austin answers 200 and quietly redirects to /search/us/plumber — the country-wide list, city dropped. Same for ?page=2 and for /austin/2. And /search/us/texas/plumber lands on /search/us/texas, losing the search word instead. So this actor never puts a place in a Hotfrog URL: Hotfrog searches a whole country, and the input form says so.

PagineGialle has no website links. Not "rarely" — zero outbound website links on 25 cards. The buttons on an Italian listing are phone, WhatsApp, request-a-quote and contact-us. website is therefore always null for Italian rows, and the requireWebsite filter refuses a PagineGialle-only run before it makes a single request rather than returning an empty dataset.

Empty results always say why

A successful run with an empty dataset is the worst failure a paid scraper can have: it looks fine, you are charged for the start, and nobody can tell whether the source broke or the input was impossible. Every empty run here writes a named reason into the run report and a warning into the log. There are twelve of them:

no-search-terms, no-sources-selected, email-not-published-by-selected-sources, accreditation-only-in-bbb, website-not-published-by-selected-sources, rating-not-published-by-selected-sources, years-in-business-only-in-yellowpages-us, filtered-out, nothing-changed, nothing-collected, source-returned-nothing, and charge-limit-reached.

The first seven are decided before a single network request. If you ask for BBB accreditation but do not select BBB, or for e-mail addresses without Gelbe Seiten, the answer was already fixed by your input: the run says so in one line and stops, instead of downloading megabytes and throwing all of it away.

Filters run before you are charged

Every filter — phone present, website present, e-mail present, no paid placements, BBB-accredited only, minimum star rating, minimum years in business, name contains, name excludes, city contains, category contains — is applied to a row before it is written to the dataset. A row a filter removes is never billed, and the run report lists exactly how many rows each filter removed.

Two rules are wired into those filters, and both were bought with empty datasets on other actors. First, maxItems limits rows you receive, not pages crawled: with a filter on, the crawl keeps going past the limit until enough rows survive, because stopping at n collected rows and then filtering them all away is how a green run delivers zero. Second, a missing value is not a mismatch: a row where the directory printed no city passes cityContains, while a row with no phone number fails requirePhone — because there the empty value is the answer.

Monitoring the same search day after day

Turn on Only new and changed businesses and the actor remembers what it has already delivered in a named key-value store that survives between runs. The next run returns only rows that are new or whose name, phone, website, address or rating changed, each tagged change_type — NEW or UPDATED. A daily monitor over a 500-row search therefore pays for a handful of rows a day instead of 500.

When nothing changed the dataset is legitimately empty, and the run says nothing-changed rather than staying silent, so a watchdog can tell that apart from a breakage.

Cost and speed

The actor runs on 1024 MB and only ever makes plain HTTP requests: no browser, no proxy fees. Measured on the cloud: a 150-row Yellow Pages US run takes five page fetches; a 524-row BBB bucket run takes 188 page fetches and finished well inside a 900-second timeout. The default input — plumber, Austin TX, Yellow Pages US plus BBB, 100 rows — completes in well under the five-minute window Apify uses for its daily health check.

Limitations, named honestly

• No company profile pages. Everything comes from search result pages. The BBB profile page is HTTP 403 from the Apify cloud, so complaint counts, review text and years-in-business-per-BBB are not available at any price.
• No start URLs. The leading Yellow Pages actor accepts a list of search URLs as an alternative to a search term; this one takes search terms, locations, countries and directory names only.
• No review text. Ratings and review counts yes, individual reviews no.
• No Australia, United Kingdom, France, Ireland or Singapore yellow pages — every one of those is blocked or, in the Australian case, lying. See above.
• Europages has no location input. It searches by product and country, so the locations field is ignored for it.
• Hotfrog is country-wide, not city-wide, for the reason given above.
• No website column for BBB, PagineGialle, Europages or Hotfrog, and no e-mail column outside Gelbe Seiten: those directories do not publish them on the search page.
• No proxy support. It is not needed today — all seven answer the Apify cloud directly — but if one of them starts blocking datacentre addresses, that directory will fail rather than fall back.

FAQ

Can I run several trades and several cities in one go? Yes. searchTerms and locations are both lists, and the actor runs every combination of the two against every selected directory.

Which directory should I start with? For United States lead lists, Yellow Pages US for contact details and BBB for quality signals — that pair is the default input. For Germany, Gelbe Seiten, because it is the only one with e-mail addresses. For European B2B suppliers, Europages.

Why is the website column empty for so many BBB rows? Because BBB does not print a website on its search results at all. It is empty for every BBB row, not for some of them, and the same is true of PagineGialle, Europages and Hotfrog. The table above says which directory publishes what.

Does expandBuckets cost more? It makes more requests to BBB — 188 page fetches instead of 15 in the measured run — and it returns three times as many businesses. Turn it off if you only want the first 175 and want the run to finish in seconds.

What happens if I ask for more rows than exist? The run stops when the directories stop returning new businesses and reports how many it found. You are charged for the rows you receive.

Do repeated runs return the same rows twice? Only if you want them to. With Only new and changed businesses off, every run returns the full result set; with it on, you get new and changed rows only.

Is a row that appears in two directories returned twice? Yes, once per directory, each with that directory's own fields and its own source. They are deliberately not merged: a BBB record and a Yellow Pages record for the same firm carry different facts, and silently collapsing them would throw half of those facts away. Deduplicate on phone number or website if you want one row per company.

Can I get only the contact columns? Turn on Compact rows for eleven contact-only columns, or Drop empty fields to leave out whatever is null on a given row. Both are meant for AI agents and for CSV exports where 38 columns are too many.

Does it need a proxy or an API key? No. Neither.

What else is this called?

Names buyers use: Yellow Pages scraper, BBB scraper, BBB accreditation data, Gelbe Seiten scraper, Pagine Gialle scraper, Europages scraper, Hotfrog scraper, business directory leads. Seven directories, one run, and the BBB rows carry the accreditation flag and the rating.

Also asked for as directory bucket expansion, which is the expandBuckets input and the bucket field.