Profi.ru scraper — headless Apify actor

A browser-based (Playwright) actor for Profi.ru — a direct HTTP scrape is anti-bot-blocked, so this runs a real browser over Apify residential proxies. Its output matches src/modules/profi/parser.py::parse_apify_item (url, name, specialty, location, rate, phone).

Deploy (one time)

npm i -g apify-cli
apify login                       # paste your Apify API token
cd apify-actors/profi-scraper
apify push                        # builds + uploads; prints the actor id (user/name)

Then in the app's .env:

PROFI_APIFY_ACTOR=<your-user>/profi-ru-scraper

and recreate the app container. The Profi apify mode will then appear in the search panel.

How it scrapes

Three phases: resolve the query to a taxonomy path, read the PROFILES listing, then open each card's profile for what the listing does not carry (the full review list and the education/experience block).

The listing serves 20 cards and appends the next 20 per click on button[data-shmid="pagination_next"] ("Показать ещё 20") — there is no page-number URL, so maxItems above 20 is only reachable by clicking through. Cards hydrate one by one, so the actor waits for the count to stop climbing before it extracts or paginates; extracting on the first card that appeared used to return 3 of 20.

Prices come from the card, not from the profile. The card lists only the services filed under the searched trade and prints the real total next to them ("Все услуги и цены (15)" — that link opens the full profile). The profile page lists every trade the specialist sells, led by whatever they do most, which is how a search for "электрик" came back with a plumbing price list. So prices is the card's rows and pricesTotal the count from that link.

The profile pass also clicks "Все услуги и цены (34)" — it expands the table in place, no modal and no navigation — and returns every row as pricesAll, each tagged with the trade it sits under (group). The app shows prices by default and pricesAll behind a toggle, so the full portfolio is available without the searched category drowning in it.

Selectors are taken from the live markup and use data-shmid / data-testid hooks (profileCard, profileIndicators_ratingBlock, profileOIO, priceRow, reviewItem, pagination_next, …). Class names are hashed CSS modules and are never matched whole.

Output schema

Two files, because the Console counts them separately:

• .actor/dataset_schema.json — what the dataset looks like: two table views (specialist overview, prices/reviews) instead of raw JSON. It ships fields without required, so a specialist whose profile omits half the fields still lands in the dataset.
• .actor/output_schema.json — what the run's Output tab points at ({{links.apiDefaultDatasetUrl}}/items). This is the one the Store's "Add Output schema" gate checks; a dataset schema alone leaves it red.

Cost and speed

One specialist = one profile page load in a headless browser over residential proxies: a 50-item run took 8 minutes and $0.40. The app therefore caps Profi at 20 items (src/modules/profi/parser.py::MAX_ITEMS) — about $0.16 and one "show more" click.

Images, media, fonts and the analytics hosts are aborted in preNavigationHooks: the scraper reads image URLs from the markup and never the pixels, so downloading them only spends residential-proxy GB — the quota a browser actor exhausts first.

fullProfiles: false

Known limits

Relevance is Profi's, not ours. Profi returns whoever it matched to the category, and the price row for the searched service is generated even when the specialist does something else — a violin tutor comes back for "тестировщики" with a "Тестировщики" price row. Cards carrying the "Нет отзывов по вашей услуге" warning are the tell; that warning is deliberately not surfaced as a trust badge.

Phones are not scrapeable. Profi gates contacts behind task creation, so no phone appears anywhere and phone is always null.

The query is resolved by Profi, not by us. Profi has no keyword-search endpoint — every trade is a taxonomy path — and smart-search cannot be called directly, since the page before it hands it the answer. So the actor types the query into Profi's own search box and reads the path off wherever the site navigates. That keeps resolution correct when Profi reshuffles its taxonomy; the only thing to maintain is SEARCH_BOX, and the log line search box resolved "…" to … says whether it worked.

/smart-search/ in the log is a waiting room, not a result: Profi keeps matching there for about ten seconds and then redirects to the wizard with ?pathname=<category>. The actor waits for that redirect (or a taxonomy path in the URL), which is what the log line search box resolved "…" to … reports. A query Profi cannot classify parks at /raznoe/raznoe_other/ — treated as a failed resolve, since its listing is every trade at once.

CATEGORY_BY_KEYWORD is a fallback used only when the search box cannot be driven, and categoryPath overrides everything:

$apify run -i '{ "query": "сварщик", "categoryPath": "/remont/svarochnye-raboty/", "region": "Кемерово" }'

Region mapping is partial. region is matched against the cities the app targets (see GEO_BY_CITY); anything unknown falls back to Moscow, Profi's own default locale.

Local test

$apify run -i '{ "query": "электрик", "region": "Москва", "maxItems": 10 }'