Indeed Jobs Scraper (21 countries)

Scrapes live job listings from Indeed across 21 country sites — US, UK, Canada, Australia, Germany, France, India, Singapore, Netherlands, Spain, Italy, Brazil, Japan, Mexico, Ireland, Philippines, South Africa, UAE, Poland, Sweden and Switzerland.

Public data only. No login, no cookies, no browser, HTTP-only. There is no bot challenge to bypass here — every TLS profile tried returned the full page cold, with no warm-up and no session.

What makes this one different: it tells you when it is missing data

Indeed has three ways of handing you a result that looks right and is not. This actor is built around all three, and reports each one in the data rather than hiding it.

1. It respects Indeed's stated crawl depth — and says so

Indeed's robots.txt names the Anthropic agents explicitly and permits &start=0 through &start=90, while disallowing deeper paging. That is 10 pages, roughly 150 organic jobs per query, and this actor stops there even though deeper pages do answer.

That ceiling is the honest headline limit. To cover a large board, split into narrower queries — per city, per job type, per datePosted window — rather than paging deeper. Every summary row carries totalJobCount, estimatedReachable and robotsPageCeilingReached so you always know how much of the result set you actually have.

The same policy also means there is no job-description pass: /viewjob is disallowed for us. Job rows carry a jobUrl you can open yourself, but the actor never fetches it.

2. A mistyped location does not fail — it silently returns the whole country

Searching Austin, TX returns Austin jobs. Searching Zzzqqxx, ZZ returns 11,078 matching jobs from Denver, Charlotte, Los Angeles, Philadelphia and Atlanta — and Indeed echoes your typo straight back as the search location, so upstream's own response cannot be trusted to catch it.

Every summary row therefore carries locationMatchRatio: the fraction of returned jobs that are actually near what you asked for. 1.0 means the filter applied; 0.0 with locationLooksIgnored: true means Indeed dropped it and served the nation. The run logs a warning naming the problem.

3. Most of page one is advertising

On the US site, page 1 returns 40 cards — 25 of them paid placements, not organic results. Pages 2–10 return 15 cards each, all organic.

includeSponsored is therefore off by default. Sponsored cards are still counted, so the summary always tells you the split (organicCount vs sponsoredCount), and when you do include them every row carries isSponsored.

Plus: an intermittent sign-in wall, handled rather than ignored

Roughly 20% of page-2-and-beyond requests come back as HTTP 200 with a "create an account or sign in" page and no jobs at all. It is probabilistic — the same URL usually succeeds on retry. This actor retries it with backoff, and if a page never clears, records it in loginWalledPages instead of quietly returning a short result set.

Bad filter values are refused before the run starts

Indeed does not reject filter values it does not understand — it ignores them and returns unfiltered results. Measured against a 3,386-job baseline:

You send	Indeed does	This actor does
jobType=zzzz	returns 3,362 — the baseline	refuses, naming the valid values
datePosted=99	returns the baseline	refuses
sort=zzz	returns the baseline, in relevance order	refuses
radius=7	not honoured predictably	refuses (Indeed's own steps only)
maxPagesPerQuery=11	would answer	refuses — past the robots ceiling

Checked in code, not only in the input schema, so an API or CLI run gets the same protection as a Console run.

What you get

Three record types share one dataset, told apart by recordType.

JOB — one row per vacancy

Indeed's own card object passed through verbatim (~117 fields), plus our envelope. Highlights: job title, company, company rating and review count, formatted location, city/state, parsed salary range with period (extractedSalary), salary text, job types, remote-work model, ranked benefits, taxonomy attributes, snippet, posting age, apply flags, jobkey and a jobUrl.

One upstream quirk worth knowing: the top-level jobTypes array is often empty even when the job is full-time — the real value lives in taxonomyAttributes under the job-types label. Both are passed through untouched.

SEARCH_SUMMARY — one row per query

Indeed's own totalJobCount, rows returned, organic/sponsored split, pages fetched, loginWalledPages, locationMatchRatio, robotsPageCeilingReached and estimatedReachable.

ERROR — one row per query that failed

So every input maps to at least one row.

Filters

searchQueries (fanned out independently), country, location, jobType, datePosted (1/3/7/14 days), radius, sort (relevance or date), remoteOnly, includeSponsored, maxItems, maxPagesPerQuery, minRequestInterval, maxConcurrency.

All of them were verified to actually move Indeed's result count — see CRAWLING_METHOD.md for the measurements.

remoteOnly uses Indeed's own Remote facet and is very aggressive: a query matching 3,386 jobs dropped to 8 with it on.

Known limits

• ~150 organic jobs per query, by Indeed's stated policy, not by choice.
• No job descriptions — the detail page is robots-disallowed for us. The snippet field carries Indeed's own summary.
• No company-profile pass — /company/* is disallowed too.
• Non-English country sites want queries in the local language.
• Indeed's totalJobCount wobbles by ~1% between identical requests; it is Indeed's estimate, passed through as-is.

Proxy — please read before running

This actor needs a residential proxy to be reliable. Indeed gates on IP reputation, not on browser fingerprint: from a residential exit it answers 200 on every request with no challenge at all, while much of a datacentre proxy pool gets 403.

Measured on the Apify platform against a US datacentre pool: uk, au and de came through, www.indeed.com (US) came through intermittently, and in and ca did not come through at all. The actor rotates its exit IP on every retry (up to 12 attempts) because some IPs in a pool are accepted and some are not — but rotation cannot manufacture a clean IP if the pool has none.

• With Apify Residential proxy — recommended, and what the default input selects.
• With datacentre proxy — expect intermittent fetch_failed rows, especially on the US, India and Canada hosts.
• With no proxy on the platform — will not work at all. Apify containers have no direct outbound route; curl fails to connect in milliseconds.

Pass an empty proxyConfiguration ({}) only for local runs.