Indeed Jobs Scraper

Scrape job search results into a clean dataset: title, company, location, salary, canonical job URL and posted date — plus the depth the Indeed search feed already carries on the same request (full job description, apply URL, employer profile fields, job type, remote flag). Search many queries per run, filter by date posted, job type and remote-only, and cap records per query.

sources also covers Glassdoor job listings with the same row shape (one extra selectable board; the default run is Indeed-only, byte-identical to the previous release).

{
  "queries": ["registered nurse", "warehouse associate", "python developer"],
  "location": "United States",
  "datePosted": "7d",
  "maxRecords": 20
}

Output

One dataset row per job (no summary row), with a per-row query so a multi-query run stays traceable. The sample row below is an Indeed feed row with all ten depth columns populated: every column is emitted only when the listing actually has it:

{
  "title": "Registered Nurse - ICU",
  "company": "Ascension",
  "location": "Austin, TX 78705",
  "salary": "$78,000 - $102,000 a year",
  "description": "Seton Healthcare Family, a member of Ascension, is seeking a Registered Nurse ICU to deliver direct patient care to ensure the well-being of patients…",
  "applyUrl": "https://careers.example.com/apply/rn-icu-3822",
  "jobType": "fulltime",
  "isRemote": false,
  "companyLogo": "https://logo.example.com/ascension.png",
  "companyWebsite": "https://employer.example.com",
  "companyEmployees": "1,001 to 5,000",
  "companyRevenue": "$1B to $5B (USD)",
  "companyIndustry": "Health Care",
  "companyDescription": "Ascension is one of the largest private healthcare systems in the United States…",
  "url": "https://www.indeed.com/viewjob?jk=1a2b3c4d5e6f7890",
  "postedDate": "2026-10-19",
  "source": "indeed",
  "jobKey": "1a2b3c4d5e6f7890",
  "query": "registered nurse",
  "ok": true,
  "country": "US"
}

• salary appears only when the listing publishes one; empty fields are never emitted, so no column is ever undefined.
• postedDate is an ISO 8601 date (YYYY-MM-DD) whenever Indeed reports one it can trust. When it does not — for example on listings older than about a year — the key is left out of the row altogether, rather than reporting a possibly wrong date.
• country is the uppercase ISO code of the country site the row came from (US, GB, …) — the run's country code, uppercased. It is present on every row, including a failed query's row.
• Depth from the same search request. The Indeed feed that already returns the row also carries the full description — HTML stripped to plain text and not capped; the ≤600-character descriptionSnippet stays as the card preview — plus applyUrl, the external/ATS apply URL of a listing that sends applicants away from Indeed (absent when the listing applies on Indeed itself). No per-job detail request is ever made to fill either column.
• Job type and remote markers — on both Indeed routes. jobType is the structured employment type — fulltime, parttime, contract, internship, temporary, perdiem and/or volunteer, comma-joined in that fixed order when a listing carries several. The feed decodes the listing's own attribute codes; the search-page fallback maps the card's own labels to the same vocabulary. isRemote is true for a remote listing and false when it is marked hybrid or on-site (Indeed lists remote/hybrid explicitly, so an attribute list with no remote marker is an on-site listing); the key is absent only when the listing carries no remote/hybrid information at all.
• Employer profile columns, feed-only. companyLogo, companyWebsite, companyEmployees, companyRevenue, companyIndustry and companyDescription come from the company profile the feed itself carries with the search result — Indeed's own labels, verbatim (e.g. "1,001 to 5,000" employees, "$1B to $5B (USD)" revenue), and companyDescription is the employer's blurb, distinct from the job's own description. The search-page fallback omits these six.
• Supplied-when-supplied. A row carries exactly as much depth as its listing has — the same absent-is-absent rule as salary: fields the search does not return are never emitted (no column is ever undefined), never invented. Error rows carry none of the ten depth columns; a row served by the search-page fallback carries only jobType / isRemote of them.

• A job that matches several of your queries appears once per matching query (the query column tells you which search found it). It is only billed once per run.
• A query that cannot be served still produces exactly one row: {"query": "...", "ok": false, "err": "blocked by the site's anti-bot check", "source": "indeed", "country": "US"}. A run never ends with an empty dataset and never fails because of the site.

Glassdoor rows (sources includes glassdoor)

Glassdoor listings use the same columns and the same query/source/ok contract; source is glassdoor and jobKey is gd-<listingId>. The search answer already carries the card fields, and one batched depth request per page (up to 25 listings per request) fills the rest — no per-job page is ever fetched:

{
  "title": "Staff Software Engineer - Monetization Platform",
  "company": "Indeed",
  "location": "Texas",
  "salary": "$140,000 - $293,000 a year",
  "description": "Our Mission As the world's number 1 job site, our mission is to help people get jobs…",
  "applyUrl": "https://www.glassdoor.com/partner/jobListing.htm?pos=101&ao=…",
  "easyApply": false,
  "jobType": "fulltime",
  "companyRating": 3.9,
  "companyReviewCount": 5506,
  "companyLogo": "https://media.glassdoor.com/sql/100561/indeed-squareLogo-1671046570073.png",
  "companyWebsite": "https://www.indeed.com",
  "companyEmployees": "10000+ Employees",
  "companyRevenue": "$1 to $5 billion (USD)",
  "companyIndustry": "Internet & Web Services",
  "companyDescription": "Indeed is the #1 job site in the world…",
  "companyCeo": "Hisayuki Deko Idekoba",
  "companyFounded": 2004,
  "sponsored": true,
  "salaryMin": 140000,
  "salaryMedian": 216500,
  "salaryMax": 293000,
  "salaryType": "YEARLY",
  "salaryCurrency": "USD",
  "url": "https://www.glassdoor.com/job-listing/staff-software-engineer-…?jl=1010302858380",
  "postedDate": "2026-10-21",
  "source": "glassdoor",
  "jobKey": "gd-1010302858380",
  "query": "software engineer",
  "ok": true,
  "country": "US"
}

• Salary is the site's own pay estimate when it publishes one: salary text plus the decoded salaryMin / salaryMedian / salaryMax, salaryCurrency and salaryType (YEARLY, MONTHLY, HOURLY). A listing with no published pay emits none of these keys.
• Company dossier: companyRating + companyReviewCount, companyLogo, companyWebsite, companyEmployees, companyRevenue, companyIndustry, companyDescription, companyCeo, companyFounded — the employer fields Glassdoor publishes for the listing's own employer id (the review count is resolved by one batched employer request, not a per-job page).
• Apply / listing facts: applyUrl (the employer's apply destination when the listing sends applicants away from Glassdoor), easyApply, sponsored, and url (the public permalink built from the listing id).
• postedDate is ISO YYYY-MM-DD, derived from the listing's own age in days; the relative text the site prints ("3d") is never published.
• Glassdoor search is keyword-only in this version: location is not applied to Glassdoor queries (the site's own search API is keyword-based), so a Glassdoor row's location is the listing's own location string.
• Every Glassdoor query that cannot be served still produces one row: {"query": "...", "ok": false, "err": "…", "source": "glassdoor", "country": "US"}.

Pricing

$0.1 per 1,000 job results — the first 50 results of a run are free. Glassdoor rows are $0.4 per 1,000 (GLASSDOOR_JOB_RESULT, $0.0004/job); the same 50 free results cover both boards in a mixed run.

That price is the JOB_RESULT event ($0.0001 per result).

Two facts drive the price, and both are measured on this actor's own runs:

1. the data path is a lightweight HTTP client (~1.5 s and <100 MB per query), not a browser — a browser session would cost 10-20x the compute for the same bytes;
2. the market anchor is an all-in $0.0001/job actor (compute absorbed by the developer), and Apify's developer share of event revenue is 80%.

Active branch: A — compute included.

Volume ladder (per job, JOB_RESULT): FREE $0.0001 · BRONZE $0.00009 · SILVER $0.00008 · GOLD $0.00007 · PLATINUM $0.00007 · DIAMOND $0.00007.

The Glassdoor price is 4× Indeed's because one Glassdoor page costs a warmed session plus two batched enrichment requests (depth + employer review counts) per 25 listings, and the market anchor for job-listing parity is $0.29-$4.75 per 1,000. The run-level first-50 counter spans both boards — a mixed run's first 50 delivered rows are free whichever board produced them, then each board bills its own event.

The last two tiers are floored at $0.00007 by the platform: the pricing API silently clamps any tiered price below that value, so a deeper discount than 30% cannot be expressed. The ladder above is the price list as it actually exists on the account, not as it was designed.

Pricing math (why branch A, and when B applies)

All numbers below are measured on a production-shaped run (20 queries, 387 jobs, 512 MB, 49 s) and use the rate the platform actually bills, not a reference rate:

So the job price is ~11× the usage it causes: compute-included pricing is safe on this data path, and the 50 free results stay genuinely free (an error row, or a query that returns nothing, is never charged at all).

A very small run does cost the developer a fraction of a cent: a single-query run bills ≈$0.0002-0.0003 of usage and at most $0.00004 of revenue. That is the price of the free tier, and it is capped by maxTotalChargeUsd.

Branch B ($0.0004/job) exists for a browser-only path, where 1 GB × 15 s per SERP costs ≈$0.0005-0.001/job at the measured rate: at that cost a $0.0004 price is NEGATIVE margin unless platform usage is billed to the user (isPPEPlatformUsagePaidByUser). Both pricing branches are pre-wired in the actor's configuration; the actor ships branch A because the measurement supports it.

Status (2026-10-21): the JOB_RESULT and apify-actor-start events are active on the account (JOB_RESULT $0.0001 on the FREE plan, ladder as above; apify-actor-start $0.00005). The first 50 results of a run are free because the actor simply does not charge for them — the platform cannot carry a zero-price event, so no paid event is emitted for results 1-50. Events are charged per delivered job (deduplicated by job key), never for a failed query. Verified end-to-end on a production run: 60 rows delivered were billed as 1 start event + 9 job events ($0.00095), with the first 50 results free.

Billing

• An error row or a failed query is never charged; the per-run start event ($0.00005) is the only charge a run that delivered nothing can ever produce.
• One delivered job is one JOB_RESULT event (deduplicated by job key), never a failed query. The first 50 results of a run are free — the platform cannot carry a zero-price event, so nothing is emitted for results 1-50.
• A delivered Glassdoor row bills GLASSDOOR_JOB_RESULT ($0.0004) after the same run-level 50 free results; a mixed run's billing line reports the per-board split (boards=indeed:…,glassdoor:…). Error rows — a query that could not be served, an unsupported country — are never charged.
• At the published maximum (50 queries × 200 = 10,000 rows) an Indeed run bills ≈$0.995 — under the maxTotalChargeUsd $2 cap the actor is configured with, so the platform never has to end a full-size run early. A Glassdoor run at that same maximum would bill ≈$3.98, so for Glassdoor-heavy runs keep queries × maxRecords at ≈5,000 rows per run to stay under the cap.

Inputs

sources is the one field that turns a Glassdoor run on; leaving it unset (or ["indeed"]) is byte-identical to the previous release. Glassdoor runs are 100% anonymous — there is no cookie input, and the session the site hands out is harvested per run and never logged or stored. Glassdoor search is keyword-only (see Output) and serves 19 country storefronts today: us, gb, ca, au, de, fr, nl, in, sg, be, ch, nz, es, it, br, mx, at, ar, ie. jp stays selectable: its Glassdoor origin is in a 530 outage, so those queries return ONE explanation row and the country joins automatically when the origin recovers.

country is one shared input for both boards, so a country only one board serves still succeeds: the serving board returns its rows and the other returns ONE explanation row (ok: false, err naming the reason). That is the case for the 10 Glassdoor-only codes (be, ch, nz, es, it, br, mx, at, ar, ie) when sources includes indeed.

maxRecords is capped at 200. Indeed's primary feed is cursor-paginated (bounded pages, de-duplicated by job key), so one query can return up to the requested cap; a query whose real supply is smaller is reported as complete when the feed itself says there is no next page. The search-page fallback is single-page by design (~45 cards in the US, 15 elsewhere): it delivers what its page holds and reports the query as complete rather than as a short failure. To cover more of the market, use more queries.

jobType and remoteOnly are filter inputs, not post-run slices. On the primary feed Indeed applies the filter on the site itself (server-side — the result set is narrowed before it reaches the run: fewer rows, less bandwidth); a query that falls back to the search page is filtered on its cards instead, where each card carries the job-type labels and a remote flag. Honesty is the rule: a listing that publishes no job type — or no remote status — is not returned by a set filter (unconfirmed is never treated as a match), so a filtered query can return fewer rows than the unfiltered supply of the same query, and a filter that narrows a query to zero rows comes back as the usual normalized row while the run itself still succeeds.

Non-US locations are served where Indeed serves them — pass the country name ("Vietnam") rather than a city that only exists on a country subdomain.

country selects which country site the run reads. Jobs and URLs come from that country's own domain on each selected board: Indeed has 10 storefronts (us, gb, ca, au, de, fr, nl, in, sg, jp), e.g. gb reads uk.indeed.com and de reads de.indeed.com; Glassdoor has the 19 live storefronts listed above. With a non-US country and location left at its default (United States), the search covers the whole country — the US prefill is dropped instead of filtering for it.

Reliability and the one external dependency

Job sites block datacenter and repeated traffic, so the actor reads search pages through a rotating pool of third-party network exits and keeps only the ones that answer normally. That pool is the actor's single external dependency, and it is genuinely variable: on some boots a large share of the exits is refused, and the actor re-boots the pool (bounded, a few seconds) before it gives up.

What that means for you:

• a query that cannot be served comes back as one row with ok: false and a readable err — never a crashed run, never an empty dataset, never a charge;
• the Glassdoor route is anonymous and self-paced: at most 1 request/s per exit, a session re-warm after 100 requests, and one re-warm + retry on a challenge before the run rotates to another exit. A Glassdoor query that still cannot be served returns the same normalized row;
• the run itself always succeeds (exit 0); only an unusable input exits 2;
• measured reliability on the release build: 20/20 queries (100%) on the acceptance run, 39/40 (97.5%) across all staging runs — those are observations from our own runs, not a promised rate: the ok-rate is reported, never guaranteed and never used as a pass threshold.

How it works

1. Build the search request and fetch the results. Pages are fetched with a browser-grade TLS fingerprint; the primary Indeed route is not a page, however — it is the job feed the Indeed app itself uses (see Routes), and its search request already carries the row's depth data (full description, apply URL, employer dossier), so no per-job detail request is made anywhere. If the feed refuses, the public search page is read as before: Indeed embeds its entire result set in the page as window.mosaic.providerData["mosaic-provider-jobcards"] — we parse that JSON, never the HTML.
2. If the page is an anti-bot interstitial instead, rotate to an alternative route (bounded to 3 routes per query, with a bounded pool re-boot), then emit a normalized error row rather than failing.
3. De-duplicate by job key inside a query, apply maxRecords, clean empty fields and push the rows.

Exactly two log levels: info (public wording, progress + summary) and raw (diagnostics, only when explicitly requested by input).

Routes

Indeed is read over two routes: the job feed its own mobile app uses (primary) and the public search page (fallback). The feed returns the same job data — title, company, location, pay, rating, description, post date — and is cursor-paginated (a small, bounded number of requests per query), so it is tried first, on the same network exit the page route would use. If the feed refuses or comes back empty, the actor falls back to the search page exactly as before. Everything after the fetch (de-duplication, maxRecords, cleaning, billing) is shared, so a row looks the same whichever route served it, except for the columns only one route can give: sponsored and isTopRatedEmployer are search-page-only, while the depth dossier — description, applyUrl and the six company* profile fields — is feed-only, read from the feed's own search request with no per-job detail request. jobType and isRemote are the two columns both routes populate. The route is chosen per query, never per run: one query may come from the feed while the next comes from the page.

Volume & paging

The published maxRecords ceiling is 200 per query. The primary feed pages: each request asks for

limit = min(remaining, GQL_PAGE_LIMIT)

Everything is configuration, never a code edit:

A non-production target must never be pointed at production volume: the low env values exist so a fixture run stays a handful of requests.

Filters

jobType and remoteOnly are not scraped-then-sliced: they ride the primary feed as a filter Indeed applies itself — a remote-only query only ever asks the feed for remote listings, so the narrowing happens on the site's side (fewer rows over the wire, less bandwidth). When a query falls back to the search page, the site-side filter cannot apply there; the actor then keeps a fallback row only when the card's own job-type labels, or its remote flag, genuinely satisfy the filter. A card that carries no job type — or no remote status — is not returned by a set filter: unconfirmed is never treated as a match. A filter that narrows a query to zero rows comes back as the usual normalized row and the run itself still succeeds (the never-fail contract above).

Glassdoor route

Glassdoor is read over the JSON search endpoint its own job-search web app calls: one warmed anonymous session per network exit, a cursor walk over the search answer, then ONE batched depth request per page (full description, apply URL, employer dossier). The review count behind the rating rides on the search answer itself; the batched employer request only fires for a row the search answer left without one (a drift tripwire, not a routine request). There is no per-job page request anywhere, and no browser.

• The session is the whole credential: ONE public page request sets the site's session cookies; the JSON endpoint refuses a cold client with an anti-bot challenge. Those cookies are never an input, never logged and never stored.
• Rate discipline: at most one request per second per exit; a session is re-warmed after 100 requests; a challenge is answered with exactly one re-warm + retry before the run rotates to another exit. The ceiling was measured, not guessed: a graduated live campaign ran 30/30 clean at 0.5, 1.0 and 2.0 requests/s with flat latency and zero refusals, so production keeps a 2x margin under the fastest clean step.
• The Glassdoor search page is kept as an opportunistic secondary route: when its zone is open it produces the same row shape; when it refuses, the JSON route's answer (or its normalized error row) stands. The run never dies either way.
• maxRecords is served by cursor paging (100/page, at most 2 pages), and a short supply counts as complete only when the site says there is no next page.

Diagnostic inputs (not part of the published schema)

Our own verification runs use a few inputs that are deliberately absent from the input schema: logLevel (info/raw), set (run a bundled test matrix from sets/), locationOverrides (per-query location for a set) and indeedRoute (auto/gql/serp — forces the feed, forces the page, or leaves the automatic order alone; used to verify the fallback is intact). They exist for testing only and are not documented as product features.

Development

python3 -m unittest discover -s tests -t .   # offline suite
python3 -m pyflakes main.py ij/*.py tests/*.py

Local run (the Pi's harness mirrors the platform surface):

INPUT='{"queries":["welder"],"location":"Denver, CO","set":""}' \
  IJ_OUT=/tmp/rows.jsonl python3 main.py

Monitoring

tools/monitor.py is our daily health check of the published actor. Because of the never-fail contract above, a finished run is only healthy when ≥ 90% of its rows are ok: true and nothing came back empty — the script measures exactly that over the recent runs (it reads them from the Apify API and needs an Apify API token) and prints exactly one line:

[indeed-monitor] 24h: runs=3 finished, rows=774, ok=100.0%, empty=0, worstQuery=100.0%(n=3), alertRuns=0 | OK

(That 90% is our own alert threshold — it flags a run for inspection. The ok-rate it watches is an observed metric, never a promised one.)

(worstQuery is the lowest per-query ok-rate in the window; ok= is the window-wide rate, while the threshold is applied to each run separately. A running=N field appears when some runs have not finished yet — those are left out of the rates.)

• | OK — every finished run is at or above the threshold and none ended with an empty dataset.
• | ALERT: … — the violating runs and the threshold, e.g. | ALERT: run <id> ok=82.5%<90; run <id> empty; the script exits 1 on an alert. Next step: open the run in the Apify console — every failed query carries its own readable err.
• Flags: --hours sets the window (default 24), --min-ok-rate the alert threshold (default 90), --actor points the check at another actor id (default: the published one), --verbose prints the per-run detail — one line per run — instead of the one-line summary, and --demo-alert prints one fabricated ALERT line without a network call — a preview of the format (it exits 1, like a real alert).

Run it by hand:

$python3 tools/monitor.py --hours 24

To keep a daily record, append one line to your user crontab (08:30 in this example); the one-line verdict is then appended to .bench/monitor.log:

$30 8 * * * cd /path/to/indeed-jobs-scraper && python3 tools/monitor.py --hours 24 >> .bench/monitor.log 2>&1

Limits

• Indeed's primary feed pages up to maxRecords (200 max) per query; the search-page fallback is one page — about 45 records in the US, 15 outside it — and a smaller result is a complete answer, not a failure.
• The depth columns (description, applyUrl, jobType, isRemote plus the six company* fields) are read out of the search request itself: rows get as much depth as Indeed's feed carries for them — no per-job detail requests, and an absent field stays absent.
• Salary is whatever Indeed shows on its result card — no detail-page parsing.
• Location strings must be ones Indeed resolves (invalid ones still produce a normalized error row).