Versions follow MAJOR.MINOR.PATCH (src/version.py); Apify shows MAJOR.MINOR from .actor/actor.json.
Every run logs its version and records it in the RUN_STATS key-value record.
About future failures: these scrapers read each ATS's public job feed. ATS vendors change those feeds
without notice. If a company that used to work starts failing, or RUN_STATS shows skippedRecords,
suspect a site/format change first, not a regression in this code. Check the run log: it names the
company and says what didn't parse. One broken site never affects the other companies in a run.
- No new platforms: 18 more were checked and none passed. BambooHR, Jobvite, iCIMS, JazzHR, Pinpoint, Taleo,
SuccessFactors, Comeet, Freshteam, Homerun, JOIN, Dover, Gem, Polymer, Manatal, Zoho Recruit, Trakstar Hire and
JobScore were each checked for a documented public job feed, their terms of service and their robots.txt. Every one
failed at least one of these: the feed needs a key or token, is a private per-company link, or is documented only for
the company's own careers page, or the terms forbid automated access or reuse (the README FAQ quotes each reason).
Their URLs, and short forms such as
bamboohr:acme, are now recognised and reported in RUN_STATS.unresolved with
the reason. A careers page that links only to one of them says why instead of "no supported job board". Nothing is
requested from these platforms, and they're never guessed from a company name.
- README: says that jobs are fetched live during the run (no pre-crawled index), so every job returned is posted at
that moment.
- Workday, Workable, Rippling and Breezy HR are no longer read (10 → 6 platforms: Greenhouse, Lever, Ashby,
Recruitee, Personio, Teamtailor). A terms-of-service review of every platform kept only those that publish their job
feed for machines and whose terms allow this use. Workday career sites are only readable through the internal
backend of Workday's own career-site app, not a documented public API, and Workday's terms require a separate
agreement for its APIs. Workable's widget endpoint is documented only for embedding a company's jobs on its own
website. Breezy HR's terms forbid access other than through its publicly supported interfaces and name scraping.
Rippling's board endpoint isn't a documented public API and the review didn't clear it for automated third-party
use. Their URLs (and
workday:/workable:/rippling:/breezy: short forms) are still recognised and reported
in RUN_STATS.unresolved with the reason, like SmartRecruiters since 1.0.1; company names are no longer looked up
there. The rest of a run is unaffected.
- Default input: 6 small real boards, one per platform (Linear, bunq, Greenhouse, Outreach, ottonova, Teamtailor),
instead of examples that included a trial account and a 1-job board.
- Teamtailor: career sites with more than 100 open jobs return every job. The feed lists 100 at a time and only the
first 100 were read; the rest are now read page by page (Teamtailor's
offset parameter).
- Personio: jobs written only in another language keep their description. Personio's English feed leaves that text
out, so it's read from the company's own-language feed (one extra request, only for companies that need it).
"Full or part time" jobs now have no employment type (
null) instead of PART_TIME.
- Recruitee: salary from Recruitee's own fields comes back as numbers (Recruitee sends strings), and a range left at
0 is no salary instead of
0-0 with a currency.
- Shared safety code (
mms_common, the same in every actor of this portfolio): robots.txt per RFC 9309 (byte-order
marks, product-token matching, percent-encoding, a 500 KiB cap counted while streaming), a Crawl-delay above 30 s
reports that site instead of stalling the run, Retry-After honoured (seconds or a date; above 30 s reported),
the private-network guard on every request and redirect hop (resolve-and-check, then connect to the checked
address; ports 80 and 443 only; IPv6 forms that wrap an address, 6to4 and site-local addresses refused), a time
limit per attempt and one for the whole request (redirects, retries and robots.txt included), and decompression in
small steps with a capped size. robots.txt rules are matched in linear time, so a hostile file can't stall a run;
a robots.txt cut at the size cap loses its partial last line, and its retries honour Retry-After. A malformed
address or redirect target is reported for that company instead of crashing the run.
- Company names: a guess that fails for robots.txt or network reasons (e.g. an unknown Personio name redirecting to
Personio's site, whose robots.txt answered 429) now just means "not on that platform", with no retries and no
misleading robots.txt message; an unknown name is answered in seconds.
- A careers domain that doesn't exist is reported as such (it was reported as a robots.txt problem).
- Jobs reserved against the run's limit for a company whose job details robots.txt then disallows are given back to
the other companies in the run.
- Honest identity and robots.txt: requests now carry an identifying User-Agent
(
HumbleEchidnaApify/1.0 (+https://apify.com/humble-echidna)) instead of a desktop-browser one, and every host is
checked against its robots.txt first (job APIs, board pages and careers pages alike). A page a site disallows is
never requested: that company is reported as failed with the reason. A site's Crawl-delay is respected (Lever asks
for 1 second between requests, so large Lever runs are a little slower).
- SmartRecruiters is no longer read: its robots.txt disallows its job API for every automated client except
LinkedIn's. SmartRecruiters URLs are recognised and reported with that reason; names are no longer looked up there.
10 platforms remain. The default input drops its SmartRecruiters example.
- The ATS code now lives in a shared package (
common/mms_ats) that the single-platform
actors (Workday, Greenhouse, Lever, Ashby Jobs Scraper) also use, so a fix to one platform reaches all of them.
- An empty
companies list now also writes the RUN_STATS record before failing the run.
- The
proxy input is removed. Every platform read here is a public API that needs no proxy, and a proxied
connection can't be checked against private networks. Saved inputs and tasks that still contain proxy keep
working: the field is ignored, with a note in the log.
- A run where no company can be read now ends successfully with 0 jobs, with a status message saying so, instead
of failing: for example when every company is on SmartRecruiters (not read, see above) or no name could be found.
RUN_STATS (unresolved) and the log say why for each company.
- "Only jobs new since my last run": a job is remembered as returned only once it's in your dataset. A run that
stopped after collecting jobs but before writing them, or that cut them at the max cost per run, used to mark them
as seen, so later runs never returned them.
- One malformed company line (e.g.
http://[bad) is reported as unresolved with the reason and the rest of the run
continues; it used to crash the whole run.
RUN_STATS has the same keys on every exit path (dropped_invalid, failure and error included), so a run
that stops early is still readable.- Once every job the run can pay for is reserved, companies still being fetched are stopped right away instead of
finishing for nothing; if a company holding reserved jobs then fails, the stopped ones get another turn.
- A job record with a missing required field no longer uses up a place under max jobs / max cost per run.
- Personio and Teamtailor feeds are parsed with DTDs forbidden (no entity expansion, no external entities).
- Workday: when a title keyword itself matches more than Workday's 2,000-job list limit, the log says so.
First release.
- 11 ATS platforms: Workday, Greenhouse, Lever, Ashby, SmartRecruiters, Workable, Recruitee, Personio,
Rippling, Breezy, Teamtailor.
- "Max jobs per run" input, and the maximum cost per run is honoured: once either is covered, companies
still being fetched are cancelled and the rest are skipped (listed as
boardsSkipped in RUN_STATS).
- Workday: an empty first page is retried once, and a missing job total no longer stops paging (a busy board
could otherwise return 0 jobs with no error).
- Companies can be given as a name (looked up on every platform except Workday; only boards whose own
company name agrees are used), a careers page (scanned for a link to its job board), or a job-board URL.
What each resolved to, and what couldn't be found and why, is in the log and
RUN_STATS.
- Board slugs are normalized to lowercase (fixes capitalized Rippling URLs, which Rippling rejects, and stops
"Linear"/"linear" being fetched twice).
- Charged per job returned, through Apify's standard
apify-default-dataset-item event (no custom event, so
nothing is billed twice).
- One output schema for all of them; employment type (
FULL_TIME, PART_TIME, CONTRACT, TEMPORARY,
INTERNSHIP, VOLUNTEER) and workplace type (remote, hybrid, onsite) normalized across platforms.
- Salary: structured where the ATS provides it, otherwise extracted from pay-transparency text in the
description (
salarySource says which).
- Company display name filled in for Lever, Ashby and Rippling (their job APIs omit it).
- Filters: title keywords (pushed down to Workday's server-side search), exclude keywords, locations, remote,
posted within N days, max per company; "only jobs new since my last run" (remembered per company and per
filter combination in the user's own
ats-jobs-seen store; keys are restricted to Apify's allowed characters).
- Failure isolation: a failing company, a malformed job record, or a failed detail request only affects itself.
- Performance: parallel pagination (SmartRecruiters 4,800 jobs: 23s → ~3s) and descriptions fetched only for jobs
the run returns.
