The Muse Jobs Scraper (USA)

Scrapes job listings from TheMuse.com — a popular US career site that pairs job listings with company-culture content (perks, benefits, leadership attributes). Never covered by this portfolio before.

Public data only. No login, no cookies, no browser.

The one thing you need to know before using this

No WAF was found — 8 TLS profiles across search and detail surfaces, all clean, cold. The real work is the data layer: TheMuse renders search via Next.js App Router RSC streaming (self.__next_f.push(...)), but the job description lives in neither JSON-LD nor the RSC flight tree — it is server-rendered as plain HTML instead. This actor extracts it via a CSS attribute-contains selector (div[class*="mainContent"]), the lowest-priority tier in this portfolio's extraction hierarchy, used only because nothing more stable carries that one field. See CRAWLING_METHOD.md for the full trail.

What you get

Three record types share one dataset, told apart by recordType.

JOB — one row per listing

Search rows (listing) already carry company info (name, logo, a long perks/benefits list, leadership attributes), categories, seniority level, employment type and posting date. Turn on Fetch job detail pages to also attach jobDescriptionHtml — the full job posting, which is not present in search results at all.

SEARCH_SUMMARY — one row per (keyword, location) query

Pages fetched, rows returned, upstream's own count, and locationApplied — whether a requested location actually narrowed the result set.

ERROR — one row per input that failed

So every entry in Keywords maps to at least one output row.

Input

Field	What it does
Keywords	free-text search (e.g. 'software engineer') — one search per entry
Location filter	optional, applies to every keyword. An unrecognised location is silently ignored by TheMuse (verified live) — this actor catches that by comparing against a keyword-only baseline and reports locationApplied honestly
Fetch job detail pages	adds jobDescriptionHtml (off by default — one extra request per job)
Max jobs / max pages per search	pagination caps — TheMuse pages honestly (verified to page 500 with no clamp)
Max concurrent requests / Min seconds between requests	tuned generously — no bot-mitigation gate observed

Example

{
  "keywords": ["software engineer", "product designer"],
  "location": "new-york-ny",
  "includeJobDetails": true,
  "maxItems": 100
}

Notes on reliability

• Bogus keyword → genuine zero results (real full-text search, not a silent fallback).
• Bogus location → SILENT-WIDEN, a real trap verified live: count and the first page's job ids come back identical to the keyword-only baseline. This actor detects it by fetching that baseline once per keyword when a location is set, and reports locationApplied: false rather than pretending the filter worked.
• count is display-capped at 10,000 for broad queries (same caveat as zoopla-properties-scraper's totalResultsWasLimited) — not a structured true total, never used for pagination logic.
• Pagination is genuinely honest, verified live to page 500 of a large query with no clamp/repeat — the real has_more flag drives the stop condition (backed by a "no new ids" guard as a safety net regardless).
• A dead/renamed job slug answers a clean HTTP 404 — the search row is still emitted, with jobDescriptionHtml: null.

Output envelope

Every record carries _input, _source and _scrapedAt. Upstream field names pass through verbatim under listing — no renaming.

See CRAWLING_METHOD.md for the full reverse-engineering trail.