# The Muse Jobs Scraper (`scrapyx/themuse-jobs-scraper`) Actor

Scrapes job listings from TheMuse.com, a popular US career site pairing job listings with company-culture content. Search by keyword and optional location; returns company info, perks/benefits, employment type and posting date, with an optional full-description detail pass.

- **URL**: https://apify.com/scrapyx/themuse-jobs-scraper.md
- **Developed by:** [Ibnu Adzim](https://apify.com/scrapyx) (community)
- **Categories:** Jobs, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.35 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## The Muse Jobs Scraper (USA)

Scrapes job listings from **[TheMuse.com](https://www.themuse.com)** — a
popular US career site that pairs job listings with company-culture
content (perks, benefits, leadership attributes). Never covered by this
portfolio before.

Public data only. No login, no cookies, no browser.

### The one thing you need to know before using this

**No WAF was found** — 8 TLS profiles across search and detail surfaces,
all clean, cold. The real work is the data layer: TheMuse renders search
via Next.js App Router RSC streaming (`self.__next_f.push(...)`), but the
job **description** lives in neither JSON-LD nor the RSC flight tree — it
is server-rendered as plain HTML instead. This actor extracts it via a
CSS attribute-contains selector (`div[class*="mainContent"]`), the
lowest-priority tier in this portfolio's extraction hierarchy, used only
because nothing more stable carries that one field. See
[`CRAWLING_METHOD.md`](CRAWLING_METHOD.md) for the full trail.

### What you get

Three record types share one dataset, told apart by `recordType`.

#### `JOB` — one row per listing

Search rows (`listing`) already carry company info (name, logo, a long
perks/benefits list, leadership attributes), categories, seniority level,
employment type and posting date. Turn on **Fetch job detail pages** to
also attach `jobDescriptionHtml` — the full job posting, which is **not
present in search results at all**.

#### `SEARCH_SUMMARY` — one row per (keyword, location) query

Pages fetched, rows returned, upstream's own `count`, and
`locationApplied` — whether a requested location actually narrowed the
result set.

#### `ERROR` — one row per input that failed

So every entry in **Keywords** maps to at least one output row.

### Input

| Field | What it does |
| --- | --- |
| **Keywords** | free-text search (e.g. `'software engineer'`) — one search per entry |
| **Location filter** | optional, applies to every keyword. An unrecognised location is silently ignored by TheMuse (verified live) — this actor catches that by comparing against a keyword-only baseline and reports `locationApplied` honestly |
| **Fetch job detail pages** | adds `jobDescriptionHtml` (off by default — one extra request per job) |
| **Max jobs / max pages per search** | pagination caps — TheMuse pages honestly (verified to page 500 with no clamp) |
| **Max concurrent requests / Min seconds between requests** | tuned generously — no bot-mitigation gate observed |

#### Example

```json
{
  "keywords": ["software engineer", "product designer"],
  "location": "new-york-ny",
  "includeJobDetails": true,
  "maxItems": 100
}
```

### Notes on reliability

- **Bogus keyword → genuine zero results** (real full-text search, not a
  silent fallback).
- **Bogus location → SILENT-WIDEN**, a real trap verified live: `count`
  and the first page's job ids come back identical to the keyword-only
  baseline. This actor detects it by fetching that baseline once per
  keyword when a location is set, and reports `locationApplied: false`
  rather than pretending the filter worked.
- **`count` is display-capped at 10,000** for broad queries (same caveat
  as `zoopla-properties-scraper`'s `totalResultsWasLimited`) — not a
  structured true total, never used for pagination logic.
- **Pagination is genuinely honest**, verified live to page 500 of a large
  query with no clamp/repeat — the real `has_more` flag drives the stop
  condition (backed by a "no new ids" guard as a safety net regardless).
- **A dead/renamed job slug answers a clean HTTP 404** — the search row is
  still emitted, with `jobDescriptionHtml: null`.

### Output envelope

Every record carries `_input`, `_source` and `_scrapedAt`. Upstream field
names pass through **verbatim** under `listing` — no renaming.

See [`CRAWLING_METHOD.md`](CRAWLING_METHOD.md) for the full
reverse-engineering trail.

# Actor input Schema

## `keywords` (type: `array`):

Free-text keyword search (e.g. 'software engineer', 'product designer') -- one search per entry, each with its own SEARCH\_SUMMARY row.

## `location` (type: `string`):

Optional. Applies to every keyword in this run. An unrecognised location is silently ignored by TheMuse rather than erroring -- this actor detects that by comparing against a keyword-only baseline and reports it as locationApplied=false on the SEARCH\_SUMMARY row, rather than pretending the filter worked.

## `includeJobDetails` (type: `boolean`):

Also fetch each job's detail page for the full job description (server-rendered HTML). Search results already carry company info, perks/benefits, employment type, level and posting date -- but NOT the description at all. Off by default -- one extra request per job.

## `maxItems` (type: `integer`):

Stop paginating a search after this many jobs. Set to 0 for unlimited (still bounded by Max pages).

## `maxPages` (type: `integer`):

Hard cap on pagination depth, independent of maxItems (20 jobs/page). TheMuse pages honestly (verified live to page 500 with no clamp/repeat -- a real has\_more flag drives the stop condition), so this is purely a cost guard for large searches.

## `maxConcurrency` (type: `integer`):

Upper bound on requests in flight at once, across searches and detail fetches. No bot-mitigation gate was observed for this target in recon, so this can be raised more freely than most actors in this portfolio.

## `minRequestInterval` (type: `integer`):

Paces request starts (not held inside a concurrency slot) rather than raw concurrency. Kept small by default as a courtesy -- no request-velocity sensitivity was observed for this target.

## `proxyConfiguration` (type: `object`):

No WAF or bot-mitigation gate was observed for TheMuse.com in recon. Residential proxy pinned to the US is still the default, as this portfolio's standard baseline for a single-country public site.

## Actor input object example

```json
{
  "keywords": [
    "software engineer"
  ],
  "location": "",
  "includeJobDetails": false,
  "maxItems": 100,
  "maxPages": 20,
  "maxConcurrency": 5,
  "minRequestInterval": 1,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ],
    "apifyProxyCountry": "US"
  }
}
```

# Actor output Schema

## `items` (type: `string`):

One row per scraped record. See the dataset's default view for field definitions.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "keywords": [
        "software engineer"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("scrapyx/themuse-jobs-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "keywords": ["software engineer"] }

# Run the Actor and wait for it to finish
run = client.actor("scrapyx/themuse-jobs-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "keywords": [
    "software engineer"
  ]
}' |
apify call scrapyx/themuse-jobs-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,scrapyx/themuse-jobs-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/176qjmDmVxLabBa7h/builds/XMPago6arFP5Lu2ca/openapi.json