# WordPress Content Scraper (`scrapyx/wordpress-content-scraper`) Actor

Scrapes any self-hosted WordPress site via its public REST API — posts, pages, media, categories, tags, comments, users, even custom post types. No login. Raw JSON passthrough, honest pagination, and a clear error row for sites that disable their REST API.

- **URL**: https://apify.com/scrapyx/wordpress-content-scraper.md
- **Developed by:** [Ibnu Adzim](https://apify.com/scrapyx) (community)
- **Categories:** News, AI, Marketing
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.10 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## WordPress Content Scraper

Scrapes **any self-hosted WordPress site** through its public
`wp-json/wp/v2` REST API — the same API WordPress core exposes,
unauthenticated, on every install by default. No login, no cookies, no
browser.

Point it at one or more site URLs and pick which content types to pull —
posts, pages, media, comments, categories, tags, users, or even custom post
types your theme/plugins registered. Every pick shares one client, so
fetching several types from the same site costs no extra setup.

### Why one actor, not one per content type

Every WordPress collection endpoint — built-in or custom — answers the
identical shape:

```
GET {site}/wp-json/wp/v2/{resource}?page=N&per_page=M
-> 200, a JSON array, with X-WP-Total / X-WP-TotalPages response headers
```

`posts`, `categories`, `comments`, a WooCommerce `product` — they are not
different *modes* of one linear chain, they are independent resources a
user may reasonably want several of from the same site in the same run.
Splitting that into N actors would re-pay the per-site setup N times for no
benefit; the `resources` input is a checklist, not a mode switch.

### What you get

Three record types share one dataset, told apart by `recordType`.

#### `POST` / `PAGE` / `MEDIA` / `CATEGORY` / `TAG` / `COMMENT` / `USER` / `CUSTOM_<slug>`

Upstream's JSON object, **passed through verbatim** — every field WordPress
returns, unrenamed. With "Embed related objects" on (the default), each row
also carries an `_embedded` object with the author, featured image and
taxonomy terms already inlined, so you are not making a second request per
row to resolve them.

#### `SEARCH_SUMMARY` — one row per (site, resource type)

WordPress's own true total (`X-WP-Total`) and page count
(`X-WP-TotalPages`), how many items this run actually pulled, the filters
that were sent, and whether the per-resource item cap was hit.

#### `ERROR` — one row per (site, resource) that failed, so nothing vanishes silently

Every failure is classified by *why*, not just *that* it failed:

| `_error` | Meaning |
| --- | --- |
| `rest_api_disabled` | This install turned off unauthenticated REST access site-wide (HTTP 401). A site-owner setting, not a block. |
| `forbidden` | WordPress's own policy answer for this specific resource (HTTP 403 with a `rest_*` code) — e.g. comments turned off. Also not a block; retrying would not help. |
| `not_found` | The resource route doesn't exist on this install (wrong resource/custom-type slug, or the URL isn't a WordPress site at all). |
| `bad_request` | The query itself was malformed (shouldn't happen in normal use — the actor stops paging before this can occur). |
| `fetch_failed` | Genuine transient failure (DNS, timeout, connection reset, rate limiting) that persisted across every retry. |
| `unexpected_shape` | Upstream returned 200 but not a JSON array — the API may have changed. |
| `invalid_input` | The run's own input failed validation (e.g. `perPage` out of range) before any request was made. |

### Input

| Field | What it does |
| --- | --- |
| **WordPress site URLs** | One or more base URLs, e.g. `https://example.com` |
| **Content types to fetch** | Any combination of posts / pages / media / comments / categories / tags / users |
| **Custom post type REST base slugs** | Extra `show_in_rest` post types your theme/plugins registered |
| **Search query** | WordPress's own full-text `search` param, applied to every selected resource |
| **Published after / before** | ISO 8601 date range (posts/pages/media/comments/custom types only — categories/tags/users have no date field) |
| **Embed related objects** | `_embed=true` — inlines author/featured-image/terms. On by default. |
| **Items per request** | `per_page`, capped at 100 by the API |
| **Max items per (site, resource) pair** | `0` = unlimited (walks every page WordPress reports) |
| **Max concurrent requests** | Across every site and resource this run touches |
| **Min seconds between request starts, per site** | Politeness pacing applied separately to each site — see "Notes on reliability" |

#### Example

```json
{
  "startUrls": ["https://wptavern.com"],
  "resources": ["posts", "categories", "tags"],
  "search": "gutenberg",
  "maxItemsPerResource": 100,
  "perPage": 100
}
```

### Notes on reliability

- **No WAF observed anywhere tested** — including sites sitting behind
  Cloudflare. The retry/rotation ladder exists for transport flakiness and
  real (if occasional) rate-limiting, not bot mitigation — see
  `CRAWLING_METHOD.md`.
- **WordPress's own 401/403 policy answers are told apart from blocks.** A
  site that disabled its REST API, or turned off comments, gets one clear
  diagnostic row instead of four wasted retries and a generic failure.
- **Pagination is honest on every site tested**: `X-WP-TotalPages` is
  trustworthy, per\_page/page out-of-range answers are a clean 400 (not a
  silent clamp or a re-served page 1), and a bogus filter value returns an
  honest zero rather than the unfiltered baseline.
- **Per-site politeness pacing, not global.** Inputs here are arbitrary
  user-supplied sites of unknown scale — a small blog on shared hosting
  next to a large outlet on enterprise infrastructure — so the pacing floor
  applies separately to each site rather than being shared across unrelated
  domains.
- **Public content only, by construction.** An unauthenticated request can
  never see drafts, private posts, or unapproved comments — there is
  nothing to accidentally over-collect.

### Known limits

- **No per-site taxonomy filtering by name.** WordPress's `categories`/
  `tags` filters take numeric IDs, which differ on every install, so a
  generic multi-site actor can't ship a fixed slug list the way a
  single-publisher scraper can. Use **Search query** or the **Published
  after/before** date range instead, or fetch the `categories`/`tags`
  resource first to look up a site's own IDs.
- **A site can turn its REST API off entirely**, or disable specific
  resources (comments is the common one). That surfaces as a clear `ERROR`
  row (`rest_api_disabled` / `forbidden`), not a silent empty result — but
  there is no bypass, by design: this is a deliberate site-owner setting,
  not a technical restriction to route around.
- **Custom post type slugs must actually be registered with
  `show_in_rest`.** A guessed slug that isn't gets a clean `not_found` row
  for that resource — check the site's own `/wp-json/` route index if
  unsure which custom types it exposes.
- You are responsible for using this actor consistent with each target
  site's terms of use and `robots.txt` — it queries whatever URLs you
  provide via WordPress's own public API and applies no site-specific
  policy of its own.

# Actor input Schema

## `startUrls` (type: `array`):

Base URL of each self-hosted WordPress site to scrape, e.g. `https://example.com` or `example.com/blog`. Every site is queried through its own public `/wp-json/wp/v2/` REST API -- the same one WordPress core exposes by default, with no login. A site that has disabled its REST API, or is not WordPress at all, gets a single ERROR row explaining why rather than stopping the whole run.

## `resources` (type: `array`):

Which built-in WordPress collections to pull from each site, in the SAME run -- these are independent resources sharing one client, not exclusive modes, so picking several is normal (e.g. posts + categories + tags together). Each gets its own SEARCH\_SUMMARY row per site.

## `customPostTypes` (type: `array`):

Extra REST base slugs for non-core content this site's theme/plugins registered with `show_in_rest` (e.g. a WooCommerce `product`, a portfolio `project`). Fetched exactly like the built-in resources above, under a `CUSTOM_<SLUG>` record type. A slug that is not actually registered on the site fails with a clear `not_found` ERROR row for just that resource, not the whole site -- check the site's `/wp-json/` route index first if unsure.

## `search` (type: `string`):

Full-text search applied to every selected resource on every site, using WordPress's own `search` parameter. Leave empty to fetch each resource's full listing (newest first) instead.

## `after` (type: `string`):

ISO 8601 datetime, e.g. `2026-01-01T00:00:00`. Only applies to date-bearing resources (posts, pages, media, comments, custom post types) -- silently skipped for categories/tags/users, which have no date field. Leave empty for no lower bound.

## `before` (type: `string`):

ISO 8601 datetime. Same scope as 'Published after'. Leave empty for no upper bound.

## `embed` (type: `boolean`):

Adds WordPress's own `_embed=true` flag, which inlines the author, featured image and taxonomy terms directly into each post/page/media/comment row under `_embedded` -- avoiding a separate request per related object. On by default; turn off only to shrink payload size.

## `perPage` (type: `integer`):

WordPress's own `per_page` parameter. Hard-capped at 100 by the API -- any higher value fails that resource's fetch with a 400.

## `maxItemsPerResource` (type: `integer`):

Stop paging one resource on one site after this many items. Set to 0 for unlimited (walks every page WordPress reports via X-WP-TotalPages). With multiple sites/resources selected, this cap applies separately to each pair.

## `maxConcurrency` (type: `integer`):

Upper bound on requests in flight at once, across every site and resource this run touches.

## `minRequestInterval` (type: `number`):

Pacing floor applied separately to each site (not shared globally across different sites), independent of 'Max concurrent requests'. No rate-limit or WAF gate was observed on any site tested, but inputs here are arbitrary user-supplied installs -- some may be small blogs on modest shared hosting, unlike this portfolio's usual large fixed targets. Set to 0 to disable.

## `proxyConfiguration` (type: `object`):

The WordPress REST API is public by design and no bot-mitigation gate was observed on any site tested, including ones behind Cloudflare. Apify's automatic proxy is the cloud default; residential is not forced since this actor targets arbitrary sites worldwide rather than one region.

## Actor input object example

```json
{
  "startUrls": [
    "https://wptavern.com"
  ],
  "resources": [
    "posts"
  ],
  "customPostTypes": [],
  "search": "",
  "after": "",
  "before": "",
  "embed": true,
  "perPage": 100,
  "maxItemsPerResource": 200,
  "maxConcurrency": 5,
  "minRequestInterval": 0.5,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `items` (type: `string`):

One row per scraped record. See the dataset's default view for field definitions.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://wptavern.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("scrapyx/wordpress-content-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": ["https://wptavern.com"] }

# Run the Actor and wait for it to finish
run = client.actor("scrapyx/wordpress-content-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://wptavern.com"
  ]
}' |
apify call scrapyx/wordpress-content-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,scrapyx/wordpress-content-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/3yq0LRXKcwf17hdeA/builds/XDZNIgsZ353TnYx0H/openapi.json