WordPress Content Scraper

Scrapes any self-hosted WordPress site through its public wp-json/wp/v2 REST API — the same API WordPress core exposes, unauthenticated, on every install by default. No login, no cookies, no browser.

Point it at one or more site URLs and pick which content types to pull — posts, pages, media, comments, categories, tags, users, or even custom post types your theme/plugins registered. Every pick shares one client, so fetching several types from the same site costs no extra setup.

Why one actor, not one per content type

Every WordPress collection endpoint — built-in or custom — answers the identical shape:

GET {site}/wp-json/wp/v2/{resource}?page=N&per_page=M
-> 200, a JSON array, with X-WP-Total / X-WP-TotalPages response headers

posts, categories, comments, a WooCommerce product — they are not different modes of one linear chain, they are independent resources a user may reasonably want several of from the same site in the same run. Splitting that into N actors would re-pay the per-site setup N times for no benefit; the resources input is a checklist, not a mode switch.

What you get

Three record types share one dataset, told apart by recordType.

POST / PAGE / MEDIA / CATEGORY / TAG / COMMENT / USER / CUSTOM_<slug>

Upstream's JSON object, passed through verbatim — every field WordPress returns, unrenamed. With "Embed related objects" on (the default), each row also carries an _embedded object with the author, featured image and taxonomy terms already inlined, so you are not making a second request per row to resolve them.

SEARCH_SUMMARY — one row per (site, resource type)

WordPress's own true total (X-WP-Total) and page count (X-WP-TotalPages), how many items this run actually pulled, the filters that were sent, and whether the per-resource item cap was hit.

ERROR — one row per (site, resource) that failed, so nothing vanishes silently

Every failure is classified by why, not just that it failed:

_error	Meaning
rest_api_disabled	This install turned off unauthenticated REST access site-wide (HTTP 401). A site-owner setting, not a block.
forbidden	WordPress's own policy answer for this specific resource (HTTP 403 with a rest_* code) — e.g. comments turned off. Also not a block; retrying would not help.
not_found	The resource route doesn't exist on this install (wrong resource/custom-type slug, or the URL isn't a WordPress site at all).
bad_request	The query itself was malformed (shouldn't happen in normal use — the actor stops paging before this can occur).
fetch_failed	Genuine transient failure (DNS, timeout, connection reset, rate limiting) that persisted across every retry.
unexpected_shape	Upstream returned 200 but not a JSON array — the API may have changed.
invalid_input	The run's own input failed validation (e.g. perPage out of range) before any request was made.

Input

Field	What it does
WordPress site URLs	One or more base URLs, e.g. https://example.com
Content types to fetch	Any combination of posts / pages / media / comments / categories / tags / users
Custom post type REST base slugs	Extra show_in_rest post types your theme/plugins registered
Search query	WordPress's own full-text search param, applied to every selected resource
Published after / before	ISO 8601 date range (posts/pages/media/comments/custom types only — categories/tags/users have no date field)
Embed related objects	_embed=true — inlines author/featured-image/terms. On by default.
Items per request	per_page, capped at 100 by the API
Max items per (site, resource) pair	0 = unlimited (walks every page WordPress reports)
Max concurrent requests	Across every site and resource this run touches
Min seconds between request starts, per site	Politeness pacing applied separately to each site — see "Notes on reliability"

Example

{
  "startUrls": ["https://wptavern.com"],
  "resources": ["posts", "categories", "tags"],
  "search": "gutenberg",
  "maxItemsPerResource": 100,
  "perPage": 100
}

Notes on reliability

• No WAF observed anywhere tested — including sites sitting behind Cloudflare. The retry/rotation ladder exists for transport flakiness and real (if occasional) rate-limiting, not bot mitigation — see CRAWLING_METHOD.md.
• WordPress's own 401/403 policy answers are told apart from blocks. A site that disabled its REST API, or turned off comments, gets one clear diagnostic row instead of four wasted retries and a generic failure.
• Pagination is honest on every site tested: X-WP-TotalPages is trustworthy, per_page/page out-of-range answers are a clean 400 (not a silent clamp or a re-served page 1), and a bogus filter value returns an honest zero rather than the unfiltered baseline.
• Per-site politeness pacing, not global. Inputs here are arbitrary user-supplied sites of unknown scale — a small blog on shared hosting next to a large outlet on enterprise infrastructure — so the pacing floor applies separately to each site rather than being shared across unrelated domains.
• Public content only, by construction. An unauthenticated request can never see drafts, private posts, or unapproved comments — there is nothing to accidentally over-collect.

Known limits

• No per-site taxonomy filtering by name. WordPress's categories/ tags filters take numeric IDs, which differ on every install, so a generic multi-site actor can't ship a fixed slug list the way a single-publisher scraper can. Use Search query or the Published after/before date range instead, or fetch the categories/tags resource first to look up a site's own IDs.
• A site can turn its REST API off entirely, or disable specific resources (comments is the common one). That surfaces as a clear ERROR row (rest_api_disabled / forbidden), not a silent empty result — but there is no bypass, by design: this is a deliberate site-owner setting, not a technical restriction to route around.
• Custom post type slugs must actually be registered with show_in_rest. A guessed slug that isn't gets a clean not_found row for that resource — check the site's own /wp-json/ route index if unsure which custom types it exposes.
• You are responsible for using this actor consistent with each target site's terms of use and robots.txt — it queries whatever URLs you provide via WordPress's own public API and applies no site-specific policy of its own.