WordPress Scraper: Posts, Pages & Categories via REST

WordPress Blog Scraper pulls posts, pages or any public custom post type from a WordPress site through its REST API, title, excerpt, full plain-text content, author, categories, tags, featured image and timestamps.

Roughly 43% of the web runs WordPress, and a stock install publishes its content for programmatic use. This Actor reads that directly: no browser, no proxies, no login.

Agent skill: SKILL.md

https://api.apify.com/v2/key-value-stores/t7YoTxpZEJOWvw4Ug/records/wordpress-blog-scraper.md

What you get

Output field	Meaning
site	Site the record came from
postId, slug, link	Identity and the public URL
title	Post title, entity-decoded
excerpt	Excerpt as clean plain text
contentPlain	Full body as clean plain text, HTML stripped, entities decoded
status	publish, and whatever else the site exposes publicly
author, authorName	Author ID, and the display name where the site publishes it
categories, tags	Slug lists, resolved from term IDs
publishedAt, modifiedAt	UTC ISO-8601 timestamps
featuredImage	Full-size featured image URL
commentCount	Comment count where the comments route is open, see the caveat below
scrapedAt	Run timestamp

A RUN_SUMMARY record in the key-value store holds per-run counts, the filters used, and any site that failed.

Common use cases

• Migration and archiving. Export a whole blog to structured JSON before a replatform.
• Competitive content analysis. Track what rivals publish, how often, and under which categories.
• Multi-site content index. Ingest dozens of publisher sites into one searchable dataset.
• Newsletter and aggregation. Pull the latest posts from a curated list of sites on a schedule.

Quick start

Two sites, the 100 most recent posts each:

{
 "sites": ["wordpress.org/news", "techcrunch.com"]
}

One category, published this year, capped:

{
 "sites": ["wordpress.org/news"],
 "categories": ["releases"],
 "sinceDate": "2026-01-01",
 "maxPostsPerSite": 50
}

Pages instead of posts:

{
 "sites": ["wordpress.org/news"],
 "postType": "pages",
 "maxPostsPerSite": 0
}

Input

Field	Type	Default	Notes
sites	array	-	Required. Domains, sub-directory installs (example.com/blog) or full article URLs.
postType	string	posts	posts, pages, or any public custom type registered with show_in_rest.
categories	array	[]	Category slugs. Resolved to term IDs automatically. posts only.
searchQuery	string	""	Full-text search, executed by the site.
sinceDate	string	""	2026-01-01 or a full ISO-8601 timestamp.
maxPostsPerSite	integer	100	Cap per site. 0 = whole archive.

categories, searchQuery and sinceDate all run server-side and combine with AND. Set maxPostsPerSite deliberately: a large publisher can have 250,000+ posts, and 0 means all of them.

Output example

{
 "site": "https://wordpress.org/news",
 "postId": 21248,
 "title": "WordPress 7.1 Release Candidate 1",
 "slug": "wordpress-7-1-release-candidate-1",
 "excerpt": "The first Release Candidate (\"RC1\") for WordPress 7.1 is ready for download and testing!",
 "contentPlain": "The first Release Candidate (\"RC1\") for WordPress 7.1 is ready…",
 "status": "publish",
 "author": 14679285,
 "authorName": "Benjamin Zekavica",
 "categories": ["development", "general", "releases"],
 "tags": ["7-1", "development", "release"],
 "publishedAt": "2026-08-05T15:37:33Z",
 "modifiedAt": "2026-08-05T15:37:33Z",
 "featuredImage": "https://wordpress.org/news/files/2026/07/wordpress-7-1-RC1.png",
 "commentCount": 0,
 "link": "https://wordpress.org/news/2026/08/wordpress-7-1-release-candidate-1/",
 "scrapedAt": "2026-08-06T12:00:00.000Z"
}

API example

curl -X POST "https://api.apify.com/v2/acts/arman-bd~wordpress-blog-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{
 "sites": ["wordpress.org/news"],
 "categories": ["releases"],
 "maxPostsPerSite": 25
 }'

JavaScript example

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_TOKEN' });
const run = await client.actor('arman-bd/wordpress-blog-scraper').call({
 sites: ['wordpress.org/news', 'techcrunch.com'],
 sinceDate: '2026-01-01',
 maxPostsPerSite: 100,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const post of items) console.log(`${post.publishedAt}, ${post.title}`);

Limits and behaviour

• It does not use ?_embed, on purpose. On a site with custom taxonomies, WordPress embeds whole post objects, including the entire yoast_head blob, for every related term. Measured on a large publisher, _embed returned 32 MB for 25 posts and blew the request timeout. This Actor requests only the fields it maps and resolves authors, terms and media in batched include= lookups instead: the same 100 posts come back in ~1.7 MB and under two seconds.
• A closed sub-route degrades. It does not fail. Plenty of publishers return 404 on /wp/v2/users or 403 on /wp/v2/comments. Those fields become null, a warning is logged, and the posts are still saved.
• commentCount is honest. Core WordPress publishes no comment count on the post object, so it is tallied from the comments collection. Where that route is closed, or holds more comments than can be counted in the page budget, the value is null rather than a guess.
• Bad sites don't kill the run. A disabled REST API, a security-plugin block or a DNS failure is recorded in RUN_SUMMARY.failures; the Actor only errors out if every site fails.
• HTML 404 pages are rejected. Sites that disable the REST API often answer with the themed 404 page at HTTP 200. The content type is checked, so that is treated as "REST disabled", not as data.
• Transient errors are retried. 429, 5xx and network faults get three attempts with exponential backoff and a 30-second timeout.
• Pagination is read, not guessed. Page count comes from the x-wp-totalpages response header.
• Public data only. No authentication, no personal data, no access-control bypass.

FAQ

Does it work on WordPress.com sites? Yes, WordPress.com serves the same /wp-json/wp/v2/ routes on custom-domain and *.wordpress.com sites.

What if a site disabled the REST API? It is reported in RUN_SUMMARY.failures with the reason, and the run continues with the other sites.

How do I find a category slug? Open a category archive on the site, /category/releases/ means the slug is releases. Unknown slugs are logged and skipped.

Can I scrape a custom post type? Yes, if it was registered with show_in_rest: true. Put its REST base in postType (for example events).

Why is authorName null on some sites? Those sites block /wp/v2/users, usually via a security plugin, or attribute posts through a plugin such as Co-Authors Plus. The numeric author ID is still returned.

Can I get only new posts? Run on a schedule with sinceDate set to your last run, or diff on postId plus modifiedAt.

Can I integrate it with something else? Yes, Apify API, client libraries, webhooks, scheduled runs, dataset exports (JSON/CSV/Excel) or MCP. Output is structured JSON.