Washington Post News Scraper

Extract Washington Post articles — headline, byline, authors, publication date, section, tags, lead image and article text — from washingtonpost.com. Give it URLs, or let it pull the latest stories from any section.

Why Use This Actor?

• Complete article text. Returns the full body the site renders for a story — commonly 5,000–30,000 characters on long-form reporting, not just the opening paragraph.
• Clean, typed output. Authors as a list, ISO timestamps, section and tags as separate fields — ready for a database or an analysis notebook without post-processing.
• Discovery built in. mode: "latest" walks the sections you name (and the public RSS feeds) and collects recent article URLs for you, so you don't have to supply a list.
• Honest about what it got. Every row carries contentChars and isTruncated, so a teaser is never silently mistaken for a full article.
• No browser. Plain HTTP requests. Runs finish in seconds and stay well under 256 MB of RAM, which keeps them cheap.

How It Works

The actor requests the article page, reads the page's own data payload, and then re-requests the same story through the site's Next.js data route to collect the complete content_elements array. That build identifier changes every time the Post deploys, so it is read fresh from each article rather than hard-coded — no pinned value to go stale.

Requests carry a browser-accurate TLS fingerprint, so no browser engine is needed. If one fingerprint stops being accepted, the actor advances through a ladder of alternatives instead of failing the run.

Input

Parameter	Type	Default	Description
mode	string	article	article scrapes only the URLs you pass. latest discovers recent articles from sections first.
startUrls	array	–	Washington Post article URLs.
sections	array	["business","technology"]	Sections to pull from in latest mode — e.g. business, technology, politics, world, opinions, sports, climate-environment.
maxItems	integer	50	Maximum articles per run.
proxyConfiguration	object	none	Optional. The site answers plain HTTP from most IPs; a US residential proxy helps if you hit rate limits at volume.

Example input

{
  "mode": "latest",
  "sections": ["technology", "business"],
  "maxItems": 25
}

Output

One row per article:

{
  "url": "https://www.washingtonpost.com/technology/2026/08/24/homelessness-billionaire-crusade/",
  "canonicalUrl": "https://www.washingtonpost.com/technology/2026/08/24/homelessness-billionaire-crusade/",
  "source": "The Washington Post",
  "articleId": "6NOMLKZTVBHORPXRREFP4BBQRU",
  "title": "How one billionaire's crusade against homelessness reshaped a city",
  "subheadline": null,
  "description": "The tech investor spent years and millions of dollars...",
  "authors": ["Elizabeth Dwoskin"],
  "publishedAt": "2026-08-24T16:00:57.673Z",
  "updatedAt": "2026-08-24T18:22:10.004Z",
  "section": "Technology",
  "tags": ["homelessness", "san francisco", "philanthropy"],
  "leadImage": "https://www.washingtonpost.com/wp-apps/imrs.php?src=...",
  "content": "California lawmakers are poised to pass some of the country's most significant restrictions...",
  "contentChars": 30878,
  "paragraphCount": 98,
  "isTruncated": false,
  "_source": "next_data_route",
  "_scrapedAt": "2026-08-26T11:44:02.113Z"
}

Field reference

Field	Type	Description
title	string	Article headline.
authors	string[]	Bylined authors. Empty for wire copy and unsigned editorials.
publishedAt / updatedAt	string	ISO-8601 timestamps.
section	string	Primary section, e.g. Technology.
tags	string[]	Editorial tags.
content	string	Article body text, paragraphs separated by blank lines.
contentChars	integer	Length of content. Use it to filter out stubs.
paragraphCount	integer	Number of prose blocks recovered.
isTruncated	boolean	true when only the teaser was reachable for that story.
_source	string	Which render supplied the body — next_data_route or article_page.
_error	string	Present only on failures (not_found, blocked, no_next_data, http_*).

Known Limits

• Some stories are genuinely short. Sports listings, briefs and photo posts return a few hundred characters because that is the whole item — check contentChars rather than assuming a failure.
• A small share of articles return the teaser only. Those rows are flagged isTruncated: true and are not padded with invented text.
• Live blogs and interactive features use custom embeds that carry little prose; expect a low contentChars on those.
• Discovery is recency-based. latest mode reaches current section fronts and RSS, not a historical archive; to go back in time, supply URLs.
• Volume. At a few hundred articles per run a US residential proxy is recommended to avoid rate limiting.

Scope & Compliance

• Public content only. This actor requests pages the same way an ordinary anonymous visitor's browser does. It uses no login, no subscriber credentials, no cookies from a paid account, and does not attempt to obtain content the publisher withholds from anonymous visitors.
• Subscriber-only material is not retrieved. Where the site serves only a headline, summary or intro to anonymous visitors, that is what the actor returns — and it flags the result via isTruncated so your pipeline can tell the difference.
• No security control is defeated. The actor sends ordinary HTTPS requests with a browser-accurate TLS fingerprint so it does not need a browser engine. It solves no CAPTCHAs and forges no authentication.
• Copyright stays with the publisher. Output is intended for research, monitoring, archiving and analysis. You are responsible for using it in line with the publisher's Terms of Service, copyright law and any applicable data-protection rules — republishing article text is generally not permitted.
• Rate limits are respected. Requests are paced and retried politely; the actor is not designed for denial-of-service-scale traffic.

Related Actors

Actor	Source
nytimes-scraper	New York Times
reuters-scraper	Reuters
ft-scraper	Financial Times
forbes-scraper	Forbes
guardian-scraper	The Guardian