Smart News Loader

Fetch any news article URL and extract its full content using adaptive, rotating HTTP methods. Automatically escalates from lightweight standard requests to advanced TLS impersonation, then reports exactly why a page failed if it cannot be loaded.

How It Works

For each URL, the actor tries HTTP methods in order from fastest/lightest to most capable

The first method that returns HTTP 200 with usable content wins. Once HTML is retrieved, content is extracted using trafilatura (best-in-class article extractor) with an HTML fallback using Open Graph tags and <p> text.

If all methods fail, the actor reports the exact reason - anti-bot protection, authentication, rate limits, network errors, and more.

Input

{
  "url": "https://www.theguardian.com/world/article/2026/...",
  "urls": [
    "https://apnews.com/article/...",
    "https://www.bbc.com/news/articles/..."
  ],
  "timeout": 20
}

Parameter	Type	Default	Description
url	string	-	Single article URL to fetch.
urls	array	[]	Multiple article URLs to process in sequence.
timeout	integer	20	Per-method request timeout in seconds (5–60).

Output

Successful extraction

{
  "url": "https://apnews.com/article/...",
  "status": "ok",
  "method_used": "httpx",
  "http_status": 200,
  "title": "Pope Leo XIV holds first public Mass at St. Peter's Square",
  "author": "Nicole Winfield",
  "date": "2026-04-19",
  "description": "Thousands gathered in St. Peter's Square...",
  "content": "VATICAN CITY (AP) - Thousands of faithful gathered...",
  "word_count": 742,
  "error": null
}

Blocked / protected page

{
  "url": "https://www.ft.com/content/...",
  "status": "blocked",
  "method_used": null,
  "http_status": 403,
  "title": "",
  "author": "",
  "date": "",
  "description": "",
  "content": "",
  "word_count": 0,
  "error": "Access blocked (HTTP 403) - anti-bot protection active (e.g. Cloudflare, DataDome, PerimeterX). All HTTP methods exhausted; browser automation (Playwright/Camoufox) may be required."
}

Other error states

status	Cause	Example error
ok	Successfully fetched and extracted	null
blocked	Anti-bot / auth protection on all methods	"Access blocked (HTTP 403) - ..."
error	URL gone, network failure, timeout	"Not found (HTTP 404) - URL may be invalid..."

Supported Failure Reasons

The actor identifies and explains these failure modes:

• HTTP 403 - Cloudflare, DataDome, PerimeterX, or other WAF; browser required
• HTTP 401 - Login / subscription paywall
• HTTP 429 - Rate limiting; back off before retrying
• HTTP 404 - Article deleted or URL invalid
• HTTP 410 - Article permanently removed
• SSL/TLS error - Certificate failure or TLS handshake rejection
• Timeout - Server unresponsive within the timeout window
• Connection failed - DNS failure, network unreachable, or domain offline

Notes

• Works on any publicly accessible news URL
• For hard-paywalled or JS-rendered sites, consider using a dedicated scraper with browser support
• word_count reflects the extracted article body, not the full page HTML