News Source Crawler

Turn public website content into a clean, ready-to-use dataset.

Add a website, section, or exact page URL and this Actor returns useful content in Markdown or plain text, together with the page details you need for monitoring, analysis, publishing workflows, and AI applications. Use site discovery for broad coverage, or direct URL mode when you already know the pages to fetch.

Best for

• Building a clean, permission-aware knowledge base from a public newsroom, blog, or documentation site
• Extracting a known set of pages cheaply with direct URL mode
• Monitoring content changes using contentHash, ETags, and Last-Modified values

What it is for

• Build a reliable content library from websites and publisher sources.
• Monitor a knowledge base, blog, newsroom, documentation hub, or resource center.
• Feed clean website content into search, RAG, analysis, or editorial workflows.
• Track content changes with content hashes, ETags, and Last-Modified values.
• Collect only a section of a site, such as /news, /guides, or /research.

What it does

• Starts with published sitemaps and RSS/Atom feeds to find the right pages efficiently.
• Can fetch only the supplied URLs when direct URL mode is selected, avoiding discovery requests for fast, low-cost page extraction.
• Stays on the same website host you supplied.
• Respects each website’s robots.txt rules before requesting content.
• Keeps successful content pages separate from skipped URLs and source issues.
• Gives every saved page clean content, title, page metadata, canonical URL, fetch date, and change-tracking details.
• Works with public pages only. It does not sign in, use proxies, or try to work around access controls.

Get started

1. Add one or more website or section URLs.
2. Set the maximum number of content pages you need.
3. Optionally include or exclude paths to focus the crawl.
4. Choose site discovery or direct URL mode.
5. Choose Markdown or plain text, then run the Actor.

For example, to collect a website’s guide pages:

{
  "sourceUrls": [{ "url": "https://example.com/guides" }],
  "includePaths": ["/guides"],
  "excludePaths": ["/guides/archive"],
  "maxPages": 200,
  "contentFormat": "markdown"
}

For a known article, use direct URL mode to skip sitemap and feed discovery:

{
  "sourceUrls": [{ "url": "https://blog.apify.com/introducing-x402-agentic-payments/" }],
  "discoveryMode": "direct",
  "maxPages": 1,
  "contentFormat": "markdown"
}

For a newsroom section with a small first pass:

{
  "sourceUrls": [{ "url": "https://example.com/news" }],
  "includePaths": ["/news"],
  "excludePaths": ["/news/archive"],
  "maxPages": 25,
  "contentFormat": "markdown"
}

What you get

Each saved row is one clean public content page. It includes:

• Page URL and canonical URL
• Title and useful page metadata
• Markdown or plain-text content
• Word count and a content hash for change detection
• ETag and Last-Modified values when the website provides them
• Fetch timestamp and discovery details, including whether it came from a sitemap, feed, start page, or supplied URL

The default dataset contains only successful content pages. The separate Source results dataset explains skipped pages, robots.txt decisions, path filters, and source errors, so your usable export stays clean.

Example output

{
  "schemaVersion": 1,
  "recordType": "content-page",
  "url": "https://example.com/news/launch",
  "canonicalUrl": "https://example.com/news/launch",
  "title": "Product launch",
  "contentFormat": "markdown",
  "content": "# Product launch\n\nClean page content...",
  "wordCount": 842,
  "contentHash": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
  "etag": null,
  "lastModified": null,
  "fetchedAt": "2026-08-13T12:00:00.000Z",
  "provenance": {
    "sourceRootUrl": "https://example.com/news",
    "discoveryMethod": "sitemap",
    "discoveryUrl": "https://example.com/sitemap.xml",
    "requestedUrl": "https://example.com/news/launch",
    "finalUrl": "https://example.com/news/launch",
    "robotsStatus": "ALLOWED",
    "contentType": "text/html"
  }
}

Tips

• Add a sitemap URL when you know it for the most complete coverage.
• Use a section URL and an include path to focus on a single topic area.
• For recurring monitoring, schedule the Actor and compare contentHash, etag, or lastModified between runs.
• Start with a small page limit when trying a new source, then raise it once you are happy with the coverage.

Troubleshooting

• No pages saved: inspect the Source results dataset for robots, path-filter, HTTP, or content-type decisions.
• A known page is missing: use discoveryMode: "direct" with that URL and check whether the source permits it in robots.txt.
• The crawl is broader than expected: set includePaths and excludePaths, and start with a low maxPages value.
• The output is not suitable for your pipeline: choose contentFormat: "markdown" for publishing/RAG or "text" for simpler search and analysis.

Related research workflows

• Google Ads Transparency Scraper — compare public ad activity with the content a brand publishes.
• Google Keyword Scraper Pro — compare search interest with news coverage.