Website Content Crawler | Markdown for RAG

Build a content collection from public newsrooms, blogs and documentation sites. Supply known page URLs for direct extraction, or discover pages through sitemaps and feeds. Export Markdown or plain text with page metadata and content hashes for your search index, RAG pipeline or content-monitoring workflow. The crawler respects robots.txt and stays within the supplied website host. It does not sign in or bypass access controls.

Best first run: open the Website Content Crawler, start with one public documentation page and a $0.01 cap, then expand to sitemap or feed discovery after confirming the Markdown fits your RAG pipeline.

Workflow: website content to Markdown for RAG

1. Open the published direct-extraction example and inspect one public documentation page. Keep the page limit at one and the example run cap at $0.01.
2. Confirm the Markdown contains the content you need, then replace the URL with a permitted page from your knowledge source. Use sitemap/feed discovery when you need multiple pages.
3. Export content and canonical URLs to your own indexing pipeline. Use content hashes to compare snapshots before deciding what to re-index. This Actor prepares content; it does not create embeddings or host a vector database.

Pricing is approximately $1.5625 per 1,000 successful pages, plus $0.00005 per start event per GB of memory (minimum one). Check the Pricing tab for the current rate.

Best for

• Building a clean, permission-aware knowledge base from a public newsroom, blog, or documentation site
• Extracting a known set of pages cheaply with direct URL mode
• Monitoring content changes using contentHash, ETags, and Last-Modified values

What it is for

• Build a reliable content library from websites and publisher sources.
• Monitor a knowledge base, blog, newsroom, documentation hub, or resource center.
• Feed clean website content into search, RAG, analysis, or editorial workflows.
• Track content changes with content hashes, ETags, and Last-Modified values.
• Collect only a section of a site, such as /news, /guides, or /research.

What it does

• Starts with published sitemaps and RSS/Atom feeds to find the right pages efficiently.
• Can fetch only the supplied URLs when direct URL mode is selected, avoiding discovery requests for fast, low-cost page extraction.
• Stays on the same website host you supplied.
• Respects each website’s robots.txt rules before requesting content.
• Keeps successful content pages separate from skipped URLs and source issues.
• Gives every saved page clean content, title, page metadata, canonical URL, fetch date, and change-tracking details.
• Works with public pages only. It does not sign in, use proxies, or try to work around access controls.

Get started

1. Add one or more website or section URLs.
2. Set the maximum number of content pages you need.
3. Optionally include or exclude paths to focus the crawl.
4. Choose site discovery or direct URL mode.
5. Choose Markdown or plain text, then run the Actor.

For a small first run, extract this public documentation page directly:

{
  "sourceUrls": [{ "url": "https://docs.apify.com/actors/development/actor-definition/actor-json" }],
  "discoveryMode": "direct",
  "maxPages": 1,
  "maxConcurrency": 1,
  "contentFormat": "markdown"
}

This input returned useful content in a validation run; live pages can change. Keep maxPages small while checking the content you need. The following discovery example is a template: replace example.com/guides with your own public section URL.

For API clients, start with the same one-page request and set a maximum run charge in the Console before expanding the crawl:

curl -X POST "https://api.apify.com/v2/acts/xtech~news-source-crawler/runs?token=$APIFY_TOKEN&maxTotalChargeUsd=0.01" \
  -H "Content-Type: application/json" \
  -d '{"sourceUrls":[{"url":"https://docs.apify.com/actors/development/actor-definition/actor-json"}],"discoveryMode":"direct","maxPages":1,"maxConcurrency":1,"contentFormat":"markdown"}'

After confirming the first page, move to a bounded monitoring run by changing only the source URL and page limit:

{
  "sourceUrls": [{ "url": "https://example.com/docs" }],
  "includePaths": ["/docs"],
  "maxPages": 25,
  "contentFormat": "markdown"
}

This two-step workflow keeps the first paid run cheap and makes the next scheduled or API run repeatable.

For example, to collect a website’s guide pages:

{
  "sourceUrls": [{ "url": "https://example.com/guides" }],
  "includePaths": ["/guides"],
  "excludePaths": ["/guides/archive"],
  "maxPages": 200,
  "contentFormat": "markdown"
}

For a known article, use direct URL mode to skip sitemap and feed discovery:

{
  "sourceUrls": [{ "url": "https://blog.apify.com/introducing-x402-agentic-payments/" }],
  "discoveryMode": "direct",
  "maxPages": 1,
  "contentFormat": "markdown"
}

For a newsroom section with a small first pass:

{
  "sourceUrls": [{ "url": "https://example.com/news" }],
  "includePaths": ["/news"],
  "excludePaths": ["/news/archive"],
  "maxPages": 25,
  "contentFormat": "markdown"
}

What you get

Each saved row is one clean public content page. It includes:

• Page URL and canonical URL
• Title and useful page metadata
• Markdown or plain-text content
• Word count and a content hash for change detection
• ETag and Last-Modified values when the website provides them
• Fetch timestamp and discovery details, including whether it came from a sitemap, feed, start page, or supplied URL

The default dataset contains only successful content pages. The separate Source results dataset explains skipped pages, robots.txt decisions, path filters, and source errors, so your usable export stays clean.

Example output

{
  "schemaVersion": 1,
  "recordType": "content-page",
  "url": "https://example.com/news/launch",
  "canonicalUrl": "https://example.com/news/launch",
  "title": "Product launch",
  "contentFormat": "markdown",
  "content": "# Product launch\n\nClean page content...",
  "wordCount": 842,
  "contentHash": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
  "etag": null,
  "lastModified": null,
  "fetchedAt": "2026-08-13T12:00:00.000Z",
  "provenance": {
    "sourceRootUrl": "https://example.com/news",
    "discoveryMethod": "sitemap",
    "discoveryUrl": "https://example.com/sitemap.xml",
    "requestedUrl": "https://example.com/news/launch",
    "finalUrl": "https://example.com/news/launch",
    "robotsStatus": "ALLOWED",
    "contentType": "text/html"
  }
}

Tips

• Add a sitemap URL when you know it for the most complete coverage.
• Use a section URL and an include path to focus on a single topic area.
• For recurring monitoring, schedule the Actor and compare contentHash, etag, or lastModified between runs.
• Start with a small page limit when trying a new source, then raise it once you are happy with the coverage.

Troubleshooting

• No pages saved: inspect the Source results dataset for robots, path-filter, HTTP, or content-type decisions.
• A known page is missing: use discoveryMode: "direct" with that URL and check whether the source permits it in robots.txt.
• The crawl is broader than expected: set includePaths and excludePaths, and start with a low maxPages value.
• The output is not suitable for your pipeline: choose contentFormat: "markdown" for publishing/RAG or "text" for simpler search and analysis.

Related research workflows

• Google Ads Transparency Scraper — compare public ad activity with the content a brand publishes.
• Google Keyword Scraper Pro — compare search interest with news coverage.

Pricing and run budget

At the current base price, one saved content page costs $0.001562536477 (about $1.56 per 1,000 pages). A run-start fee of $0.00005 per allocated GB of memory applies, with a minimum of one event. Discounts, when available, are shown in the Console.

Start with one direct page and a $0.01 maximum run charge. Set the spending limit in the Console or pass maxTotalChargeUsd when starting through the API. Skipped pages, robots.txt decisions, and source errors are kept in Source results rather than billed as content pages.