Sitemap Content Extractor — Crawl a Sitemap, Extract Clean Content

Point this Actor at any website's sitemap.xml and it crawls every listed URL and returns clean full-text content plus metadata for each page — title, meta description, keywords, H1 headings, word count, and last-modified date. Perfect for SEO audits, content inventories, site migrations, and building AI training corpora from documentation sites.

No browser automation to configure, no page-by-page URL lists to maintain — the sitemap is the input.

Free to use — no per-page charges, no start fee. You only pay Apify's standard platform usage for the compute your run consumes.

Why this Actor

• Sitemap index aware — handles both plain sitemaps (<urlset>) and sitemap indexes (<sitemapindex>), recursively following every child sitemap.
• Gzip support — reads .xml.gz sitemaps transparently.
• Clean extraction — uses trafilatura to strip nav/ads/boilerplate and return just the article text.
• Precise scoping — include/exclude URL regex patterns so you crawl only /blog/ or skip /tag/ pages.
• Structured output — one dataset item per page, ready for search indexing, embeddings, or a content spreadsheet.

How it works

Give it a sitemapUrl. The Actor fetches and parses the sitemap (following index files and gzip automatically), applies your include/exclude filters, then visits up to maxUrls pages and extracts clean content and metadata from each. Set extractContent: false to inventory URLs and metadata only (faster, no page fetches).

Budget semantics: maxUrls is a hard cap on pages processed, applied after filtering — the Actor keeps scanning child sitemaps until it has collected maxUrls URLs that match your includePatterns (or the sitemap index is exhausted), so a broad pattern can never push the crawl over your budget, a narrow pattern still fills it where matches exist, and duplicate URLs across child sitemaps are collected once.

Input

{
  "sitemapUrl": "https://apify.com/sitemap.xml",
  "maxUrls": 50,
  "extractContent": true,
  "includePatterns": ["/blog/"],
  "excludePatterns": ["/tag/", "/author/"],
  "proxyConfiguration": { "useApifyProxy": true }
}

Field	Type	Required	Default	Description
sitemapUrl	string	Yes	—	URL to sitemap.xml or a sitemap index
maxUrls	integer	No	50	Maximum URLs to process
extractContent	boolean	No	true	Fetch each page and extract full text
includePatterns	array	No	[]	Only process URLs matching these regex patterns
excludePatterns	array	No	[]	Skip URLs matching these regex patterns
proxyConfiguration	object	No	Apify Proxy	Proxy settings for page fetches

Output

One item per page:

{
  "url": "https://apify.com/blog/web-scraping-guide",
  "title": "The Complete Web Scraping Guide",
  "content": "Web scraping is the process of ...",
  "wordCount": 2184,
  "metaDescription": "Learn web scraping from scratch...",
  "metaKeywords": "web scraping, crawling",
  "h1Headings": ["The Complete Web Scraping Guide"],
  "lastmod": "2026-06-30",
  "extractedAt": "2026-07-18T09:12:44Z"
}

Use cases

• 🔍 SEO audits — inventory every indexable page, spot missing titles/meta descriptions, and measure content depth by word count.
• 🚚 Site migrations — pull all content from a legacy site into a structured dataset before rebuilding.
• 🤖 AI training data — collect clean text from documentation and blog sitemaps to feed LLM fine-tuning or RAG.
• 📚 Documentation indexing — build a searchable index of a docs site from its sitemap in one run.
• 🕵️ Competitor analysis — map a competitor's content coverage and structure across their whole site.

Run it from your code

Python (Apify SDK):

from apify_client import ApifyClient

client = ApifyClient("YOUR-APIFY-TOKEN")

run_input = {
    "sitemapUrl": "https://apify.com/sitemap.xml",
    "maxUrls": 100,
    "includePatterns": ["/blog/"],
}

run = client.actor("darknezz/sitemap-content-extractor").call(run_input=run_input)
for page in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(page["url"], page["wordCount"], page["title"])

cURL (one-liner, sync):

curl -X POST "https://api.apify.com/v2/acts/darknezz~sitemap-content-extractor/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"sitemapUrl":"https://apify.com/sitemap.xml","maxUrls":100,"includePatterns":["/blog/"]}'

Worked example — pointing the actor at a documentation site's sitemap with includePatterns: ["/docs/"] returns every docs URL with its clean article text and word count; the lastmod field shows which pages changed since your last crawl, so a scheduled daily run doubles as a change detector.

Scheduling: attach an Apify Schedule (e.g. daily) to re-crawl a sitemap and keep a content inventory or search index continuously fresh — the lastmod field lets you detect which pages changed.

Pricing

Free. This Actor carries no per-page charges — no page-extracted fee, no start fee. You pay only Apify's standard platform usage (compute + storage) for your runs.

Companion tools

No sitemap, or need LLM-ready text from sites that don't publish one? AI Web Content Crawler (pay-per-event) crawls arbitrary websites and returns clean, token-estimated chunks with heading structure, tables and JSON-LD — one dataset item per page, no post-processing.

FAQ

Does it follow sitemap index files? Yes — nested <sitemapindex> files are followed recursively, so a single index URL crawls the whole site.

Can I crawl only part of a site? Yes — use includePatterns / excludePatterns with regex to scope to specific sections (e.g. only /blog/, skip /tag/).

What if a site has no sitemap? This Actor requires a sitemap URL. For arbitrary link-following crawls, use a general web crawler instead.

Do I need a proxy? Most sitemaps and pages fetch fine over Apify Proxy (the default). Sites with heavy anti-bot protection may need residential proxies.

How do I get metadata without full text? Set extractContent: false — you still get title, meta tags, and lastmod per URL, much faster.

How is the "clean content" extracted? The actor runs trafilatura, which scores HTML blocks and keeps only the main article content — nav, sidebars, cookie banners and boilerplate are dropped before you see the text.

Limitations

• Requires a real sitemap — sites without sitemap.xml (or a sitemap index) cannot be crawled by this actor.
• maxUrls is a hard cap — very large sites (100k+ URLs) need multiple runs or a raised maxUrls; the actor stops at the cap.
• JS-rendered content — pages that render their text with client-side JavaScript may yield little or no text without a browser; plain HTML pages extract best.
• Proxy-dependent pages — heavily protected sites may need proxyConfiguration with residential proxies (extra cost applies on Apify's side, not ours).

Enjoying the extractor? A quick review on the Apify Store helps others find it.