Sitemap + RSS Finder: every page URL and feed on a site

Extract every sitemap URL (.xml, .xml.gz, nested indexes) and RSS/Atom feed item from any website. Built for SEO audits, RAG ingestion, and AI agents.

What it does

Reads robots.txt Sitemap lines (or common paths), expands nested indexes and gzip sitemaps, finds feeds from the homepage, and returns one row per URL. Export clean, deduplicated data as JSON, CSV, Excel, XML, or RSS, or connect directly via the Apify REST API, Python/JavaScript SDKs, CLI, MCP Server, n8n, Make, or Zapier — with no login, cookies, or browser overhead.

Use cases

• RAG & LLM Knowledge Base Ingestion: Discover every public page URL (sitemap-url) and blog post (feed-item) on a domain before scraping or embedding.
• Technical SEO & Sitemap Audits: Parse robots.txt sitemap directives, nested sitemapindex files, and gzipped .xml.gz sitemaps with lastmod, changefreq, and priority.
• RSS/Atom Feed Discovery & Monitoring: Automatically detect <link rel="alternate"> feeds or probe common feed paths (/feed, /rss.xml, /atom.xml) and track new articles with onlyNew: true.
• Competitor Content Tracking: Monitor when competitors publish new landing pages or blog posts on a daily schedule.

Why use this scraper?

Feature	This Actor	Generic Browser / HTML Scrapers
Architecture	Direct structured API / JSON / feed requests (curl_cffi)	Heavy headless browser automation
Speed & reliability	2–25 seconds for 50–200 items	Minutes of high compute burn
Pricing model	Pay per result ($0.00 actor-start fee)	Monthly rental or high CU overhead
Error handling	Failed inputs logged in ERRORS key-value record and never charged	Bills compute time even on failures

Input parameters

• Websites (sites, array): Domains or homepage URLs, e.g. 'example.com' or 'https://blog.example.com'.
• Max URLs per site (maxUrlsPerSite, integer): Cap on rows per site (charged per row).
• Read sitemaps (includeSitemaps, boolean): Find sitemaps via robots.txt (or common paths) and expand nested sitemap indexes, including .xml.gz.
• Max sitemap-index depth (maxSitemapDepth, integer): How many levels of nested sitemap indexes to follow.
• Find RSS/Atom feeds (includeFeeds, boolean): Find feeds declared on the homepage ().
• Try common feed paths (guessFeedPaths, boolean): If the homepage declares no feed, try /feed, /rss.xml, /atom.xml and similar.
• Include feed items (includeFeedItems, boolean): Also return each item (title, link, date) in every feed found.
• Only new URLs (onlyNew, boolean): On scheduled runs, return only URLs not seen before. Good for change monitoring.
• Delay between requests (delaySeconds, number): Polite pause between requests (default 0.5).

Example input

{
  "sites": [
    "https://astralcodexten.substack.com",
    "https://www.python.org"
  ],
  "maxUrlsPerSite": 150
}

Output & example output

One dataset row per result. Default table view columns: site, type, url, title, lastmod, sourceUrl. Full structured JSON is stored in the run's default dataset.

Example output item

{
  "site": "https://astralcodexten.substack.com",
  "type": "sitemap-url",
  "url": "https://www.astralcodexten.com/archive",
  "lastmod": null,
  "changefreq": "daily",
  "priority": null,
  "title": null,
  "sourceUrl": "https://www.astralcodexten.com/sitemap.xml",
  "feedUrl": null,
  "scrapedAt": "2026-10-25T04:23:38+00:00"
}

Pricing (Pay-Per-Result)

This Actor uses transparent Pay-Per-Event pricing with a $0.00 actor-start fee:

Event	What is charged	Price
url-result	URL discovered (One sitemap URL, feed or feed item)	$0.001 / result ($1.00 / 1,000 results)
apify-actor-start	Actor start fee	$0.00

Concrete pricing math

• url-result (URL discovered): 1,000 results = $1.00 (50 results = $0.05, 10,000 results = $10.00). Apify's $5.00/month Free Tier covers up to 5,000 results/month at $0.00 out-of-pocket.
• Empty runs (0 results) & skipped duplicates (onlyNew / onlyChanged): $0.00.

Data and compliance

Reads only files that sites publish for crawlers (robots.txt, sitemaps, feeds). Requests are rate-limited and routed through Apify Proxy. Per-input errors are recorded in the run's ERRORS key-value record and are never charged.

FAQ

Do I need an account, login, or API key?
No. This Actor queries public endpoints directly and requires zero credentials.

What happens if I start a run with empty {} input via API or CLI?
Every required input defines both prefill and default values in .actor/input_schema.json, so an empty {} run automatically returns 50+ live records out of the box.