Sitemap Extractor & RSS Feed Finder: All URLs on a Site avatar

Sitemap Extractor & RSS Feed Finder: All URLs on a Site

Pricing

Pay per event

Go to Apify Store
Sitemap Extractor & RSS Feed Finder: All URLs on a Site

Sitemap Extractor & RSS Feed Finder: All URLs on a Site

Get every URL listed in a site's sitemaps (robots.txt Sitemap lines, nested indexes,.xml.gz) with lastmod, changefreq and priority, plus its RSS/Atom feeds and latest items. Reads sitemaps, doesn't crawl. Only-new mode for monitoring. Two sites in a test run produced 270 records.

Pricing

Pay per event

Rating

0.0

(0)

Developer

Almandine Data

Almandine Data

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

10 days ago

Last modified

Features

Limited permissions

Share

Sitemap Extractor & RSS Feed Finder: Every URL on a Site

Extract every URL in a website's sitemaps and find its RSS/Atom feeds in one run. The Actor reads robots.txt Sitemap: lines (or common sitemap paths), expands nested sitemap indexes and gzipped .xml.gz files, and returns one row per URL with lastmod, changefreq and priority. It also detects feeds declared on the homepage or at common paths (/feed, /rss.xml) and returns their latest items. It reads sitemaps and doesn't crawl, so it's fast and cheap. Set onlyNew: true to track newly published pages. Export as JSON, CSV or Excel, or use the Apify API, MCP, n8n, Make or Zapier.

What it does

Reads robots.txt Sitemap lines (or common paths), expands nested indexes and gzip sitemaps, finds feeds from the homepage, and returns one row per URL. Export clean, deduplicated data as JSON, CSV, Excel, XML, or RSS, or connect directly via the Apify REST API, Python/JavaScript SDKs, CLI, MCP Server, n8n, Make, or Zapier — with no login, cookies, or browser overhead.

Use cases

  • RAG & LLM Knowledge Base Ingestion: Discover every public page URL (sitemap-url) and blog post (feed-item) on a domain before scraping or embedding.
  • Technical SEO & Sitemap Audits: Parse robots.txt sitemap directives, nested sitemapindex files, and gzipped .xml.gz sitemaps with lastmod, changefreq, and priority.
  • RSS/Atom Feed Discovery & Monitoring: Automatically detect <link rel="alternate"> feeds or probe common feed paths (/feed, /rss.xml, /atom.xml) and track new articles with onlyNew: true.
  • Competitor Content Tracking: Monitor when competitors publish new landing pages or blog posts on a daily schedule.

Why use this scraper?

FeatureThis ActorGeneric Browser / HTML Scrapers
ArchitectureDirect structured API / JSON / feed requests (curl_cffi)Heavy headless browser automation
Speed & reliability2–25 seconds for 50–200 itemsMinutes of high compute burn
Pricing modelPay per result ($0.00 actor-start fee)Monthly rental or high CU overhead
Error handlingFailed inputs logged in ERRORS key-value record and never chargedBills compute time even on failures

Input parameters

  • Websites (sites, array): Domains or homepage URLs, e.g. 'example.com' or 'https://blog.example.com'.
  • Max URLs per site (maxUrlsPerSite, integer): Cap on rows per site (charged per row).
  • Read sitemaps (includeSitemaps, boolean): Find sitemaps via robots.txt (or common paths) and expand nested sitemap indexes, including .xml.gz.
  • Max sitemap-index depth (maxSitemapDepth, integer): How many levels of nested sitemap indexes to follow.
  • Find RSS/Atom feeds (includeFeeds, boolean): Find feeds declared on the homepage ().
  • Try common feed paths (guessFeedPaths, boolean): If the homepage declares no feed, try /feed, /rss.xml, /atom.xml and similar.
  • Include feed items (includeFeedItems, boolean): Also return each item (title, link, date) in every feed found.
  • Only new URLs (onlyNew, boolean): On scheduled runs, return only URLs not seen before. Good for change monitoring.
  • Delay between requests (delaySeconds, number): Polite pause between requests (default 0.5).

Example input

{
"sites": [
"https://astralcodexten.substack.com",
"https://www.python.org"
],
"maxUrlsPerSite": 150
}

Output & example output

One dataset row per result. Default table view columns: site, type, url, title, lastmod, sourceUrl. Full structured JSON is stored in the run's default dataset.

Example output item

{
"site": "https://astralcodexten.substack.com",
"type": "sitemap-url",
"url": "https://www.astralcodexten.com/archive",
"lastmod": null,
"changefreq": "daily",
"priority": null,
"title": null,
"sourceUrl": "https://www.astralcodexten.com/sitemap.xml",
"feedUrl": null,
"scrapedAt": "2026-10-25T04:23:38+00:00"
}

Pricing (Pay-Per-Result)

This Actor uses transparent Pay-Per-Event pricing with a $0.00 actor-start fee:

EventWhat is chargedPrice
url-resultURL discovered (One sitemap URL, feed or feed item)$0.001 / result ($1.00 / 1,000 results)
apify-actor-startActor start fee$0.00

Concrete pricing math

  • url-result (URL discovered): 1,000 results = $1.00 (50 results = $0.05, 10,000 results = $10.00). Apify's $5.00/month Free Tier covers up to 5,000 results/month at $0.00 out-of-pocket.
  • Empty runs (0 results) & skipped duplicates (onlyNew / onlyChanged): $0.00.

Data and compliance

Reads only files that sites publish for crawlers (robots.txt, sitemaps, feeds). Requests are rate-limited and routed through Apify Proxy. Per-input errors are recorded in the run's ERRORS key-value record and are never charged.

FAQ

Do I need an account, login, or API key?
No. This Actor queries public endpoints directly and requires zero credentials.

What happens if I start a run with empty {} input via API or CLI?
Every required input defines both prefill and default values in .actor/input_schema.json, so an empty {} run automatically returns 50+ live records out of the box.