Sitemap Extractor & RSS Feed Finder: All URLs on a Site
Pricing
Pay per event
Sitemap Extractor & RSS Feed Finder: All URLs on a Site
Get every URL listed in a site's sitemaps (robots.txt Sitemap lines, nested indexes,.xml.gz) with lastmod, changefreq and priority, plus its RSS/Atom feeds and latest items. Reads sitemaps, doesn't crawl. Only-new mode for monitoring. Two sites in a test run produced 270 records.
Pricing
Pay per event
Rating
0.0
(0)
Developer
Almandine Data
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
10 days ago
Last modified
Categories
Features
Share
Sitemap Extractor & RSS Feed Finder: Every URL on a Site
Extract every URL in a website's sitemaps and find its RSS/Atom feeds in one run. The Actor reads robots.txt Sitemap: lines (or common sitemap paths), expands nested sitemap indexes and gzipped .xml.gz files, and returns one row per URL with lastmod, changefreq and priority. It also detects feeds declared on the homepage or at common paths (/feed, /rss.xml) and returns their latest items. It reads sitemaps and doesn't crawl, so it's fast and cheap. Set onlyNew: true to track newly published pages. Export as JSON, CSV or Excel, or use the Apify API, MCP, n8n, Make or Zapier.
What it does
Reads robots.txt Sitemap lines (or common paths), expands nested indexes and gzip sitemaps, finds feeds from the homepage, and returns one row per URL. Export clean, deduplicated data as JSON, CSV, Excel, XML, or RSS, or connect directly via the Apify REST API, Python/JavaScript SDKs, CLI, MCP Server, n8n, Make, or Zapier — with no login, cookies, or browser overhead.
Use cases
- RAG & LLM Knowledge Base Ingestion: Discover every public page URL (
sitemap-url) and blog post (feed-item) on a domain before scraping or embedding. - Technical SEO & Sitemap Audits: Parse
robots.txtsitemap directives, nestedsitemapindexfiles, and gzipped.xml.gzsitemaps withlastmod,changefreq, andpriority. - RSS/Atom Feed Discovery & Monitoring: Automatically detect
<link rel="alternate">feeds or probe common feed paths (/feed,/rss.xml,/atom.xml) and track new articles withonlyNew: true. - Competitor Content Tracking: Monitor when competitors publish new landing pages or blog posts on a daily schedule.
Why use this scraper?
| Feature | This Actor | Generic Browser / HTML Scrapers |
|---|---|---|
| Architecture | Direct structured API / JSON / feed requests (curl_cffi) | Heavy headless browser automation |
| Speed & reliability | 2–25 seconds for 50–200 items | Minutes of high compute burn |
| Pricing model | Pay per result ($0.00 actor-start fee) | Monthly rental or high CU overhead |
| Error handling | Failed inputs logged in ERRORS key-value record and never charged | Bills compute time even on failures |
Input parameters
- Websites (
sites,array): Domains or homepage URLs, e.g. 'example.com' or 'https://blog.example.com'. - Max URLs per site (
maxUrlsPerSite,integer): Cap on rows per site (charged per row). - Read sitemaps (
includeSitemaps,boolean): Find sitemaps via robots.txt (or common paths) and expand nested sitemap indexes, including .xml.gz. - Max sitemap-index depth (
maxSitemapDepth,integer): How many levels of nested sitemap indexes to follow. - Find RSS/Atom feeds (
includeFeeds,boolean): Find feeds declared on the homepage (). - Try common feed paths (
guessFeedPaths,boolean): If the homepage declares no feed, try /feed, /rss.xml, /atom.xml and similar. - Include feed items (
includeFeedItems,boolean): Also return each item (title, link, date) in every feed found. - Only new URLs (
onlyNew,boolean): On scheduled runs, return only URLs not seen before. Good for change monitoring. - Delay between requests (
delaySeconds,number): Polite pause between requests (default0.5).
Example input
{"sites": ["https://astralcodexten.substack.com","https://www.python.org"],"maxUrlsPerSite": 150}
Output & example output
One dataset row per result. Default table view columns: site, type, url, title, lastmod, sourceUrl. Full structured JSON is stored in the run's default dataset.
Example output item
{"site": "https://astralcodexten.substack.com","type": "sitemap-url","url": "https://www.astralcodexten.com/archive","lastmod": null,"changefreq": "daily","priority": null,"title": null,"sourceUrl": "https://www.astralcodexten.com/sitemap.xml","feedUrl": null,"scrapedAt": "2026-10-25T04:23:38+00:00"}
Pricing (Pay-Per-Result)
This Actor uses transparent Pay-Per-Event pricing with a $0.00 actor-start fee:
| Event | What is charged | Price |
|---|---|---|
url-result | URL discovered (One sitemap URL, feed or feed item) | $0.001 / result ($1.00 / 1,000 results) |
apify-actor-start | Actor start fee | $0.00 |
Concrete pricing math
url-result(URL discovered):1,000 results = $1.00(50 results = $0.05,10,000 results = $10.00). Apify's $5.00/month Free Tier covers up to 5,000 results/month at $0.00 out-of-pocket.- Empty runs (
0results) & skipped duplicates (onlyNew/onlyChanged): $0.00.
Data and compliance
Reads only files that sites publish for crawlers (robots.txt, sitemaps, feeds). Requests are rate-limited and routed through Apify Proxy. Per-input errors are recorded in the run's ERRORS key-value record and are never charged.
FAQ
Do I need an account, login, or API key?
No. This Actor queries public endpoints directly and requires zero credentials.
What happens if I start a run with empty {} input via API or CLI?
Every required input defines both prefill and default values in .actor/input_schema.json, so an empty {} run automatically returns 50+ live records out of the box.