# Sitemap & RSS Finder: Every Sitemap URL and Feed (`informed_almandine_vzb/sitemap-rss-finder`) Actor

Point it at a domain and get every URL listed in the site's sitemaps, plus its RSS/Atom feeds and their latest items. It doesn't crawl. Two sites in a test run produced 270 records.

- **URL**: https://apify.com/informed_almandine_vzb/sitemap-rss-finder.md
- **Developed by:** [Almandine Data](https://apify.com/informed_almandine_vzb) (community)
- **Categories:** Developer tools, SEO tools, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$1.00 / 1,000 url discovereds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap + RSS Finder: every page URL and feed on a site

Extract every sitemap URL (.xml, .xml.gz, nested indexes) and RSS/Atom feed item from any website. Built for SEO audits, RAG ingestion, and AI agents.

### What it does

Reads robots.txt Sitemap lines (or common paths), expands nested indexes and gzip sitemaps, finds feeds from the homepage, and returns one row per URL. Export clean, deduplicated data as **JSON, CSV, Excel, XML, or RSS**, or connect directly via the **Apify REST API, Python/JavaScript SDKs, CLI, MCP Server, n8n, Make, or Zapier** — with **no login, cookies, or browser overhead**.

### Use cases

- **RAG & LLM Knowledge Base Ingestion:** Discover every public page URL (`sitemap-url`) and blog post (`feed-item`) on a domain before scraping or embedding.
- **Technical SEO & Sitemap Audits:** Parse `robots.txt` sitemap directives, nested `sitemapindex` files, and gzipped `.xml.gz` sitemaps with `lastmod`, `changefreq`, and `priority`.
- **RSS/Atom Feed Discovery & Monitoring:** Automatically detect `<link rel="alternate">` feeds or probe common feed paths (`/feed`, `/rss.xml`, `/atom.xml`) and track new articles with `onlyNew: true`.
- **Competitor Content Tracking:** Monitor when competitors publish new landing pages or blog posts on a daily schedule.

### Why use this scraper?

| Feature | This Actor | Generic Browser / HTML Scrapers |
|---|---|---|
| **Architecture** | Direct structured API / JSON / feed requests (`curl_cffi`) | Heavy headless browser automation |
| **Speed & reliability** | **2–25 seconds** for 50–200 items | Minutes of high compute burn |
| **Pricing model** | **Pay per result (`$0.00` actor-start fee)** | Monthly rental or high CU overhead |
| **Error handling** | Failed inputs logged in `ERRORS` key-value record and **never charged** | Bills compute time even on failures |

### Input parameters

- **Websites** (`sites`, `array`): Domains or homepage URLs, e.g. 'example.com' or 'https://blog.example.com'.
- **Max URLs per site** (`maxUrlsPerSite`, `integer`): Cap on rows per site (charged per row).
- **Read sitemaps** (`includeSitemaps`, `boolean`): Find sitemaps via robots.txt (or common paths) and expand nested sitemap indexes, including .xml.gz.
- **Max sitemap-index depth** (`maxSitemapDepth`, `integer`): How many levels of nested sitemap indexes to follow.
- **Find RSS/Atom feeds** (`includeFeeds`, `boolean`): Find feeds declared on the homepage (<link rel=alternate>).
- **Try common feed paths** (`guessFeedPaths`, `boolean`): If the homepage declares no feed, try /feed, /rss.xml, /atom.xml and similar.
- **Include feed items** (`includeFeedItems`, `boolean`): Also return each item (title, link, date) in every feed found.
- **Only new URLs** (`onlyNew`, `boolean`): On scheduled runs, return only URLs not seen before. Good for change monitoring.
- **Delay between requests** (`delaySeconds`, `number`): Polite pause between requests (default `0.5`).

#### Example input

```json
{
  "sites": [
    "https://astralcodexten.substack.com",
    "https://www.python.org"
  ],
  "maxUrlsPerSite": 150
}
```

### Output & example output

One dataset row per result. Default table view columns: `site, type, url, title, lastmod, sourceUrl`. Full structured JSON is stored in the run's default dataset.

#### Example output item

```json
{
  "site": "https://astralcodexten.substack.com",
  "type": "sitemap-url",
  "url": "https://www.astralcodexten.com/archive",
  "lastmod": null,
  "changefreq": "daily",
  "priority": null,
  "title": null,
  "sourceUrl": "https://www.astralcodexten.com/sitemap.xml",
  "feedUrl": null,
  "scrapedAt": "2026-10-25T04:23:38+00:00"
}
```

### Pricing (Pay-Per-Result)

This Actor uses transparent **Pay-Per-Event** pricing with a **`$0.00` actor-start fee**:

| Event | What is charged | Price |
|---|---|---|
| `url-result` | URL discovered (One sitemap URL, feed or feed item) | **$0.001 / result** (**$1.00 / 1,000 results**) |
| `apify-actor-start` | Actor start fee | **$0.00** |

#### Concrete pricing math

- **`url-result` (URL discovered):** `1,000 results = $1.00` (`50 results = $0.05`, `10,000 results = $10.00`). Apify's **$5.00/month Free Tier** covers up to **5,000 results/month** at **$0.00** out-of-pocket.
- **Empty runs (`0` results) & skipped duplicates (`onlyNew` / `onlyChanged`):** **$0.00**.

### Data and compliance

Reads only files that sites publish for crawlers (robots.txt, sitemaps, feeds). Requests are rate-limited and routed through Apify Proxy. Per-input errors are recorded in the run's `ERRORS` key-value record and are never charged.

### FAQ

**Do I need an account, login, or API key?**\
No. This Actor queries public endpoints directly and requires zero credentials.

**What happens if I start a run with empty `{}` input via API or CLI?**\
Every required input defines both `prefill` and `default` values in `.actor/input_schema.json`, so an empty `{}` run automatically returns **50+ live records** out of the box.

# Actor input Schema

## `sites` (type: `array`):

Domains or homepage URLs, e.g. 'example.com' or 'https://blog.example.com'.

## `maxUrlsPerSite` (type: `integer`):

Cap on rows per site (charged per row).

## `includeSitemaps` (type: `boolean`):

Find sitemaps via robots.txt (or common paths) and expand nested sitemap indexes, including .xml.gz.

## `maxSitemapDepth` (type: `integer`):

How many levels of nested sitemap indexes to follow.

## `includeFeeds` (type: `boolean`):

Find feeds declared on the homepage (<link rel=alternate>).

## `guessFeedPaths` (type: `boolean`):

If the homepage declares no feed, try /feed, /rss.xml, /atom.xml and similar.

## `includeFeedItems` (type: `boolean`):

Also return each item (title, link, date) in every feed found.

## `onlyNew` (type: `boolean`):

On scheduled runs, return only URLs not seen before. Good for change monitoring.

## `delaySeconds` (type: `number`):

Polite pause between requests.

## `proxyConfiguration` (type: `object`):

Apify Proxy settings.

## Actor input object example

```json
{
  "sites": [
    "https://astralcodexten.substack.com",
    "https://www.python.org"
  ],
  "maxUrlsPerSite": 150,
  "includeSitemaps": true,
  "maxSitemapDepth": 3,
  "includeFeeds": true,
  "guessFeedPaths": true,
  "includeFeedItems": true,
  "onlyNew": false,
  "delaySeconds": 0.5,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `overview` (type: `string`):

No description

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sites": [
        "https://astralcodexten.substack.com",
        "https://www.python.org"
    ],
    "maxUrlsPerSite": 150
};

// Run the Actor and wait for it to finish
const run = await client.actor("informed_almandine_vzb/sitemap-rss-finder").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "sites": [
        "https://astralcodexten.substack.com",
        "https://www.python.org",
    ],
    "maxUrlsPerSite": 150,
}

# Run the Actor and wait for it to finish
run = client.actor("informed_almandine_vzb/sitemap-rss-finder").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sites": [
    "https://astralcodexten.substack.com",
    "https://www.python.org"
  ],
  "maxUrlsPerSite": 150
}' |
apify call informed_almandine_vzb/sitemap-rss-finder --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,informed_almandine_vzb/sitemap-rss-finder"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/x4sfy061dfTSeofMa/builds/ccwEN2ztOkjJ3FJat/openapi.json