# Site Discovery: sitemaps, robots.txt, RSS & llms.txt (`steadyline/site-discovery`) Actor

Map any website in seconds without crawling: robots.txt rules for your user agent (incl. AI-bot blocks), every sitemap and sitemap index (.gz too), RSS/Atom/JSON feeds, llms.txt and llms-full.txt, plus a filtered URL list with lastmod. By Steadyline Data.

- **URL**: https://apify.com/steadyline/site-discovery.md
- **Developed by:** [Steadyline Data](https://apify.com/steadyline) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 domain summaries

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Site Discovery for AI Agents: sitemaps, robots.txt, RSS & llms.txt

Give it a list of domains. It maps each site **without crawling a single page**: what robots.txt allows for *your* user agent, every sitemap (including sitemap indexes and gzipped `.xml.gz` files), RSS/Atom/JSON feeds, and whether the site publishes `llms.txt` / `llms-full.txt` for AI agents. You can also get a clean, filtered URL list with `lastmod`, `changefreq` and `priority`.

It is built for teams that feed websites to LLMs, RAG pipelines, AI agents and SEO tools. You find out what to fetch, what has changed since your last run, and what you are allowed to fetch before you spend money on a crawler.

### What you get per domain

- **robots.txt summary** for the user agent you choose (for example `GPTBot`, `Googlebot` or your own bot). It shows the matched group, the allow and disallow rules, crawl-delay, whether the whole site is blocked for you, and which well-known AI bots (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot…) are blocked. Rules are matched per RFC 9309: longest match wins, with `*` and `$` wildcards.
- **Sitemaps** listed in robots.txt. When robots.txt lists none, the common paths are tried. Sitemap indexes are followed, newest first, and gzip is handled.
- **Feeds** from `<link rel="alternate">` on the homepage, or from common paths such as `/feed`, `/rss.xml` and `/atom.xml`. Each feed shows its format, title, item count and latest item date.
- **llms.txt and llms-full.txt**: whether each is present, its size, and an excerpt.
- **URL counts**: URLs scanned, URLs that matched your filters, URLs filtered out, URLs disallowed by robots.txt, and URLs output.

There is also an optional **per-URL record**: `url`, `lastmod`, `changefreq`, `priority`, `sourceSitemap` and `allowedByRobots`.

### Filters

- `includePatterns` / `excludePatterns` match the URL path. Use a glob (`/blog/**`, `/docs/*/intro`) or a regex with the `re:` prefix (`re:^/p/\d+`).
- `lastmodSince` keeps only URLs modified on or after a date. Child sitemaps older than that date are skipped. URLs without a lastmod are dropped when this is set.
- `maxUrls` caps the URL records per domain. `maxSitemaps` caps the sitemap files fetched per domain.

### Output sample (real test run, trimmed)

```json
{
 "recordType": "domain-summary", "domain": "vercel.com", "homepageStatus": 200,
 "robots": { "status": 200, "present": true, "userAgent": "GPTBot", "matchedGroup": "*",
   "disallow": ["/api/", "/oauth", "/confirm", "..."], "allow": ["/api/og/*", "..."],
   "crawlDelay": null, "blocksEverything": false, "aiBotsBlocked": [] },
 "sitemaps": [ { "url": "https://vercel.com/sitemap.xml", "status": 200, "type": "urlset", "entries": 9269, "error": null } ],
 "feeds": [ { "url": "https://vercel.com/atom", "foundVia": "common-path", "format": "atom", "title": "Vercel News", "itemCount": 2102, "latestItem": "2026-10-29T18:44:00.000Z" } ],
 "llmsTxt": { "present": true, "bytes": 5397, "excerpt": "# Vercel\n\n> Vercel is a cloud platform for building, deploying, and scaling web applications and AI workloads. ..." },
 "llmsFullTxt": { "present": false },
 "urlCounts": { "matchedFilters": 30, "disallowedByRobots": 0, "output": 30, "truncatedAtMaxUrls": true },
 "error": null
}
{ "recordType": "url", "domain": "vercel.com", "url": "https://vercel.com/connect/ocean-io",
  "lastmod": "2026-10-25T14:10:15.167Z", "changefreq": null, "priority": null,
  "sourceSitemap": "https://vercel.com/sitemap.xml", "allowedByRobots": true }
```

The **Domain summaries** and **URLs** dataset views split the two record types.

### Pricing (pay per event)

| Event | Price |
|---|---|
| `domain-summary`: one reachable domain analyzed | $0.002 |
| `url-discovered`: one per-URL record written | $0.0003 |

**Failures are never charged.** A domain where we could not get robots.txt, the homepage or any sitemap is still listed with an `error`, but it is free. URLs dropped by your filters or by robots.txt are free too.
**Cost example:** 100 domains with 500 URLs each = 100 × $0.002 + 50,000 × $0.0003 = **$15.20**. With summaries only (`outputUrls: false`), the same run costs **$0.20**.

### Limitations

- Only one origin is analyzed per input. `wordpress.org/news` analyzes `wordpress.org`, so enter subdomains such as `blog.example.com` separately.
- Sites that block datacenter IPs may return 403 for sitemaps. Those sitemaps are listed with their error.
- When `lastmodSince` is set and a site has no lastmod values (common), every URL is filtered out. Check `urlCounts.filteredOut`.

### Acceptable use

This Actor reads only public, machine-readable files that sites publish for robots (robots.txt, sitemaps, feeds, llms.txt), plus the homepage. It identifies itself with a descriptive user agent. Respecting robots.txt is on by default. Use the results in line with each site's terms and robots.txt, and don't use them to overload or attack sites. You are responsible for what you do with the discovered URLs.

Made by **Steadyline Data**.

# Actor input Schema

## `domains` (type: `array`):

Domains (example.com) or URLs. Each origin is analyzed once.

## `outputUrls` (type: `boolean`):

Write one dataset row per discovered URL (charged per URL). Off = domain summaries only.

## `maxUrls` (type: `integer`):

Stop collecting URLs for a domain after this many (after filters).

## `includePatterns` (type: `array`):

Only keep URLs whose path matches one of these. Glob on path (/blog/\*\*, /docs/\*/intro) or regex with re: prefix (re:^/p/\d+).

## `excludePatterns` (type: `array`):

Drop URLs whose path matches (same syntax).

## `lastmodSince` (type: `string`):

Only keep URLs with a sitemap lastmod on/after this date. URLs without lastmod are dropped when set.

## `followSitemapIndexes` (type: `boolean`):

Recurse into sitemap index files (including gzipped .xml.gz).

## `maxSitemaps` (type: `integer`):

Safety cap on sitemap files fetched per domain.

## `discoverFeeds` (type: `boolean`):

From <link rel=alternate> on the homepage, else common paths (/feed, /rss.xml, /atom.xml...).

## `fetchLlmsTxt` (type: `boolean`):

Check /llms.txt and /llms-full.txt and include an excerpt.

## `llmsExcerptChars` (type: `integer`):

Characters of llms.txt content to include.

## `userAgent` (type: `string`):

User-agent token whose robots.txt rules are evaluated (e.g. GPTBot, Googlebot, MyAgent).

## `respectRobots` (type: `boolean`):

Drop URLs disallowed for the user agent above (counted in urlCounts.disallowedByRobots). Off = output them with allowedByRobots=false.

## `maxConcurrency` (type: `integer`):

Max 20.

## `timeoutSecs` (type: `integer`):

Per request.

## Actor input object example

```json
{
  "domains": [
    "apify.com",
    "vercel.com",
    "blog.cloudflare.com"
  ],
  "outputUrls": true,
  "maxUrls": 100,
  "includePatterns": [],
  "excludePatterns": [],
  "followSitemapIndexes": true,
  "maxSitemaps": 50,
  "discoverFeeds": true,
  "fetchLlmsTxt": true,
  "llmsExcerptChars": 2000,
  "userAgent": "GPTBot",
  "respectRobots": true,
  "maxConcurrency": 5,
  "timeoutSecs": 20
}
```

# Actor output Schema

## `results` (type: `string`):

One row per domain (rows with recordType=url are blank here).

## `urls` (type: `string`):

Per-URL records.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "apify.com",
        "vercel.com",
        "blog.cloudflare.com"
    ],
    "maxUrls": 100,
    "userAgent": "GPTBot"
};

// Run the Actor and wait for it to finish
const run = await client.actor("steadyline/site-discovery").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "domains": [
        "apify.com",
        "vercel.com",
        "blog.cloudflare.com",
    ],
    "maxUrls": 100,
    "userAgent": "GPTBot",
}

# Run the Actor and wait for it to finish
run = client.actor("steadyline/site-discovery").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "apify.com",
    "vercel.com",
    "blog.cloudflare.com"
  ],
  "maxUrls": 100,
  "userAgent": "GPTBot"
}' |
apify call steadyline/site-discovery --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,steadyline/site-discovery"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/TJEhxMowD2D4hBrIb/builds/kYo1hpMtSB2IrYsvU/openapi.json