Robots.txt & Sitemap Extractor by Domain

Robots.txt & Sitemap Extractor Fetch and parse robots.txt for any domain — user-agents, allow/disallow rules, crawl-delay and sitemap URLs — in one bulk call. No API key. Built for developers who need fast, reliable, bulk results without managing servers or proxies.

Features

• One structured JSON record per domain.
• Bulk + concurrent — process up to 100 domains per run.
• No proxy, no browser, no API key — fast and low-cost.
• Graceful failures: unreachable inputs return ok: false with an error; the run never crashes.

Input

{"domains": ["github.com", "apify.com"], "maxItems": 100}

Output

One record per domain:

{
  "domain": "github.com",
  "ok": true,
  "status": 200,
  "sitemaps": ["https://github.com/sitemap.xml"],
  "groupCount": 2,
  "crawlDelay": null,
  "groups": [{"userAgents": ["*"], "allow": [], "disallow": ["/search"], "crawlDelay": null}]
}

Who it's for

• SEO teams auditing crawl rules
• Crawler/scraper developers respecting robots
• Sitemap discovery pipelines
• Site migration & QA checks

How to use

1. Add your domains to the domains input field.
2. Run the Actor and read one structured record per input from the dataset.
3. Export as JSON, CSV, or Excel, or call the Actor's API from your backend.

FAQ

Do I need an API key? No. You provide domains; the Actor does the rest. Does it return personal data? No — only public infrastructure/metadata, never personal contact data. What formats can I export? JSON, CSV, and Excel via the dataset, or the Actor API.

Pricing

Pay-per-event: a tiny per-run start fee plus a small per-result charge. You pay only for successful results. See the Pricing tab.

Further reading

• Keyword: robots.txt sitemap extractor.

Use Robots.txt & Sitemap Extractor by Domain with AI agents (MCP)

Give Claude, ChatGPT, Cursor, VS Code or any MCP client this Actor as a tool through the hosted Apify MCP server. The agent sees the input fields with descriptions and example values, runs the Actor, and reads the results.

MCP server URL (this Actor only):

https://mcp.apify.com/?tools=technicaldost/robots-sitemap-extractor

Claude Code:

$claude mcp add --transport http apify "https://mcp.apify.com/?tools=technicaldost/robots-sitemap-extractor" --header "Authorization: Bearer YOUR_APIFY_TOKEN"

Cursor, VS Code, Claude Desktop and other clients (mcp.json or the client's MCP settings):

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com/?tools=technicaldost/robots-sitemap-extractor",
      "headers": { "Authorization": "Bearer YOUR_APIFY_TOKEN" }
    }
  }
}

Clients that support sign-in (such as Claude.ai connectors) can use the URL without a token and log in to Apify instead. Get your token in Apify Console → Settings → API & Integrations, or generate a ready config for your client at mcp.apify.com. In the agent the tool is named technicaldost--robots-sitemap-extractor; ask it, for example, to "run Robots.txt & Sitemap Extractor by Domain with {"domains": ["github.com", "apify.com"]}".

AI agents without an Apify account can also run and pay for this Actor per request with x402 agentic payments.

Run Robots.txt & Sitemap Extractor by Domain via API

Run the Actor and get its results in one call (replace YOUR_APIFY_TOKEN):

curl -X POST "https://api.apify.com/v2/acts/technicaldost~robots-sitemap-extractor/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"domains": ["github.com", "apify.com"]}'

Python (pip install apify-client):

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("technicaldost/robots-sitemap-extractor").call(run_input={"domains": ["github.com", "apify.com"]})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

JavaScript / TypeScript (npm install apify-client):

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });
const run = await client.actor('technicaldost/robots-sitemap-extractor').call({"domains": ["github.com", "apify.com"]});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Long runs: start them with POST /v2/acts/technicaldost~robots-sitemap-extractor/runs and fetch the dataset when the run finishes, or receive a webhook.

Integrations

Robots.txt & Sitemap Extractor by Domain works with Make, Zapier, n8n, LangChain, Google Sheets, Slack, GitHub and webhooks. Schedule runs in Apify Console and send new results straight to your apps, spreadsheets or AI pipelines.