Robots.txt & Sitemap Extractor by Domain avatar

Robots.txt & Sitemap Extractor by Domain

Pricing

from $2.00 / 1,000 results

Try for free
Go to Apify Store
Robots.txt & Sitemap Extractor by Domain

Robots.txt & Sitemap Extractor by Domain

Try for free

Fetch and parse robots.txt and XML sitemaps for any domain. Extract allowed and disallowed paths, sitemap URLs and every listed page. Great for SEO audits and crawl planning. JSON output.

Pricing

from $2.00 / 1,000 results

Rating

0.0

(0)

Developer

Technical Dost Solutions

Technical Dost Solutions

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

a day ago

Last modified

Categories

SEO tools

Developer tools

Share

Robots.txt & Sitemap Extractor Fetch and parse robots.txt for any domain — user-agents, allow/disallow rules, crawl-delay and sitemap URLs — in one bulk call. No API key. Built for developers who need fast, reliable, bulk results without managing servers or proxies.

Features

  • One structured JSON record per domain.
  • Bulk + concurrent — process up to 100 domains per run.
  • No proxy, no browser, no API key — fast and low-cost.
  • Graceful failures: unreachable inputs return ok: false with an error; the run never crashes.

Input

{"domains": ["github.com", "apify.com"], "maxItems": 100}

Output

One record per domain:

{
  "domain": "github.com",
  "ok": true,
  "status": 200,
  "sitemaps": ["https://github.com/sitemap.xml"],
  "groupCount": 2,
  "crawlDelay": null,
  "groups": [{"userAgents": ["*"], "allow": [], "disallow": ["/search"], "crawlDelay": null}]
}

Who it's for

  • SEO teams auditing crawl rules
  • Crawler/scraper developers respecting robots
  • Sitemap discovery pipelines
  • Site migration & QA checks

How to use

  1. Add your domains to the domains input field.
  2. Run the Actor and read one structured record per input from the dataset.
  3. Export as JSON, CSV, or Excel, or call the Actor's API from your backend.

FAQ

Do I need an API key? No. You provide domains; the Actor does the rest. Does it return personal data? No — only public infrastructure/metadata, never personal contact data. What formats can I export? JSON, CSV, and Excel via the dataset, or the Actor API.

Pricing

Pay-per-event: a tiny per-run start fee plus a small per-result charge. You pay only for successful results. See the Pricing tab.

Further reading

  • Keyword: robots.txt sitemap extractor.

You might also like

Sitemap URL Extractor - XML Sitemap Scraper avatar

Sitemap URL Extractor - XML Sitemap Scraper

benthepythondev/sitemap-url-extractor

Extract URLs from XML sitemaps and sitemap indexes. Get URL, lastmod, changefreq, priority and source sitemap.

User avatar

Ben

3

Find Sitemap From Url avatar

Find Sitemap From Url

scrapers-hub/find-sitemap-from-url

🔎 Find Sitemap From Url tool instantly locates a website’s sitemap from its URL, helping you discover indexable pages faster. ⚡ Perfect for SEO audits, crawling, indexing analysis & link research. 🚀 Boost efficiency with accurate sitemap detection.

User avatar

Scrapers Hub

2

Website Metadata Extractor(sitemap, socialLinks, robotsTxt) avatar

Website Metadata Extractor(sitemap, socialLinks, robotsTxt)

codescraper/website-metadata-extractor

A very fast metadata extractor to get all meta tags, robots.txt, sitemaps, social links, H1s, word count, and JSON-LD data. Also provides technology detection for a full analysis. Get your data fast for just $3/month.

User avatar

CodeScraper

104

5.0

(2)

Sitemap Scraper avatar

Sitemap Scraper

scrapers-hub/sitemap-scraper

Sitemap scraper to crawl and extract URLs, pages, and structure from website sitemaps 🌐📊 Perfect for SEO analysis, website auditing, and data extraction. Fast, reliable, and scalable.

User avatar

Scrapers Hub

5

Company Domain Tech, DNS & Security Audit avatar

Company Domain Tech, DNS & Security Audit

taroyamada/company-enrichment-intelligence

Turn company domains into structured website, tech stack, DNS, SPF, DMARC, RDAP, TLS certificate, robots.txt, and sitemap audit records. Public, keyless sources only: the company's own website plus public registries. Pay per audited domain.

User avatar

naoki anzai

2

Website Metadata Extractor avatar

Website Metadata Extractor

scrapers-hub/website-metadata-extractor

Website metadata extractor to extract titles, descriptions, keywords, and meta tags from any website 🌐📊 Perfect for SEO analysis, auditing, and research. Fast, accurate, and scalable extraction.

User avatar

Scrapers Hub

2

SU

Sitemap URL Extractor — robots.txt + sitemap.xml Crawl

v0iddo/sitemap-url-extractor

Discover every URL a site exposes via its public sitemap chain. Reads robots.txt, follows Sitemap declarations, recursively descends sitemap-index files, extracts URLs with lastmod, changefreq, priority.

User avatar

vøiddo

2

Sitemap Url Extractor avatar

Sitemap Url Extractor

scrapers-hub/sitemap-url-extractor

Sitemap URL extractor to extract all URLs from XML sitemaps quickly and efficiently 🌐📄 Ideal for SEO audits, site analysis, and indexing workflows. Fast, accurate, and easy to use.

User avatar

Scrapers Hub

3

AT

AI Training Blocker Checker

bipbip-apis/ai-training-blocker-checker

Check robots.txt, llms.txt, sitemap hints, and AI crawler blocking rules for websites and domains.

User avatar

Daniel Christensen

2

Robots Sitemap Analyzer - SEO Crawl Rules avatar

Robots Sitemap Analyzer - SEO Crawl Rules

benthepythondev/robots-sitemap-analyzer

Analyze robots.txt files and discover sitemap URLs, user-agent groups, allow rules, disallow rules and crawl-delay directives.

User avatar

Ben

2