Pricing

$0.40 / 1,000 results

Try for free

Go to Apify Store

Sitemap URL Extractor + Robots.txt & llms.txt Validator

Try for free

Extract every URL from a site's sitemaps and check the sitemap files, robots.txt (rules for AI bots) and the structure of llms.txt. Streaming parser, flat memory; sites that cannot be read are free.

Pricing

$0.40 / 1,000 results

Rating

0.0

(0)

Developer

Francesco Antonio Russo

Actor stats

Bookmarked

Total users

Monthly active users

4 days ago

Last modified

Sitemap URL Extractor + Robots.txt & llms.txt Validator API through CLI

The Apify CLI is the official tool that allows you to use Sitemap URL Extractor + Robots.txt & llms.txt Validator locally, providing convenience functions and automatic retries on errors.

Install Apify CLI

Using installation script (macOS/Linux):

$curl -fsSL https://apify.com/install-cli.sh | bash

Using installation script (Windows):

$irm https://apify.com/install-cli.ps1 | iex

Using Homebrew:

$brew install apify-cli

Using npm:

$npm install -g apify-cli

Other API clients include:

Sitemap URL Extractor + Robots.txt & llms.txt Validator API in Python

Sitemap URL Extractor + Robots.txt & llms.txt Validator API in JavaScript

Sitemap URL Extractor + Robots.txt & llms.txt Validator OpenAPI definition

Sitemap URL Extractor + Robots.txt & llms.txt Validator API

SEO Site Audit — Website SEO Checker & Crawler

saliga/seo-site-audit

Crawl a website and audit every page for technical and on-page SEO: a 0-100 score, issues with fix hints, broken internal links, duplicate titles, orphan pages, canonical and robots problems, plus a site summary. $3 per 1,000 pages.

Salih Küçükkayış

SiteVerdict — robots.txt & Sitemap Monitor

siteverdict/siteverdict-robots-sitemap-monitor

Catch SEO disasters before Google does: watches robots.txt, XML sitemaps and key pages for Disallow: /, noindex, canonical and status changes, sitemap URL drops and errors. Diffs vs last run, Slack/webhook alerts, HTML report.

Aman Thakran

Lighthouse Batch Audits & Regression Alerts

peerless_columbine/lighthouse-batch-audit-monitor

Audit public pages with Lighthouse13.5.0, export raw and HTML reports, and compare persistent category baselines.

tingyou333 zhuang

sitemap-api.com

sitemap-api/sitemap-api-com

sitemap-api

Sitemap URL Extractor — recursive, filters, .gz

x402opklaar/sitemap-url-extractor

Extract every URL from any sitemap: recursive sitemap-index expansion, gzipped sitemaps, robots.txt auto-discovery, include/exclude filters, lastmod output. Feed the URL list to any crawler or audit. Pay per sitemap file processed.

Opklaar

Redirect Checker & Bulk URL Status: Chains, Loops, 301 Map QA

cygnix_labs/redirect-checker

Bulk redirect checker and HTTP status code checker. Traces every hop (301/302/307/308, meta refresh) and flags redirect chains, loops, 404s, soft 404s, noindex targets and redirect-map mismatches after a site migration. First 20 URLs free, then $0.0008/URL.

Umair Saleem Mohammad Saleem

Shopify Store Auditor: SEO & Product Catalog Audit

siftwright_data/shopify-store-auditor

Audit any Shopify store's catalog: thin or missing descriptions, images without alt text, duplicate titles, zero prices, missing product types and tags. 0-100 score per store plus every flagged product. No login or app needed.

Siftwright Data

Bulk URL Redirect, Status & SSL Health Checker

reaperoak/bulk-url-redirect-status-ssl-health-checker

Check hundreds of URLs at once: HTTP status, the full redirect chain (301/302, http to https, www), security headers and SSL certificate expiry. One health verdict and issue list per URL. No login, no proxy.

Owais Khan

Broken Link Checker (internal and external)

nitsuahlabs/broken-link-checker

Crawl a website and list every broken link with the pages it appears on.

Austin Hardy

AI Agent Web Readiness Auditor

unrivaled_fortress/agent-web-readiness-auditor

Score any site for AI-agent visibility: llms.txt, robots.txt AI-crawler matrix (GPTBot, ClaudeBot, PerplexityBot+), and JSON-LD presence. The GEO-era successor to robots.txt audits.