Pricing

from $2.00 / 1,000 domain summaries

Try for free

Go to Apify Store

Sitemap Extractor & Site Map: robots.txt, RSS & llms.txt

Try for free

Map any website without crawling: robots.txt rules for your user agent (incl. AI-bot blocks like GPTBot), every sitemap and sitemap index (.gz too), RSS/Atom/JSON feeds, llms.txt and llms-full.txt, plus a filtered URL list with lastmod. By Steadyline Data.

Pricing

from $2.00 / 1,000 domain summaries

Rating

0.0

(0)

Developer

Steadyline Data

Actor stats

Bookmarked

Total users

Monthly active users

6 hours ago

Last modified

Sitemap Extractor: Website → All URLs (sitemap.xml parser)

boxbox10/sitemap-extractor

Give it a website. Get every URL from its sitemap — loc, lastmod, changefreq, priority — as one clean record per URL. Auto-discovers sitemap.xml, robots.txt Sitemap: directives, and nested sitemap indexes. Perfect for SEO audits, crawl seeding, and URL discovery.

Marvin Eguilos

Sitemap Extractor

apify/sitemap-extractor

This Apify Actor extracts all URLs from a website's sitemaps and checks their status codes via lightweight HTTP requests. It provides a clean list of valid links, acting as an ideal pre-processor to ensure your larger crawling projects target only active URLs.

Apify

436

3.2

(8)

GEO Site Audit - AI Readiness Checker

dltik/geo-site-audit

Audit your website for AI crawler accessibility: robots.txt (GPTBot, ClaudeBot, Perplexity), llms.txt, sitemap, Schema.org, meta tags, content extractability, TTFB. Get an AI-readiness score 0-100 with prioritized recommendations.

Walid

5.0

(1)

Site Researcher

rahulhinduja/site-researcher

Stop wasting hours on competitor research. Paste any URL → get every image, video, and data point in 60 seconds. Full site crawl, tech stack detection, media download. Perfect for agencies, creators, and marketers.

Rahul Hinduja

XML Sitemap Scraper & URL Extractor API - SEO Crawler

pink_comic/sitemap-url-extractor

Parse XML website sitemaps and extract URLs for SEO crawls, audits, migrations, and RAG indexing. Auto-discovers sitemap files from robots.txt, follows nested indexes, and exports URL, lastmod, priority, changefreq, and image metadata.

Ava Torres

Sitemap & URL Discovery - Find All URLs on Any Site

santamaria-automations/sitemap-url-discovery

Discover every URL on any website by parsing sitemap.xml, robots.txt, and sitemap indexes. Extract URLs with last modified dates, change frequency, and priority. Perfect for SEO audits, content analysis, crawling preparation, and site mapping.

NanoScrape

Sitemap Generator

alizarin_refrigerator-owner/sitemap-generator

Generate XML sitemaps by crawling any website. Discover all pages, images, & videos with configurable crawl depth, URL filters, & multiple output formats. Full Site Crawling ,Image Sitemap, Video Sitemap, Multiple Output Formats, URL Filtering, Configurable Depth, Last Modified, Webhook Integration

The Howlers

llms.txt Generator: llms-full.txt for Any Website (AI SEO)

fayoussef/llms-txt-generator

Generate a spec-compliant llms.txt and llms-full.txt for any website in one run. Crawls your sitemap or internal links, extracts every page's title, meta description and full Markdown content, and outputs two ready-to-upload files for ChatGPT, Claude, Perplexity and Gemini.

Youssef Farhan

Site to LLM Knowledge Base

adambounhar/site-to-knowledge-base

Turn any website or docs into clean, LLM-ready Markdown for RAG and AI agents — one record per page, each with a token count. Sitemap- and robots.txt-aware, with predictable per-page pricing (no token credits). Simple knowledge-base ingestion.