Pricing

from $0.16 / 1,000 url extracteds

Try for free

Go to Apify Store

Sitemap URL Extractor - All URLs from robots.txt & sitemaps

Try for free

Extract every URL of a website from its sitemaps: robots.txt discovery, sitemap indexes, .xml.gz, lastmod, hreflang. Filters by pattern and date. $0.20 per 1,000 URLs.

Pricing

from $0.16 / 1,000 url extracteds

Rating

0.0

(0)

Developer

Emmanuel Vingadassalon

Actor stats

Bookmarked

Total users

Monthly active users

3 hours ago

Last modified

Categories

SEO tools

Developer tools

Automation

You can access the Sitemap URL Extractor - All URLs from robots.txt & sitemaps programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1import { ApifyClient } from 'apify-client';
2
3// Initialize the ApifyClient with your Apify API token
4// Replace the '<YOUR_API_TOKEN>' with your token
5const client = new ApifyClient({
6    token: '<YOUR_API_TOKEN>',
7});
8
9// Prepare Actor input
10const input = {
11    "startUrls": [
12        "apify.com"
13    ]
14};
15
16// Run the Actor and wait for it to finish
17const run = await client.actor("propped_cloak/sitemap-url-extractor").call(input);
18
19// Fetch and print Actor results from the run's dataset (if any)
20console.log('Results from dataset');
21console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
22const { items } = await client.dataset(run.defaultDatasetId).listItems();
23items.forEach((item) => {
24    console.dir(item);
25});
26
27// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

Sitemap URL Extractor - All URLs of a Website API in JavaScript

The Apify API client for JavaScript is the official library that allows you to use Sitemap URL Extractor - All URLs from robots.txt & sitemaps API in JavaScript or TypeScript, providing convenience functions and automatic retries on errors.

Install the apify-client

$npm install apify-client

Other API clients include:

Sitemap URL Extractor - All URLs from robots.txt & sitemaps API in Python

Sitemap URL Extractor - All URLs from robots.txt & sitemaps API through CLI

Sitemap URL Extractor - All URLs from robots.txt & sitemaps OpenAPI definition

Sitemap URL Extractor - All URLs from robots.txt & sitemaps API

AI Search Visibility Tracker: ChatGPT, Perplexity

adyg/ai-search-visibility-tracker

Track brand mentions, share of voice and cited URLs in ChatGPT, Perplexity, Gemini, Google AI Overviews, AI Mode and Claude. Per prompt x engine: full answer, mention rank, sentiment, citations, your domain cited or not. Monitoring mode flags gained/lost mentions. GEO/AEO rank tracker.

Adyanth Ganesh

AI Visibility Tracker

blueskyscraper/ai-visibility-tracker

[from $0.0375/check] Track brand mentions in ChatGPT, Google AI Overview, AI Mode, Gemini, Perplexity and Claude: mentioned, position, cited domains, competitors, tone, snippet. Share of voice. Export, run via API or use with AI agents.

BlueskyScraper

Indonesia KBLI 2025 Converter & Search

lokalapi/indonesia-kbli

Convert KBLI 2020 business codes to KBLI 2025 (mandatory in OSS since June 2026) and find KBLI 2025 codes for a business activity. Official BPS data, flags splits and merges that need review.

lokalapi Data

Static SEO Audit - Score and Fixes for URL Lists

quillray/static-seo-audit

Audit a list of pages for 18 on-page SEO issues. Get a 0-100 score, a fix for every issue and what changed since your last audit. No browser, pay per page.

Quillray

Web Search

apify/web-search

Search the web and get organic results as JSON with title, URL, snippet, and position. You can filter by date, country, language, and domain. The Actor provides a real-time API.

Apify

llms.txt Generator & Checker - llms-full.txt, AI SEO (GEO)

tidytools/llms-txt-generator

Generate llms.txt and llms-full.txt for any website from its sitemap or a crawl, with sections and AI descriptions, or validate existing llms.txt files in bulk. $2 per 1,000 pages or sites.

Yukai Lin

Sitemap Extractor: All Website URLs + New/Removed Pages

pewterdesk/sitemap-url-inventory-diff

Sitemap extractor that lists every URL a website publishes, with lastmod, images and hreflang. Finds sitemaps via robots.txt; handles nested indexes, gzip and text sitemaps. Compare runs to see new, updated and removed pages, and optionally check HTTP status.

PewterDesk

Google Trends Agent Summary

nickatnite79/google-trends-agent-summary

Google Trends interest over time, top and rising related queries, and a plain-English rising/falling/flat/mixed verdict per keyword, ready for AI agents. Pay only for successful keyword reports.

Nicholas Tamagni

Sitemap Generator

datawinder/sitemap-generator

Automatically crawl a website and generate an SEO-ready sitemap in XML, HTML, or TXT format. Supports crawl depth limits, URL include/exclude patterns, and optional merging with an existing sitemap.xml. Ideal for SEO audits, site migrations, and automation.

DatawinderLabs

Keyword Cannibalization & SEO Collision Monitor

quanmatrix/keyword-cannibalization-seo-collision-intelligence

Compare recurring SERP and ranked-keyword datasets to detect pages competing for the same query, ranking collisions, cannibalization drift and SEO consolidation opportunities.