Pricing

from $7.00 / 1,000 results

Try for free

Go to Apify Store

Sitemap Generator

Try for free

Pricing

from $7.00 / 1,000 results

Rating

5.0

(1)

Developer

Sameer Pun

Actor stats

Bookmarked

Total users

Monthly active users

a day ago

Last modified

Sitemap Generator Actor

Python Apify Actor that crawls a single hostname and generates sitemap files:

sitemap.xml (always an XML sitemap index)
sitemap-00001.xml, sitemap-00002.xml, ... (50,000 URLs max per chunk)
sitemap.html (optional)
sitemap.txt (optional)
sitemap-summary.json (run summary and output key references)

The Actor respects robots.txt, includes only canonical URLs, deduplicates by normalized URL, and supports regex include/exclude filters.

Input

Field	Type	Default	Description
`startUrl`	string	required	Start page (`http`/`https`)
`maxDepth`	integer	`3`	Max crawl depth (`startUrl` is depth `0`)
`maxPages`	integer	`1000`	Max fetched pages
`concurrency`	integer	`10`	Concurrent HTTP workers (`1-50`)
`allowNoindex`	boolean	`false`	If true, includes pages with `noindex` directives
`sitemapSeedUrls`	string[]	`[]`	Optional sitemap XML URLs to seed discovery (in addition to `robots.txt` `Sitemap:` entries)
`includePatterns`	string[]	`[]`	Regex allow-list for URLs
`excludePatterns`	string[]	`[]`	Regex deny-list for URLs
`outputFormats`	string[]	`["html","txt"]`	Optional extra outputs (`html`, `txt`)
`lastmodStrategy`	string	`headers`	`headers` or `crawl_time`
`changefreq`	string	`weekly`	Default sitemap `changefreq`
`priorityRules.defaultPriority`	number	`0.5`	Default sitemap priority
`priorityRules.rules`	object[]	`[]`	Ordered regex overrides (`pattern`, optional `priority`, optional `changefreq`)

Run Locally (Apify)

Put your JSON input into storage/key_value_stores/default/INPUT.json.
Run:

$apify run

Example INPUT.json:

{
  "startUrl": "https://example.com/",
  "maxDepth": 2,
  "maxPages": 500,
  "concurrency": 10,
  "allowNoindex": false,
  "includePatterns": [],
  "excludePatterns": ["/private", "/preview"],
  "outputFormats": ["html", "txt"],
  "lastmodStrategy": "headers",
  "changefreq": "weekly",
  "priorityRules": {
    "defaultPriority": 0.5,
    "rules": [
      { "pattern": "/docs/", "priority": 0.8, "changefreq": "daily" }
    ]
  }
}

Run Locally (CLI)

The module also supports direct CLI flags:

python -m src \
  --start-url https://example.com/ \
  --max-depth 2 \
  --max-pages 500 \
  --concurrency 10 \
  --allow-noindex \
  --sitemap-seed-url https://example.com/sitemap.xml \
  --exclude-pattern /private \
  --output-format html \
  --output-format txt \
  --lastmod-strategy headers \
  --changefreq weekly

Priority rules from CLI can be provided as JSON string or file path:

$python -m src --start-url https://example.com/ --priority-rules-json priority-rules.json

CLI precedence is higher than Actor input: CLI > INPUT JSON > defaults.

Output Locations

Dataset: one item per included canonical URL (url, lastmod, changefreq, priority, depth, sourceUrl, statusCode, discoveredAt)
Key-value store records:
- sitemap.xml
- sitemap-00001.xml, sitemap-00002.xml, ...
- sitemap.html (if enabled)
- sitemap.txt (if enabled)
- sitemap-summary.json

Run Tests

$python -m unittest discover -s tests -p "test_*.py"

Integration fixture site is under fixtures/site/.

Sitemap Generator

igview-owner/sitemap-generator

Automatically crawl any website and generate XML, HTML, and text sitemaps for SEO optimization. Perfect for submitting to Google Search Console, Bing Webmaster Tools, and improving search engine indexing. no manual work required. Free sitemap generator tool for WordPress, Blogger, and all website.

Sachin Kumar Yadav

Fast Sitemap Generator

eunit/sitemap-generator

Boost SEO with this automatic Sitemap Generator. Crawl any site to create XML, HTML, & TXT sitemaps. Supports custom depth, regex filters, & robots.txt. Compatible with Google Search Console.

Emmanuel Uchenna

5.0

(1)

Sitemap Generator

alizarin_refrigerator-owner/sitemap-generator

Generate XML sitemaps by crawling any website. Discover all pages, images, & videos with configurable crawl depth, URL filters, & multiple output formats. Full Site Crawling ,Image Sitemap, Video Sitemap, Multiple Output Formats, URL Filtering, Configurable Depth, Last Modified, Webhook Integration

John Rippy

Sitemap Generator - Crawl Website & Create XML Sitemap

scrappy_garden/sitemap-generator

Generate an XML sitemap for any website. Crawls internal pages from start URLs (with depth + page limits), deduplicates URLs, and stores a ready-to-submit sitemap.xml plus a structured dataset and summary for SEO audits.

Bikram Adhikari

Sitemap Generator - Creates sitemap.xml for any domain

wisteria_banjo/sitemap-generator---creates-sitemap-xml-for-any-domain

Generate a clean, standards-compliant sitemap.xml for a website. This actor crawls a single website, discovers all indexable pages, and produces: ✅ A ready-to-submit sitemap.xml (Google-compliant) ✅ A structured JSON dataset of discovered URLs (for auditing, reporting, and billing)

Chris Xavier

Sitemap Generator

datawinder/sitemap-generator

Automatically crawl a website and generate an SEO-ready sitemap in XML, HTML, or TXT format. Supports crawl depth limits, URL include/exclude patterns, and optional merging with an existing sitemap.xml. Ideal for SEO audits, site migrations, and automation.

Datawinder

Sitemap Generator

salman_bareesh/sitemap-generator

A powerful Apify actor that generates XML sitemaps for websites. Perfect for SEO optimization and website indexing.

Salman Bareesh

Sitemap URL Extractor

onescales/sitemap-url-extractor

Provide a website link to a sitemap.xml and the app will extract and list all URLs in the sitemap as well as additional data in the sitemap (i.e. https://onescales.com/sitemap.xml).

One Scales

377

5.0

(2)

YouTube Trending Videos by Categories Scraper

eunit/youtube-trending-videos-by-categories

Scrape real-time YouTube trending videos by country and category. Get detailed metrics like rank, title, views, likes, and channel name from 25+ countries. Monitor viral content, music, gaming, and more.

Emmanuel Uchenna

5.0

(2)

Sqoosh Image Compressor

eunit/sqoosh-image-compressor

Optimize images for SEO with the Squoosh Actor on Apify. Batch compress, resize, and convert images to WebP, AVIF, and MozJPEG to boost site speed and Core Web Vitals. Automate high-performance image optimization for web scraping and developer workflows with ease.

Emmanuel Uchenna

5.0

(3)