Sitemap Finder

A powerful Apify Actor that finds sitemap URLs for any website. This Actor helps you discover XML sitemaps by checking common locations, robots.txt files, and analyzing HTML content for sitemap links.

Features

• Checks common sitemap locations (e.g., /sitemap.xml, /sitemap-index.xml, etc.)
• Parses robots.txt for Sitemap directives
• Analyzes website HTML for sitemap links
• Verifies discovered URLs are valid XML sitemaps
• Option to find either the primary sitemap or all available sitemaps
• Configurable request timeout and verbose logging

Input Parameters

Parameter	Type	Description
url	String	The URL of the website to search for sitemaps (required)
findAll	Boolean	If true, find all available sitemaps. If false, find only the primary sitemap (default: true)
noVerify	Boolean	If true, skip verification of sitemap XML format (default: false)
timeout	Integer	Timeout in seconds for HTTP requests (default: 5)
verbose	Boolean	Enable detailed logging of the sitemap discovery process (default: false)

Output

The Actor saves results to the default dataset with the following structure:

When findAll = true

{
  "url": "example.com",
  "sitemaps": [
    "https://example.com/sitemap.xml",
    "https://example.com/post-sitemap.xml"
  ],
  "count": 2
}

When findAll = false

{
  "url": "example.com",
  "sitemap": "https://example.com/sitemap.xml"
}

Example Usage

You can run the Actor on the Apify Platform or integrate it into your own applications using the Apify API.

Running on Apify Platform

1. Navigate to the Actor page
2. Enter the website URL you want to search for sitemaps
3. Configure optional parameters as needed
4. Run the Actor and wait for results

API Example

import apify_client

# Initialize the ApifyClient with your API token
client = apify_client.ApifyClient(token="YOUR_API_TOKEN")

# Prepare the Actor input
run_input = {
    "url": "example.com",
    "findAll": True,
    "verbose": True
}

# Run the Actor and wait for it to finish
run = client.actor("YOUR_ACTOR_ID").call(run_input=run_input)

# Fetch and print Actor results from the default dataset
results = client.dataset(run["defaultDatasetId"]).list_items().items
print(results)

Use Cases

• SEO Analysis: Quickly locate all sitemaps to analyze website structure and content organization
• Web Scraping: Find sitemaps to extract URLs for crawling
• Website Auditing: Check if sitemaps are properly configured and accessible
• Content Discovery: Identify pages and sections included in sitemaps

Resources

• Apify SDK for Python documentation
• Apify Platform documentation
• Join our developer community on Discord