List of start URLs to scrape. These can be direct sitemap urls or website on which the sitemaps are going to be found if the discoverSitemaps is enabled.

Discover sitemaps

discoverSitemapsbooleanOptional

If enabled, the actor will fetch each start URL's robots.txt and enqueue any sitemap URL it finds. This is useful if you don't want to enter direct sitemap URLs. Please note that this will only work if the website has robots.txt.

Default value of this property is true

Change types

changeTypesarrayOptional

Maximum number of URLs the Website Content Crawler will process in each run. Increasing this decreases the total number of runs needed to process all URLs from the Sitemap Change Detector. Note that the run datasets will be merged and output afterwards.

Default value of this property is ["NEW","UPDATED"]

Snapshot key prefix

snapshotKeyPrefixstringOptional

Prefix for the snapshot record key stored in the snapshots key-value store, to separate runs by website or project.

Default value of this property is "DEFAULT"

URL filter regex

urlFilterRegexstringOptional

Regex pattern to filter which URLs are included in the output and snapshot. This filter applies only to the final URLs and not to intermediate sitemap URLs.

Add removed URLs to key-value store

addRemovedUrlsToKvsbooleanOptional

If enabled, the actor will always also include URLs that were removed compared to the previous snapshot to the key-value store.

Default value of this property is false

Proxy configuration

proxyConfigurationobjectOptional

Proxy configuration used for crawling.

Default value of this property is {"useApifyProxy":true}

Sitemap Change Orchestrator

tri_angle/sitemap-change-orchestrator

Monitor website sitemaps for new, updated, or removed URLs. Integration with the Website Content Crawler (WCC) allows feeding only relevant URLs. This ensures your web crawls are efficient, targeted, and resource-optimized, keeping your datasets fresh for any application.

Tri⟁angle

Sitemap URL Extractor

onescales/sitemap-url-extractor

Provide a link to a sitemap.xml and the app will extract and list all URLs in the sitemap as well as additional data in the sitemap (i.e. https://onescales.com/sitemap.xml).

One Scales

5.0

Website Changes Detector

tri_angle/website-changes-detector

This actor uses Apify’s Website Content Crawler to track website changes by comparing new and previous crawl results, highlighting only relevant updates to save time and resources.

Tri⟁angle

Website Change Detector

eloquent_mountain/website-change-detector

Monitors websites for changes. Detects modifications to HTML structure and visual differences via screenshots. Provides detailed change reports including HTML diff. Track multiple URLs. Use tasks for recurring runs. Integrate as API

Paco

Sitemap Scraper

pvillalva/sitemap-scraper

The Sitemap Scraper extracts and outputs all URLs from a given sitemap.

Percival Villalva

5.0

Find Sitemap from url

eesti/find-sitemap-from-url

A powerful [Apify Actor] that finds sitemap URLs for any website. This Actor helps you discover XML sitemaps by checking common locations, robots.txt files, and analyzing HTML content for sitemap links.

ando

Sitemap Sniffer

vaclavrut/sitemap-sniffer

Sitemap sniffer will check the most used variants of sitemaps and you can use that for crawling. This will just save you time so you don't have to check manually.

Vaclav Rut

630

5.0

Example Sitemap Cheerio

jancurn/example-sitemap-cheerio

An example actor that first downloads a sitemap in XML format and the crawls each page from the sitemap using the fast CheerioCrawler from Apify SDK.

Jan Čurn

Sitemap To Request Queue

pocesar/sitemap-to-request-queue

Download sitemap XMLs and put them in a RequestQueue

Paulo Cesar

111

LLM Dataset Processor

dusan.vystrcil/llm-dataset-processor

Allows you to process output of other actors or stored dataset with single LLM prompt. It's useful if you need to enrich data, summarize content, extract specific information, or manipulate data in a structured way using AI.