Pricing
Pay per usage
Monitor website sitemaps for new, updated, or removed URLs. Integration with the Website Content Crawler (WCC) allows feeding only relevant URLs. This ensures your web crawls are efficient, targeted, and resource-optimized, keeping your datasets fresh for any application.
0.0 (0)
Pricing
Pay per usage
1
8
3
Last modified
2 months ago
startUrlsarrayOptional
List of start URLs to scrape. These can be direct sitemap urls or website on which the sitemaps are going to be found if the discoverSitemaps is enabled.
discoverSitemapsbooleanOptional
If enabled, the actor will fetch each start URL's robots.txt and enqueue any sitemap URL it finds. This is useful if you don't want to enter direct sitemap URLs. Please note that this will only work if the website has robots.txt.
Default value of this property is true
changeTypesarrayOptional
Which change types to include in the output.
Default value of this property is ["NEW","UPDATED"]
snapshotKeyPrefixstringOptional
Prefix for the snapshot record key stored in the snapshots key-value store, to separate runs by website or project.
Default value of this property is "DEFAULT"
urlFilterRegexstringOptional
Regex pattern to filter which URLs are included in the output and snapshot. This filter applies only to the final URLs and not to intermediate sitemap URLs.
addRemovedUrlsToKvsbooleanOptional
If enabled, the actor will always also include URLs that were removed compared to the previous snapshot to the key-value store.
Default value of this property is false
proxyConfigurationobjectOptional
Proxy configuration used for crawling.
Default value of this property is {"useApifyProxy":true}
scdMemoryEnumOptional
Amount of memory (RAM) allocated to the actor run in megabytes.
"32768": string"16384": string"8192": string"4096": string"2048": string"1024": string"512": stringDefault value of this property is "4096"
scdTimeoutintegerOptional
Timeout for the actor run in seconds. Zero value means there is no timeout, and the Actor runs until completion, or maybe infinitely. Default is 360,000 seconds (100 hours).
Default value of this property is 360000
addWccUrlsToScdbooleanOptional
If enabled, start URLs from Website Content Crawler will be treated as start URLs defined within this orchestrator.
Default value of this property is true
wccMaxUrlsPerRunintegerOptional
How many URLs from the Sitemap Change Detecter is there going to be in a single Website Content Crawler run. Note that each run's default dataset will be merged and output after all runs complete.
Default value of this property is 50000
wccMemoryEnumOptional
Amount of memory (RAM) allocated to the actor run in megabytes.
"32768": string"16384": string"8192": string"4096": string"2048": string"1024": string"512": stringDefault value of this property is "4096"