Site URL Mapper & SEO Crawler
Pricing
from $2.00 / 1,000 url mappeds
Go to Apify Store
Site URL Mapper & SEO Crawler
Pricing
from $2.00 / 1,000 url mappeds
For example https://www.example.com/. Pages on the same domain are followed.
Stop after this many pages have been checked.
How many link clicks away from the start page to follow. 0 checks only the start page.
Also crawl subdomains of the site, such as blog.example.com.
Skips pages the site asks crawlers not to visit. Turn off only for your own site.
Read the site's XML sitemap to find pages, including pages that nothing links to.
Treat the same page with different ?query parameters as separate pages instead of merging them.
Regular expressions, for example /blog/.
Regular expressions, for example /tag/ or ?page=.
Up to 500 outbound links, listed in SITE_SUMMARY.