Website Crawl Map & Sitemap Diff

Map the public URLs on one or more websites, preserve the internal links between them, and compare the crawl with public XML sitemaps. Website Crawl Map & Sitemap Diff returns a migration-ready URL inventory with HTTP status, redirects, canonicals, indexability, depth, every observed parent page, anchor text, and honest sitemap-only “orphan candidate” labels.

For a useful first run, enter one website, keep Auto rendering, set Max URL records to 100 and Max pages crawled to 50, then inspect the URL Inventory and Crawl Summary outputs before increasing the limits.

What does Website Crawl Map & Sitemap Diff do?

The Actor starts from public HTTP or HTTPS pages and follows links within the selected website scope. At the same time, it looks for public XML sitemaps, including sitemap indexes declared in robots.txt. It merges both sources into one clean URL inventory.

Each result explains not only which URL was found, but also:

• whether it came from links, a sitemap, or both;
• the minimum observed click depth from the submitted start page;
• every observed source page and anchor text, subject to your evidence-list limit;
• whether the URL was fetched, redirected, failed, or remained sitemap-only;
• its HTTP status, final URL, response type, canonical, and robots directives;
• whether it looks indexable, canonicalized elsewhere, blocked by noindex, or not crawled within the selected page budget;
• the sitemap file and lastmod, changefreq, and priority values when supplied;
• stable issue codes for filtering and downstream automation.

The Actor does not extract page-body content, log in, submit forms, or claim access to search-engine indexes. It maps public URL and link evidence. It can be called from the Apify API, run on a schedule, connected to webhooks, or used as the first step in a larger scraping or website-migration pipeline.

Why use this website URL crawler?

Build a defensible URL inventory

A normal link crawler misses URLs that are declared only in sitemaps. A sitemap parser misses reachable pages that were never added to the sitemap. This Actor combines both and reports the evidence source for every URL.

Preserve the internal-link graph

Many URL tools keep one parent URL and discard the rest. Website Crawl Map records all unique observed source-target-anchor-rel link edges up to an explicit graph limit. The URL row contains convenient parent and anchor arrays, while LINK_GRAPH.csv contains one row per observed edge.

Find honest orphan candidates

A URL is labeled orphan_candidate only when it appears in a sitemap but no internal HTML link to it was observed during this bounded crawl. That is useful evidence, but it is not proof that the page has no inbound link anywhere. Links behind login pages, forms, unvisited pages, scripts, or a tighter crawl budget may still exist.

Control cost and crawl size before starting

The input exposes separate limits for stored URL records, crawled pages, sitemap URLs, sitemap files, graph edges, depth, links per page, redirects, retries, response size, browser fallbacks, concurrency, and wall-clock work time. You can sample a website without accidentally requesting an unbounded crawl.

Export operational files, not only dataset rows

Alongside the standard Apify dataset, the Actor can create a flat URL inventory, link edge list, sitemap-diff file, run summary, and a conservative generated XML sitemap. These files are designed for spreadsheets, SQL imports, migration workbooks, QA tools, and downstream Actors.

What data can this website crawler extract?

Field	Type	What it tells you
url	string	Normalized URL. This is the portfolio-standard item URL field.
source_classification	string	both, crawl_only, sitemap_only, or external_link.
orphan_candidate	boolean	True for internal sitemap-only URLs other than the start URL.
crawl_status	string	Whether the URL was crawled, discovered, or failed.
depth	integer	Minimum observed hyperlink distance from the start URL.
parent_urls	array	Unique pages that linked to this URL, capped by input.
anchor_texts	array	Unique observed anchor text values, capped by input.
inbound_link_count	integer	Count of unique source-target-anchor-rel observations.
unique_parent_count	integer	Count of distinct parent pages.
status_code	integer	Final HTTP status when the URL was fetched.
final_url	string	Final normalized URL after redirects.
redirect_chain	array	Requested and redirected URLs observed during navigation.
content_type	string	Returned response type, such as HTML or PDF.
response_time_ms	integer	End-to-end response time for the navigation.
title	string	Page title when HTML was parsed.
meta_description	string	Public meta description when present.
canonical_url	string	Resolved canonical link when present.
robots_meta	string	Page-level robots directives.
x_robots_tag	string	HTTP-header robots directives.
indexability	string	Evidence-based status such as indexable, noindex, or redirect.
sitemap_url	string	First sitemap file that declared the URL.
sitemap_lastmod	string	Last-modified value supplied by the sitemap.
issue_codes	array	Stable flags for filtering and downstream automation.

Issue codes include orphan_candidate, missing_from_sitemap, deep_page, http_4xx, http_5xx, redirect, redirect_chain, redirect_off_scope, fetch_failed, response_too_large, non_html, noindex, canonical_missing, canonicalized_elsewhere, nofollow_only, and sitemap_url_not_crawled.

How to crawl a website and compare its sitemap

1. Open the Actor and select the Input tab.
2. Add one public website, page URL, or bare domain under Website URLs.
3. Keep Crawl scope set to Same registrable domain when the site uses useful subdomains. Choose Same host for a stricter inventory.
4. For a first run, set Max URL records to 100, Max pages crawled per website to 50, and Max crawl depth to 3.
5. Leave Discover XML sitemaps enabled.
6. Use Auto page rendering. Choose HTTP only for predictable server-rendered sites, or browser only when you know navigation links appear after scripts run.
7. Start the Actor and open URL Records.
8. Review source_classification, indexability, and issue_codes before broadening the limits.
9. Download URL_INVENTORY.csv, LINK_GRAPH.csv, or SITEMAP_DIFF.csv from the output links when you need a spreadsheet or migration artifact.

Minimal API input

{
  "startUrls": [{"url": "https://example.com"}],
  "maxResults": 100,
  "maxPagesPerStartUrl": 50,
  "maxDepth": 3
}

Multi-site migration input

{
  "startUrls": [
    {"url": "https://www.example.com"},
    {"url": "https://docs.example.org"}
  ],
  "crawlScope": "same-domain",
  "includeUrlPatterns": ["*/products/*", "*/docs/*", "*/blog/*"],
  "excludeUrlPatterns": ["*/logout*", "*/cart*", "*?replytocom=*"],
  "queryParameterPolicy": "drop-tracking",
  "assetPolicy": "pages-and-documents",
  "discoverSitemaps": true,
  "renderMode": "auto",
  "maxResults": 5000,
  "maxPagesPerStartUrl": 2000,
  "maxDepth": 8,
  "generateXmlSitemap": true
}

Input options

See the Input tab for the complete form and current bounds.

Website and URL controls

Input	Default	Guidance
startUrls	https://example.com	Add 1–20 public HTTP/HTTPS seeds.
crawlScope	same-domain	Includes subdomains that share the registrable domain.
includeUrlPatterns	empty	Restricts saved URL records but allows traversal through intermediate internal pages.
excludeUrlPatterns	empty	Matching URLs are neither saved nor followed. Exclusions win over inclusions.
queryParameterPolicy	drop-tracking	Removes common analytics parameters and sorts the rest for stable deduplication.
assetPolicy	pages-and-documents	Saves HTML-style pages and common document links, but not images/scripts/media.

Patterns use shell-style matching. */blog/* matches a blog section. *?preview=* matches URLs containing that query form. Test filters with a small result limit before running a large crawl.

Sitemap and rendering controls

Input	Default	Guidance
discoverSitemaps	true	Combines declared sitemap URLs with crawl discoveries.
maxSitemapsPerStartUrl	50	Bounds nested sitemap-index work.
maxSitemapUrlsPerStartUrl	5,000	Bounds URL entries merged from sitemaps.
renderMode	auto	Uses fast page requests first, then a small browser fallback for likely script shells.
browserFallbackMaxPages	20	Prevents auto mode from silently rendering a large site.
followNofollow	true	Nofollow links are always recorded; this controls whether internal ones are followed.

Page, graph, and time budgets

maxResults caps stored URL records across the run; failed-fetch rows inside that cap remain uncharged. maxPagesPerStartUrl caps unique internal pages navigated for each seed. They are not the same: a sitemap can add discovered records without those pages being fetched, and a narrow include pattern can require traversal through pages that are not saved.

maxEdges caps link-graph observations. Parent and anchor arrays on each URL are separately capped by maxParentUrlsPerRecord, while aggregate counts remain complete up to the graph-edge cap.

maxRunSeconds is a work budget, not the container timeout. The Actor stops crawl and sitemap work before this budget ends so it can store partial useful output and create the requested files. Set the Apify run timeout higher than maxRunSeconds.

Output examples

You can download the dataset in formats such as JSON, HTML, CSV, XML, or Excel through Apify’s standard dataset export controls.

URL found through both navigation and sitemap

{
  "result_type": "url_record",
  "diagnostic_item": false,
  "start_url": "https://example.com/",
  "url": "https://example.com/docs",
  "relationship": "internal",
  "link_type": "page",
  "discovery_sources": ["link", "sitemap"],
  "source_classification": "both",
  "orphan_candidate": false,
  "crawl_status": "crawled",
  "fetch_mode": "http",
  "depth": 1,
  "parent_url": "https://example.com/",
  "parent_urls": ["https://example.com/", "https://example.com/help"],
  "anchor_texts": ["Documentation", "Read the docs"],
  "inbound_link_count": 2,
  "unique_parent_count": 2,
  "status_code": 200,
  "final_url": "https://example.com/docs",
  "redirect_chain": ["https://example.com/docs"],
  "content_type": "text/html; charset=utf-8",
  "response_time_ms": 84,
  "title": "Documentation",
  "canonical_url": "https://example.com/docs",
  "indexability": "indexable",
  "sitemap_url": "https://example.com/sitemap.xml",
  "sitemap_lastmod": "2026-08-31",
  "issue_codes": [],
  "scraped_at": "2026-09-02T12:00:00Z"
}

Sitemap-only orphan candidate

{
  "result_type": "url_record",
  "diagnostic_item": false,
  "start_url": "https://example.com/",
  "url": "https://example.com/old-campaign",
  "discovery_sources": ["sitemap"],
  "source_classification": "sitemap_only",
  "orphan_candidate": true,
  "crawl_status": "discovered",
  "depth": null,
  "inbound_link_count": 0,
  "indexability": "not_crawled",
  "issue_codes": ["orphan_candidate", "sitemap_url_not_crawled"]
}

No-charge diagnostic

{
  "result_type": "diagnostic",
  "diagnostic_item": true,
  "diagnostic_type": "source_unavailable",
  "start_url": "https://example.invalid/",
  "reason": "no_chargeable_urls",
  "message": "No chargeable URL records were found for this start URL.",
  "recommended_action": "Check filters, increase limits, try browser mode, or enable an appropriate proxy."
}

Diagnostic rows are operational evidence. They do not create url-record charges.

Downloadable crawl artifacts

SUMMARY

JSON with duration, selected rendering mode, stored record count, charge-eligible and platform-confirmed charged events, diagnostics, classification totals, issue totals, artifact errors, and per-site page, edge, sitemap, failure, warning, and limit information.

URL_INVENTORY.csv

A flat version of the accepted URL dataset suitable for a migration workbook, content inventory, SQL import, or spreadsheet review.

LINK_GRAPH.csv

One row per observed unique link edge with start URL, source URL, target URL, anchor text, rel values, internal/external relationship, and target depth. This is the most complete output for internal-link analysis.

SITEMAP_DIFF.csv

A focused internal-URL export showing whether each URL appeared in the crawl, a sitemap, or both. Filter source_classification=sitemap_only for orphan candidates and crawl_only for pages that may be missing from the sitemap.

GENERATED_SITEMAP.xml

Optional conservative sitemap containing deduplicated internal pages that finished with HTTP 200, returned HTML, and were classified as indexable. Review it before publishing: the Actor cannot decide your site’s complete editorial or SEO policy.

How much does it cost to crawl a website?

The repository pricing source proposes pay-per-event pricing:

• URL record: $0.005 after one useful unique, non-failed URL row is stored— equivalent to $5.00 per 1,000 chargeable records before any future plan-tier changes.
• Browser-rendered page: $0.003 after a successful non-empty rendered page is processed and represented by a stored URL row.

HTTP 4xx and 5xx rows are useful audit data and can be billable URL records because they answer whether a discovered URL works. Invalid-input, blocked/empty, unsafe- destination, and other diagnostic-only rows are not charged. Failed-fetch URL rows may preserve discovery/error evidence, but create neither a url-record nor a browser render charge.

The live Store Pricing tab is the source of truth for current prices. Set a maximum charge when starting a run if you need a hard billing ceiling.

Tips for accurate and efficient URL maps

Start with a sample

Use 50 crawled pages, 100 URL records, depth 3, and HTTP or Auto mode. Inspect the summary for sitemap counts, browser fallbacks, failures, and limit flags. Increase one budget at a time.

Prefer HTTP mode when it works

HTTP mode is normally faster and cheaper. Auto mode is a safe default because it only escalates pages that resemble empty script-driven shells, and browser fallback has its own explicit cap. Browser-only mode is best reserved for sites whose navigation truly requires rendering.

Separate URL inventory size from page-audit size

If you need a large sitemap inventory but only a small technical sample, use a high maxSitemapUrlsPerStartUrl and a lower maxPagesPerStartUrl. Sitemap-only records will have indexability=not_crawled because no HTTP claim was manufactured.

Use include patterns for focused exports

An include pattern restricts saved records without necessarily blocking traversal. This allows the Actor to cross category or navigation pages on the way to matching product, blog, or documentation URLs. Use exclusions for URLs that must not be visited.

Be deliberate with query parameters

The default removes common tracking parameters while retaining functional parameters. Choose strip-all only when parameters never identify distinct pages on the target. Choose keep-all when product variants, filters, pagination, or application state in the query string must remain distinct. A poor query policy can either duplicate pages or merge legitimately different URLs.

Understand partial output

The Actor finalizes partial useful results when a page, sitemap, edge, response-size, browser, result, or run-time limit is reached. Check SUMMARY.limits before treating an inventory as complete. “No issue found” within a bounded crawl is not proof that no issue exists elsewhere.

FAQ

Can it find every URL on a website?

No crawler can guarantee every URL. Pages may be private, unlinked, generated only after form submissions, blocked, outside the selected scope, or beyond a configured limit. The Actor reports its sources and truncation state so you can judge coverage.

What is an orphan candidate?

It is an internal URL declared in a parsed sitemap for which this bounded crawl did not observe an HTML link. It is a review candidate, not proof of a globally orphaned page.

Does it check broken links?

It records status and redirect evidence for URLs that were actually fetched. Sitemap- only or budget-limited discovered URLs may not have HTTP facts. Use crawl_status and indexability to distinguish audited pages from discoveries.

Why is a reachable URL marked missing from sitemap?

The Actor parsed at least one sitemap, found the URL through the start page or an HTML link, and did not see it in those parsed sitemaps. The site may publish another sitemap outside the discovery paths or beyond your sitemap limits.

Does it obey robots.txt?

The Actor reads public sitemap declarations from robots.txt; it does not use the file as an automatic crawl blocker. You are responsible for choosing lawful targets and settings and for complying with website terms and applicable rules.

Can it crawl external websites linked from my site?

No. The include-external scope saves outbound URLs as discovered evidence but does not follow them. This prevents one submitted site from expanding into an open-ended web crawl.

Can it crawl private networks or localhost?

No. Seeds, sitemap documents, queued links, and redirects are restricted to publicly resolvable Internet destinations. This safety boundary is not configurable.

Why did Auto mode not render a page?

Auto mode uses a conservative heuristic to control cost. Select browser-only mode when you know that navigation links require rendering, and set an appropriate page budget.

Where can I report a problem?

Open the Actor’s Issues tab and include the run ID, start URL, selected mode, and the relevant diagnostic or summary warning. Do not include credentials, cookies, or private URLs. Use the API tab for current code examples in Python, JavaScript, and direct HTTP requests.

Legal and compliance

This Actor collects public URLs and technical page metadata. It does not intentionally extract private user data, bypass logins, or submit forms. You are responsible for reviewing the target website’s terms, crawl policies, intellectual-property rights, and applicable laws before running it. Use conservative concurrency and delay settings and avoid disrupting target services.

Public pages can still contain personal data in URLs, titles, descriptions, or anchor text. Personal data is protected by the GDPR in the European Union and by other laws around the world. Do not collect or process personal data unless you have a legitimate purpose and an appropriate legal basis. Apply retention, access-control, deletion, and data-minimization practices to exported datasets and files. If you are unsure whether your use is lawful, consult qualified legal counsel.

Website Crawl Map provides technical evidence, not legal advice, an SEO guarantee, or proof of global site structure. Review output before using it to publish sitemaps, remove pages, change canonicals, or deploy redirects.