Website URL Crawler & Link Extractor

🔗 Build a website URL inventory from links and sitemaps

Website URL Crawler is for SEO specialists, developers, and site owners who need a structured map of a public website. It collects accepted URLs from rendered navigation and public sitemaps, with hierarchy, classification, crawl state, HTTP facts for loaded pages, and available sitemap metadata.

• Website crawler tool — create a URL inventory for a public website.
• Website migration URL inventory — collect paths before planning redirects or comparing site structures.
• Internal linking review — inspect parent pages, anchor text, and crawl depth.
• Sitemap URL extractor — collect URLs and source-provided metadata from public sitemaps.
• Rendered link crawler — find links exposed by page navigation after rendering.
• Broken link workflow crawler — export loaded-page HTTP facts for downstream checks.

📦 Returned data

Each saved row is one accepted URL found through rendered navigation, sitemap discovery, or both. Fields include:

• startUrl and normalized url
• discoverySources: rendered, sitemap, or both
• parentUrl, depth, and anchorText when rendered navigation provides them
• relationship and linkType
• crawlStatus: crawled or discovered
• httpStatusCode, finalUrl, and contentType for loaded pages
• sitemapUrl, lastmod, priority, and changefreq when supplied by a sitemap

Unavailable source metadata remains empty.

🚀 Running the Actor

1. Add one or more public website URLs, page URLs, or bare domains.
2. Choose the crawl scope.
3. Set row, page, depth, and per-page link limits.
4. Run the Actor and open its dataset.

Bare domains such as example.com are normalized to HTTPS. The Actor crawls public pages only; it does not log in, submit forms, scrape full page content, or check search-engine index status.

🧾 Input

Website URLs is required. The other public fields are:

• URL keywords — words or path parts used by the URL filter setting.
• Crawl scope — controls which links can be followed and saved. Same host and same domain save only internal URLs. The external option can also save external URLs, but does not follow them.
• Asset links — selects pages, pages and documents, or all link types.
• Ignored extensions — skips listed extensions unless all links are included.
• Max URL rows — caps accepted rows across the run; 0 uses the other crawl limits only.
• Max pages per website, Max crawl depth, and Max links per page — control traversal breadth.

{
  "urls": [{ "url": "https://example.com" }],
  "keywords": ["docs"],
  "crawlScope": "same-domain",
  "assetPolicy": "include-documents",
  "maxResults": 100,
  "maxPagesPerStartUrl": 10,
  "maxDepth": 2,
  "maxLinksPerPage": 120
}

📤 Output

A realistic dataset row looks like this:

{
  "startUrl": "https://example.com/",
  "url": "https://example.com/docs",
  "discoverySources": ["rendered", "sitemap"],
  "parentUrl": "https://example.com/",
  "depth": 1,
  "anchorText": "Documentation",
  "relationship": "internal",
  "linkType": "page",
  "crawlStatus": "crawled",
  "httpStatusCode": 200,
  "finalUrl": "https://example.com/docs",
  "contentType": "text/html; charset=utf-8",
  "sitemapUrl": "https://example.com/sitemap.xml",
  "lastmod": "2026-05-14",
  "priority": 0.5,
  "changefreq": "weekly"
}

Rows are available through the Apify dataset and its standard export formats.

💳 Pricing

This Actor uses pay-per-event pricing. The Website URL event is charged once for each accepted URL saved to the dataset. Discovered URLs that are not accepted or saved do not create this event. Start with a small Max URL rows value, inspect the output, and then broaden the limits if needed.

🔌 Integrations

Use Apify datasets, the API, schedules, webhooks, and platform integrations to process or deliver the URL inventory. This walkthrough shows how to connect an Actor to other services:

❓ FAQ

Why can a URL be marked discovered without HTTP fields?

A URL can come from a sitemap or an unvisited link. Only pages that the crawl loads receive crawled state and available HTTP response facts.

When are external URLs saved?

Same host and Same domain save only internal URLs. Include external URLs as discovered also saves external URLs, but the Actor does not crawl them.

Can I crawl only one submitted page?

Set Max crawl depth to 0 and use a low Max pages per website. Public sitemap discovery can still add sitemap-backed URLs.

Does it parse sitemap indexes?

Yes. It reads public XML URL sitemaps and nested sitemap indexes. Sitemap metadata is included only when the source supplies it.

Is this a full broken-link checker?

No. It provides HTTP facts for pages actually loaded during the crawl, which can support a downstream broken-link workflow. It does not promise an HTTP check for every discovered URL.

Does it scrape page text or private pages?

No. It extracts URL and link evidence from public pages and sitemaps. It does not return full page content, log in, or access private pages.

📝 Changelog

• 1.0: Added sitemap discovery, sitemap metadata, URL keyword filtering, total row limits, and the merged URL inventory output.

🆘 Support

For issues, questions, or feature requests, file a ticket and I'll fix or implement it in less than 24h 🫡

🔗 Related Actors

• Sitemap Sniffer — discover sitemap files and optionally export their listed URLs without crawling site navigation.
• XML Sitemap Validator — check sitemap-listed URLs for HTTP status, redirects, response time, and issues.
• Robots.txt Generator — create and validate crawler directives after reviewing a site's crawl surface.
• Seobility SEO Checker — run public single-page SEO checks on selected URLs from your inventory.
• SEMrush Free Website Stats Scraper — add public SEMrush domain metrics to website research workflows.

Made with ❤️ by Maxime Dupré