Technical SEO Site Crawler: Broken Links, Redirects, Duplicates and Orphan Pages

Crawl a whole website like a search engine and get one row per URL with everything a technical SEO audit needs: HTTP status code, redirect target and chain, indexability (and why not), title, meta description, H1, canonical, noindex, hreflang, word count, structured data, response time, broken internal links on every page, duplicate titles and meta descriptions, orphan pages, pages missing from the XML sitemap, click depth, inlinks and a 0–100 SEO score with a prioritised list of issues.

A site-wide SITE_REPORT sums it all up: status codes, issue counts, every broken link with the page it sits on, redirect chains, duplicate groups, orphans and the slowest pages. Export to CSV, Excel or JSON, or call it from the API, Make, Zapier or n8n. Pay only $0.002 per page, with no subscription and no desktop crawler licence.

Who is this SEO crawler for?

• SEO consultants and agencies who need a quick technical audit of a client site before a pitch or a monthly report.
• Web developers who want to catch 404s, redirect chains and noindex mistakes before or after a migration or relaunch.
• Content and marketing teams looking for duplicate titles, missing meta descriptions, thin pages and pages that nothing links to.
• Site owners who want a scheduled weekly health check without installing Screaming Frog or paying for a SaaS seat.

What does Technical SEO Site Crawler do?

1. Reads robots.txt and the XML sitemap(s) of the site.
2. Starts at your URL and follows internal links (www and non-www count as the same site), breadth-first, up to your Max pages and Max click depth.
3. Requests every URL without following redirects automatically, so each 301/302/307/308 is listed with its target, and chains and loops are detected.
4. Analyses every HTML page: on-page SEO checks plus the links on the page.
5. When the link crawl is done, it also checks sitemap URLs that no page links to (orphan pages), as long as Max pages allows.
6. Combines everything into site-wide checks: broken links, links to redirects, duplicates, sitemap gaps and depth.

Plain HTTP crawler: it reads the HTML the server sends and does not render JavaScript, which keeps it fast and cheap. Sites that build all their content in the browser (pure SPAs) will show up as thin pages.

How to use it

1. Enter your homepage (or a section such as https://example.com/blog/) under Website to crawl.
2. Set Max pages. This is your cost cap.
3. Optional: add Only crawl URLs containing or Skip URLs containing patterns (/blog/, ?sort=, /tag/), turn on Ignore query strings for shops with filters, or Include subdomains.
4. Click Start and open the Overview, Issues, Titles & meta or Links & redirects tab, or download SITE_REPORT from the key-value store.

Input example

{
  "startUrls": ["https://www.example.com/"],
  "maxPages": 500,
  "excludePatterns": ["/tag/", "?replytocom="],
  "ignoreQueryStrings": true,
  "onlyPagesWithIssues": false
}

Output example (one page, real run on crawlee.dev, 28 Oct 2026)

{
  "status": "ok",
  "url": "https://crawlee.dev/blog",
  "statusCode": 200,
  "indexable": true,
  "indexabilityReason": null,
  "depth": 1,
  "foundOn": "https://crawlee.dev/",
  "inlinks": 96,
  "inSitemap": true,
  "title": "Crawlee Blog - learn how to build better scrapers | Crawlee for JavaScript · Build reliable crawlers. Fast.",
  "titleLength": 107,
  "metaDescriptionLength": 151,
  "h1Count": 0,
  "canonicalIsSelf": true,
  "wordCount": 1464,
  "ttfbMs": 3,
  "brokenLinks": [],
  "linksToRedirects": 3,
  "duplicateTitlePages": 1,
  "seoScore": 77,
  "issueCodes": [
    "h1-missing",
    "duplicate-title",
    "title-too-long",
    "duplicate-meta-description",
    "links-to-redirects"
  ],
  "issues": [
    {
      "severity": "error",
      "code": "h1-missing",
      "message": "No H1 heading on the page."
    },
    {
      "severity": "warning",
      "code": "duplicate-title",
      "message": "Same title as 1 other page(s)."
    },
    {
      "severity": "warning",
      "code": "title-too-long",
      "message": "Title is long (107 characters; Google usually cuts off around 60)."
    },
    {
      "severity": "warning",
      "code": "duplicate-meta-description",
      "message": "Same meta description as 1 other page(s)."
    },
    {
      "severity": "notice",
      "code": "links-to-redirects",
      "message": "3 internal link(s) point to redirecting URLs; link to the final URL."
    }
  ]
}

Redirects get redirectTo, redirectFinalUrl and redirectHops; 404 pages list the page that links to them in foundOn, and every page shows its own broken links in brokenLinks ({url, statusCode}, or {url, statusCode: null, error} when the target could not be loaded at all, e.g. a timeout after two attempts).

Which issues are detected?

• Errors: broken internal links (4xx/5xx/unreachable), HTTP 4xx/5xx pages, redirect loops, redirects to error pages, HTTPS redirecting to HTTP, missing title, missing meta description, missing H1, noindex, missing viewport, no HTTPS, pages blocked for Googlebot in robots.txt.
• Warnings: duplicate titles and meta descriptions, orphan pages (in the sitemap, not linked), sitemap URLs that are not indexable, redirect chains, titles and descriptions too long or short, multiple H1s, canonical missing, invalid hreflang or JSON-LD, images without alt text, thin content, slow server response, HTML over 2 MB (Googlebot only reads the first 2 MB).
• Notices: links to redirects, temporary redirects, indexable pages missing from the sitemap, pages deeper than 4 clicks, missing structured data or Open Graph tags, hreflang without self-reference or x-default.

The SEO score starts at 100 and deducts points per failed check, so you can sort pages by priority.

Pricing

Pay per event:

• $0.005 per run start
• $0.002 per page on the Free plan; $0.0018 Bronze, $0.0016 Silver, $0.0015 Gold and higher

Every crawled URL counts as one page, including redirects and 404s, because those are findings too. Free: pages that could not be reached at all, URLs blocked by robots.txt, and pages left out by Only save pages with errors or warnings.

Pages	Free plan cost
100	$0.21
500	$1.01
1,000	$2.01
5,000	$10.01

If your spending limit for the run is lower, the crawl stops early at that page count.

FAQ

Does it respect robots.txt?

Yes, always. Disallowed URLs are not fetched (they are counted in SITE_REPORT), a Crawl-delay is honoured, and requests are spaced at least 500 ms apart by default with up to 3 in parallel. The crawler identifies itself as TechnicalSeoCrawler.

How long does a crawl take?

About 1–2 pages per second with the default politeness settings: 100 pages in ~1 minute, 1,000 pages in ~10 minutes. You can lower the delay for your own site.

Can it crawl JavaScript-rendered sites?

No, it reads the server HTML, like most technical SEO crawlers in their default mode. For performance and rendering checks, use our Bulk PageSpeed & SEO Audit.

Why are there pages from the sitemap with depth null?

Those are orphan pages: listed in the sitemap but not linked from any crawled page. They are crawled after the link crawl if Max pages allows.

Can I monitor my site every week?

Yes. Save your input as a task and add a schedule; combine it with Only save pages with errors or warnings to get a short list of problems each week.

Is it legal?

It only reads publicly available pages, like a search engine, identifies itself, respects robots.txt and keeps the load low. Only crawl sites you are allowed to audit and respect their terms.

Related actors by benelux-signals

Combine a full-site crawl with speed and news checks:

• Bulk PageSpeed & SEO Audit – Core Web Vitals, Lighthouse scores and on-page SEO checks for many URLs or a sitemap.
• EU & Dutch News Monitor – keyword alerts across 80+ built-in Dutch, Belgian and EU news and official RSS feeds.

More tools from the same maintainer:

• EU Tender Radar – TED + TenderNed public tenders with keyword/CPV filters and only-new alerts.
• Benelux Trade Fair Exhibitors – exhibitor lists from Dutch and Belgian trade fairs (Horecava, METSTRADE, Batibouw…) for B2B lead lists.
• Career Site Jobs API – jobs straight from Greenhouse, Lever, Recruitee, Personio, Ashby and other company career sites.
• Workday Jobs API – jobs from any myworkdayjobs.com career site, with a European employer seed list.
• PDF & Image to Text OCR – text and Markdown from PDFs and scans, ready for RAG and AI agents.
• Arbeitsagentur Jobs API – German job ads from the Bundesagentur für Arbeit Jobbörse.
• Remote Jobs Aggregator – Europe-friendly remote jobs from RemoteOK, We Work Remotely, Jobicy and more, deduplicated.

Questions or a source we should add? Open an issue on the Actor page.