Site SEO Crawler: Broken Links & On-Page Audit

What does Site SEO Crawler & Audit do?

Site SEO Crawler & Audit crawls a whole website and tells you what is wrong, in one run:

• 🔗 Broken links, internal and external, with the pages that link to them (up to 20 per link) and the anchor text, so you know where to fix each one.
• ↪️ Redirects, chains and loops: every hop of every redirect, and the internal links that still point to a redirecting URL.
• 📄 On-page issues per page: missing or duplicate titles and meta descriptions, missing or multiple H1, noindex, canonical problems (missing, several, pointing to a broken, redirecting or noindex page), thin pages, images without alt text and slow responses, each with a severity.
• 📊 A free summary row per site with the number of pages affected by each issue type.

It starts at your start URL and reads sitemap.xml, follows internal links breadth-first (nearest pages first), stays on the site (and its subdomains if you ask), and respects robots.txt and its Crawl-delay. HTTP only with Cheerio: no browser, no proxy, no login.

Who is it for?

• SEO and content teams: a site-wide to-do list: which pages have no title, which share one, which are noindex by mistake.
• Web agencies and freelancers: audit a client's site before and after a relaunch or migration (redirects, broken links).
• Developers and site owners: find 404s and redirect chains after a release.
• Marketers: check landing pages for thin content, missing H1s and slow responses.

How to use it

1. Enter the Start URL (the home page of the site). You can add several sites.
2. Set Max pages per site (default 200, up to 5,000). The crawl goes breadth-first, so the pages closest to the start come first.
3. Keep Check external links on to find dead links to other sites. Switch on Include subdomains to crawl blog.example.com too.
4. Click Start. Pages shows one row per page with its issues, Broken links shows each broken link with the pages that contain it, Site summary shows the totals.

Input example

{
    "startUrls": [{ "url": "https://www.example.com/" }],
    "maxPages": 500,
    "checkExternalLinks": true,
    "includeSubdomains": false,
    "respectRobots": true,
    "userAgent": "MySiteAudit"
}

Output example

One row per page (real run on quotes.toscrape.com, 7 October 2026, a public practice site):

{
    "type": "page",
    "site": "quotes.toscrape.com",
    "url": "https://quotes.toscrape.com/login",
    "status": 200,
    "depth": 1,
    "discoveredVia": "link",
    "inSitemap": null,
    "finalUrl": "https://quotes.toscrape.com/login",
    "redirectChain": [],
    "contentType": "text/html",
    "title": "Quotes to Scrape",
    "metaDescription": null,
    "h1s": ["Quotes to Scrape"],
    "canonical": null,
    "robotsMeta": null,
    "indexable": true,
    "wordCount": 13,
    "responseMs": 10,
    "imagesWithoutAlt": 0,
    "internalLinks": 2,
    "externalLinks": 2,
    "brokenLinks": [],
    "issues": [
        { "type": "meta-description-missing", "severity": "warning", "detail": "The page has no meta description." },
        { "type": "canonical-missing", "severity": "info", "detail": "The page has no canonical link." },
        { "type": "thin-page", "severity": "warning", "detail": "Only 13 words of visible text (under 200)." },
        { "type": "title-duplicate", "severity": "warning", "detail": "The same title is used on 24 pages, for example https://quotes.toscrape.com/." }
    ],
    "issueCount": 4
}

A URL that redirects gets its own free row with every hop:

{
    "type": "page",
    "url": "https://quotes.toscrape.com/author/Albert-Einstein",
    "status": 308,
    "finalUrl": "http://quotes.toscrape.com/author/Albert-Einstein/",
    "redirectChain": [
        { "url": "https://quotes.toscrape.com/author/Albert-Einstein", "status": 308 },
        { "url": "http://quotes.toscrape.com/author/Albert-Einstein/", "status": 200 }
    ],
    "issues": [{ "type": "redirect", "severity": "info", "detail": "Redirects (308) to http://quotes.toscrape.com/author/Albert-Einstein/." }]
}

A broken link row (from the Actor's test site, host replaced by example.com):

{
    "type": "broken-link",
    "site": "example.com",
    "url": "https://example.com/missing",
    "status": 404,
    "error": "http-error",
    "errorMessage": "The URL answers HTTP 404.",
    "isExternal": false,
    "linkedFrom": ["https://example.com/", "https://example.com/a"],
    "linkedFromCount": 2,
    "anchorText": "Missing page",
    "checkedVia": "crawl"
}

And the free summary row, for example: "quotes.toscrape.com: 30 pages crawled (stopped at the page limit), 24 audited; 0 broken links (0 internal, 0 external); 0 redirect chains or loops; 0 errors and 66 warnings on pages; most common: meta-description-missing (24), title-duplicate (24), thin-page (18)."

Issue types

Severity	Types
error	title-missing, canonical-multiple, canonical-broken, broken-links, redirect-loop, redirect-to-error, http-error, fetch-failed
warning	meta-description-missing, title-duplicate, meta-description-duplicate, h1-missing, noindex, thin-page (under 200 words), images-missing-alt, slow-response (over 2,000 ms), canonical-redirects, canonical-noindex, canonical-chain, canonical-invalid, redirect-chain, redirect-to-http
info	title-too-long (over 60 characters), title-too-short, meta-description-too-long (over 160), h1-multiple, canonical-missing, canonical-other-url, canonical-other-domain, redirect, redirect-temporary, links-to-redirect, not-html

How much does it cost?

Pay per event: $0.001 per audited page (page-audited): a page that answers HTTP 200 with HTML and gets the full on-page audit. 1,000 pages cost $1.00. Free: broken-link rows, the summary row, and rows for URLs that were not audited (redirects, 404 and other error pages, non-HTML files). Use Max pages or Apify's "maximum cost per run" to cap spend.

Politeness and robots.txt

• robots.txt is respected by default: URLs it disallows for the crawler's name are not fetched (they are counted in the summary), and its Crawl-delay is honoured (up to 10 s between requests). Respect robots.txt is on by default; switch it off only for sites you own or may crawl.
• At most two requests at a time to the site, 300 ms apart; retries on 429 and 5xx with backoff.
• External links are checked once each per run, one at a time per host: HEAD first, GET when the server answers 405 or 501. Sites that refuse automated checks (HTTP 401, 403, 429, 999) are not reported as broken.
• The crawler announces itself with a name you can set (User-Agent name), so site owners can recognise it.

Limitations

• HTTP only, no JavaScript rendering. Links and content that only appear after scripts run are not seen. Single-page apps may look empty.
• Page limit: the crawl stops at Max pages per site; the summary says so. Internal links beyond the limit are not checked.
• Word count is the visible text of the page, including menus and footers, so "thin" is a hint, not a verdict.
• Response time is the download time of the page from the Actor's server, not a performance score.
• Duplicate titles and descriptions are compared across the crawled pages only.
• Query-string URLs are different pages (tracking parameters such as utm_* are ignored). Sites with endless filter URLs can use up the page limit: set a lower limit.
• External link checks can show false alarms for sites that block automated requests with unusual answers; the row says what was returned.

FAQ

Does it check external links?

Yes, by default (switch off with Check external links). Each distinct link is checked once.

How do I find the pages that link to a broken URL?

Open the Broken links tab: every row lists up to 20 pages that contain the link (linkedFrom), the number of all pages (linkedFromCount) and the anchor text. Each page row also lists its own broken links.

Why was a page not crawled?

It may be disallowed by robots.txt, outside the site (another host or a subdomain you did not include), beyond Max pages per site, or a file (PDF, image ...), which is checked as a link but not crawled.

Does it use a proxy or a browser?

No.

More tools from the same developer

All pay-per-result, no proxy or login needed, built and maintained by the same developer:

Website audits

• Website Accessibility Checker: WCAG 2.2 & EAA: accessibility issues with fixes, SEO basics and security headers.
• Cookie & Tracker Audit: GDPR Consent Checker: cookies and tracking tags that load before consent.
• AI Crawler Access Checker: robots.txt & llms.txt: which AI crawlers a site allows, plus llms.txt.
• Structured Data Validator: Schema & Rich Results: JSON-LD and Microdata checked against Google's rich result rules.
• Website Change Monitor: Diffs, Prices & Alerts: get a row only when a page changes, with a clean diff.
• Policy Change Monitor: Privacy & Terms Tracker: privacy policy and terms changes, classified clause by clause.
• SaaS Pricing Monitor: Track Competitor Plans: competitor pricing plans as structured data, with price-change diffs.
• Sitemap Change Monitor: New & Removed URLs: new, removed and changed URLs from a site's sitemaps.
• Website Tech Stack Detector: Bulk Lead Filters: technologies used by websites, with filters for lead lists.
• Domain Availability & Expiry Checker (RDAP): domain availability, registrar and expiry dates from RDAP.
• Status Page SLA Monitor: Uptime & Incidents: SaaS vendor uptime, incidents and a monthly SLA table from public status pages.

Company data and compliance

• Company Contact Finder: Emails, Phones & Socials: contact details published on company websites.
• UK New Companies Feed: Companies House Daily: newly incorporated UK companies with sector filters.
• EU VAT Number Validator: Bulk VIES Checker: bulk VAT checks with name, address and consultation number.
• LEI Corporate Tree: GLEIF Parents & Subsidiaries: LEI lookup with parents, subsidiaries and a KYC summary.
• UK Company Distress Watch: Insolvency Alerts: administrations, liquidations and winding-up petitions from The Gazette.
• IBAN Validator: Bulk Check & Bank Lookup: IBAN format and checksum, plus bank name and BIC where open data exists.
• Sanctions Screening: OFAC, EU, UN & UK Lists: names screened against official sanctions lists, with match scores.
• OSHA & EPA Violation Monitor: Fines by Company: US OSHA and EPA enforcement cases and penalties by company.
• Email List Precheck: Syntax, MX & Disposable Check: cheap email list hygiene before a real verification.
• UK Company Officer Changes: Director & PSC Alerts: director, owner, address and status changes for a watchlist of UK companies.
• FDA Warning Letters Monitor: By Company & Subject: FDA warning letters by company and subject, with cited regulations.
• Supplier Risk Report: Sanctions, VAT & Insolvency: one risk row per supplier from eight official registers, with changes since last run.
• Brand Impersonation Domain Monitor: Look-alikes: look-alike domains and certificates that imitate your brand, with a risk score.

Media and AI

• Audio & Video Transcriber: Podcasts, SRT, Speakers: audio, video, podcasts and social links to text, SRT and VTT.
• Podcast Mention Alerts: Brand Mentions by Second: where your brand is said in new podcast episodes, with the exact second.
• News Brand Mentions Monitor: New Articles Only: new news articles about your brand, people or topics.
• YouTube Channel Monitor: New Videos & Growth: new videos and view growth of the channels you watch.

Market signals

• US WARN Layoff Notices: 12 States Daily Feed: layoff and plant closure notices from official state sources.
• US Product Recalls Monitor: FDA & CPSC Feed: FDA and CPSC recalls in one feed, with severity.
• EU Tenders Feed: TED Public Procurement Alerts: EU public tenders from TED with CPV, country and value filters.
• Liquor License Openings: New Bars & Stores (US): new US liquor licences from official state open data.
• Clinical Trials Change Monitor: Status & Updates: status, enrollment and results changes on ClinicalTrials.gov studies.
• Career Site Jobs Monitor: ATS Jobs & New Roles: open roles from company career sites, with new and closed roles since the last run.
• App Reviews Monitor: App Store & Google Play: app reviews from both stores, only new ones in monitor mode.
• US Federal Contracts Feed: SAM.gov & USAspending: SAM.gov contract opportunities with the buyer's award history.
• Grants.gov Opportunities: US Federal Grants Feed: US federal grant opportunities by keyword, eligibility and agency.
• Federal Register Monitor: US Rules & Notices Feed: new US rules, proposed rules and notices by agency or topic.
• B2B Sales Trigger Feed: Hiring + Tech Leads: companies hiring for a role and using (or not using) a technology, with contacts.
• Local Business Leads: OpenStreetMap, No Website: local businesses by city and category, including those with no website.
• Solana Token Pair Monitor: New & Trending Pairs: new and trending Solana pairs with liquidity, volume and safety flags.

Feedback

Missing an issue type, or a page that is wrongly flagged? Open an issue on the Issues tab.