Site SEO Crawler: Broken Links & On-Page Audit

What does Site SEO Crawler & Audit do?

Site SEO Crawler & Audit crawls a whole website and tells you what is wrong, in one run:

• 🔗 Broken links, internal and external, with the pages that link to them (up to 20 per link) and the anchor text, so you know where to fix each one.
• ↪️ Redirects, chains and loops: every hop of every redirect, and the internal links that still point to a redirecting URL.
• 📄 On-page issues per page: missing or duplicate titles and meta descriptions, missing or multiple H1, noindex, canonical problems (missing, several, pointing to a broken, redirecting or noindex page), thin pages, images without alt text and slow responses, each with a severity.
• 📊 A free summary row per site with the number of pages affected by each issue type.

It starts at your start URL and reads sitemap.xml, follows internal links breadth-first (nearest pages first), stays on the site (and its subdomains if you ask), and respects robots.txt and its Crawl-delay. HTTP only with Cheerio: no browser, no proxy, no login.

Who is it for?

• SEO and content teams: a site-wide to-do list: which pages have no title, which share one, which are noindex by mistake.
• Web agencies and freelancers: audit a client's site before and after a relaunch or migration (redirects, broken links).
• Developers and site owners: find 404s and redirect chains after a release.
• Marketers: check landing pages for thin content, missing H1s and slow responses.

How to use it

1. Enter the Start URL (the home page of the site). You can add several sites.
2. Set Max pages per site (default 200, up to 5,000). The crawl goes breadth-first, so the pages closest to the start come first.
3. Keep Check external links on to find dead links to other sites. Switch on Include subdomains to crawl blog.example.com too.
4. Click Start. Pages shows one row per page with its issues, Broken links shows each broken link with the pages that contain it, Site summary shows the totals.

Input example

{
    "startUrls": [{ "url": "https://www.example.com/" }],
    "maxPages": 500,
    "checkExternalLinks": true,
    "includeSubdomains": false,
    "respectRobots": true,
    "userAgent": "MySiteAudit"
}

Output example

One row per page (real run on quotes.toscrape.com, 7 October 2026, a public practice site):

{
    "type": "page",
    "site": "quotes.toscrape.com",
    "url": "https://quotes.toscrape.com/login",
    "status": 200,
    "depth": 1,
    "discoveredVia": "link",
    "inSitemap": null,
    "finalUrl": "https://quotes.toscrape.com/login",
    "redirectChain": [],
    "contentType": "text/html",
    "title": "Quotes to Scrape",
    "metaDescription": null,
    "h1s": ["Quotes to Scrape"],
    "canonical": null,
    "robotsMeta": null,
    "indexable": true,
    "wordCount": 13,
    "responseMs": 10,
    "imagesWithoutAlt": 0,
    "internalLinks": 2,
    "externalLinks": 2,
    "brokenLinks": [],
    "issues": [
        { "type": "meta-description-missing", "severity": "warning", "detail": "The page has no meta description." },
        { "type": "canonical-missing", "severity": "info", "detail": "The page has no canonical link." },
        { "type": "thin-page", "severity": "warning", "detail": "Only 13 words of visible text (under 200)." },
        { "type": "title-duplicate", "severity": "warning", "detail": "The same title is used on 24 pages, for example https://quotes.toscrape.com/." }
    ],
    "issueCount": 4
}

A URL that redirects gets its own free row with every hop:

{
    "type": "page",
    "url": "https://quotes.toscrape.com/author/Albert-Einstein",
    "status": 308,
    "finalUrl": "http://quotes.toscrape.com/author/Albert-Einstein/",
    "redirectChain": [
        { "url": "https://quotes.toscrape.com/author/Albert-Einstein", "status": 308 },
        { "url": "http://quotes.toscrape.com/author/Albert-Einstein/", "status": 200 }
    ],
    "issues": [{ "type": "redirect", "severity": "info", "detail": "Redirects (308) to http://quotes.toscrape.com/author/Albert-Einstein/." }]
}

A broken link row (from the Actor's test site, host replaced by example.com):

{
    "type": "broken-link",
    "site": "example.com",
    "url": "https://example.com/missing",
    "status": 404,
    "error": "http-error",
    "errorMessage": "The URL answers HTTP 404.",
    "isExternal": false,
    "linkedFrom": ["https://example.com/", "https://example.com/a"],
    "linkedFromCount": 2,
    "anchorText": "Missing page",
    "checkedVia": "crawl"
}

And the free summary row, for example: "quotes.toscrape.com: 30 pages crawled (stopped at the page limit), 24 audited; 0 broken links (0 internal, 0 external); 0 redirect chains or loops; 0 errors and 66 warnings on pages; most common: meta-description-missing (24), title-duplicate (24), thin-page (18)."

Issue types

Severity	Types
error	title-missing, canonical-multiple, canonical-broken, broken-links, redirect-loop, redirect-to-error, http-error, fetch-failed
warning	meta-description-missing, title-duplicate, meta-description-duplicate, h1-missing, noindex, thin-page (under 200 words), images-missing-alt, slow-response (over 2,000 ms), canonical-redirects, canonical-noindex, canonical-chain, canonical-invalid, redirect-chain, redirect-to-http
info	title-too-long (over 60 characters), title-too-short, meta-description-too-long (over 160), h1-multiple, canonical-missing, canonical-other-url, canonical-other-domain, redirect, redirect-temporary, links-to-redirect, not-html

How much does it cost?

Pay per event: $0.001 per audited page (page-audited): a page that answers HTTP 200 with HTML and gets the full on-page audit. 1,000 pages cost $1.00. Free: broken-link rows, the summary row, and rows for URLs that were not audited (redirects, 404 and other error pages, non-HTML files). Use Max pages or Apify's "maximum cost per run" to cap spend.

Politeness and robots.txt

• robots.txt is respected by default: URLs it disallows for the crawler's name are not fetched (they are counted in the summary), and its Crawl-delay is honoured (up to 10 s between requests). Respect robots.txt is on by default; switch it off only for sites you own or may crawl.
• At most two requests at a time to the site, 300 ms apart; retries on 429 and 5xx with backoff.
• External links are checked once each per run, one at a time per host: HEAD first, GET when the server answers 405 or 501. Sites that refuse automated checks (HTTP 401, 403, 429, 999) are not reported as broken.
• The crawler announces itself with a name you can set (User-Agent name), so site owners can recognise it.

Limitations

• HTTP only, no JavaScript rendering. Links and content that only appear after scripts run are not seen. Single-page apps may look empty.
• Page limit: the crawl stops at Max pages per site; the summary says so. Internal links beyond the limit are not checked.
• Word count is the visible text of the page, including menus and footers, so "thin" is a hint, not a verdict.
• Response time is the download time of the page from the Actor's server, not a performance score.
• Duplicate titles and descriptions are compared across the crawled pages only.
• Query-string URLs are different pages (tracking parameters such as utm_* are ignored). Sites with endless filter URLs can use up the page limit: set a lower limit.
• External link checks can show false alarms for sites that block automated requests with unusual answers; the row says what was returned.

FAQ

Does it check external links?

Yes, by default (switch off with Check external links). Each distinct link is checked once.

How do I find the pages that link to a broken URL?

Open the Broken links tab: every row lists up to 20 pages that contain the link (linkedFrom), the number of all pages (linkedFromCount) and the anchor text. Each page row also lists its own broken links.

Why was a page not crawled?

It may be disallowed by robots.txt, outside the site (another host or a subdomain you did not include), beyond Max pages per site, or a file (PDF, image ...), which is checked as a link but not crawled.

Does it use a proxy or a browser?

No.

More tools from the same developer

All pay-per-result, no proxy or login needed, built and maintained by the same developer:

Website audits

• Website Accessibility Checker: WCAG 2.2 & EAA: accessibility issues with fixes, SEO basics and security headers.
• Cookie & Tracker Audit: GDPR Consent Checker: cookies and tracking tags that load before consent.
• AI Crawler Access Checker: robots.txt & llms.txt: which AI crawlers a site allows, plus llms.txt.
• Structured Data Validator: Schema & Rich Results: JSON-LD and Microdata checked against Google's rich result rules.
• Website Change Monitor: Diffs, Prices & Alerts: get a row only when a page changes, with a clean diff.
• Policy Change Monitor: Privacy & Terms Tracker: privacy policy and terms changes, classified clause by clause.
• SaaS Pricing Monitor: Track Competitor Plans: competitor pricing plans as structured data, with price-change diffs.
• Sitemap Change Monitor: New & Removed URLs: new, removed and changed URLs from a site's sitemaps.
• Website Tech Stack Detector: Bulk Lead Filters: technologies used by websites, with filters for lead lists.
• Domain Availability & Expiry Checker (RDAP): domain availability, registrar and expiry dates from RDAP.
• Status Page SLA Monitor: Uptime & Incidents: SaaS vendor uptime, incidents and a monthly SLA table from public status pages.

Company data and compliance

• Company Contact Finder: Emails, Phones & Socials: contact details published on company websites.
• UK New Companies Feed: Companies House Daily: newly incorporated UK companies with sector filters.
• EU VAT Number Validator: Bulk VIES Checker: bulk VAT checks with name, address and consultation number.
• LEI Corporate Tree: GLEIF Parents & Subsidiaries: LEI lookup with parents, subsidiaries and a KYC summary.
• UK Company Distress Watch: Insolvency Alerts: administrations, liquidations and winding-up petitions from The Gazette.
• IBAN Validator: Bulk Check & Bank Lookup: IBAN format and checksum, plus bank name and BIC where open data exists.
• Sanctions Screening: OFAC, EU, UN & UK Lists: names screened against official sanctions lists, with match scores.
• OSHA & EPA Violation Monitor: Fines by Company: US OSHA and EPA enforcement cases and penalties by company.
• Email List Precheck: Syntax, MX & Disposable Check: cheap email list hygiene before a real verification.
• UK Company Officer Changes: Director & PSC Alerts: director, owner, address and status changes for a watchlist of UK companies.
• FDA Warning Letters Monitor: By Company & Subject: FDA warning letters by company and subject, with cited regulations.
• Supplier Risk Report: Sanctions, VAT & Insolvency: one risk row per supplier from eight official registers, with changes since last run.
• Brand Impersonation Domain Monitor: Look-alikes: look-alike domains and certificates that imitate your brand, with a risk score.

Media and AI

• Audio & Video Transcriber: Podcasts, SRT, Speakers: audio, video, podcasts and social links to text, SRT and VTT.
• Document to Audiobook: EPUB, PDF, DOCX to MP3: EPUB, PDF, DOCX, MOBI and 150+ formats to MP3 or M4B audiobooks with chapters.
• Podcast Mention Alerts: Brand Mentions by Second: where your brand is said in new podcast episodes, with the exact second.
• News Brand Mentions Monitor: New Articles Only: new news articles about your brand, people or topics.
• YouTube Channel Monitor: New Videos & Growth: new videos and view growth of the channels you watch.

Market signals

• US WARN Layoff Notices: 12 States Daily Feed: layoff and plant closure notices from official state sources.
• US Product Recalls Monitor: FDA & CPSC Feed: FDA and CPSC recalls in one feed, with severity.
• EU Tenders Feed: TED Public Procurement Alerts: EU public tenders from TED with CPV, country and value filters.
• Liquor License Openings: New Bars & Stores (US): new US liquor licences from official state open data.
• Clinical Trials Change Monitor: Status & Updates: status, enrollment and results changes on ClinicalTrials.gov studies.
• Career Site Jobs Monitor: ATS Jobs & New Roles: open roles from company career sites, with new and closed roles since the last run.
• App Reviews Monitor: App Store & Google Play: app reviews from both stores, only new ones in monitor mode.
• US Federal Contracts Feed: SAM.gov & USAspending: SAM.gov contract opportunities with the buyer's award history.
• Grants.gov Opportunities: US Federal Grants Feed: US federal grant opportunities by keyword, eligibility and agency.
• Federal Register Monitor: US Rules & Notices Feed: new US rules, proposed rules and notices by agency or topic.
• B2B Sales Trigger Feed: Hiring + Tech Leads: companies hiring for a role and using (or not using) a technology, with contacts.
• Local Business Leads: OpenStreetMap, No Website: local businesses by city and category, including those with no website.
• Solana Token Pair Monitor: New & Trending Pairs: new and trending Solana pairs with liquidity, volume and safety flags.

Social media

• Instagram Search Scraper: Find Accounts by Keyword: find Instagram accounts for any keyword or niche.
• Instagram Profile Scraper: Followers, Bio & Latest Posts: scrape public Instagram profiles.
• Instagram Hashtag Scraper: Top Posts, Views & Related Tags: get the top posts Instagram shows for any hashtag (shortcode, URL, type, full caption, play count, owner...
• LinkedIn Public Profile Scraper: Headline & Current Role: get the public headline data of LinkedIn profiles.
• TikTok Profile Scraper: Followers, Likes & Bio: get public TikTok profile data in bulk.
• Facebook Pages Search: Find Business Pages by Keyword: find Facebook business Pages for any keyword or local niche.
• TikTok Comments Scraper: Comments & Replies: scrape comments (and optionally replies) from TikTok videos.
• YouTube Transcript Scraper: Captions to Text: get the transcript of YouTube videos and Shorts.
• YouTube Comments Scraper: Text, Likes & Authors: scrape comments from YouTube videos and Shorts.
• YouTube Channel Search: Find Channels by Keyword: find YouTube channels for any keyword with YouTube's own Channels filter.
• TikTok Video Scraper: Views, Likes, Shares & Saves: get full data for TikTok videos by URL.
• YouTube Shorts Scraper: Channel Shorts & Views: get the Shorts of YouTube channels.
• Threads Search Scraper: Posts by Keyword or Hashtag: search Meta Threads for keywords or hashtags and get the posts.
• Instagram Posts & Reels Scraper: Latest Posts: get the 12 newest posts and reels of public Instagram accounts.
• Threads Replies Scraper: Comments on Threads Posts: get the replies under Meta Threads posts.
• YouTube Channel Videos Scraper: All Videos & Views: get every video of YouTube channels.
• Google Ads Transparency Scraper: Competitor Ads: get every Google ad of an advertiser or website from the Google Ads Transparency Center.
• Threads Profile Scraper: Followers, Bio & Latest Posts: scrape public Meta Threads profiles.
• Instagram Bio Email & Phone Scraper: get the emails and phone numbers Instagram accounts write in their public bio, plus their website, bio...
• Facebook Page Contact Scraper: Emails, Phones & Websites: get the contact details Facebook Pages publish.
• TikTok Sound Scraper: Song Usage & Top Videos: track TikTok sounds and songs: title, artist, total number of videos made with the sound, cover, and the...
• Telegram Channel Scraper: Posts, Views & History: scrape public Telegram channels.
• YouTube Channel Info & Email Scraper: About Page: get YouTube channel About data in bulk.
• Telegram Channel Keyword Search: Posts by Keyword: search public Telegram channels for keywords and get every matching post.
• Facebook Page Followers Checker: Bulk Follower Counts: check exact follower counts of Facebook Pages in bulk.

Jobs and talent

• LinkedIn Jobs Scraper: Descriptions, Criteria & Applicants: scrape LinkedIn job listings by keywords or a LinkedIn search URL.
• Wellfound Jobs Scraper: Startup Jobs & Salaries: scrape startup jobs from Wellfound (formerly AngelList Talent) by role, remote or by city.
• SEEK Jobs Scraper: Australia & New Zealand: scrape job ads from SEEK Australia and New Zealand by keywords and location.
• Naukrigulf Jobs Scraper: Gulf Jobs (UAE, Saudi, Qatar): scrape Naukrigulf job listings for the Gulf.
• Dice Jobs Scraper: US Tech Jobs & Salaries: scrape tech job listings from Dice.com by keyword and location.
• XING Jobs Scraper: Germany, Austria & Switzerland: scrape XING job listings (DACH region) by keyword and location.
• AI & ML Jobs Scraper: Data Science Roles: the newest ~100 AI, machine learning and data jobs (aijobs.net / Foorilla), optionally filtered by keyword.
• foundit Jobs Scraper: India (formerly Monster India): scrape job listings from foundit.in (formerly Monster India) by keyword and city.
• Malt Freelancers Scraper: Profiles & Day Rates: search Malt (France, Spain, Germany, Belgium, Netherlands...) for freelancers.
• Cybersecurity Jobs Scraper: InfoSec & Privacy Roles: the newest ~100 cybersecurity and privacy jobs (infosec-jobs.com / Foorilla), optionally filtered by keyword.

E-commerce and marketplaces

• Facebook Marketplace Scraper: Listings & Prices: search Facebook Marketplace in any city.
• AliExpress Search Scraper: Products, Prices & Sales: scrape AliExpress search results.
• TikTok Shop Scraper: Best-Selling Products by Category: scrape TikTok Shop category best-sellers.
• Whatnot Search Scraper: Listings, Bids & Sellers: search Whatnot (live shopping marketplace) listings.
• App Store Search Scraper: Apps, Ratings & Publishers: search the Apple App Store for apps or publishers.
• USA Scientific Lab Products Scraper: Prices & Specs: search USA Scientific's lab supply catalog.

Travel, sports and places

• Airbnb Search Scraper: Prices, Ratings & Listings: scrape Airbnb search results for any place and dates.
• Flashscore Scraper: Live Scores, Results & Fixtures: get Flashscore matches for 18 sports (football, tennis, basketball, hockey, cricket, esports...) for any...
• Google Hotels Prices Scraper: Lowest Rates by City: get Google Hotels "prices starting from" for any city and dates.
• Google Flights Scraper: Cheapest Fares by Route & Date: get Google Flights one-way fares for many routes and dates.
• PropertyPal Scraper: Northern Ireland Property Listings: scrape PropertyPal (Northern Ireland) listings for sale or rent.
• Live Flight Status by Area: Position & Ground Flag: live flight status inside a latitude/longitude box.
• Airbnb Listing Ratings Scraper: Scores & Review Themes: get the rating profile of Airbnb listings.
• Live Aircraft Tracker: ADS-B Positions in an Area: live aircraft positions inside a latitude/longitude box from OpenSky Network ADS-B receivers.
• Flights in an Area: Live Callsigns & Altitudes: flights currently in the air inside a latitude/longitude box (OpenSky ADS-B).

Finance and crypto

• Pump.fun New Tokens: Launches & Migrations: newest pump.fun Solana token launches.
• Bybit Funding Rates & Open Interest: Bybit perpetual futures data per symbol.
• Ethereum ETF Flows: Daily Net Inflows by Fund: daily net flows of US spot Ether ETFs (ETHA, FETH, ETHE...).
• Bitcoin ETF Flows: Daily Net Inflows by Fund: daily net flows of US spot Bitcoin ETFs (IBIT, FBTC, GBTC, ARKB...).
• California Cap-and-Trade Auction Results: California-Quebec cap-and-trade auction results from CARB.

Research and public data

• LinkedIn Company Scraper: Size, Industry & HQ: get public LinkedIn company page data in bulk.
• Google News Scraper: Headlines, Sources & Real URLs: search Google News for any query, language and country.
• Indie Hackers Posts Scraper: Trending & Newest: get Indie Hackers posts (trending, featured, newest or throwbacks).
• UNGM Tenders Scraper: UN Procurement Notices: search United Nations Global Marketplace (UNGM) procurement notices.
• Global Patent Search: US, EP, WO & More: search patents worldwide by keywords, assignee, inventor, country and dates.
• AlienVault OTX Threat Lookup: IPs, Domains & Hashes: look up IPs, domains, URLs or file hashes in AlienVault OTX threat intelligence.
• China Patents Search (CNIPA): Titles, Assignees & Dates: search Chinese patents (CNIPA) by keywords, assignee or inventor.
• California License Lookup: Doctors, Nurses & 33 Boards (DCA): look up California professional licenses from the Department of Consumer Affairs' official monthly licensee...

Feedback

Missing an issue type, or a page that is wrongly flagged? Open an issue on the Issues tab.