SEO Site Audit Crawler: On-Page SEO, Broken Links & Redirects

Crawl a whole website the way a search engine does, starting from its home page or sitemap, and get an on-page SEO audit of every page: title, meta description, headings, canonical, noindex, hreflang, structured data, Open Graph tags, image alt text and word count, plus broken links, redirect chains and duplicate titles and descriptions across the site. Each address is one row with its status code and a plain-English list of issues (error, warning or notice); each site gets a summary with a site score. The crawler uses plain HTTP, no browser, respects robots.txt and keeps a gentle pace, so runs are fast and cheap.

• 🕷️ Crawls the whole site from a start page and its sitemap: same site only (subdomains optional), with page, depth and pattern limits.
• 🏷️ On-page SEO per page: title and meta description (missing, too long or short, duplicate), H1 and H2, canonical, meta robots and X-Robots-Tag, hreflang, <html lang>, viewport, Open Graph and Twitter cards, JSON-LD types and errors, images without alt text, word count.
• 🔗 Broken links and redirects: every 404 and 500 with the pages that link to it, redirect chains and loops hop by hop, links that go through redirects, and optionally links to other sites.
• 🗺️ Sitemap checks: orphan pages, sitemap entries that redirect, break or are noindex, and indexable pages missing from the sitemap.
• 🤖 Polite by design: robots.txt and its Crawl-delay are always respected, one request at a time per site, an honest user agent, and a site that blocks crawlers is left alone.
• 🔔 Monitoring: schedule it with a monitor ID and every row shows the issues that are new or fixed since the last crawl.
• 💸 $0.003 per audited page ($3 per 1,000). Redirects, broken addresses, files, addresses robots.txt blocks and link checks are listed free.

Try it: the input comes pre-filled with crawlee.dev and a limit of 20 pages. Click Start; the audit takes under a minute. Then put in your own site.

Who it's for

• SEO consultants and agencies: a technical audit of a client's site in minutes, as a spreadsheet you can sort and filter, with the summary for the report.
• Site owners and marketers: find the broken links, missing titles and duplicate descriptions before Google does, and watch for new ones every week.
• Developers: check a release or a migration didn't break links, redirects, canonicals or hreflang.
• Content teams: list thin pages, missing meta descriptions and images without alt text, page by page.
• Data teams: feed site health into dashboards through the Apify API.

What you get

One row per address the crawl found. The type says what it is:

Type	What it is	Charged
page	An HTML page that answered 200, fully audited	Yes
redirect	An address that redirects, with the whole chain	Free
error	An address that answered 4xx or 5xx, with the pages that link to it	Free
failed	An address that didn't answer (time-out, DNS, refused connection)	Free
file	A PDF, image or other file that isn't HTML	Free
disallowed	An address robots.txt blocks; not fetched	Free
blocked	Where a site refused the crawler (bot check, HTTP 403 or 429)	Free

For every page: status code, response time, size, depth (clicks from the start page), the pages that link to it, whether it's in the sitemap and whether it's indexable; title and meta description with their lengths; the first H1, the number of H1s and H2s; canonical (self, other, missing or conflicting) and the canonical target's status; meta robots, X-Robots-Tag, noindex and nofollow; hreflang links with their status and whether they link back; <html lang>, viewport, Open Graph tags, Twitter card; structured data types (JSON-LD and microdata) and JSON-LD errors; images and images without alt text (with examples); word count; internal, external and nofollow link counts; broken links on the page; links to redirects; mixed content (http resources on https pages); and the issues list with counts by severity.

Issues come with a code, a severity and a message, for example title-too-long (warning): "The title is 72 characters; search results usually show about 60." There are 68 kinds (16 errors, 31 warnings, 21 notices); the most common are missing or duplicate titles and descriptions, missing H1, broken internal links, redirect chains, canonical and hreflang problems, noindex pages in the sitemap, orphan pages, images without alt text, thin pages and mixed content.

The run summary (RUN_SUMMARY in the run's key-value store), per site: pages audited, status code counts, issue counts by kind, groups of duplicate titles, descriptions and H1s, broken addresses with the pages that link to them, redirect chains, addresses robots.txt blocks, sitemap addresses the crawl didn't reach (and why), external link results, and a site score out of 100. Each audited page starts at 100 and loses 15 points for each kind of error and 5 for each kind of warning it has (notices don't count), down to 0; the site score is the average over the audited pages.

How to use it

In Apify Console

1. Enter a start page, usually the site's home page. You can add more sites; they're crawled side by side.
2. Set Pages per site (100 by default). Pages nearest the start page are audited first.
3. Optionally narrow the crawl with Link depth, Only addresses matching or Skip addresses matching (for example */tag/*), or include subdomains.
4. Click Start. At the default pace a site is crawled at about one page a second.
5. Open the Pages, Issues, Links and redirects or Tags view, or download the results as Excel, CSV or JSON. The Storage → Key-value store → RUN_SUMMARY record has the site summary.

Check links to other sites

Turn on Check links to other sites to check, by status only, the links that lead off the site (up to 300 by default). Each of those sites' robots.txt and pace are respected. A site that turns the check away (HTTP 401, 403, 429) is listed as "refused" in the summary, not as broken, because many sites answer bots that way.

Get alerts for new issues (monitoring)

1. Enter a Monitor ID, for example my-site-weekly.
2. Schedule the Actor (Schedules → create, e.g. weekly) with the same input.

Each run then compares every address with the last crawl of the same site: rows get isNewPage, newIssues and fixedIssues, and the summary counts them. Use the Changes since last crawl view, or connect the run to Slack or email with Apify's integrations.

With the Apify API

curl -X POST "https://api.apify.com/v2/acts/subtleasianmerch~seo-site-audit/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://www.example.com"], "maxPagesPerSite": 200}'

Example input

{
  "startUrls": ["https://www.example.com"],
  "maxPagesPerSite": 500,
  "maxDepth": 5,
  "sitemapPages": "crawl",
  "excludePatterns": ["*/tag/*", "?sort="],
  "checkExternalLinks": true,
  "maxExternalLinks": 300,
  "delayMs": 500,
  "monitorId": "example-weekly"
}

Output

A real row from a test run on Apify (2 November 2026, docs.apify.com), shortened:

{
  "url": "https://docs.apify.com/proxy",
  "type": "page",
  "statusCode": 200,
  "finalUrl": "https://docs.apify.com/proxy",
  "redirectHops": 0,
  "responseTimeMs": 4,
  "sizeBytes": 76878,
  "depth": 1,
  "inlinks": 99,
  "inSitemap": true,
  "indexable": true,
  "indexability": "indexable",
  "title": "Proxy | Platform | Apify Documentation",
  "titleLength": 38,
  "metaDescription": "Learn to anonymously access websites in scraping/automation jobs. Improve data outputs and efficiency of bots, and access websites from various geographies.",
  "metaDescriptionLength": 156,
  "h1": "Proxy",
  "h1Count": 1,
  "h2Count": 9,
  "canonicalUrl": "https://docs.apify.com/proxy",
  "canonicalStatus": "self",
  "noindex": false,
  "hreflang": [
    { "lang": "en", "url": "https://docs.apify.com/proxy", "statusCode": 200, "returnLink": true },
    { "lang": "x-default", "url": "https://docs.apify.com/proxy", "statusCode": 200, "returnLink": true }
  ],
  "htmlLang": "en",
  "viewport": true,
  "twitterCard": "summary_large_image",
  "structuredDataTypes": [],
  "images": 3,
  "imagesWithoutAlt": 0,
  "wordCount": 1747,
  "internalLinks": 38,
  "externalLinks": 34,
  "brokenLinks": [
    { "url": "https://docs.apify.com/residential_proxy", "statusCode": 404, "error": null, "internal": true }
  ],
  "brokenLinkCount": 1,
  "linksToRedirects": 2,
  "issues": [
    { "code": "broken-internal-links", "severity": "error", "message": "1 link to broken pages on the site: /residential_proxy (HTTP 404)." },
    { "code": "structured-data-missing", "severity": "notice", "message": "No structured data (JSON-LD or microdata)." },
    { "code": "links-to-redirects", "severity": "notice", "message": "2 links to addresses that redirect (e.g. /cli, /sdk/js/docs/guides/proxy-management); linking to the final address saves a hop." }
  ],
  "issueCount": 3,
  "errorCount": 1,
  "warningCount": 0,
  "noticeCount": 2,
  "site": "docs.apify.com",
  "crawledAt": "2026-11-02T18:37:37.748Z"
}

The broken address has a row of its own (free): { "url": "https://docs.apify.com/residential_proxy", "type": "error", "statusCode": 404, "linkedFrom": ["https://docs.apify.com/proxy"], "issues": [{ "code": "http-4xx", "severity": "error", "message": "HTTP 404. 1 crawled page links to it (e.g. /proxy)." }] }. A redirect row lists its chain, for example "redirectChain": [{ "url": "https://crawler-test.com/redirects/redirect_2", "statusCode": 301, "redirectsTo": "…" }, …], with finalUrl and finalStatusCode.

Pricing

• $0.003 per audited page ($3 per 1,000 pages). 500 pages cost $1.50.
• Free: redirects, error pages, addresses that don't answer, files, addresses robots.txt blocks, sites that refuse the crawler, status checks of links beyond your page limit, and external link checks.
• A run stops cleanly at your maximum cost per run. Apify's platform usage is included in the price.

Good to know

• No browser. The crawler reads the HTML the server sends, as search engines do on their first pass. Pages built only with JavaScript (some single-page apps) show little text, few links and missing tags here, even if they look fine in a browser. For speed and Core Web Vitals, use the Lighthouse Audit Actor (below).
• robots.txt always applies. Addresses it blocks are listed but never fetched. If robots.txt can't be read (the server answers 5xx), the site is treated as closed to crawlers, as the robots.txt standard says. A Crawl-delay is honoured up to 60 seconds; a site asking for more isn't crawled.
• Gentle pace. One request at a time per site, at least 500 ms apart by default (you can choose 250 ms to 60 s), and longer if robots.txt asks. At the default pace a site goes at roughly 1,000 to 4,000 pages an hour, depending on how fast it answers (in tests: 100 pages of docs.apify.com in 87 seconds). The run's time limit is 1 hour by default; for big sites, raise it under Run options → Timeout. The crawl stops about 2–3 minutes before the limit and delivers what it found.
• Sites that refuse crawlers are left alone. A bot check (Cloudflare, DataDome and similar), a 403 on the first page, five 401/403 answers in a row, or HTTP 429 that continues after waiting stop that site's crawl at once; it's listed as blocked, free. The Actor doesn't use proxies or disguises to get past blocks.
• Limits: up to 5,000 audited pages per run in all, and up to twice your page limit in requests per site. Once the page limit is reached, the links that the audited pages point to are still checked for status (HEAD requests, free), so their broken links are found; pages beyond the limit aren't audited. Sitemaps: up to 20 files, 100,000 addresses, 30 MB or 60 seconds of reading. HTML pages larger than 5 MB are read up to 5 MB.
• Addresses are normalised: the #fragment is dropped, and tracking parameters (utm_*, gclid, fbclid and similar) are removed unless you turn that off. /page and /page/ stay different addresses, because servers treat them differently; when both answer, the duplicate title check shows it.
• Duplicates are counted among indexable pages only: a page that is noindex or whose canonical points elsewhere is meant to be a copy.
• Length limits for titles (30–60 characters) and meta descriptions (70–160) are the usual guides; search engines actually cut by pixel width.
• Nofollow links are followed within the site, so broken links behind them are found too.

FAQ

Does it work on any website? On public sites that allow crawling. Pages behind a login, and sites that block crawlers, can't be audited.

Will it slow my site down? No more than a single visitor: one request at a time, with a pause after each.

Can I audit just one section? Yes: start at that section and use Only addresses matching, for example https://www.example.com/blog/*.

What about orphan pages? With Sitemap pages set to "crawl" (the default), pages only the sitemap lists are audited after the linked pages, and flagged as orphans when the crawl finished without reaching the page limit.

Why is a page I can see in my browser almost empty here? It's probably built with JavaScript in the browser. The crawler sees the HTML the server sends.

What does a broken link cost? Nothing: only audited HTML pages are charged.

Can I get the results in Google Sheets? Yes: export the dataset as CSV or Excel and open it in Google Sheets, or connect it with Make, Zapier or n8n.

Related Actors

More Actors from the same developer that work well with this one:

Actor	Use it to
Lighthouse Audit	score the same pages for speed, Core Web Vitals and accessibility

Questions, bugs or ideas? Open an issue on this Actor's Issues tab.