# Site SEO Crawler: Broken Links & On-Page Audit (`offerastudio/site-seo-crawler-audit`) Actor

Crawl a whole site and find broken internal and external links with the pages that link to them, redirect chains and loops, and on-page SEO issues per page: titles, meta descriptions, H1, noindex, canonical, thin pages, missing alt text and slow responses. Respects robots.txt.

- **URL**: https://apify.com/offerastudio/site-seo-crawler-audit.md
- **Developed by:** [Offera Studio](https://apify.com/offerastudio) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 page auditeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### What does Site SEO Crawler & Audit do?

**Site SEO Crawler & Audit** crawls a **whole website** and tells you what is wrong, in one run:

- 🔗 **Broken links, internal and external, with the pages that link to them** (up to 20 per link) and the anchor text, so you know where to fix each one.
- ↪️ **Redirects, chains and loops**: every hop of every redirect, and the internal links that still point to a redirecting URL.
- 📄 **On-page issues per page**: missing or duplicate titles and meta descriptions, missing or multiple H1, `noindex`, canonical problems (missing, several, pointing to a broken, redirecting or noindex page), thin pages, images without alt text and slow responses, each with a severity.
- 📊 **A free summary row per site** with the number of pages affected by each issue type.

It starts at your start URL and reads `sitemap.xml`, follows internal links breadth-first (nearest pages first), stays on the site (and its subdomains if you ask), and **respects `robots.txt`** and its Crawl-delay. HTTP only with Cheerio: no browser, no proxy, no login.

### Who is it for?

- **SEO and content teams**: a site-wide to-do list: which pages have no title, which share one, which are noindex by mistake.
- **Web agencies and freelancers**: audit a client's site before and after a relaunch or migration (redirects, broken links).
- **Developers and site owners**: find 404s and redirect chains after a release.
- **Marketers**: check landing pages for thin content, missing H1s and slow responses.

### How to use it

1. Enter the **Start URL** (the home page of the site). You can add several sites.
2. Set **Max pages per site** (default 200, up to 5,000). The crawl goes breadth-first, so the pages closest to the start come first.
3. Keep **Check external links** on to find dead links to other sites. Switch on **Include subdomains** to crawl `blog.example.com` too.
4. Click **Start**. **Pages** shows one row per page with its issues, **Broken links** shows each broken link with the pages that contain it, **Site summary** shows the totals.

### Input example

```json
{
    "startUrls": [{ "url": "https://www.example.com/" }],
    "maxPages": 500,
    "checkExternalLinks": true,
    "includeSubdomains": false,
    "respectRobots": true,
    "userAgent": "MySiteAudit"
}
```

### Output example

One row per page (real run on `quotes.toscrape.com`, 7 October 2026, a public practice site):

```json
{
    "type": "page",
    "site": "quotes.toscrape.com",
    "url": "https://quotes.toscrape.com/login",
    "status": 200,
    "depth": 1,
    "discoveredVia": "link",
    "inSitemap": null,
    "finalUrl": "https://quotes.toscrape.com/login",
    "redirectChain": [],
    "contentType": "text/html",
    "title": "Quotes to Scrape",
    "metaDescription": null,
    "h1s": ["Quotes to Scrape"],
    "canonical": null,
    "robotsMeta": null,
    "indexable": true,
    "wordCount": 13,
    "responseMs": 10,
    "imagesWithoutAlt": 0,
    "internalLinks": 2,
    "externalLinks": 2,
    "brokenLinks": [],
    "issues": [
        { "type": "meta-description-missing", "severity": "warning", "detail": "The page has no meta description." },
        { "type": "canonical-missing", "severity": "info", "detail": "The page has no canonical link." },
        { "type": "thin-page", "severity": "warning", "detail": "Only 13 words of visible text (under 200)." },
        { "type": "title-duplicate", "severity": "warning", "detail": "The same title is used on 24 pages, for example https://quotes.toscrape.com/." }
    ],
    "issueCount": 4
}
```

A URL that redirects gets its own free row with every hop:

```json
{
    "type": "page",
    "url": "https://quotes.toscrape.com/author/Albert-Einstein",
    "status": 308,
    "finalUrl": "http://quotes.toscrape.com/author/Albert-Einstein/",
    "redirectChain": [
        { "url": "https://quotes.toscrape.com/author/Albert-Einstein", "status": 308 },
        { "url": "http://quotes.toscrape.com/author/Albert-Einstein/", "status": 200 }
    ],
    "issues": [{ "type": "redirect", "severity": "info", "detail": "Redirects (308) to http://quotes.toscrape.com/author/Albert-Einstein/." }]
}
```

A broken link row (from the Actor's test site, host replaced by `example.com`):

```json
{
    "type": "broken-link",
    "site": "example.com",
    "url": "https://example.com/missing",
    "status": 404,
    "error": "http-error",
    "errorMessage": "The URL answers HTTP 404.",
    "isExternal": false,
    "linkedFrom": ["https://example.com/", "https://example.com/a"],
    "linkedFromCount": 2,
    "anchorText": "Missing page",
    "checkedVia": "crawl"
}
```

And the free **summary row**, for example: *"quotes.toscrape.com: 30 pages crawled (stopped at the page limit), 24 audited; 0 broken links (0 internal, 0 external); 0 redirect chains or loops; 0 errors and 66 warnings on pages; most common: meta-description-missing (24), title-duplicate (24), thin-page (18)."*

#### Issue types

| Severity | Types |
| --- | --- |
| error | `title-missing`, `canonical-multiple`, `canonical-broken`, `broken-links`, `redirect-loop`, `redirect-to-error`, `http-error`, `fetch-failed` |
| warning | `meta-description-missing`, `title-duplicate`, `meta-description-duplicate`, `h1-missing`, `noindex`, `thin-page` (under 200 words), `images-missing-alt`, `slow-response` (over 2,000 ms), `canonical-redirects`, `canonical-noindex`, `canonical-chain`, `canonical-invalid`, `redirect-chain`, `redirect-to-http` |
| info | `title-too-long` (over 60 characters), `title-too-short`, `meta-description-too-long` (over 160), `h1-multiple`, `canonical-missing`, `canonical-other-url`, `canonical-other-domain`, `redirect`, `redirect-temporary`, `links-to-redirect`, `not-html` |

### How much does it cost?

Pay per event: **$0.001 per audited page** (`page-audited`): a page that answers HTTP 200 with HTML and gets the full on-page audit. 1,000 pages cost $1.00. **Free**: broken-link rows, the summary row, and rows for URLs that were not audited (redirects, 404 and other error pages, non-HTML files). Use **Max pages** or Apify's "maximum cost per run" to cap spend.

### Politeness and robots.txt

- **robots.txt is respected by default**: URLs it disallows for the crawler's name are not fetched (they are counted in the summary), and its **Crawl-delay** is honoured (up to 10 s between requests). **Respect robots.txt is on by default**; switch it off only for sites you own or may crawl.
- At most **two requests at a time** to the site, **300 ms apart**; retries on 429 and 5xx with backoff.
- External links are checked **once each per run**, one at a time per host: `HEAD` first, `GET` when the server answers 405 or 501. Sites that refuse automated checks (HTTP 401, 403, 429, 999) are not reported as broken.
- The crawler announces itself with a name you can set (**User-Agent name**), so site owners can recognise it.

### Limitations

- **HTTP only, no JavaScript rendering.** Links and content that only appear after scripts run are not seen. Single-page apps may look empty.
- **Page limit**: the crawl stops at **Max pages per site**; the summary says so. Internal links beyond the limit are not checked.
- **Word count** is the visible text of the page, including menus and footers, so "thin" is a hint, not a verdict.
- **Response time** is the download time of the page from the Actor's server, not a performance score.
- **Duplicate titles and descriptions** are compared across the crawled pages only.
- Query-string URLs are different pages (tracking parameters such as `utm_*` are ignored). Sites with endless filter URLs can use up the page limit: set a lower limit.
- **External link checks** can show false alarms for sites that block automated requests with unusual answers; the row says what was returned.

### FAQ

#### Does it check external links?

Yes, by default (switch off with **Check external links**). Each distinct link is checked once.

#### How do I find the pages that link to a broken URL?

Open the **Broken links** tab: every row lists up to 20 pages that contain the link (`linkedFrom`), the number of all pages (`linkedFromCount`) and the anchor text. Each page row also lists its own broken links.

#### Why was a page not crawled?

It may be disallowed by `robots.txt`, outside the site (another host or a subdomain you did not include), beyond **Max pages per site**, or a file (PDF, image ...), which is checked as a link but not crawled.

#### Does it use a proxy or a browser?

No.

### More tools from the same developer

All pay-per-result, no proxy or login needed, built and maintained by the same developer:

**Website audits**

- [Website Accessibility Checker: WCAG 2.2 & EAA](https://apify.com/offerastudio/website-accessibility-audit): accessibility issues with fixes, SEO basics and security headers.
- [Cookie & Tracker Audit: GDPR Consent Checker](https://apify.com/offerastudio/cookie-tracker-audit): cookies and tracking tags that load before consent.
- [AI Crawler Access Checker: robots.txt & llms.txt](https://apify.com/offerastudio/ai-crawler-access-audit): which AI crawlers a site allows, plus llms.txt.
- [Structured Data Validator: Schema & Rich Results](https://apify.com/offerastudio/structured-data-validator): JSON-LD and Microdata checked against Google's rich result rules.
- [Website Change Monitor: Diffs, Prices & Alerts](https://apify.com/offerastudio/website-change-monitor): get a row only when a page changes, with a clean diff.
- [Policy Change Monitor: Privacy & Terms Tracker](https://apify.com/offerastudio/policy-change-monitor): privacy policy and terms changes, classified clause by clause.
- [SaaS Pricing Monitor: Track Competitor Plans](https://apify.com/offerastudio/saas-pricing-monitor): competitor pricing plans as structured data, with price-change diffs.
- [Sitemap Change Monitor: New & Removed URLs](https://apify.com/offerastudio/sitemap-change-monitor): new, removed and changed URLs from a site's sitemaps.
- [Website Tech Stack Detector: Bulk Lead Filters](https://apify.com/offerastudio/website-tech-stack-detector): technologies used by websites, with filters for lead lists.
- [Domain Availability & Expiry Checker (RDAP)](https://apify.com/offerastudio/domain-whois-availability): domain availability, registrar and expiry dates from RDAP.
- [Status Page SLA Monitor: Uptime & Incidents](https://apify.com/offerastudio/status-page-sla-monitor): SaaS vendor uptime, incidents and a monthly SLA table from public status pages.

**Company data and compliance**

- [Company Contact Finder: Emails, Phones & Socials](https://apify.com/offerastudio/company-contact-finder): contact details published on company websites.
- [UK New Companies Feed: Companies House Daily](https://apify.com/offerastudio/uk-new-companies-feed): newly incorporated UK companies with sector filters.
- [EU VAT Number Validator: Bulk VIES Checker](https://apify.com/offerastudio/eu-vat-number-validator): bulk VAT checks with name, address and consultation number.
- [LEI Corporate Tree: GLEIF Parents & Subsidiaries](https://apify.com/offerastudio/gleif-lei-corporate-tree): LEI lookup with parents, subsidiaries and a KYC summary.
- [UK Company Distress Watch: Insolvency Alerts](https://apify.com/offerastudio/uk-company-distress-watch): administrations, liquidations and winding-up petitions from The Gazette.
- [IBAN Validator: Bulk Check & Bank Lookup](https://apify.com/offerastudio/iban-bank-validator): IBAN format and checksum, plus bank name and BIC where open data exists.
- [Sanctions Screening: OFAC, EU, UN & UK Lists](https://apify.com/offerastudio/sanctions-screening): names screened against official sanctions lists, with match scores.
- [OSHA & EPA Violation Monitor: Fines by Company](https://apify.com/offerastudio/osha-epa-violation-monitor): US OSHA and EPA enforcement cases and penalties by company.
- [Email List Precheck: Syntax, MX & Disposable Check](https://apify.com/offerastudio/email-list-precheck): cheap email list hygiene before a real verification.
- [UK Company Officer Changes: Director & PSC Alerts](https://apify.com/offerastudio/uk-company-officer-changes): director, owner, address and status changes for a watchlist of UK companies.
- [FDA Warning Letters Monitor: By Company & Subject](https://apify.com/offerastudio/fda-warning-letters-monitor): FDA warning letters by company and subject, with cited regulations.
- [Supplier Risk Report: Sanctions, VAT & Insolvency](https://apify.com/offerastudio/supplier-risk-report): one risk row per supplier from eight official registers, with changes since last run.
- [Brand Impersonation Domain Monitor: Look-alikes](https://apify.com/offerastudio/brand-impersonation-domain-monitor): look-alike domains and certificates that imitate your brand, with a risk score.

**Media and AI**

- [Audio & Video Transcriber: Podcasts, SRT, Speakers](https://apify.com/offerastudio/audio-video-transcriber): audio, video, podcasts and social links to text, SRT and VTT.
- [Podcast Mention Alerts: Brand Mentions by Second](https://apify.com/offerastudio/podcast-mention-alerts): where your brand is said in new podcast episodes, with the exact second.
- [News Brand Mentions Monitor: New Articles Only](https://apify.com/offerastudio/news-brand-mentions-monitor): new news articles about your brand, people or topics.
- [YouTube Channel Monitor: New Videos & Growth](https://apify.com/offerastudio/youtube-channel-monitor): new videos and view growth of the channels you watch.

**Market signals**

- [US WARN Layoff Notices: 12 States Daily Feed](https://apify.com/offerastudio/us-warn-layoff-notices): layoff and plant closure notices from official state sources.
- [US Product Recalls Monitor: FDA & CPSC Feed](https://apify.com/offerastudio/us-product-recalls-monitor): FDA and CPSC recalls in one feed, with severity.
- [EU Tenders Feed: TED Public Procurement Alerts](https://apify.com/offerastudio/eu-tender-feed): EU public tenders from TED with CPV, country and value filters.
- [Liquor License Openings: New Bars & Stores (US)](https://apify.com/offerastudio/liquor-license-openings): new US liquor licences from official state open data.
- [Clinical Trials Change Monitor: Status & Updates](https://apify.com/offerastudio/clinicaltrials-change-monitor): status, enrollment and results changes on ClinicalTrials.gov studies.
- [Career Site Jobs Monitor: ATS Jobs & New Roles](https://apify.com/offerastudio/career-site-jobs-monitor): open roles from company career sites, with new and closed roles since the last run.
- [App Reviews Monitor: App Store & Google Play](https://apify.com/offerastudio/app-store-reviews-monitor): app reviews from both stores, only new ones in monitor mode.
- [US Federal Contracts Feed: SAM.gov & USAspending](https://apify.com/offerastudio/us-federal-contracts-feed): SAM.gov contract opportunities with the buyer's award history.
- [Grants.gov Opportunities: US Federal Grants Feed](https://apify.com/offerastudio/grants-gov-opportunities): US federal grant opportunities by keyword, eligibility and agency.
- [Federal Register Monitor: US Rules & Notices Feed](https://apify.com/offerastudio/federal-register-monitor): new US rules, proposed rules and notices by agency or topic.
- [B2B Sales Trigger Feed: Hiring + Tech Leads](https://apify.com/offerastudio/b2b-sales-trigger-feed): companies hiring for a role and using (or not using) a technology, with contacts.
- [Local Business Leads: OpenStreetMap, No Website](https://apify.com/offerastudio/local-business-leads-osm): local businesses by city and category, including those with no website.
- [Solana Token Pair Monitor: New & Trending Pairs](https://apify.com/offerastudio/solana-token-pair-monitor): new and trending Solana pairs with liquidity, volume and safety flags.

### Feedback

Missing an issue type, or a page that is wrongly flagged? Open an issue on the **Issues** tab.

# Changelog

This Actor's version history is a separate document: https://apify.com/offerastudio/site-seo-crawler-audit/changelog.md

# Actor input Schema

## `startUrls` (type: `array`):

One or more sites to crawl (the home page of each is the usual start). Each start URL is crawled on its own host (and its subdomains if you ask for it), up to "Max pages" each.

## `maxPages` (type: `integer`):

Pages crawled per start URL (up to 5,000). The crawl goes breadth-first, so the pages nearest the start come first, and sitemap URLs are added.

## `checkExternalLinks` (type: `boolean`):

Check every link that leaves the site once per run (HEAD, then GET when the server does not allow HEAD) and report the broken ones.

## `includeSubdomains` (type: `boolean`):

Also crawl subdomains of the start host (blog.example.com when starting at example.com).

## `respectRobots` (type: `boolean`):

Skip URLs the site's robots.txt disallows and honour its Crawl-delay. Keep this on unless you own the site or have permission.

## `userAgent` (type: `string`):

Name the crawler announces and robots.txt rules are matched against, e.g. MySiteAudit. Site owners see it in their logs.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://quotes.toscrape.com/"
    }
  ],
  "maxPages": 30,
  "checkExternalLinks": true,
  "includeSubdomains": false,
  "respectRobots": true,
  "userAgent": "OfferaStudioBot"
}
```

# Actor output Schema

## `pages` (type: `string`):

No description

## `brokenLinks` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://quotes.toscrape.com/"
        }
    ],
    "maxPages": 30
};

// Run the Actor and wait for it to finish
const run = await client.actor("offerastudio/site-seo-crawler-audit").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://quotes.toscrape.com/" }],
    "maxPages": 30,
}

# Run the Actor and wait for it to finish
run = client.actor("offerastudio/site-seo-crawler-audit").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://quotes.toscrape.com/"
    }
  ],
  "maxPages": 30
}' |
apify call offerastudio/site-seo-crawler-audit --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,offerastudio/site-seo-crawler-audit"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/u2LfPEFyhQfyoddWP/builds/DKB5hpoDgi4knHr2o/openapi.json