# Technical SEO Site Crawler – Broken Links & Audit (`benelux-signals/technical-seo-site-crawler`) Actor

Crawl a website within robots.txt and get one row per page: status code, redirects, title, meta description, H1, canonical, noindex, broken internal links, duplicates, orphan pages, sitemap gaps and a 0–100 SEO score.

- **URL**: https://apify.com/benelux-signals/technical-seo-site-crawler.md
- **Developed by:** [Benelux Signals](https://apify.com/benelux-signals) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.50 / 1,000 page crawleds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Technical SEO Site Crawler: Broken Links, Redirects, Duplicates and Orphan Pages

**Crawl a whole website like a search engine and get one row per URL with everything a technical SEO audit needs**: HTTP status code, redirect target and chain, indexability (and why not), title, meta description, H1, canonical, noindex, hreflang, word count, structured data, response time, **broken internal links on every page**, **duplicate titles and meta descriptions**, **orphan pages**, **pages missing from the XML sitemap**, click depth, inlinks and a **0–100 SEO score** with a prioritised list of issues.

A site-wide **SITE_REPORT** sums it all up: status codes, issue counts, every broken link with the page it sits on, redirect chains, duplicate groups, orphans and the slowest pages. Export to CSV, Excel or JSON, or call it from the API, Make, Zapier or n8n. Pay only **$0.002 per page**, with no subscription and no desktop crawler licence.

### Who is this SEO crawler for?

- **SEO consultants and agencies** who need a quick technical audit of a client site before a pitch or a monthly report.
- **Web developers** who want to catch 404s, redirect chains and noindex mistakes before or after a migration or relaunch.
- **Content and marketing teams** looking for duplicate titles, missing meta descriptions, thin pages and pages that nothing links to.
- **Site owners** who want a scheduled weekly health check without installing Screaming Frog or paying for a SaaS seat.

### What does Technical SEO Site Crawler do?

1. Reads **robots.txt** and the **XML sitemap(s)** of the site.
2. Starts at your URL and follows internal links (www and non-www count as the same site), breadth-first, up to your **Max pages** and **Max click depth**.
3. Requests every URL **without following redirects automatically**, so each 301/302/307/308 is listed with its target, and chains and loops are detected.
4. Analyses every HTML page: on-page SEO checks plus the links on the page.
5. When the link crawl is done, it also checks **sitemap URLs that no page links to** (orphan pages), as long as Max pages allows.
6. Combines everything into site-wide checks: broken links, links to redirects, duplicates, sitemap gaps and depth.

Plain HTTP crawler: it reads the HTML the server sends and does **not render JavaScript**, which keeps it fast and cheap. Sites that build all their content in the browser (pure SPAs) will show up as thin pages.

### How to use it

1. Enter your homepage (or a section such as `https://example.com/blog/`) under **Website to crawl**.
2. Set **Max pages**. This is your cost cap.
3. Optional: add **Only crawl URLs containing** or **Skip URLs containing** patterns (`/blog/`, `?sort=`, `/tag/`), turn on **Ignore query strings** for shops with filters, or **Include subdomains**.
4. Click **Start** and open the **Overview**, **Issues**, **Titles & meta** or **Links & redirects** tab, or download SITE_REPORT from the key-value store.

#### Input example

```json
{
  "startUrls": ["https://www.example.com/"],
  "maxPages": 500,
  "excludePatterns": ["/tag/", "?replytocom="],
  "ignoreQueryStrings": true,
  "onlyPagesWithIssues": false
}
```

#### Output example (one page, real run on crawlee.dev, 28 Oct 2026)

```json
{
  "status": "ok",
  "url": "https://crawlee.dev/blog",
  "statusCode": 200,
  "indexable": true,
  "indexabilityReason": null,
  "depth": 1,
  "foundOn": "https://crawlee.dev/",
  "inlinks": 96,
  "inSitemap": true,
  "title": "Crawlee Blog - learn how to build better scrapers | Crawlee for JavaScript · Build reliable crawlers. Fast.",
  "titleLength": 107,
  "metaDescriptionLength": 151,
  "h1Count": 0,
  "canonicalIsSelf": true,
  "wordCount": 1464,
  "ttfbMs": 3,
  "brokenLinks": [],
  "linksToRedirects": 3,
  "duplicateTitlePages": 1,
  "seoScore": 77,
  "issueCodes": [
    "h1-missing",
    "duplicate-title",
    "title-too-long",
    "duplicate-meta-description",
    "links-to-redirects"
  ],
  "issues": [
    {
      "severity": "error",
      "code": "h1-missing",
      "message": "No H1 heading on the page."
    },
    {
      "severity": "warning",
      "code": "duplicate-title",
      "message": "Same title as 1 other page(s)."
    },
    {
      "severity": "warning",
      "code": "title-too-long",
      "message": "Title is long (107 characters; Google usually cuts off around 60)."
    },
    {
      "severity": "warning",
      "code": "duplicate-meta-description",
      "message": "Same meta description as 1 other page(s)."
    },
    {
      "severity": "notice",
      "code": "links-to-redirects",
      "message": "3 internal link(s) point to redirecting URLs; link to the final URL."
    }
  ]
}
```

Redirects get `redirectTo`, `redirectFinalUrl` and `redirectHops`; 404 pages list the page that links to them in `foundOn`, and every page shows its own broken links in `brokenLinks` (`{url, statusCode}`, or `{url, statusCode: null, error}` when the target could not be loaded at all, e.g. a timeout after two attempts).

### Which issues are detected?

- **Errors**: broken internal links (4xx/5xx/unreachable), HTTP 4xx/5xx pages, redirect loops, redirects to error pages, HTTPS redirecting to HTTP, missing title, missing meta description, missing H1, noindex, missing viewport, no HTTPS, pages blocked for Googlebot in robots.txt.
- **Warnings**: duplicate titles and meta descriptions, orphan pages (in the sitemap, not linked), sitemap URLs that are not indexable, redirect chains, titles and descriptions too long or short, multiple H1s, canonical missing, invalid hreflang or JSON-LD, images without alt text, thin content, slow server response, HTML over 2 MB (Googlebot only reads the first 2 MB).
- **Notices**: links to redirects, temporary redirects, indexable pages missing from the sitemap, pages deeper than 4 clicks, missing structured data or Open Graph tags, hreflang without self-reference or x-default.

The SEO score starts at 100 and deducts points per failed check, so you can sort pages by priority.

### Pricing

Pay per event:

- **$0.005** per run start
- **$0.002 per page** on the Free plan; **$0.0018** Bronze, **$0.0016** Silver, **$0.0015** Gold and higher

Every crawled URL counts as one page, including redirects and 404s, because those are findings too. **Free**: pages that could not be reached at all, URLs blocked by robots.txt, and pages left out by **Only save pages with errors or warnings**.

| Pages | Free plan cost |
|---|---|
| 100 | $0.21 |
| 500 | $1.01 |
| 1,000 | $2.01 |
| 5,000 | $10.01 |

If your spending limit for the run is lower, the crawl stops early at that page count.

### FAQ

#### Does it respect robots.txt?

Yes, always. Disallowed URLs are not fetched (they are counted in SITE_REPORT), a Crawl-delay is honoured, and requests are spaced at least 500 ms apart by default with up to 3 in parallel. The crawler identifies itself as `TechnicalSeoCrawler`.

#### How long does a crawl take?

About 1–2 pages per second with the default politeness settings: 100 pages in ~1 minute, 1,000 pages in ~10 minutes. You can lower the delay for your own site.

#### Can it crawl JavaScript-rendered sites?

No, it reads the server HTML, like most technical SEO crawlers in their default mode. For performance and rendering checks, use our Bulk PageSpeed & SEO Audit.

#### Why are there pages from the sitemap with depth null?

Those are orphan pages: listed in the sitemap but not linked from any crawled page. They are crawled after the link crawl if Max pages allows.

#### Can I monitor my site every week?

Yes. Save your input as a task and add a schedule; combine it with **Only save pages with errors or warnings** to get a short list of problems each week.

#### Is it legal?

It only reads publicly available pages, like a search engine, identifies itself, respects robots.txt and keeps the load low. Only crawl sites you are allowed to audit and respect their terms.

### Related actors by benelux-signals

Combine a full-site crawl with speed and news checks:

- [Bulk PageSpeed & SEO Audit](https://apify.com/benelux-signals/bulk-pagespeed-seo-audit) – Core Web Vitals, Lighthouse scores and on-page SEO checks for many URLs or a sitemap.
- [EU & Dutch News Monitor](https://apify.com/benelux-signals/eu-news-rss-monitor) – keyword alerts across 80+ built-in Dutch, Belgian and EU news and official RSS feeds.

More tools from the same maintainer:

- [EU Tender Radar](https://apify.com/benelux-signals/eu-tender-radar) – TED + TenderNed public tenders with keyword/CPV filters and only-new alerts.
- [Benelux Trade Fair Exhibitors](https://apify.com/benelux-signals/benelux-trade-fair-exhibitors) – exhibitor lists from Dutch and Belgian trade fairs (Horecava, METSTRADE, Batibouw…) for B2B lead lists.
- [Career Site Jobs API](https://apify.com/benelux-signals/eu-career-site-jobs-api) – jobs straight from Greenhouse, Lever, Recruitee, Personio, Ashby and other company career sites.
- [Workday Jobs API](https://apify.com/benelux-signals/workday-jobs-api-europe) – jobs from any myworkdayjobs.com career site, with a European employer seed list.
- [PDF & Image to Text OCR](https://apify.com/benelux-signals/pdf-image-to-text-ocr) – text and Markdown from PDFs and scans, ready for RAG and AI agents.
- [Arbeitsagentur Jobs API](https://apify.com/benelux-signals/arbeitsagentur-jobs-api) – German job ads from the Bundesagentur für Arbeit Jobbörse.
- [Remote Jobs Aggregator](https://apify.com/benelux-signals/remote-jobs-aggregator) – Europe-friendly remote jobs from RemoteOK, We Work Remotely, Jobicy and more, deduplicated.

Questions or a source we should add? Open an issue on the Actor page.

# Actor input Schema

## `startUrls` (type: `array`):

Homepage or section URL to start from. The crawler follows internal links on the same host (www and non-www count as the same site). Add several URLs of the same site to crawl several sections; for different websites start separate runs.

## `maxPages` (type: `integer`):

Stop after this many pages (HTML pages, redirects and error pages each count as one). You pay per page, so this is your cost cap.

## `maxDepth` (type: `integer`):

How many clicks away from the start URL the crawler goes. 0 = only the start URL(s) (orphan pages from the sitemap are then not crawled either).

## `useSitemap` (type: `boolean`):

Read robots.txt sitemaps (or /sitemap.xml) to flag indexable pages missing from the sitemap, sitemap URLs that are not indexable, and orphan pages (in the sitemap but not linked). Orphan pages are crawled once the link crawl is done and pages are left within Max pages.

## `includePatterns` (type: `array`):

Optional. Only follow URLs that contain one of these texts (e.g. /blog/). Use \* as wildcard for a full-URL pattern (e.g. https://example.com/nl/\*). Start URLs are always crawled.

## `excludePatterns` (type: `array`):

Optional. Do not crawl URLs that contain one of these texts, e.g. /tag/, ?sort=, /wp-admin/. \* is a wildcard.

## `ignoreQueryStrings` (type: `boolean`):

Treat /page?a=1 and /page?a=2 as the same URL. Turn on for shops and sites with filter or tracking parameters to avoid crawling thousands of variants.

## `includeSubdomains` (type: `boolean`):

Also crawl subdomains such as blog.example.com and shop.example.com.

## `followNofollowLinks` (type: `boolean`):

By default links with rel=nofollow (and pages with meta robots nofollow) are not followed, like search engines.

## `onlyPagesWithIssues` (type: `boolean`):

Skip pages without errors or warnings in the dataset (they are not charged). The SITE_REPORT still covers all crawled pages.

## `minDelayMs` (type: `integer`):

Minimum time between two requests to the site. A Crawl-delay in robots.txt is respected when it is longer (up to 30 s). Keep this polite for small sites.

## `maxConcurrency` (type: `integer`):

Requests in flight at the same time. The delay above still spaces them out.

## Actor input object example

```json
{
  "startUrls": [
    "https://crawlee.dev/"
  ],
  "maxPages": 50,
  "maxDepth": 20,
  "useSitemap": true,
  "includePatterns": [],
  "excludePatterns": [],
  "ignoreQueryStrings": false,
  "includeSubdomains": false,
  "followNofollowLinks": false,
  "onlyPagesWithIssues": false,
  "minDelayMs": 500,
  "maxConcurrency": 3
}
```

# Actor output Schema

## `overview` (type: `string`):

All crawled pages in the 'overview' table view.

## `siteReport` (type: `string`):

Site-wide JSON report: status codes, issue counts, broken links, redirects, duplicate titles/descriptions, orphan pages, slowest pages.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://crawlee.dev/"
    ],
    "maxPages": 50
};

// Run the Actor and wait for it to finish
const run = await client.actor("benelux-signals/technical-seo-site-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://crawlee.dev/"],
    "maxPages": 50,
}

# Run the Actor and wait for it to finish
run = client.actor("benelux-signals/technical-seo-site-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://crawlee.dev/"
  ],
  "maxPages": 50
}' |
apify call benelux-signals/technical-seo-site-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,benelux-signals/technical-seo-site-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/uoVha9O70AN1wx3cV/builds/evRQaQnDV5LNJsf0u/openapi.json