# SEO Site Audit Scraper (`datawatcher/seo-site-audit-scraper`) Actor Audit any website for SEO: crawl pages, check titles, meta tags, headings, canonicals, noindex, status codes, redirect chains and broken links, read robots.txt and sitemaps. One row per page with issue codes. - **URL**: https://apify.com/datawatcher/seo-site-audit-scraper.md - **Developed by:** [DataWatcher](https://apify.com/datawatcher) (community) - **Categories:** SEO tools, Developer tools - **Security:** Limited permissions - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $1.70 / 1,000 page audits This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## SEO Site Audit Scraper: technical SEO audit, broken links and redirect checker Scrape **website data** for a full **technical SEO audit**: every page checked for on-page SEO issues, **broken links**, redirect chains, the internal link graph with anchor text, and your **sitemap and robots.txt** validated. Get one clean JSON row per finding, with a plain-language fix for each issue. Run it from the Console or the API, schedule it and monitor your site over time. It is a pay-per-event site audit alternative to desktop crawlers and monthly SEO suites: you pay only for the rows you get. ### ⭐ Why this one - **One schema across SEO actors.** Page audits, links, redirects, sitemap URLs and robots.txt come as typed rows (`recordType`) with the same envelope, so one parser reads every row. - **Every issue comes with a fix.** Each finding has a stable `code`, a `severity`, a plain `message`, the `evidence` found on the page and a `fix` you can hand to a developer. - **Pay per event, no subscription.** Audit one site on demand or on a schedule; there is no seat or monthly licence. - **API and JSON first.** Feed dashboards, spreadsheets, Slack alerts or a CI check through the Apify API and integrations. - **Polite by design.** It respects robots.txt, waits between requests and audits public sites only. ### πŸ” What you get | You want to find | Row type | What it holds | |---|---|---| | On-page SEO problems | `pageAudit` | Title, meta description, h1, canonical, indexability, hreflang, Open Graph, security headers, content hash, health score and an `issues` list with fixes | | Broken links checker results | `brokenLink` | Source page, target URL, HTTP status, anchor text, internal or external | | Redirect chains and loops | `redirectChain` | Start URL, final URL, every hop with status and location | | Internal links and anchor text | `internalLink` | Source, target, anchor text, `rel`, followed or not, source depth | | Sitemap validator | `sitemapUrl` | Each URL listed in the sitemap and the sitemap that lists it | | robots.txt validator | `robotsReport` | Status, rules found and the sitemaps it declares | | Site-wide health | `auditSummary` | Pages crawled, status groups, issue counts by code and severity, health score | Checks include missing, short, long and duplicate titles and meta descriptions, missing or multiple h1, missing or wrong canonical, noindex and canonical conflicts, orphan pages, duplicate content, thin content, missing image alt text, mixed content, non-https pages, redirect loops and long chains, links to 4xx and 5xx pages, deep pages, hreflang errors, missing viewport and lang, and missing security headers. **What this actor is not.** It audits the site you give it: its pages, its internal link graph and (if you switch it on) its outbound links. It does not hold a backlink index and returns no data about who links to you from other sites. Page-speed scores such as LCP are not measured here either. **Fields that are not measured.** The actor makes no image requests, so `largeImageCount` is always `null` and the `IMAGE_TOO_LARGE` code is never raised. Missing data is `null`, never a guess. ### πŸš€ How to scrape a site for SEO data 1. Open the actor and enter the website address in **Start URLs**. 2. Choose a **Mode**: crawl a site, check a URL list, or read a sitemap. 3. Set **Max pages** and **Max depth**, then click **Start**. 4. Download the data as JSON, CSV or Excel, or read it through the API. #### πŸ•·οΈ 1. Crawl a site ```json { "mode": "crawl", "startUrls": ["https://example.com/"], "maxPages": 200, "maxDepth": 3, "emitLinkRows": true } ``` Follows internal links, audits every page, reads robots.txt and the sitemap, and writes broken links, redirect chains and a summary. Turn on `emitLinkRows` for the full internal link graph. #### πŸ“‹ 2. Check a URL list ```json { "mode": "urlList", "startUrls": ["https://example.com/old-page", "https://example.com/pricing"] } ``` Checks exactly the URLs you give: status, redirects, final URL and the page checks. No links are followed. Good for migration checks. #### πŸ—ΊοΈ 3. Read a sitemap ```json { "mode": "sitemap", "startUrls": ["https://example.com/"] } ``` Reads the sitemap (index files too) and checks the status of each listed URL. #### Useful options - `includeUrlPatterns` and `excludeUrlPatterns`: glob patterns such as `https://example.com/blog/**` or `**/tag/**`. - `includeSubdomains`: also crawl `blog.example.com` for `example.com`. - `checkExternalLinks`: request outbound links to find broken external links. - `respectRobotsTxt`: on by default; skipped pages still show as a finding. - Free plan: up to 1,000 rows per run (see Pricing). ### πŸ“¦ Output: SEO audit data Every row has the same envelope (`schemaVersion`, `recordType`, `url`, `dates`, `target`, `status`, `coverage`, `warnings`, `errors`) and a `data` object with the fields of its type. Real rows from a crawl, shortened: **pageAudit** ```json { "recordType": "pageAudit", "url": "https://crawler-test.com/content/mult_h1", "status": "success", "data": { "finalUrl": "https://crawler-test.com/content/mult_h1", "httpStatus": 200, "title": "Mult H1s", "titleLength": 8, "h1": ["header number 1", "header number 2"], "h1Count": 2, "indexable": true, "score": 67, "wordCount": 14, "internalLinkCount": 1, "issues": [ { "code": "MULTIPLE_H1", "severity": "low", "message": "The page has more than one h1 heading.", "fix": "Keep one h1 and make the others h2.", "evidence": "2 h1: header number 1; header number 2" } ] } } ``` **brokenLink** ```json { "recordType": "brokenLink", "url": "https://crawler-test.com/links/broken_links_internal", "data": { "sourceUrl": "https://crawler-test.com/links/broken_links_internal", "targetUrl": "https://crawler-test.com/links/not_found/foo1", "httpStatus": 404, "linkScope": "internal", "anchorText": "Broken Internal Link 1" } } ``` **redirectChain** ```json { "recordType": "redirectChain", "url": "https://httpbin.org/redirect/3", "data": { "startUrl": "https://httpbin.org/redirect/3", "finalUrl": "https://httpbin.org/get", "hopCount": 3, "loop": false, "hops": [ {"url": "https://httpbin.org/redirect/3", "httpStatus": 302, "location": "/relative-redirect/2"}, {"url": "https://httpbin.org/relative-redirect/2", "httpStatus": 302, "location": "/relative-redirect/1"}, {"url": "https://httpbin.org/relative-redirect/1", "httpStatus": 302, "location": "/get"} ] } } ``` **internalLink** ```json { "recordType": "internalLink", "url": "https://crawler-test.com/links/broken_links_internal", "data": { "sourceUrl": "https://crawler-test.com/links/broken_links_internal", "targetUrl": "https://crawler-test.com/", "anchorText": "Crawler Test two point oh!", "followed": true, "sourceDepth": 0 } } ``` **sitemapUrl** ```json { "recordType": "sitemapUrl", "url": "https://crawler-test.com/urls/page_with_hreflang_header_not_ok", "data": { "loc": "https://crawler-test.com/urls/page_with_hreflang_header_not_ok", "sitemapUrl": "https://crawler-test.com/test_sitemap.xml" } } ``` **robotsReport** ```json { "recordType": "robotsReport", "url": "https://crawler-test.com/robots.txt", "data": { "robotsUrl": "https://crawler-test.com/robots.txt", "httpStatus": 200, "sitemaps": ["https://crawler-test.com/test_sitemap.xml", "https://crawler-test.com/sitemap.xml.gz"] } } ``` **auditSummary** ```json { "recordType": "auditSummary", "url": "https://crawler-test.com/links/broken_links_internal", "data": { "pagesCrawled": 3, "pagesByStatus": {"2xx": 2, "4xx": 1}, "healthScore": 52, "issueCounts": [ {"code": "INTERNAL_LINK_TO_4XX", "severity": "high", "count": 1}, {"code": "VIEWPORT_MISSING", "severity": "medium", "count": 2} ] } } ``` ### πŸ”Œ Run it your way - **Schedule it** in the Console (daily or weekly) and watch the `auditSummary` health score and issue counts over time. - **Start it from the API** with the same input as in the Console and read the dataset of the run. - **Send the data on** to Google Sheets, Slack, a webhook, Make, Zapier or your own tool through Apify integrations. - **Use it from an AI agent** through the Apify MCP server: ```text Audit https://example.com with the seo-site-audit-scraper actor, crawl up to 200 pages and list the high severity issues with their fixes. ``` ### Pricing Pay per event. There is no start fee. Free plan: each run returns up to 1,000 rows; upgrade your Apify plan for more. | Event | Price | When it is charged | |---|---|---| | `seo-page-audited` | $0.0020 ($2.00 per 1,000) | One page audited with all its checks and issue codes. | | `seo-audit-summarized` | $0.0010 ($1.00 per 1,000) | One summary row per audited site. | | `seo-link-checked` | $0.0005 ($0.50 per 1,000) | One broken link, redirect chain or internal link row. | | `seo-sitemap-url-read` | $0.0002 ($0.20 per 1,000) | One URL read from a sitemap. | | `seo-robots-read` | $0.0010 ($1.00 per 1,000) | One parsed robots.txt file. | ### Data This actor returns only what the site shows publicly at run time. We don't store or keep that data and are not its controller β€” you decide how you use it. It audits public sites, respects robots.txt by default and keeps a polite pause between requests. This actor is not affiliated with or endorsed by any site you audit, or by any SEO tool vendor. ### 🩺 If a run fails The run health (the `RUN_HEALTH` record, `RUN_SUMMARY` and the last line of the log) names the cause, for example `site down: list request answered 500 x3`. Find that start of the message in the table. | Error in the log / run health | What it means | What you do | |---|---|---| | `site down: answered 500, 502-504, 520-526` | The site is offline or has a fault. The actor waits longer between tries, then stops the run. | Try again later. | | `blocked: answered 401, 403, 1020` or a captcha page | The site refused the request or showed a captcha page. The actor does not use a proxy. | Try again later, or ask the site owner to allow the actor's User-Agent. | | `rate limited: answered 429` | The site asks for fewer requests. The actor waits and slows down by itself. | Try again later. | | `gone: answered 404, 410` | The page was removed. The actor records it as a finding (a row with the status) and goes on. | Nothing; this is normal. | | `network error: failed with timeout` (or connection error, DNS error) | A timeout or a broken connection. The actor retries with a pause. | Try again later. | | `wrong answer: returned HTML instead of JSON` (or an empty body) | The site answered with a page that is not data (HTML instead of JSON, an empty body). The site has probably changed. | Report it in the Issues tab; we fix it. | ### ❓ FAQ #### How do I run a technical SEO audit of my website? Enter your site in Start URLs, keep the mode on crawl and start the run. Read the `pageAudit` rows for on-page issues, `brokenLink` and `redirectChain` rows for link problems and the `auditSummary` row for the overall health score. #### How do I find broken links and redirect chains? Run a crawl. Every internal link that answers 4xx or 5xx becomes a `brokenLink` row with its source page and anchor text; every redirect with two or more hops becomes a `redirectChain` row. Switch on `checkExternalLinks` for outbound links too. #### Does it show backlinks? No. It audits one site: its pages, internal links and anchor text, and the outbound links you ask it to check. It has no backlink index. #### How do I export the SEO audit data to Excel or CSV? Open the Output tab of the run and download the data in the format you need. #### How do I scrape a website with the API? Start the actor from the Apify API with the same input as in the Console and read the dataset of the run. ### πŸ’¬ Feedback and custom work Need another check, a field or a custom feed? Open the Issues tab and tell us. # Changelog This Actor's version history is a separate document: https://apify.com/datawatcher/seo-site-audit-scraper/changelog.md # Actor input Schema ## `startUrls` (type: `array`): Websites to audit, one per line. The crawl stays on the host of each URL. Public sites only; private and local addresses are refused. ## `maxPages` (type: `integer`): Maximum pages fetched per start URL. The crawl never runs unbounded. Free plan Apify users get at most 1000 rows per run. ## `maxDepth` (type: `integer`): Clicks from the start URL to follow. 0 reads only the start URL. ## `mode` (type: `string`): crawl follows links from the start URLs and audits every page. urlList checks exactly the URLs you give (status, redirects, final URL and the page checks) and follows no links. sitemap reads the sitemap and checks the status of each listed URL. ## `includeSubdomains` (type: `boolean`): Also crawl subdomains of each start host (blog.example.com for example.com). Off - only the exact host. ## `includeUrlPatterns` (type: `array`): Crawl only URLs that match one of these glob patterns, e.g. https://example.com/blog/\*\*. Empty - all URLs of the site. Start URLs are always read. ## `excludeUrlPatterns` (type: `array`): Skip URLs that match one of these glob patterns, e.g. \*\*/\*.pdf or **/tag/**. Wins over the include list. ## `userAgent` (type: `string`): The User-Agent header of every request, and the name robots.txt rules are matched against. The default says who we are. ## `requestDelayMs` (type: `integer`): Minimum pause between two requests to the same host. Never below 500 ms, so the audit stays polite. ## `emitLinkRows` (type: `boolean`): Also write one row per internal link (the full link graph, billed per row). Off - broken links and redirect chains still come as rows, and every page row carries a free broken-link summary. ## `readSitemap` (type: `boolean`): Read robots.txt and the sitemap (index files too) and add the listed URLs to the crawl. ## `checkExternalLinks` (type: `boolean`): Also request links to other sites (one request each) to find broken outbound links. ## `respectRobotsTxt` (type: `boolean`): Skip pages that robots.txt disallows. Skipped pages still show as a finding. ## `maxRequestRetries` (type: `integer`): Retries of a request that failed on a network error or a server error. A block (403, 429) is never retried. ## `requestTimeoutSecs` (type: `integer`): Time limit of one page request. ## Actor input object example ```json { "startUrls": [ "https://crawler-test.com/" ], "maxPages": 50, "maxDepth": 3, "mode": "crawl", "includeSubdomains": false, "userAgent": "SeoSiteAuditBot/1.0 (+https://apify.com/datawatcher/seo-site-audit-scraper)", "requestDelayMs": 500, "emitLinkRows": false, "readSitemap": true, "checkExternalLinks": false, "respectRobotsTxt": true, "maxRequestRetries": 2, "requestTimeoutSecs": 30 } ``` # Actor output Schema ## `results` (type: `string`): One row per finding: pages, summary, links, redirect chains, sitemap URLs, robots.txt. ## `runSummary` (type: `string`): OUTPUT: status, sites audited, pages and requests used, early stop and its reason, errors, charged events. ## `runHealth` (type: `string`): {status, reason}: ok, degraded, or failed. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ "https://crawler-test.com/" ] }; // Run the Actor and wait for it to finish const run = await client.actor("datawatcher/seo-site-audit-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": ["https://crawler-test.com/"] } # Run the Actor and wait for it to finish run = client.actor("datawatcher/seo-site-audit-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ "https://crawler-test.com/" ] }' | apify call datawatcher/seo-site-audit-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,datawatcher/seo-site-audit-scraper" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/N48m43WC2kfm7aLNv/builds/XBrrqGcO3dXBnk1aF/openapi.json