SEO Page Audit — Sitemap Crawl & On-Page Checker

Point it at a domain and get every page audited. This Actor reads a site's robots.txt, walks its sitemap — nested indexes, gzip children and relative URLs included — or crawls the site when there is no sitemap, and then audits each page: title, meta description, headings, canonical, hreflang, structured data, images, links, response time, page weight. Every page comes back with a list of issue codes and a 0–100 score built from published weights you can check against the table below.

No API key. No browser. No JavaScript execution. $1.00 per 1,000 pages, proxy included. robots.txt, sitemaps and link checks are free, pages that could not be read are free, and a run that reads nothing costs nothing at all.

Try it in 30 seconds

Input — this is the Store prefill; paste it and run:

{
  "websites": ["basecamp.com"],
  "maxPages": 25
}

A bare domain is enough. The run reads https://basecamp.com/robots.txt, follows the sitemap it finds, and audits the first 25 URLs listed there — typically well under a minute. Want specific pages instead? Put them in pageUrls and nothing is discovered or crawled.

Output — the first row is the website's site summary: its 0–100 score and the issue codes costing it the most points. Then one row per audited page; the fields you will use most are url, statusCode, title, metaDescription, issues and score (full list under Output reference), and, with checkLinks on, a broken-link row per dead internal link. Anything that could not be read comes back as a free diagnostic row (ok: false, errorType, error) instead of a charge.

Price — $1.00 per 1,000 pages on the FREE tier (+$0.002 per site summary, +$0.0002 per broken link found, +$0.001 per run); robots.txt, every sitemap file, every link check, a page that could not be read and a run that reads nothing are all free. Pay-per-event, no API key, no browser, limited permissions — works over the Apify MCP server (mcp.apify.com) and with agentic (x402) payments.

From code — client.actor("insight.solutions/seo-page-audit").call(run_input={"websites": ["basecamp.com"], "maxPages": 25}) with apify-client, or POST https://api.apify.com/v2/acts/insight.solutions~seo-page-audit/run-sync-get-dataset-items.

What comes back

Every website starts with its site row — the summary, written first so it is the first thing you see (shape shown; the numbers are illustrative):

{
  "ok": true,
  "rowType": "site",
  "website": "basecamp.com",
  "url": "https://basecamp.com",
  "score": 91,                             // mean of the page scores
  "issues": ["missing-description", "thin-content", "title-too-short"],
                                           // the codes costing the site the most points: pages × weight
  "pagesAudited": 25,
  "pagesFailed": 0,
  "discovery": "sitemap",
  "robotsFound": true,
  "robotsBlocked": 0,
  "sitemapUrls": ["https://basecamp.com/sitemap.xml"],
  "sitemapUrlCount": 174,                  // URLs read from the sitemap, not just the 25 audited
  "statusCounts": { "200": 24, "404": 1 },
  "avgResponseMs": 268,
  "issueCounts": { "missing-description": 6, "thin-content": 2, "title-too-short": 1 },
  "worstPages": [
    { "url": "https://basecamp.com/…", "score": 42, "issues": ["http-error", "missing-h1", "…"] }
  ],
  "linksChecked": 0,
  "linksUnchecked": 0,                     // links left in the queue when the time budget ran out
  "brokenLinkCount": 0,
  "proxyUsed": "datacenter"                // or "residential" — see "Sites that answer nothing"
}

Then one page row per audited URL. This one is real — read from basecamp.com/pricing on 2026-09-11:

{
  "ok": true,
  "rowType": "page",                       // "page" | "site" | "broken-link" | "diagnostic"

  "input": "basecamp.com",
  "website": "basecamp.com",
  "url": "https://basecamp.com/pricing",
  "source": "sitemap",                     // "sitemap" | "crawl" | "input"
  "sitemapLastmod": "2026-05-26T12:00:00-07:00",

  // the response
  "statusCode": 200,
  "finalUrl": "https://basecamp.com/pricing",
  "redirectChain": [],
  "responseTimeMs": 243,
  "contentType": "text/html",
  "contentLength": 65209,                  // bytes of HTML
  "compressed": true,
  "lastModified": "Wed, 02 Sep 2026 23:45:00 GMT",
  "hsts": false,
  "xRobotsTag": null,

  // the markup
  "title": "Basecamp — Pricing",           // — decoded to what a reader sees
  "titleLength": 18,
  "metaDescription": "Start free with Basecamp — one project, five users, forever free…",
  "metaDescriptionLength": 128,
  "metaRobots": "index, follow",
  "canonical": "https://basecamp.com/pricing",
  "canonicalMatchesUrl": true,
  "lang": "en",
  "h1": ["No per-user fees, everyone's included. Simple fixed prices — another Basecamp exclusive."],
  "h1Count": 1, "h2Count": 9, "h3Count": 3,
  "wordCount": 2060,
  "textToHtmlRatio": 18.88,                // visible chars per 100 chars of markup
  "ogTitle": "Pricing",
  "ogDescription": "Start free with Basecamp — one project, five users, forever free…",
  "ogImage": "https://basecamp.com/assets/images/opengraph/basecamp.png?v=1.5.1",
  "twitterCard": "summary_large_image",
  "hreflang": [],
  "structuredDataTypes": ["WebPage", "WebSite", "ItemList", "SiteNavigationElement"],
  "hasViewport": true,
  "charset": "utf-8",
  "favicon": "https://basecamp.com/assets/images/general/favicon.svg?v=1.5.1",
  "internalLinkCount": 35,
  "externalLinkCount": 12,
  "nofollowCount": 0,
  "imageCount": 1,
  "imagesMissingAlt": 0,
  "imagesMissingAltExamples": [],
  "scriptCount": 10, "stylesheetCount": 48, "inlineStyleCount": 0,
  "noindex": false, "nofollowPage": false,
  "mixedContentCount": 0,
  "mixedContentExamples": [],

  // the verdict
  "issues": ["title-too-short"],
  "issueCount": 1,
  "score": 95,                             // 100 − 5 for title-too-short

  "proxyUsed": "datacenter",
  "error": null,
  "errorType": null,
  "scrapedAt": "2026-09-11T15:44:37.120Z",
  "sourceUrl": "https://basecamp.com/pricing"
}

And, with checkLinks on, one broken-link row per dead internal link, naming the pages that point at it. The run's status message says the same thing in one line for a single-website run — Audited 25 pages across 1 website — basecamp.com scores 91/100; fix first: missing-description (6 pages), thin-content (2 pages).

The issue codes, and what each costs

Every page starts at 100. Each issue found subtracts its weight. The total is floored at 0.

score = max(0, 100 − Σ weight(issue))

That is the whole formula. No curve, no weighting by page importance, nothing calibrated against anything. It is a checklist with a running total, published here so that fixing a missing <h1> and predicting exactly ten points is possible — and so that if you disagree with a weight you can recompute the column yourself from the issues array, which is the part actually worth acting on.

Code	−pts	Fires when
http-error	40	The page answered 4xx or 5xx — a 404 in the sitemap, a 500 on a live URL. It still gets a full row: a broken page you can see is the point. (A 401, 403 or 429 is a refusal, not a page; those are free diagnostic rows instead)
noindex	25	The robots meta tag or the X-Robots-Tag header says noindex or none
missing-title	20	No <title>, or an empty one
missing-description	10	No <meta name="description">
missing-h1	10	No <h1> on the page
duplicate-title	10	Another audited page on the same site publishes this exact title
thin-content	10	Fewer than 200 words of visible body text
mixed-content	8	An https page loads at least one asset over plain http
missing-viewport	5	No viewport meta tag, so the page will not adapt to a phone
multiple-h1	5	More than one <h1>
missing-canonical	5	No rel="canonical" link
canonical-mismatch	5	The canonical points somewhere other than the URL that served the page
images-missing-alt	5	At least one <img> has no alt attribute at all
slow-response	5	The page took longer than 2,000 ms to answer
duplicate-description	5	Another audited page on the same site publishes this exact description
title-too-long	5	Title longer than 60 characters
title-too-short	5	Title shorter than 20 characters
missing-lang	3	No lang attribute on <html>
description-too-long	3	Description longer than 160 characters
no-structured-data	3	No JSON-LD on the page
not-compressed	3	The response was sent uncompressed — and only when the headers prove it
redirected	3	The URL in the sitemap (or the one you sent) redirected somewhere else
large-html	3	The HTML document alone is over 1 MB, before any assets load

Four of those are more careful than they look, and it matters:

• alt="" is not a missing alt. It is the correct, deliberate markup for a decorative image. Counting it would file a finding against every site that gets accessibility right. Only a missing attribute counts.
• mixed-content counts assets, not links. An <a href="http://…"> is the reader's choice and no browser warns about it. The captured apify.com pricing page contains exactly one http:// URL — the LinkedIn link in the footer — and correctly reports zero mixed content.
• canonical-mismatch normalises before comparing. A canonical differing only by a trailing slash or a fragment is not a mismatch.
• not-compressed fires only on proof. The HTTP client inflates compressed responses transparently and deletes content-encoding and content-length when it does, so "no encoding header" is not evidence of anything. This fires only when a declared content-length matches the body that arrived — meaning nothing was inflated. When the headers do not settle it, compressed is null and no issue is raised. A false "you are not compressing" on every page of every site would be a worse bug than missing the occasional real one.

Use cases

• Pre-launch and post-migration QA — run the sitemap after a deploy and diff the issueCounts histogram against last week's. A migration that quietly shipped noindex on 400 pages shows up as a single number.
• Agency audits and pitches — walk in with a scored page-by-page table of a prospect's site, sorted by score, with the ten worst named on the summary row.
• Content-team backlog — export the issues view, filter to missing-description and thin-content, and you have a prioritised list of pages to write.
• Broken-link sweeps — checkLinks: true on a site you are responsible for, then read the links view. Each row names the broken URL and up to ten pages that link to it, which is normally enough to identify the template at fault.
• Duplicate-content detection — duplicate-title and duplicate-description are computed across the whole site, so faceted listings and paginated archives that all share one title are found in one run.
• Competitive teardown — audit a competitor's sitemap and compare wordCount, structuredDataTypes and textToHtmlRatio distributions against your own.
• Monitoring — schedule it weekly against maxPages: 100 and watch the site score as a single trend line.

How it works, and why it keeps working

A website tells you which pages it has in two places, and this Actor reads both before it audits anything.

1. robots.txt. The first request of every website is GET /robots.txt. It does three jobs: it proves which of the four candidate addresses (https://, https://www., http://, http://www.) the host actually answers on (a site that answers on none of them is covered under Sites that answer nothing, below); it supplies the Sitemap: lines; and it supplies the Disallow rules, which are obeyed. Rule matching is the same longest-match-wins algorithm search engines use, with * and $ patterns, matched against path and query — so Disallow: /search? blocks /search?q=shoes and leaves /searching alone, and Allow: /private/public-notes correctly overrides Disallow: /private/. Only the User-agent: * group is read; obeying Googlebot's rules would be pretending to be Googlebot.

2. The sitemap. Whatever robots.txt named, or failing that /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml and /sitemap/sitemap.xml in that order. A <sitemapindex> is followed to its children, three levels deep, up to 50 sitemap files per site. Children ending .xml.gz are inflated. <loc> values are resolved against the sitemap's own URL, because the spec says they are absolute and real sitemaps disagree — basecamp.com's, captured live, lists /, /2 and /2/integrations. <lastmod>, <changefreq> and <priority> ride along onto the page row. A guessed path that is not there is not a fault — a site with no sitemap is the case the crawl fallback exists for, and it produces no diagnostic row; a sitemap that robots.txt or a sitemap index declared and that then fails does.

3. No sitemap? Crawl. A site that answers nothing useful at any of those locations is crawled breadth-first from its home page: same host with www. folded away, HTML pages only, shallowest links first, fragments and mailto: and asset paths skipped, bounded by maxPages and by robots.txt.

Then each page is fetched with a browser accept header, redirects followed, a 20-second per-request timeout, up to maxConcurrency at once — and 150–400 ms apart per host, always. A 500-page audit arriving as 500 simultaneous requests is a denial-of-service attempt with a friendly user agent; raising maxConcurrency overlaps the waiting, not the hammering.

Three decisions are worth spelling out, because they are what separates a useful audit from a noisy one:

A 404 is a row, not an error. A URL in a sitemap that answers 404 is one of the most useful things an audit can find. It comes back as a full page row — status, title, response time, the http-error issue, a score — and it is charged, because it is a finding. Two cases are deliberately not rows: a URL that answered nothing at all, and a URL that was refused (401, 403, 429, or a Cloudflare origin error). A refusal is very often about this exit IP rather than about the URL, and billing a row that scores 0 because a bot wall answered would be charging for the wall. Both are free diagnostic rows.

Duplicate checks run at the end of a site. duplicate-title and duplicate-description cannot be decided while pages are still arriving, so every page of a website is buffered until the site is finished, the title and description index is built, and then all the page rows are written at once followed by the site summary. The buffer holds extracted fields, never HTML.

The site row comes first. It is computed last — it needs every page — but it is written first, so the first row of a website in the dataset, the Overview table and an export is its score and the issues worth fixing first. It is still charged last, after the pages it summarises.

The word count is a word count. Scripts, styles, SVG, <template> and the whole <head> are removed before counting, so a client-rendered page shipping 400 KB of JSON state in a script tag does not read as 60,000 words of content. The captured apify.com pricing page is 437 KB of markup carrying 2,109 real words, and textToHtmlRatio: 2.81 says so.

Sites that answer nothing, and the residential retry

Some sites drop every connection from cloud and datacenter addresses; some are simply down. Either way the Actor used to find out the slow way — a 15–20-second timeout per request, one after another: a real run on 2026-09-21 spent 2 min 19 s on robots.txt, the sitemap and the home page of one such site and ended with nothing. Now two rules bound it:

• One waiting budget per website: 45 s. robots.txt on every rung of the ladder, the first sitemap requests, the home page and — until one page of the site has answered — the first page requests share 45 s of time spent on requests that got no answer. Each request's timeout is cut to what is left, and a timed-out request is not retried in place. Requests that answered — quickly, slowly, or with a 404 — cost the budget nothing, so a slow site that is answering is never cut short by it.
• Two strikes per host. After two consecutive connection failures on one host (a timeout, a reset, a refused connection), no further URL on that host is requested. A name that does not resolve is not a strike — it is an instant, final answer.

Then, when the run uses Apify's datacenter proxy (the default) and the site gave no usable answer — every request timed out, was reset, or met a bot wall (HTTP 403, 429 or 503, or an anti-bot interstitial, after at least one retry from a fresh datacenter IP) — the home page gets one 15-second request through a residential exit. If it answers, the rest of that website (robots.txt, sitemap, pages, link checks) is read through residential, and every row of the site says proxyUsed: "residential". If it does not, the website gets one free unreachable (or blocked) row saying exactly what was tried, how each attempt failed, and how long it all took — instead of a row per URL nobody answered. The same check runs over the first pages of a sitemap: until one page of the site answers, two pages that hang in a row stop the audit and the residential request gets its turn, and the pages that failed are re-read through residential first.

Worst cases, end to end:

The site…	Datacenter	Residential retry	Total	Before
answers nothing at all	robots.txt on https://, https://www., http:// — 15 s each, 45 s; the fourth rung is not tried	15 s	≈ 60 s, one free row	2 min 19 s
answers robots.txt, then hangs	two sitemap requests, 15 s each, then the host is not asked again — home page included	15 s	≈ 45–50 s	up to ≈ 2¾ min
answers robots.txt and its sitemap, but no page	the first pages, 20 s each, until two in a row have hung	15 s	≈ 55 s	until maxRunSecs
does not resolve	four instant DNS failures	none — no exit can make a name exist	≈ 1 s	≈ 1 s

Without a residential retry (your own proxy URLs, no proxy, or already on RESIDENTIAL) take 15 s off; the row then says why no retry was made. Either way the run finishes SUCCEEDED, and if no website produced a page it has zero results, a status message naming the diagnostic rows, and a bill of $0.00, start fee included.

What the residential retry costs. Apify prices residential proxy traffic by the gigabyte (about $8 per GB; the datacenter proxy is not metered that way). This Actor is priced per event with the proxy included, so that cost is ours, not yours: a 100-page audit that needed the residential retry costs you the same $0.103 (FREE tier: 100 pages, one site summary, one run start) as any other 100-page audit. For scale, its residential traffic is roughly 3–6 MB — about $0.02–$0.05 — absorbed by the page price. Link checks on a rescued site go through residential too, at the same unchanged prices.

How it compares

• The sitemap is the input, not a crawl budget. Most page-level auditors make you either paste URLs one at a time or buy a crawler subscription. This takes a domain and does the discovery — including the gzip and nested-index cases that break naive readers.
• The score is arithmetic you can audit. Every weight is in the table above and in src/audit.js as one object. Nothing is hidden behind a model, and the issues array is always there to recompute from.
• Priced by the page, with the plumbing free. robots.txt, every sitemap file, every gzip child and every link check cost nothing. You pay for audited pages and one summary row.
• Failures are free and legible. A dead host, a URL that answers with a PDF, a bot wall, or a site that ran out the clock produces a diagnostic row with an errorType you can branch on — and no charge. A run that audits nothing at all finishes SUCCEEDED with zero results, a status message that says so, and a free diagnostic row per URL telling you why; it finishes FAILED only when no entry was usable, or the Actor itself hit an error.
• No browser. Auditing is pure HTTP and targeted regular expressions, which is why a 5,000-page site costs single-digit dollars rather than hundreds — and why a page whose content only exists after client-side rendering is a documented limitation rather than a hidden one.

Input reference

Field	Type	Default	What it does
websites	array of strings	["basecamp.com"]	Bare domains or full URLs. Each is tried https:// → https://www. → http://. Per site: robots.txt, then sitemap discovery, then up to maxPages page audits. Duplicates are read, and billed, once
pageUrls	array of strings	[]	Audit exactly these URLs — no sitemap, no crawl. URLs on the same host as a websites entry are folded into that site's audit rather than becoming a second one
maxPages	integer 1–5000	100 (the form starts at 25)	Pages audited per website. This is the number that sets the price of a run
checkLinks	boolean	false	Also fetch every unique internal link found (a plain GET), and write a broken-link row for each one that answers 4xx, 5xx or nothing. $0.0002 per broken link; checking is free. Links the time budget cuts off are counted in linksUnchecked, never as broken
maxLinksToCheck	integer 1–5000	500	Ceiling on the link check, per website
respectRobots	boolean	true	Obey the site's own robots.txt. Skipped URLs are counted as robotsBlocked and never fetched or billed. Turn off only for a site you own
maxSitemapUrls	integer 1–5000	5000	Ceiling on URLs collected from sitemaps. Collection also stops as soon as there are enough to fill maxPages
maxConcurrency	integer 1–16	8	Pages of one website in flight at once. Each worker keeps its own proxy session. Per-host spacing of 150–400 ms applies regardless
maxRunSecs	integer 30–3600	240	Whole-run wall-clock budget. When it runs out the Actor keeps and bills what it has, and files a free diagnostic row saying how many URLs it never reached
proxyConfiguration	object	Apify datacenter	Datacenter is the default because it is cheaper and ordinary websites accept it. On it, a site that answers nothing over datacenter gets one residential request to its home page, and is read through residential if that answers (Sites that answer nothing). Switch the group to RESIDENTIAL to use residential for everything

At least one of websites and pageUrls must be non-empty. A run with neither finishes FAILED, having charged nothing.

Output reference

Every row carries the same keys, whatever its type. ok: true is a page, site or broken-link row; ok: false is a free diagnostic row.

Common to every row

Field	What it is
ok, rowType	Whether this row is a result, and which of the four kinds it is
input, website	The entry you supplied, and the registrable domain it resolved to
url, sourceUrl	The URL this row is about
source	Where the URL came from: sitemap, crawl or input on a page row; site on a summary; link-check on a broken link
proxyUsed	Which exit read this website: datacenter, residential (your setting, or the residential retry), custom (your own proxy URLs) or none. The price is the same whichever it is
error, errorType, scrapedAt	Why a row failed, if it did, and when the row was written

page rows

Field	What it is
statusCode, finalUrl, redirectChain, responseTimeMs	What the server answered, where it ended up, and how long it took
contentType, contentLength, compressed, lastModified, hsts, xRobotsTag	Response headers worth an audit column. contentLength is bytes of HTML received
title, titleLength, metaDescription, metaDescriptionLength, metaRobots	The tags search results are built from, entity-decoded
canonical, canonicalMatchesUrl, lang, charset, hasViewport, favicon	Indexing and rendering signals
h1, h1Count, h2Count, h3Count	Heading text and structure
wordCount, textToHtmlRatio	Visible body text, and how much of the document it is
ogTitle, ogDescription, ogImage, twitterCard	Social cards
hreflang, structuredDataTypes	Alternates as {lang, href}, and every JSON-LD @type including nested ones
internalLinkCount, externalLinkCount, nofollowCount	Unique links, split by host with www. folded away
imageCount, imagesMissingAlt, imagesMissingAltExamples	Images, and up to twenty src values with no alt attribute
scriptCount, stylesheetCount, inlineStyleCount	Front-end weight
noindex, nofollowPage, mixedContentCount, mixedContentExamples	Indexing directives and http assets on an https page
issues, issueCount, score	The verdict: every issue code found, heaviest first, how many, and the 0–100 score
sitemapLastmod, sitemapChangefreq, sitemapPriority	What the sitemap said about this URL, when it came from one

site rows — the first row of every website

Field	What it is
pagesAudited, pagesFailed	How many URLs produced a row, and how many could not be read
discovery	sitemap, crawl or input
robotsFound, robotsBlocked	Whether robots.txt was readable, and how many URLs it kept out
sitemapUrls, sitemapUrlCount	Which sitemap files were read, and how many URLs came out of them
statusCounts, avgResponseMs	Status histogram and mean response time
issueCounts	How many pages triggered each issue code, heaviest first
score	The rounded mean of the page scores
issues	The (at most five) issue codes costing the site the most points — pages affected × weight — worst first. issueCount stays empty here
worstPages	The ten lowest-scoring pages, each with its score and issues
linksChecked, linksUnchecked, brokenLinkCount	Link-check totals, zero unless checkLinks is on. linksUnchecked is what the time budget cut off

broken-link rows carry url, statusCode, error and foundOn — up to ten audited pages that link to the dead URL.

errorType on a diagnostic row is one of:

Value	Meaning	Charged?
unreachable	Nothing answered — a dead host, a connection that failed, a domain that does not resolve, the proxy's own "host not found" page. When a whole website answered nothing, this is one row for the site that lists every attempt, the residential retry and the time spent	No
blocked	The request was refused — 401, 403, 429, a Cloudflare origin error, or an anti-bot interstitial served in place of the page. When every page and the home page were refused over datacenter and residential, this is one row for the site	No
not-html	The URL answered with a file rather than a page — a PDF, an image, a JSON feed	No
timeout	The run's maxRunSecs budget ran out before this URL was read	No
memory	The run's memory ran low, so the audit stopped fetching and delivered the pages it had read; the row says how many URLs were left	No
invalid-input	The entry was not a fetchable website or URL — an email address, an IP, a bare word	No

Pricing

$1.00 per 1,000 pages. Pay-per-event, with the proxy already inside that number — there is no separate proxy line on your bill for this Actor.

Event	What triggers it	FREE	Starter	Scale	Business
Page audited (primary)	One audited page written to your dataset	$0.001	$0.001	$0.0008	$0.0006
Site summary	One website that produced at least one page	$0.002	$0.002	$0.002	$0.002
Broken link found	One internal link that failed, only when checkLinks is on	$0.0002	$0.0002	$0.0002	$0.0002
Run started	Once per run, after the first page is audited	$0.001	$0.001	$0.001	$0.001

Worked example. A weekly audit of a 500-page site with link checking on, finding 30 broken links:

• 500 pages × $0.001 = $0.500
• 1 site summary × $0.002 = $0.002
• 30 broken links × $0.0002 = $0.006
• 1 run start = $0.001
• Total: $0.509

What you are never charged for: robots.txt, any sitemap file however many there are, any link check (only a broken link produces a billed row), a URL that answered nothing, a URL that answered with a file, a URL that was refused with 401/403/429 or behind a bot wall, a URL robots.txt disallowed, an entry that was not a website, anything the run never reached before maxRunSecs, or the residential traffic of a site that needed the residential retry. If a whole run comes back empty it finishes SUCCEEDED with zero results and a status message naming the free diagnostic rows, and bills nothing at all, start fee included.

Set ACTOR_MAX_TOTAL_CHARGE_USD on a run and the Actor stops fetching once the ceiling is in sight, rather than handing you rows it cannot bill or billing you for rows it cannot hand over. It finishes SUCCEEDED with the ceiling named in its status message, and everything already delivered is yours.

Limits, and the ones that might bite

No browser, so the audit is of the HTML the server returned. A page whose body copy is rendered client-side will read as thin-content with a low textToHtmlRatio, and that is an honest description of what a crawler receives — but it is not what a human sees, and treating it as a content problem would be wrong. The wordCount and textToHtmlRatio columns together are the tell: a low ratio on a big document means a framework, not an empty page.

sitemapUrlCount is what was read, not everything the site publishes. Sitemap collection stops as soon as there are enough URLs to fill maxPages, at 50 sitemap files, at three levels of index nesting, or at maxSitemapUrls. Reading a news site's entire sitemap tree to audit 100 pages would be rude as well as slow. Raise maxPages and maxSitemapUrls together when you want a complete count.

redirectChain has two ends, not every hop. The HTTP client follows redirects itself and hands back only the URL it finished on, so a page that went through three hops reports the URL requested and the URL delivered. The redirected issue and finalUrl are correct; the intermediate hops are not visible.

Duplicate checks are within one run and one site. duplicate-title compares the pages this run audited on this website. Two pages sharing a title where only one was audited will not be flagged, and a title shared across two different domains in the same run is not a duplicate.

compressed can be null. See the not-compressed note above. When the response carried neither content-encoding nor content-length and the body was empty, nothing can be concluded, and the column says so rather than guessing.

Memory: the default 1 GB is enough for the largest audit this Actor allows. Page rows are buffered per website until the site finishes (the duplicate checks need every page first), and the buffer holds extracted fields, never markup — roughly 20–50 KB per page, so 5,000 pages fit comfortably in the default run memory. If the process ever does run low, the audit stops fetching rather than dying: every page already read is still scored, written and summarised, and a free memory diagnostic says how many URLs were left and that raising the run's memory or lowering maxPages will finish the job. A run that is killed by the platform delivers nothing; this one never gets there.

A crawl is not a full-site crawl. The fallback follows internal HTML links breadth-first from the home page up to maxPages. It does not submit forms, does not paginate listings by guessing URL patterns, and does not leave the host. A site with no sitemap and a JavaScript-only navigation will yield one page: its home page.

Scores are a checklist total, not a ranking prediction. Nothing here measures backlinks, domain authority, search intent, competitor positioning or whether the content is any good. A page can score 100 and rank nowhere. The score is for triage — sort by it, then read the issues.

Sites behind aggressive bot management may refuse the default proxy. The datacenter default is right for the overwhelming majority of the web and wrong for the small slice that fingerprints every request. When a whole site refuses or ignores datacenter, the residential retry above usually gets through on its own. It is one request per website and it is only made when nothing answered: a site whose home page answers over datacenter but that refuses some of its other pages keeps those pages as free blocked diagnostic rows — switch proxyConfiguration to RESIDENTIAL and re-run if you need them. A site that refuses the residential exit too (or for which none could be created) gets one free blocked row and no further requests.

A site that answers nothing still costs about a minute. That is the waiting budget plus the residential retry (see the timeline above). A list of fifty dead domains is therefore up to fifty minutes, not fifty seconds — raise maxRunSecs accordingly; a website the clock does not reach gets a free timeout row.

The upstream format may change. Sites are rebuilt, sitemaps move, and a site that published a clean sitemap.xml last month can start serving an index of gzip children this month. When that happens the discovery path changes rather than breaking — the discovery and sitemapUrls columns on the site row are how you see which path was taken.

Use it from an AI agent, or from code

One JSON object in, one flat array out — the shape agent runtimes want. The Actor runs with limited permissions, uses pay-per-event pricing and never enters Standby, so it works over the Apify MCP server and with x402 agentic payments. The Integrations tab pushes results to Slack, a webhook, Zapier, Make, Google Sheets, Snowflake or BigQuery.

curl -X POST "https://api.apify.com/v2/acts/insight.solutions~seo-page-audit/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"websites":["basecamp.com"],"maxPages":25}'

# pip install apify-client
from apify_client import ApifyClient

client = ApifyClient("<APIFY_TOKEN>")
run = client.actor("insight.solutions/seo-page-audit").call(run_input={
    "websites": ["basecamp.com"],
    "maxPages": 200,
    "checkLinks": True,
})

for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    if row["rowType"] == "page" and row["score"] < 80:
        print(row["score"], row["url"], "→", ", ".join(row["issues"]))
    elif row["rowType"] == "site":
        print("site score:", row["score"], "over", row["pagesAudited"], "pages")
    elif row["rowType"] == "broken-link":
        print("broken:", row["url"], "linked from", row["foundOn"][0])

The four dataset views — Overview, Pages with issues, Broken links and Site summaries — are set up in the Console so each row type exports as its own clean table.

FAQ

How does it find the pages? robots.txt first, for the Sitemap: lines and the crawl rules. Then those sitemaps, or /sitemap.xml and three other usual locations. Sitemap indexes are followed to their children, gzip children are inflated, and relative <loc> values are resolved. Only if none of that yields anything does it crawl the site from its home page.

Does it respect robots.txt? Yes, by default, using the User-agent: * group and the longest-match-wins rule search engines use. Disallowed URLs are never fetched, counted as robotsBlocked on the site row, and never billed. respectRobots: false exists for sites you own — for example a staging site that blocks everything.

How fast is it? Requests to one host are spaced 150–400 ms apart, so about three pages a second per site plus the time each page takes to answer, overlapped across maxConcurrency workers. The 25 pages the form starts with take seconds; 50 pages of a typical site is well under a minute. A site that answers nothing costs about a minute before its free diagnostic row (Sites that answer nothing). maxRunSecs is the hard stop.

Why did a page I know exists not get audited? Four possibilities, in order of likelihood: it is not in the sitemap; maxPages was reached before the run got to it; robots.txt disallows it; or it answered with something that is not a web page. The site row's sitemapUrlCount, pagesAudited and robotsBlocked distinguish the first three, and there is a free diagnostic row for the fourth.

What counts as a duplicate title? An exact string match, after entity decoding and whitespace collapsing, between two pages audited on the same website in the same run. Near-duplicates are not detected — "Shoes | Acme" and "Shoes - Acme" are two different titles.

Can I audit just a handful of specific pages? Yes. Put them in pageUrls and leave websites empty. No sitemap is fetched and no crawl happens; only robots.txt is read, because the Disallow rules still apply.

Does checkLinks check external links too? No. Only internal links, deduplicated across the whole site, capped at maxLinksToCheck. Checking outbound links would multiply the request count against sites that did not ask to be crawled.

Can I audit 5,000 pages? Yes, that is the per-website ceiling. Raise maxRunSecs to match — 5,000 pages at three a second is roughly half an hour — and bound the spend with ACTOR_MAX_TOTAL_CHARGE_USD. At $0.001 per page the whole site is $5.

What happens if one website in a list fails? The others still run. The failed one produces a free diagnostic row — one row for the site, saying what was tried and for how long — and the run finishes SUCCEEDED. If every website fails — a site that does not exist, one that refused us, one whose robots.txt disallows every page — the run still finishes SUCCEEDED, with zero results, a status message that says so and a free diagnostic row per failure, and you are billed nothing at all, start fee included. It finishes FAILED only when nothing you sent is a website or page URL, or the Actor itself hit an error.

Is the data fresh? Live. Every run reads the site at that moment; nothing is cached, and there is no index behind this.

Legal and data-protection notes

• Public pages only. Every URL fetched is one a browser would load without signing in. The Actor never logs in, never accepts cookies or session tokens, never takes an API key belonging to anyone else, and never touches anything behind authentication.
• It obeys the site's own instructions by default. robots.txt is read before anything else and its Disallow rules are respected unless you deliberately turn that off for a site you control.
• It is a polite crawler. One request at a time per host with 150–400 ms between them, a 20-second per-request timeout, and a whole-run budget. It does not retry a refusal in a loop.
• This is not personal data. An on-page SEO record describes a web page's markup, not a person. Nothing in the output identifies an individual, and no contact details, names or user records are collected.
• Scores are observations, not advice. "This page has no meta description" is what was seen. What to do about it depends on the page, and the issue list is there precisely so the number never has to be taken on faith.
• Not affiliated with any website audited or named in these docs. All trademarks belong to their respective owners.

Our other Actors

Every Insight Solutions Actor is pay-per-result with no browser, no login and no API key, and every one of them returns free diagnostic rows instead of billing for failures. Prices are per 1,000 results.

Video, audio & social

• YouTube Transcript API — captions as timed segments, text, SRT or VTT, with language fallback and translation.
• YouTube Comments API — comments and replies with likes, pinned and hearted flags, newest or top sort.
• YouTube Channel API — a channel's videos, Shorts and live streams, plus YouTube search.
• Podcast Search, Episodes & Charts API — Apple Podcasts search, charts and full episode feeds.
• Bluesky Scraper — profiles, posts, followers and follows from the public AT Protocol API.
• Telegram Channel Scraper — posts, views and channel stats from public Telegram channels.
• Substack Scraper — posts with full free text, comments and publication profiles.
• Hacker News API — stories, comments, users, front page and a structured "Who is hiring?" parser from the official HN APIs.
• GitHub Repos, Trending & Releases API — repository facts, trending lists, releases and search, no token needed for small runs.
• Discourse Forum API — topics, posts and categories from any Discourse community via its own JSON endpoints, usernames only.

News, documents & the web

• Google News Search, Topics & Real Article URLs — news search and topic feeds with the publisher's real URL decoded.
• Website to Markdown — Content Extractor for LLMs & RAG — any site as clean Markdown, text and heading-aware chunks.
• WordPress Posts & Pages API — posts and pages from any WordPress site as clean text or Markdown, with categories, tags and images.
• Website Metadata API — any URL's meta tags, Open Graph, Twitter cards, hreflang, feeds, JSON-LD and microdata, with normalised Product, Article, Event and FAQ entities.
• Subdomain Finder API — every subdomain of a domain from certificate-transparency logs, resolved over DNS, with an optional live HTTP check.
• Internet Archive API — archive.org search, item metadata, files and reviews.
• Wayback Machine Toolkit — archived URL inventories, snapshots and text diffs between dates.
• Website Technology Detector — the tech stack behind any site, with the evidence for each detection.
• Domain Intelligence API — DNS, RDAP registration, TLS certificate and HTTP facts in one row per domain.
• Keyword Suggestions API — Google, YouTube, Bing, Amazon and eBay autocomplete with alphabet and question expansions.
• Website Contact Extractor — emails, phone numbers and social profiles from any list of websites.
• Web Search Results API — Bing and DuckDuckGo organic results with snippets, no key, no browser.
• Company Enrichment API — a domain in, a company profile out: firmographics, contacts, tech stack, DNS and hiring signal.
• Company Dossier API — one company in, twelve sections out: profile, tech, contacts, DNS, open roles, news, SEC filings, federal awards, recalls, YC batch and apps.
• Press Releases API — GlobeNewswire and PR Newswire releases plus any newsroom feed, by keyword, company, ticker or subject.
• Federal Register API — rules, proposed rules, notices and the Public Inspection desk with dockets, comment deadlines and CFR references.
• Academic Papers Search API — OpenAlex, Crossref, arXiv and PubMed in one row per paper: abstract, citations, open-access PDF, authors and venue.
• RSS & Atom Feed Monitor — any RSS, Atom or JSON feed (or an OPML file) in, only the new items out, with keyword filters and a webhook.
• Website Change Monitor — watch any pages, diff the text between runs, get change rows with added/removed lines, keyword alerts and a webhook.
• Wikipedia & Wikidata API — article text, search, daily pageviews and Wikidata entity facts, any language edition.

Business, finance & jobs

• Congress & Insider Trades API — STOCK Act periodic transaction reports and SEC Form 4 insider trades in one schema.
• Federal Contracts, Grants & Lobbying API — SAM.gov opportunities, USAspending awards, Grants.gov notices and Senate lobbying filings in one schema.
• SEC EDGAR API — filings, XBRL financials and full-text search by ticker or CIK.
• Clinical Trials & FDA API — ClinicalTrials.gov studies plus openFDA recalls, labels, approvals, 510(k)s and adverse-event reports.
• Product & Vehicle Recalls API — CPSC, NHTSA, FDA and USDA recalls, vehicle complaints and ratings, plus a VIN decoder.
• Y Combinator Companies, Batches & Founders — the YC directory with founders and social links, filterable by batch, industry and hiring status.
• Career Site Jobs API — jobs straight from Greenhouse, Lever, Ashby, Workable and 10+ other ATS career sites.
• Workday Jobs API — every posting on any Workday career site, with facets, search, posted dates and full descriptions.
• New Job Postings Monitor — new, closed and changed postings on the career sites you watch.
• Hiring Signals API — Open Roles & Hiring Surge by Company — one row per company per run: open roles, what opened and closed, department and seniority breakdowns, and a hiring-surge flag.
• Remote Jobs API — RemoteOK, Remotive, We Work Remotely, Himalayas, Jobicy and more in one schema, deduplicated.
• Shopify Products API — any Shopify store's catalogue, variants, prices and stock signals.
• WooCommerce Products API — any WooCommerce store's catalogue, prices, variants and stock through its public Store API.
• Shopify App Store API — Shopify App Store listings by category, collection or developer: pricing plans, ratings and merchant reviews.
• Luma Events API — upcoming Luma events by city, category or calendar: dates, venue, price, capacity and organiser calendar, no host names.
• Shopify Store Monitor — price drops, sales, restocks, sell-outs and new products on any Shopify store, one row per change.
• Public Tenders API — EU TED, UK Find a Tender and Contracts Finder notices by keyword, CPV code, country, stage and deadline.
• Nonprofit & IRS 990 Lookup API — search US nonprofits and get EIN, NTEE code and multi-year Form 990 financials.
• OpenStreetMap Places API — businesses and points of interest by category and area from OpenStreetMap: name, address, coordinates, website, phone, opening hours.

Apps & games

• App Store & Google Play Reviews API — reviews from both stores with ratings, versions and developer replies.
• App Store Top Charts & App Search API — Apple top charts by country and genre, plus app search and details.
• App Store Keyword Rank Tracker — where any app ranks for any keyword on the App Store and Google Play, with rank changes and ASO suggestions.
• Steam Reviews API — Steam reviews with playtime, helpfulness and game details.
• Steam Game Data API — prices, tags, review scores, live player counts and top charts.
• Sports Scores & Schedules API — scores, schedules, standings, teams and headlines for the NFL, NBA, MLB, NHL, soccer and more, no key.
• App Store Monitor — App Store and Google Play listings watched between runs: new versions with release notes, price, rating and listing changes, one row per change.