SEO Page Audit — Sitemap Crawl & On-Page Checker

Point it at a domain and get every page audited. This Actor reads a site's robots.txt, walks its sitemap — nested indexes, gzip children and relative URLs included — or crawls the site when there is no sitemap, and then audits each page: title, meta description, headings, canonical, hreflang, structured data, images, links, response time, page weight. Every page comes back with a list of issue codes and a 0–100 score built from published weights you can check against the table below.

No API key. No browser. No JavaScript execution. $1.00 per 1,000 pages, proxy included. robots.txt, sitemaps and link checks are free, pages that could not be read are free, and a run that reads nothing costs nothing at all.

Try it in 30 seconds

{
  "websites": ["basecamp.com"],
  "maxPages": 50
}

A bare domain is enough. The run reads https://basecamp.com/robots.txt, follows the sitemap it finds, and audits the first 50 URLs listed there. Want specific pages instead? Put them in pageUrls and nothing is discovered or crawled.

What comes back

One page row per audited URL. This one is real — read from basecamp.com/pricing on 2026-09-11:

{
  "ok": true,
  "rowType": "page",                       // "page" | "site" | "broken-link" | "diagnostic"

  "input": "basecamp.com",
  "website": "basecamp.com",
  "url": "https://basecamp.com/pricing",
  "source": "sitemap",                     // "sitemap" | "crawl" | "input"
  "sitemapLastmod": "2026-05-26T12:00:00-07:00",

  // the response
  "statusCode": 200,
  "finalUrl": "https://basecamp.com/pricing",
  "redirectChain": [],
  "responseTimeMs": 243,
  "contentType": "text/html",
  "contentLength": 65209,                  // bytes of HTML
  "compressed": true,
  "lastModified": "Wed, 02 Sep 2026 23:45:00 GMT",
  "hsts": false,
  "xRobotsTag": null,

  // the markup
  "title": "Basecamp — Pricing",           // — decoded to what a reader sees
  "titleLength": 18,
  "metaDescription": "Start free with Basecamp — one project, five users, forever free…",
  "metaDescriptionLength": 128,
  "metaRobots": "index, follow",
  "canonical": "https://basecamp.com/pricing",
  "canonicalMatchesUrl": true,
  "lang": "en",
  "h1": ["No per-user fees, everyone's included. Simple fixed prices — another Basecamp exclusive."],
  "h1Count": 1, "h2Count": 9, "h3Count": 3,
  "wordCount": 2060,
  "textToHtmlRatio": 18.88,                // visible chars per 100 chars of markup
  "ogTitle": "Pricing",
  "ogDescription": "Start free with Basecamp — one project, five users, forever free…",
  "ogImage": "https://basecamp.com/assets/images/opengraph/basecamp.png?v=1.5.1",
  "twitterCard": "summary_large_image",
  "hreflang": [],
  "structuredDataTypes": ["WebPage", "WebSite", "ItemList", "SiteNavigationElement"],
  "hasViewport": true,
  "charset": "utf-8",
  "favicon": "https://basecamp.com/assets/images/general/favicon.svg?v=1.5.1",
  "internalLinkCount": 35,
  "externalLinkCount": 12,
  "nofollowCount": 0,
  "imageCount": 1,
  "imagesMissingAlt": 0,
  "imagesMissingAltExamples": [],
  "scriptCount": 10, "stylesheetCount": 48, "inlineStyleCount": 0,
  "noindex": false, "nofollowPage": false,
  "mixedContentCount": 0,
  "mixedContentExamples": [],

  // the verdict
  "issues": ["title-too-short"],
  "issueCount": 1,
  "score": 95,                             // 100 − 5 for title-too-short

  "error": null,
  "errorType": null,
  "scrapedAt": "2026-09-11T15:44:37.120Z",
  "sourceUrl": "https://basecamp.com/pricing"
}

Then one site row per website, written after the pages it summarises:

{
  "ok": true,
  "rowType": "site",
  "website": "basecamp.com",
  "url": "https://basecamp.com",
  "pagesAudited": 50,
  "pagesFailed": 0,
  "discovery": "sitemap",
  "robotsFound": true,
  "robotsBlocked": 0,
  "sitemapUrls": ["https://basecamp.com/sitemap.xml"],
  "sitemapUrlCount": 174,
  "statusCounts": { "200": 49, "404": 1 },
  "avgResponseMs": 268,
  "issueCounts": { "missing-description": 12, "thin-content": 4, "title-too-short": 1 },
  "score": 91,                             // mean of the 50 page scores
  "worstPages": [
    { "url": "https://basecamp.com/…", "score": 42, "issues": ["http-error", "missing-h1", "…"] }
  ],
  "linksChecked": 0,
  "linksUnchecked": 0,                     // links left in the queue when the time budget ran out
  "brokenLinkCount": 0
}

And, with checkLinks on, one broken-link row per dead internal link, naming the pages that point at it.

The issue codes, and what each costs

Every page starts at 100. Each issue found subtracts its weight. The total is floored at 0.

score = max(0, 100 − Σ weight(issue))

That is the whole formula. No curve, no weighting by page importance, nothing calibrated against anything. It is a checklist with a running total, published here so that fixing a missing <h1> and predicting exactly ten points is possible — and so that if you disagree with a weight you can recompute the column yourself from the issues array, which is the part actually worth acting on.

Code	−pts	Fires when
http-error	40	The page answered 4xx or 5xx — a 404 in the sitemap, a 500 on a live URL. It still gets a full row: a broken page you can see is the point. (A 401, 403 or 429 is a refusal, not a page; those are free diagnostic rows instead)
noindex	25	The robots meta tag or the X-Robots-Tag header says noindex or none
missing-title	20	No <title>, or an empty one
missing-description	10	No <meta name="description">
missing-h1	10	No <h1> on the page
duplicate-title	10	Another audited page on the same site publishes this exact title
thin-content	10	Fewer than 200 words of visible body text
mixed-content	8	An https page loads at least one asset over plain http
missing-viewport	5	No viewport meta tag, so the page will not adapt to a phone
multiple-h1	5	More than one <h1>
missing-canonical	5	No rel="canonical" link
canonical-mismatch	5	The canonical points somewhere other than the URL that served the page
images-missing-alt	5	At least one <img> has no alt attribute at all
slow-response	5	The page took longer than 2,000 ms to answer
duplicate-description	5	Another audited page on the same site publishes this exact description
title-too-long	5	Title longer than 60 characters
title-too-short	5	Title shorter than 20 characters
missing-lang	3	No lang attribute on <html>
description-too-long	3	Description longer than 160 characters
no-structured-data	3	No JSON-LD on the page
not-compressed	3	The response was sent uncompressed — and only when the headers prove it
redirected	3	The URL in the sitemap (or the one you sent) redirected somewhere else
large-html	3	The HTML document alone is over 1 MB, before any assets load

Four of those are more careful than they look, and it matters:

• alt="" is not a missing alt. It is the correct, deliberate markup for a decorative image. Counting it would file a finding against every site that gets accessibility right. Only a missing attribute counts.
• mixed-content counts assets, not links. An <a href="http://…"> is the reader's choice and no browser warns about it. The captured apify.com pricing page contains exactly one http:// URL — the LinkedIn link in the footer — and correctly reports zero mixed content.
• canonical-mismatch normalises before comparing. A canonical differing only by a trailing slash or a fragment is not a mismatch.
• not-compressed fires only on proof. The HTTP client inflates compressed responses transparently and deletes content-encoding and content-length when it does, so "no encoding header" is not evidence of anything. This fires only when a declared content-length matches the body that arrived — meaning nothing was inflated. When the headers do not settle it, compressed is null and no issue is raised. A false "you are not compressing" on every page of every site would be a worse bug than missing the occasional real one.

Use cases

• Pre-launch and post-migration QA — run the sitemap after a deploy and diff the issueCounts histogram against last week's. A migration that quietly shipped noindex on 400 pages shows up as a single number.
• Agency audits and pitches — walk in with a scored page-by-page table of a prospect's site, sorted by score, with the ten worst named on the summary row.
• Content-team backlog — export the issues view, filter to missing-description and thin-content, and you have a prioritised list of pages to write.
• Broken-link sweeps — checkLinks: true on a site you are responsible for, then read the links view. Each row names the broken URL and up to ten pages that link to it, which is normally enough to identify the template at fault.
• Duplicate-content detection — duplicate-title and duplicate-description are computed across the whole site, so faceted listings and paginated archives that all share one title are found in one run.
• Competitive teardown — audit a competitor's sitemap and compare wordCount, structuredDataTypes and textToHtmlRatio distributions against your own.
• Monitoring — schedule it weekly against maxPages: 100 and watch the site score as a single trend line.

How it works, and why it keeps working

A website tells you which pages it has in two places, and this Actor reads both before it audits anything.

1. robots.txt. The first request of every website is GET /robots.txt. It does three jobs: it proves which of the four candidate addresses (https://, https://www., http://, http://www.) the host actually answers on; it supplies the Sitemap: lines; and it supplies the Disallow rules, which are obeyed. Rule matching is the same longest-match-wins algorithm search engines use, with * and $ patterns, matched against path and query — so Disallow: /search? blocks /search?q=shoes and leaves /searching alone, and Allow: /private/public-notes correctly overrides Disallow: /private/. Only the User-agent: * group is read; obeying Googlebot's rules would be pretending to be Googlebot.

2. The sitemap. Whatever robots.txt named, or failing that /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml and /sitemap/sitemap.xml in that order. A <sitemapindex> is followed to its children, three levels deep, up to 50 sitemap files per site. Children ending .xml.gz are inflated. <loc> values are resolved against the sitemap's own URL, because the spec says they are absolute and real sitemaps disagree — basecamp.com's, captured live, lists /, /2 and /2/integrations. <lastmod>, <changefreq> and <priority> ride along onto the page row. A guessed path that is not there is not a fault — a site with no sitemap is the case the crawl fallback exists for, and it produces no diagnostic row; a sitemap that robots.txt or a sitemap index declared and that then fails does.

3. No sitemap? Crawl. A site that answers nothing useful at any of those locations is crawled breadth-first from its home page: same host with www. folded away, HTML pages only, shallowest links first, fragments and mailto: and asset paths skipped, bounded by maxPages and by robots.txt.

Then each page is fetched with a browser accept header, redirects followed, a 20-second per-request timeout, up to maxConcurrency at once — and 150–400 ms apart per host, always. A 500-page audit arriving as 500 simultaneous requests is a denial-of-service attempt with a friendly user agent; raising maxConcurrency overlaps the waiting, not the hammering.

Three decisions are worth spelling out, because they are what separates a useful audit from a noisy one:

A 404 is a row, not an error. A URL in a sitemap that answers 404 is one of the most useful things an audit can find. It comes back as a full page row — status, title, response time, the http-error issue, a score — and it is charged, because it is a finding. Two cases are deliberately not rows: a URL that answered nothing at all, and a URL that was refused (401, 403, 429, or a Cloudflare origin error). A refusal is very often about this exit IP rather than about the URL, and billing a row that scores 0 because a bot wall answered would be charging for the wall. Both are free diagnostic rows.

Duplicate checks run at the end of a site. duplicate-title and duplicate-description cannot be decided while pages are still arriving, so every page of a website is buffered until the site is finished, the title and description index is built, and then all the page rows are written at once followed by the site summary. The buffer holds extracted fields, never HTML.

The word count is a word count. Scripts, styles, SVG, <template> and the whole <head> are removed before counting, so a client-rendered page shipping 400 KB of JSON state in a script tag does not read as 60,000 words of content. The captured apify.com pricing page is 437 KB of markup carrying 2,109 real words, and textToHtmlRatio: 2.81 says so.

How it compares

• The sitemap is the input, not a crawl budget. Most page-level auditors make you either paste URLs one at a time or buy a crawler subscription. This takes a domain and does the discovery — including the gzip and nested-index cases that break naive readers.
• The score is arithmetic you can audit. Every weight is in the table above and in src/audit.js as one object. Nothing is hidden behind a model, and the issues array is always there to recompute from.
• Priced by the page, with the plumbing free. robots.txt, every sitemap file, every gzip child and every link check cost nothing. You pay for audited pages and one summary row.
• Failures are free and legible. A dead host, a URL that answers with a PDF, a bot wall, or a site that ran out the clock produces a diagnostic row with an errorType you can branch on — and no charge. A run that audits nothing at all finishes FAILED with the reason in its status message, never a green run containing an apology.
• No browser. Auditing is pure HTTP and targeted regular expressions, which is why a 5,000-page site costs single-digit dollars rather than hundreds — and why a page whose content only exists after client-side rendering is a documented limitation rather than a hidden one.

Input reference

Field	Type	Default	What it does
websites	array of strings	["basecamp.com"]	Bare domains or full URLs. Each is tried https:// → https://www. → http://. Per site: robots.txt, then sitemap discovery, then up to maxPages page audits. Duplicates are read, and billed, once
pageUrls	array of strings	[]	Audit exactly these URLs — no sitemap, no crawl. URLs on the same host as a websites entry are folded into that site's audit rather than becoming a second one
maxPages	integer 1–5000	100	Pages audited per website. This is the number that sets the price of a run
checkLinks	boolean	false	Also fetch every unique internal link found (a plain GET), and write a broken-link row for each one that answers 4xx, 5xx or nothing. $0.0002 per broken link; checking is free. Links the time budget cuts off are counted in linksUnchecked, never as broken
maxLinksToCheck	integer 1–5000	500	Ceiling on the link check, per website
respectRobots	boolean	true	Obey the site's own robots.txt. Skipped URLs are counted as robotsBlocked and never fetched or billed. Turn off only for a site you own
maxSitemapUrls	integer 1–5000	5000	Ceiling on URLs collected from sitemaps. Collection also stops as soon as there are enough to fill maxPages
maxConcurrency	integer 1–16	8	Pages of one website in flight at once. Each worker keeps its own proxy session. Per-host spacing of 150–400 ms applies regardless
maxRunSecs	integer 30–3600	240	Whole-run wall-clock budget. When it runs out the Actor keeps and bills what it has, and files a free diagnostic row saying how many URLs it never reached
proxyConfiguration	object	Apify datacenter	Switch the group to RESIDENTIAL for sites behind an aggressive bot wall. Datacenter is the default because it is cheaper and ordinary websites accept it

At least one of websites and pageUrls must be non-empty. A run with neither finishes FAILED, having charged nothing.

Output reference

Every row carries the same keys, whatever its type. ok: true is a page, site or broken-link row; ok: false is a free diagnostic row.

Common to every row

Field	What it is
ok, rowType	Whether this row is a result, and which of the four kinds it is
input, website	The entry you supplied, and the registrable domain it resolved to
url, sourceUrl	The URL this row is about
source	Where the URL came from: sitemap, crawl or input on a page row; site on a summary; link-check on a broken link
error, errorType, scrapedAt	Why a row failed, if it did, and when the row was written

page rows

Field	What it is
statusCode, finalUrl, redirectChain, responseTimeMs	What the server answered, where it ended up, and how long it took
contentType, contentLength, compressed, lastModified, hsts, xRobotsTag	Response headers worth an audit column. contentLength is bytes of HTML received
title, titleLength, metaDescription, metaDescriptionLength, metaRobots	The tags search results are built from, entity-decoded
canonical, canonicalMatchesUrl, lang, charset, hasViewport, favicon	Indexing and rendering signals
h1, h1Count, h2Count, h3Count	Heading text and structure
wordCount, textToHtmlRatio	Visible body text, and how much of the document it is
ogTitle, ogDescription, ogImage, twitterCard	Social cards
hreflang, structuredDataTypes	Alternates as {lang, href}, and every JSON-LD @type including nested ones
internalLinkCount, externalLinkCount, nofollowCount	Unique links, split by host with www. folded away
imageCount, imagesMissingAlt, imagesMissingAltExamples	Images, and up to twenty src values with no alt attribute
scriptCount, stylesheetCount, inlineStyleCount	Front-end weight
noindex, nofollowPage, mixedContentCount, mixedContentExamples	Indexing directives and http assets on an https page
issues, issueCount, score	The verdict
sitemapLastmod, sitemapChangefreq, sitemapPriority	What the sitemap said about this URL, when it came from one

site rows

Field	What it is
pagesAudited, pagesFailed	How many URLs produced a row, and how many could not be read
discovery	sitemap, crawl or input
robotsFound, robotsBlocked	Whether robots.txt was readable, and how many URLs it kept out
sitemapUrls, sitemapUrlCount	Which sitemap files were read, and how many URLs came out of them
statusCounts, avgResponseMs	Status histogram and mean response time
issueCounts	How many pages triggered each issue code, heaviest first
score	The rounded mean of the page scores
worstPages	The ten lowest-scoring pages, each with its score and issues
linksChecked, linksUnchecked, brokenLinkCount	Link-check totals, zero unless checkLinks is on. linksUnchecked is what the time budget cut off

broken-link rows carry url, statusCode, error and foundOn — up to ten audited pages that link to the dead URL.

errorType on a diagnostic row is one of:

Value	Meaning	Charged?
unreachable	Nothing answered — a dead host, a connection that failed, a domain that does not resolve	No
blocked	The request was refused — 401, 403, 429, a Cloudflare origin error, or an anti-bot interstitial served in place of the page	No
not-html	The URL answered with a file rather than a page — a PDF, an image, a JSON feed	No
timeout	The run's maxRunSecs budget ran out before this URL was read	No
invalid-input	The entry was not a fetchable website or URL — an email address, an IP, a bare word	No

Pricing

$1.00 per 1,000 pages. Pay-per-event, with the proxy already inside that number — there is no separate proxy line on your bill for this Actor.

Event	What triggers it	FREE	Starter	Scale	Business
Page audited (primary)	One audited page written to your dataset	$0.001	$0.001	$0.0008	$0.0006
Site summary	One website that produced at least one page	$0.002	$0.002	$0.002	$0.002
Broken link found	One internal link that failed, only when checkLinks is on	$0.0002	$0.0002	$0.0002	$0.0002
Run started	Once per run, after the first page is audited	$0.001	$0.001	$0.001	$0.001

Worked example. A weekly audit of a 500-page site with link checking on, finding 30 broken links:

• 500 pages × $0.001 = $0.500
• 1 site summary × $0.002 = $0.002
• 30 broken links × $0.0002 = $0.006
• 1 run start = $0.001
• Total: $0.509

What you are never charged for: robots.txt, any sitemap file however many there are, any link check (only a broken link produces a billed row), a URL that answered nothing, a URL that answered with a file, a URL that was refused with 401/403/429 or behind a bot wall, a URL robots.txt disallowed, an entry that was not a website, or anything the run never reached before maxRunSecs. If a whole run comes back empty it finishes FAILED and bills nothing at all, start fee included.

Set ACTOR_MAX_TOTAL_CHARGE_USD on a run and the Actor stops fetching once the ceiling is in sight, rather than handing you rows it cannot bill or billing you for rows it cannot hand over. It finishes SUCCEEDED with the ceiling named in its status message, and everything already delivered is yours.

Limits, and the ones that might bite

No browser, so the audit is of the HTML the server returned. A page whose body copy is rendered client-side will read as thin-content with a low textToHtmlRatio, and that is an honest description of what a crawler receives — but it is not what a human sees, and treating it as a content problem would be wrong. The wordCount and textToHtmlRatio columns together are the tell: a low ratio on a big document means a framework, not an empty page.

sitemapUrlCount is what was read, not everything the site publishes. Sitemap collection stops as soon as there are enough URLs to fill maxPages, at 50 sitemap files, at three levels of index nesting, or at maxSitemapUrls. Reading a news site's entire sitemap tree to audit 100 pages would be rude as well as slow. Raise maxPages and maxSitemapUrls together when you want a complete count.

redirectChain has two ends, not every hop. The HTTP client follows redirects itself and hands back only the URL it finished on, so a page that went through three hops reports the URL requested and the URL delivered. The redirected issue and finalUrl are correct; the intermediate hops are not visible.

Duplicate checks are within one run and one site. duplicate-title compares the pages this run audited on this website. Two pages sharing a title where only one was audited will not be flagged, and a title shared across two different domains in the same run is not a duplicate.

compressed can be null. See the not-compressed note above. When the response carried neither content-encoding nor content-length and the body was empty, nothing can be concluded, and the column says so rather than guessing.

A crawl is not a full-site crawl. The fallback follows internal HTML links breadth-first from the home page up to maxPages. It does not submit forms, does not paginate listings by guessing URL patterns, and does not leave the host. A site with no sitemap and a JavaScript-only navigation will yield one page: its home page.

Scores are a checklist total, not a ranking prediction. Nothing here measures backlinks, domain authority, search intent, competitor positioning or whether the content is any good. A page can score 100 and rank nowhere. The score is for triage — sort by it, then read the issues.

Sites behind aggressive bot management may refuse the default proxy. The datacenter default is right for the overwhelming majority of the web and wrong for the small slice that fingerprints every request. Those pages come back as free blocked diagnostic rows; switch proxyConfiguration to RESIDENTIAL and re-run if you need them.

The upstream format may change. Sites are rebuilt, sitemaps move, and a site that published a clean sitemap.xml last month can start serving an index of gzip children this month. When that happens the discovery path changes rather than breaking — the discovery and sitemapUrls columns on the site row are how you see which path was taken.

Use it from an AI agent, or from code

One JSON object in, one flat array out — the shape agent runtimes want. The Actor runs with limited permissions, uses pay-per-event pricing and never enters Standby, so it works over the Apify MCP server and with x402 agentic payments. The Integrations tab pushes results to Slack, a webhook, Zapier, Make, Google Sheets, Snowflake or BigQuery.

curl -X POST "https://api.apify.com/v2/acts/insight.solutions~seo-page-audit/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"websites":["basecamp.com"],"maxPages":50}'

# pip install apify-client
from apify_client import ApifyClient

client = ApifyClient("<APIFY_TOKEN>")
run = client.actor("insight.solutions/seo-page-audit").call(run_input={
    "websites": ["basecamp.com"],
    "maxPages": 200,
    "checkLinks": True,
})

for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    if row["rowType"] == "page" and row["score"] < 80:
        print(row["score"], row["url"], "→", ", ".join(row["issues"]))
    elif row["rowType"] == "site":
        print("site score:", row["score"], "over", row["pagesAudited"], "pages")
    elif row["rowType"] == "broken-link":
        print("broken:", row["url"], "linked from", row["foundOn"][0])

The four dataset views — Overview, Pages with issues, Broken links and Site summaries — are set up in the Console so each row type exports as its own clean table.

FAQ

How does it find the pages? robots.txt first, for the Sitemap: lines and the crawl rules. Then those sitemaps, or /sitemap.xml and three other usual locations. Sitemap indexes are followed to their children, gzip children are inflated, and relative <loc> values are resolved. Only if none of that yields anything does it crawl the site from its home page.

Does it respect robots.txt? Yes, by default, using the User-agent: * group and the longest-match-wins rule search engines use. Disallowed URLs are never fetched, counted as robotsBlocked on the site row, and never billed. respectRobots: false exists for sites you own — for example a staging site that blocks everything.

How fast is it? Requests to one host are spaced 150–400 ms apart, so about three pages a second per site plus the time each page takes to answer, overlapped across maxConcurrency workers. 50 pages of a typical site is well under a minute. maxRunSecs is the hard stop.

Why did a page I know exists not get audited? Four possibilities, in order of likelihood: it is not in the sitemap; maxPages was reached before the run got to it; robots.txt disallows it; or it answered with something that is not a web page. The site row's sitemapUrlCount, pagesAudited and robotsBlocked distinguish the first three, and there is a free diagnostic row for the fourth.

What counts as a duplicate title? An exact string match, after entity decoding and whitespace collapsing, between two pages audited on the same website in the same run. Near-duplicates are not detected — "Shoes | Acme" and "Shoes - Acme" are two different titles.

Can I audit just a handful of specific pages? Yes. Put them in pageUrls and leave websites empty. No sitemap is fetched and no crawl happens; only robots.txt is read, because the Disallow rules still apply.

Does checkLinks check external links too? No. Only internal links, deduplicated across the whole site, capped at maxLinksToCheck. Checking outbound links would multiply the request count against sites that did not ask to be crawled.

Can I audit 5,000 pages? Yes, that is the per-website ceiling. Raise maxRunSecs to match — 5,000 pages at three a second is roughly half an hour — and bound the spend with ACTOR_MAX_TOTAL_CHARGE_USD. At $0.001 per page the whole site is $5.

What happens if one website in a list fails? The others still run. The failed one produces a free diagnostic row and the run finishes SUCCEEDED. If every website fails, the run finishes FAILED and you are billed nothing at all.

Is the data fresh? Live. Every run reads the site at that moment; nothing is cached, and there is no index behind this.

Legal and data-protection notes

• Public pages only. Every URL fetched is one a browser would load without signing in. The Actor never logs in, never accepts cookies or session tokens, never takes an API key belonging to anyone else, and never touches anything behind authentication.
• It obeys the site's own instructions by default. robots.txt is read before anything else and its Disallow rules are respected unless you deliberately turn that off for a site you control.
• It is a polite crawler. One request at a time per host with 150–400 ms between them, a 20-second per-request timeout, and a whole-run budget. It does not retry a refusal in a loop.
• This is not personal data. An on-page SEO record describes a web page's markup, not a person. Nothing in the output identifies an individual, and no contact details, names or user records are collected.
• Scores are observations, not advice. "This page has no meta description" is what was seen. What to do about it depends on the page, and the issue list is there precisely so the number never has to be taken on faith.
• Not affiliated with any website audited or named in these docs. All trademarks belong to their respective owners.

Our other Actors

Every Insight Solutions Actor is pay-per-result with no browser, no login and no API key, and every one of them returns free diagnostic rows instead of billing for failures. Prices are per 1,000 results.

Video, audio & social

• YouTube Transcript API — captions as timed segments, text, SRT or VTT, with language fallback and translation.
• YouTube Comments API — comments and replies with likes, pinned and hearted flags, newest or top sort.
• YouTube Channel API — a channel's videos, Shorts and live streams, plus YouTube search.
• Podcast Search, Episodes & Charts API — Apple Podcasts search, charts and full episode feeds.
• Bluesky Scraper — profiles, posts, followers and follows from the public AT Protocol API.
• Telegram Channel Scraper — posts, views and channel stats from public Telegram channels.
• Substack Scraper — posts with full free text, comments and publication profiles.

News, documents & the web

• Google News Search, Topics & Real Article URLs — news search and topic feeds with the publisher's real URL decoded.
• Website to Markdown — Content Extractor for LLMs & RAG — any site as clean Markdown, text and heading-aware chunks.
• Internet Archive API — archive.org search, item metadata, files and reviews.
• Wayback Machine Toolkit — archived URL inventories, snapshots and text diffs between dates.
• Website Technology Detector — the tech stack behind any site, with the evidence for each detection.
• Domain Intelligence API — DNS, RDAP registration, TLS certificate and HTTP facts in one row per domain.
• Keyword Suggestions API — Google, YouTube, Bing, Amazon and eBay autocomplete with alphabet and question expansions.
• Website Contact Extractor — emails, phone numbers and social profiles from any list of websites.

Business, finance & jobs

• Congress & Insider Trades API — STOCK Act periodic transaction reports and SEC Form 4 insider trades in one schema.
• SEC EDGAR API — filings, XBRL financials and full-text search by ticker or CIK.
• Y Combinator Companies, Batches & Founders — the YC directory with founders and social links, filterable by batch, industry and hiring status.
• Career Site Jobs API — jobs straight from Greenhouse, Lever, Ashby, Workable and 10+ other ATS career sites.
• New Job Postings Monitor — new, closed and changed postings on the career sites you watch.
• Shopify Products API — any Shopify store's catalogue, variants, prices and stock signals.

Apps & games

• App Store & Google Play Reviews API — reviews from both stores with ratings, versions and developer replies.
• App Store Top Charts & App Search API — Apple top charts by country and genre, plus app search and details.
• Steam Reviews API — Steam reviews with playtime, helpfulness and game details.
• Steam Game Data API — prices, tags, review scores, live player counts and top charts.