SEO Site Audit Crawler - Title, Meta, Canonical, H1, Robots

Crawls a site and returns one row per page: what the page actually says to a search engine, what evidence that claim rests on, and what this Actor did not look at.

There is no 0–100 score. A score is a weighting somebody chose; it is not a measurement. Every finding here points at the tag, the header or the status code it came from, so you can check it yourself.

What it will not tell you, and why

This list is attached to every row as notChecked, so nobody has to guess whether a blank cell means "fine" or "not looked at".

Not checked	Why
JavaScript-rendered content	The HTML is read as the server sends it. Scripts are not executed.
Core Web Vitals	Those are field measurements collected from real visitors (CrUX). They cannot be derived from one server-side fetch, so they are not reported.
Ranking positions	No search result data is queried.
Image file sizes and dimensions	Image files are not downloaded.

responseMs is the time from sending the request to receiving the response headers, over a connection that may already be open, and the response may have come from a CDN cache — cacheAgeSeconds tells you how old it was. It is not a visitor-perceived load time and is not presented as one.

What each page row contains

Indexability — httpStatus, noindex, metaRobots, xRobotsTag, canonical, canonicalIsSelf, canonicalCount, redirectCount, redirectChain. Both the robots meta tag and the X-Robots-Tag response header are read; a page can be excluded by either.

On-page — title, titleLength, metaDescription, metaDescriptionLength, h1, h1Count, headingJumps, wordCount, lang, hasViewport, charset.

Images — imagesTotal, imagesMissingAltAttr, imagesEmptyAlt. alt="" is counted separately from a missing alt attribute, because an empty alt is the correct way to mark a decorative image, not a defect.

Links — internalLinks, externalLinks, nofollowLinks.

Structured data and sharing — jsonLdBlocks, jsonLdTypes, jsonLdInvalidBlocks, openGraphTitle, openGraphDescription, openGraphImage, twitterCard, hreflang. A JSON-LD block that does not parse is counted as invalid rather than silently ignored — search engines will drop it, so you should know it is there.

Request and crawl context — url (the final URL after redirects), requestedUrl (what was asked for), foundOn (the page the link was found on), analyzed, contentType, htmlBytes, responseMs, totalMs, cacheAgeSeconds, nofollow (the page-level directive), checkedAt.

Across the crawl — duplicateTitleWith, duplicateDescriptionWith. Each lists the other URLs in the same crawl that share the exact value.

Findings — issues, issueCount, blockingIssueCount. Each issue carries severity, code, message and evidence:

{
  "severity": "info",
  "code": "duplicate-description",
  "message": "18 pages in this crawl share this exact meta description.",
  "evidence": "https://www.python.org/psf-landing/ https://www.python.org/jobs/ https://www.python.org/community/"
}

severity is blocking, defect or info — three named categories, not a number:

• blocking — the page will not appear in search results as it stands: an HTTP error, a noindex directive, or a canonical pointing at a different URL.
• defect — something is missing or broken that has a real effect: no <title>, no <h1>, images with no alt attribute, unparseable JSON-LD, duplicate titles.
• info — a measured fact worth knowing: title length, several <h1> elements, skipped heading levels, redirect chains, and a server response that arrived nearly empty.

A page that arrives empty is reported as such, not left blank. When the HTML the server sends contains almost no text and no links to the same site, the row carries an empty-server-html finding saying so, and pointing at the usual cause: the page builds its content with JavaScript, which this Actor does not run. Measured example: nhk.or.jp returns 6 KB of HTML with 4 words and no links, while gov.uk returns 798 words and 54 links from the same kind of request.

Pages it did not analyse are still returned

A crawler that quietly drops what it could not read is indistinguishable from a crawler that found nothing. Every skipped URL comes back with analyzed: false and a reason:

Situation	What the row says
robots.txt disallows the path	Not fetched, and the row says so. Verified against wordpress.org/wp-admin/, which is disallowed for all crawlers.
The URL is not HTML	Not analysed because it is not HTML (Content-Type: image/png)
The request failed	The transport error, after the configured retries

Skipped pages are not charged.

robots.txt is obeyed by default, and obeyed exactly

The crawler requests each URL exactly as it was found, including the trailing slash. That sounds trivial and is not: stripping the trailing slash from /wp-admin/ makes it stop matching Disallow: /wp-admin/, and the crawler would fetch a path the site asked it not to. It also manufactures redirects — on a site whose links all end in a slash, requesting the slashless form produces a 301 on every page, which then looks like a site defect that does not exist. On a 20-page crawl of python.org, fixing this took the redirect findings from 19 down to 3. The remaining 3 are real.

Allow beats Disallow when it is more specific, * and $ in patterns are honoured, and comments are stripped — including trailing comments on a rule line, which otherwise corrupt the path being matched.

Input

Field	Type	Default	What it does
startUrls	array	https://www.python.org/	Where the crawl begins. Internal links found on these pages are followed up to the page limit. Links to other sites are counted, not crawled.
maxPages	number	20	How many pages to analyse before stopping. Range 1–1000.
respectRobots	boolean	true	When on, URLs that robots.txt disallows are reported, not fetched.
timeoutSecs	number	20	Timeout for one page. Range 3–120.
maxRetries	number	3	Retries for temporary failures — timeout, connection reset, HTTP 429, HTTP 5xx. A 404 is never retried, because that is already an answer. Range 1–10.

Speed and cost

Measured: 20 pages of python.org in 1.25 s, 2.01 MB of HTML. No proxies and no third-party API, so the only cost is compute time.

You are charged per page analysed. Pages that were skipped — disallowed by robots.txt, not HTML, or unreachable — are returned with a reason and are not charged.

Typical uses

• Before a migration. Record canonical, redirect chains and noindex across the site, then run it again afterwards and compare.
• Finding duplicate titles and descriptions. The crawl reports which other URLs share each exact value, so the fix is obvious.
• Checking a client site you did not build. Blocking issues first: HTTP errors, noindex, canonicals pointing elsewhere.
• Structured data review. Which JSON-LD types exist per page, and which blocks are invalid.

FAQ

Why is there no SEO score? Because a score is not a measurement. Two tools will give the same page different scores, and neither number can be checked. Facts and their evidence can be.

Why does it report Core Web Vitals as not checked instead of estimating them? Core Web Vitals come from real visitors' browsers. A number produced from one server-side fetch would not be that measurement, whatever it was called.

Does it run JavaScript? No. If a site renders its content client-side, that content is not in the HTML this reads — which is why notChecked says so on every row rather than letting a low wordCount look like a finding.

Can it crawl pages robots.txt disallows? respectRobots can be turned off, but leaving it on is the default for a reason: those URLs are returned with a reason instead, so nothing disappears silently.

Other tools by NeverEmpty

Every NeverEmpty Actor follows the same rule: it never returns an empty result to mean two different things, and it only charges for rows that actually carry an answer.

• link-checker - broken links on a site, with blocked told apart from dead
• domain-lookup - WHOIS, DNS, MX, SPF, DMARC and SSL expiry for a list of domains
• website-contacts - emails, phones and social profiles from any website, no invented addresses

Support

Found a value you believe is wrong? Open an issue on the Issues tab with the URL and the row you got. Reports about incorrect data are fixed first.