Audit a website you own or may audit: reads robots.txt and sitemaps, then checks each page for status code, redirects, title, meta description, H1, canonical and 35+ SEO issues, or lists every sitemap URL. Plain HTTP, no browser, honours robots.txt and Crawl-delay. From $1.50 per 1,000 pages.
Build 0.1.5 (QA round 2): a site behind a login wall no longer makes unbounded requests. Skipped login pages and rows that fail the output check count toward Max pages, and the run stops with login-wall (SUCCEEDED with what was saved, nothing charged for skipped pages) after 5 login pages in a row or when more than half of the first 20 checked pages are login pages. README responsible-use paragraph worded to match: individual login pages are skipped, login walls stop the run.
Build 0.1.4 (QA and legal round 1): a login popup, sidebar login box or CAPTCHA on a contact or newsletter form no longer stops the audit; such pages are saved with the new has-login-form or has-captcha issue. Only a CAPTCHA or bot-check interstitial or HTTP 401/403/407/429 stops the run; a login page reached by a link is skipped (not saved, not charged) and the crawl goes on. Sign-up, register, join and similar account URLs are not requested. www . and the bare domain share one speed limit. Bounded memory for seen URLs (hashes) and the queue, URLs over 2,048 characters skipped, checkpoint records split under 1 MB, redirect targets kept after a migration, large JSON-LD no longer reported invalid, every row checked against the output schema before it is saved. README wording from the legal review.
Build 0.1.3: fixed a count that could stop an audit a few pages before Max pages, or before the budget was used up, while a batch was being saved.
First version. Audit mode: reads robots.txt and the sitemaps, then checks the start URL, the sitemap URLs and same-site links (up to 5,000 pages), one row per requested URL with status, redirect target, title, meta description, H1, canonical, robots directives, indexability, language, hreflang, viewport, Open Graph, JSON-LD types, word count, link and image counts, and 35+ issue codes.
Sitemap mode: lists every URL of the sitemaps (up to 100,000) with lastmod, changefreq, priority and robots.txt status, without requesting any page.
Sitemaps: indexes up to 3 levels, gzip files detected by their content, plain-text sitemaps, protocol size limits, up to 50 files per run.
robots.txt per RFC 9309 on every run (own group or *, longest match, 404 = allow, 5xx = disallow); Crawl-delay honoured (one request at a time, at least the delay apart; up to 10 s in audit mode, 60 s in sitemap mode).
Politeness and safety: at most 2 requests at a time per host, 10-120 requests per minute (default 60), honest User-Agent, no proxies, no browser; stops on 401/403/407/429, CAPTCHA, bot-check or login pages (changed in 0.1.4/0.1.5: login pages are skipped, login walls stop the run); refuses local, private, link-local and reserved addresses, also after redirects and at DNS connect time.
Privacy: e-mail addresses and phone numbers (also international formats) masked in title, meta description and H1; no body text, link texts, mailto: or tel: links; URLs carrying contact details are not requested.
Pay-per-event with exactly three events: run-start (with the first saved result), page and sitemap-url. The budget is checked before every request and every charge; a run that saves nothing costs nothing.
Stops cleanly 2 minutes before the run timeout (time-limit), so long audits end as SUCCEEDED with what they saved.
Results saved in batches of up to 25 pages or every 15 seconds; migration and abort stop at a batch boundary, so no page is saved or charged twice.