SEO & Accessibility Auditor Scraper

🔍 SEO & Accessibility Auditor Scraper – Technical SEO Crawler and WCAG Compliance Checker

The SEO & Accessibility Auditor Scraper crawls the pages you give it and returns a scored technical SEO and accessibility audit for each one, plus a site-level summary that ranks the issues by how many pages they affect. Point it at a start URL, let it follow internal links, and you get back nine category scores per page — meta tags, headings, content, technical, schema, performance, links, images and accessibility — alongside a crawl-coverage breakdown showing what is indexable and what is not.

Site audits usually arrive as a PDF nobody can query. This SEO auditor produces a dataset instead. Every page becomes a row with a structured audit object, and every run produces a summary record with averageScore, categoryAverages, topIssues and a score distribution. That means you can diff two crawls, track a score over time, or feed the output straight into a reporting dashboard rather than transcribing findings by hand.

---

📊 What Data Can You Extract with This SEO Auditor Scraper?

The Actor writes two kinds of record: one per crawled page, and one site-level summary. Fields group as follows.

Category	Fields	What it gives you
🏷️ Record type	type	Distinguishes a page record from the site-summary record
📄 Page identity	pageUrl, title, httpStatus	The final URL after redirects, its title tag and the HTTP status returned
🧪 Page audit	audit	The full audit object for that page, including overall score, per-category scores, and critical, warning and info issues
📈 Site scoring	averageScore, categoryAverages, distribution	Mean score across the crawl, per-category means, and how pages fall into excellent, good, needs-improvement and poor bands
🚩 Prioritised problems	topIssues	Every distinct issue found, with a count of affected pages and the list of URLs, sorted by frequency
🕸️ Crawl reach	pagesCrawled, crawlCoverage	How many pages were audited, and how they split across indexable, noindex, canonicalised, 4xx, 5xx and redirect buckets

The field that changes how audits get actioned is topIssues. Rather than reading nine categories across fifty pages and building your own frequency table, you get a pre-sorted list where the issue affecting the most pages sits at the top, complete with the URLs where it occurs. That is the difference between an audit report and a work queue.

---

🌟 Key Features of the SEO Auditor Scraper

Feature	Description
🧭 Nine audit categories	Meta tags, headings, content, technical, schema, performance, links, images and accessibility are each scored independently and can be toggled on or off
🕸️ Internal link crawling	With crawlPages enabled the auditor discovers and follows internal links from each audited page, up to your maxPages limit
🤖 robots.txt awareness	respectRobotsTxt is on by default, so the crawler honours the rules the site publishes for automated agents
🌐 Subdomain control	includeSubdomains decides whether links to subdomains count as internal and get crawled, which matters for sites split across blog. or shop. hosts
🚫 URL exclusion patterns	excludeUrlPatterns accepts regex and wildcard patterns so you can keep faceted navigation, search pages and parameter noise out of the crawl
⚡ Concurrent crawling	maxConcurrency controls how many requests run in parallel, letting you trade crawl speed against load on the target server
📊 Site-level summary record	One site-summary item aggregates average score, category averages, issue frequency and score distribution across the whole crawl
🗂️ Crawl coverage breakdown	crawlCoverage counts indexable pages against noindex, canonicalised, 4xx, 5xx and redirect pages — the indexation picture in one object
♿ Accessibility compliance checks	The accessibility audit examines language declaration, ARIA usage and skip links, giving a WCAG-oriented score alongside the SEO ones

---

🚀 Why Choose This SEO Auditor Scraper?

Every finding is queryable data. Because both the per-page audit object and the site-summary record land in a dataset, an audit becomes something you can SELECT against. Track averageScore release over release, alert when categoryAverages.accessibility drops, or export only the pages where the technical score fell below a threshold.

Accessibility is audited alongside SEO, not bolted on. The auditAccessibility category checks language declaration, ARIA attributes and skip links in the same pass that scores your meta tags. For teams where WCAG compliance and organic performance are owned by the same person, that removes an entire second tool from the workflow.

Issue prioritisation is done for you. topIssues is sorted by the number of affected pages and carries the URL list for each issue. A missing meta description on eighty pages outranks a single malformed heading, and the output says so explicitly rather than leaving you to work it out.

Granular control over what gets audited. Nine separate boolean toggles mean you can run a narrow, fast crawl that checks only the categories you are working on this sprint. Disabling performance and schema when you are focused on heading structure makes runs quicker and the output easier to read.

---

📥 Input

The SEO auditor takes a start URL list, crawl controls and a switch for each audit category.

{
  "startUrls": [
    { "url": "https://example.com" }
  ],
  "crawlPages": true,
  "maxPages": 5,
  "maxConcurrency": 5,
  "includeSubdomains": false,
  "respectRobotsTxt": true,
  "excludeUrlPatterns": [],
  "auditMetaTags": true,
  "auditHeadings": true,
  "auditContent": true,
  "auditTechnical": true,
  "auditSchema": true,
  "auditPerformance": true,
  "auditLinks": true,
  "auditImages": true,
  "auditAccessibility": true
}

🔧 SEO Auditor Scraper Input Fields

Field	Type	Required	Default	Description
startUrls	array	✅ Yes	[{ "url": "https://example.com" }]	URLs to start the SEO and accessibility audit from.
crawlPages	boolean	❌ No	true	Crawl internal links discovered on the audited pages.
maxPages	integer	❌ No	5	Maximum number of pages to crawl and audit.
maxConcurrency	integer	❌ No	5	Maximum number of concurrent requests to execute.
includeSubdomains	boolean	❌ No	false	Include subdomains when finding internal links.
respectRobotsTxt	boolean	❌ No	true	Respect rules defined in the site's robots.txt.
excludeUrlPatterns	array	❌ No	[]	Exclude URLs matching these regex or wildcard patterns.
auditMetaTags	boolean	❌ No	true	Enable auditing of meta tags (title, description, canonical, OG/Twitter).
auditHeadings	boolean	❌ No	true	Enable auditing of headings hierarchy (H1–H6).
auditContent	boolean	❌ No	true	Enable auditing of content readability and keyword density.
auditTechnical	boolean	❌ No	true	Enable technical audits (HTTPS, mobile friendliness, robots, AMP).
auditSchema	boolean	❌ No	true	Enable schema structured data audit.
auditPerformance	boolean	❌ No	true	Enable simulated page performance audit.
auditLinks	boolean	❌ No	true	Enable link status and descriptive text audit.
auditImages	boolean	❌ No	true	Enable image alt text presence audit.
auditAccessibility	boolean	❌ No	true	Enable accessibility compliance check (lang, aria, skip links).

💡 Input Examples

Full site audit across fifty pages

{
  "startUrls": [{ "url": "https://example.com" }],
  "crawlPages": true,
  "maxPages": 50,
  "maxConcurrency": 10
}

Accessibility-only compliance sweep

{
  "startUrls": [{ "url": "https://example.com" }],
  "maxPages": 25,
  "auditMetaTags": false,
  "auditContent": false,
  "auditSchema": false,
  "auditPerformance": false,
  "auditLinks": false,
  "auditAccessibility": true
}

Single-page check with no crawling

{
  "startUrls": [{ "url": "https://example.com/pricing" }],
  "crawlPages": false,
  "maxPages": 1
}

---

📤 Output

The auditor emits two record shapes. Page records are written as each page is audited; the summary record is written once at the end of the crawl.

Page record

{
  "type": "page",
  "pageUrl": "https://example.com/pricing",
  "title": "Pricing – Example",
  "httpStatus": 200,
  "audit": {
    "title": "Pricing – Example",
    "overallScore": 82,
    "categoryScores": {
      "metaTags": 90,
      "headings": 75,
      "content": 80,
      "technical": 95,
      "schema": 60,
      "performance": 78,
      "links": 88,
      "images": 70,
      "accessibility": 84
    },
    "issues": {
      "critical": [],
      "warnings": ["Missing image alt text"],
      "info": ["No Twitter card tags"]
    },
    "metaTags": { "canonical": { "url": "https://example.com/pricing" } }
  }
}

Site summary record

{
  "type": "site-summary",
  "pagesCrawled": 25,
  "averageScore": 79,
  "categoryAverages": {
    "metaTags": 86, "headings": 71, "content": 77, "technical": 92,
    "schema": 55, "performance": 74, "links": 85, "images": 68, "accessibility": 81
  },
  "topIssues": [
    { "issue": "Missing image alt text", "count": 18, "pages": ["https://example.com/pricing"] }
  ],
  "distribution": { "excellent": 3, "good": 9, "needsImprovement": 11, "poor": 2 },
  "crawlCoverage": {
    "pagesCrawled": 25,
    "indexable": 21,
    "nonIndexable": { "noindex": 2, "canonicalized": 1, "errors4xx": 1, "errors5xx": 0, "redirects": 0 }
  }
}

🧾 SEO Auditor Output Fields — Page Records

Field	Type	Description
type	string | null	Record type, page for per-page audits
pageUrl	string | null	Final URL of the audited page after redirects
title	string | null	Title of the audited page
httpStatus	string | null	HTTP status code returned by the page
audit	—	Full audit object, containing the overall score, per-category scores and the critical, warning and info issue lists

🧾 SEO Auditor Output Fields — Site Summary Record

Field	Type	Description
type	string | null	Record type, site-summary for the aggregate record
pagesCrawled	—	Number of pages crawled and audited in this run
averageScore	number | null	Mean overall audit score across all crawled pages
categoryAverages	—	Mean score for each of the nine audit categories
topIssues	—	Distinct issues sorted by the number of affected pages, each with a count and the list of page URLs
distribution	—	Count of pages falling into the excellent, good, needs-improvement and poor score bands
crawlCoverage	object | null	Indexation breakdown: pages crawled, indexable count, and non-indexable counts for noindex, canonicalised, 4xx, 5xx and redirects

Filter on type to separate the two shapes. Most reporting workflows read the single site-summary record for headline numbers and the page records for the detail behind them.

---

💻 How to Use the SEO Auditor Scraper (Step by Step)

Step 1: Open the SEO Auditor on Apify

Sign in to Apify and open the SEO & Accessibility Auditor Scraper Actor. The crawl runs entirely on Apify's infrastructure, so there is nothing to install locally and no crawler configuration to maintain. The Actor page gives you a structured input form, a live log and a dataset view where audit records appear as pages are processed.

Step 2: Set Your Start URLs

Enter the URLs you want the audit to begin from. A single homepage is the normal starting point for a full site audit; a list of specific templates — one product page, one category page, one article — is better when you want representative coverage without a large crawl. Use the canonical HTTPS version of each URL to avoid spending crawl budget on redirects.

Step 3: Configure the Crawl Boundaries

maxPages is your crawl budget and defaults to 5, which is deliberately conservative — raise it for a real audit. Set includeSubdomains to true if your blog or shop lives on a separate host and you want it covered. Use excludeUrlPatterns to keep faceted navigation, internal search results, pagination and tracking-parameter URLs out of the crawl, since those inflate page counts without adding insight.

Step 4: Choose Which Audits to Run

All nine audit categories are enabled by default, which is right for a first baseline. On follow-up runs, turn off the categories you are not working on: an accessibility remediation sprint only needs auditAccessibility and perhaps auditImages, and a narrower run finishes faster and produces output your team can read without filtering.

Step 5: Tune Concurrency and Robots Handling

maxConcurrency defaults to 5. Raise it for large crawls on infrastructure that can absorb the load; lower it to 1 or 2 when auditing a small shared-hosting site you do not want to overwhelm. Leave respectRobotsTxt on unless you are auditing a site you own and specifically need to inspect paths that are disallowed for crawlers.

Step 6: Run and Read the Summary First

Press Start. Page records stream into the dataset as the crawl progresses, and the site-summary record is written at the end. Read the summary first: averageScore tells you where the site stands, distribution tells you whether the problem is a few terrible pages or uniformly mediocre ones, and topIssues tells you what to fix first.

Step 7: Work the Issue Queue and Re-Audit

Take topIssues in order. Each entry carries the affected page URLs, so the fix list writes itself. Once changes ship, re-run the auditor with identical settings and compare averageScore and categoryAverages against the previous run — that comparison is the evidence that the work moved the needle, and it is why keeping every run's dataset is worthwhile.

---

🔌 API Access & Integrations

Run the SEO auditor and receive the audit dataset in the same request:

curl -X POST "https://api.apify.com/v2/acts/scrapers-hub~seo-accessibility-auditor/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": [{ "url": "https://example.com" }],
    "crawlPages": true,
    "maxPages": 25,
    "maxConcurrency": 5
  }'

The same audit with the official Python client:

from apify_client import ApifyClient

client = ApifyClient("YOUR_TOKEN")

run = client.actor("scrapers-hub/seo-accessibility-auditor").call(
    run_input={
        "startUrls": [{"url": "https://example.com"}],
        "crawlPages": True,
        "maxPages": 25,
    }
)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    if item.get("type") == "site-summary":
        print("Average score:", item["averageScore"])
        for issue in item["topIssues"][:5]:
            print(" -", issue)

Audit results can also be pushed automatically to Zapier, Make, Google Sheets and Slack, or delivered to any endpoint you control using Apify webhooks fired on run completion — which is how most teams post a score summary into a channel after each deploy.

---

💡 Best Use Cases for SEO and Accessibility Audit Data

🏗️ Pre-Launch and Pre-Migration Site Checks

Run the auditor across a staging build before it goes live. The crawlCoverage object immediately exposes accidental noindex tags, unintended canonicals and broken internal links, which are the three mistakes that most reliably destroy organic traffic during a migration.

📉 Regression Monitoring After Deploys

Schedule an audit after each release and compare averageScore and categoryAverages with the previous run. A sudden drop in the technical or meta-tags category points straight at a templating change, and topIssues names the specific problem and the pages it hit.

♿ WCAG and Accessibility Compliance Programmes

Organisations with accessibility obligations can track the accessibility entry in categoryAverages as a programme metric. Because the check covers language declaration, ARIA usage and skip links across every crawled page, progress is measurable rather than anecdotal.

🧾 Agency Audit Deliverables

Agencies can turn a run into a client-ready report in minutes. The distribution object gives the headline picture, topIssues gives the prioritised recommendations with affected URL counts, and per-page audit records provide the appendix detail clients ask for.

🕵️ Competitor Technical Benchmarking

Point the auditor at a competitor's site with the same maxPages setting you used for your own, then compare categoryAverages side by side. Differences in the schema and technical categories in particular reveal where a rival has invested in structured data or performance.

🗂️ Content Inventory and Indexation Analysis

Because every page record includes pageUrl, title and httpStatus, a crawl doubles as a content inventory. Combined with crawlCoverage, it answers the question most CMS reports cannot: how many of these pages are genuinely eligible to rank?

🔗 Internal Linking and Broken Link Cleanup

With auditLinks enabled, link status and descriptive anchor text are scored per page. Pages scoring poorly in the links category are where broken references and unhelpful anchors such as "click here" cluster, giving you a targeted cleanup list.

---

⚙️ Tips for Better SEO Auditing Results

• Raise maxPages before your first real audit. The default of 5 is a sample, not an audit. Set it to match the size of the site section you actually care about, and remember it is a hard ceiling on the crawl.
• Use excludeUrlPatterns aggressively. Faceted navigation and internal search can generate thousands of near-duplicate URLs that consume your entire crawl budget. Excluding them early is the single highest-impact setting change.
• Keep settings identical between comparison runs. Score comparisons are only meaningful when maxPages, startUrls and the audit toggles match. Save your configuration as a task so repeat runs are exact.
• Lower maxConcurrency for small or shared hosting. Five parallel requests is fine for most infrastructure, but a small site on cheap hosting can start returning 5xx errors under load — which will then be recorded as audit failures rather than crawler pressure.
• Read distribution before averageScore. A mean of 70 can mean every page is mediocre or that most are excellent and a handful are broken. The distribution tells you which, and the two situations need completely different responses.
• Disable categories you are not acting on. Narrower runs are faster and produce reports your team will actually read. Turn everything back on for periodic baselines.

---

🛠️ Troubleshooting

The crawl stopped after only a few pages. maxPages defaults to 5, so this is almost always the cause. Raise it, and confirm crawlPages is true — with crawling disabled the auditor only audits the URLs you supplied.

No internal links were discovered on a page. The crawler reads links from server-returned HTML. Sites that render navigation entirely client-side will expose few or no crawlable links, so supply a fuller startUrls list — from a sitemap, for example — rather than relying on link discovery.

Pages are being skipped that I expected to be audited. Check three things in order: whether respectRobotsTxt is blocking them, whether an entry in excludeUrlPatterns matches, and whether they live on a subdomain while includeSubdomains is false.

I am seeing 5xx errors that do not occur in a browser. This usually indicates the target server is struggling with concurrent requests. Reduce maxConcurrency to 1 or 2 and re-run — if the errors disappear, they were crawler-induced rather than genuine faults.

The performance scores do not match my lab tooling. The performance audit is a simulated check based on the page's returned markup and resources, not a browser-rendered measurement. Treat it as a comparative signal across pages in the same crawl rather than as a substitute for real user monitoring.

---

❓ Frequently Asked Questions About SEO and Accessibility Auditing

What does the SEO & Accessibility Auditor Scraper do? It crawls the pages you specify and produces a scored SEO and accessibility audit for each one across nine categories, plus a site-level summary record containing average scores, category averages, a prioritised issue list and a crawl-coverage breakdown.

Which audit categories are included? Meta tags, headings, content, technical, schema, performance, links, images and accessibility. Each has its own boolean input so it can be enabled or disabled independently.

How many pages will the auditor crawl? Up to maxPages, which defaults to 5. Raise it for a real site audit — it is a hard limit on the number of pages crawled and audited.

Does the crawler follow internal links automatically? Yes, when crawlPages is true, which is the default. It discovers internal links on each audited page and queues them, subject to your exclusion patterns and subdomain setting.

Does it respect robots.txt? Yes by default. respectRobotsTxt is true unless you turn it off, so the crawler honours the rules the site publishes for automated agents.

What is the difference between the two record types? Records with type of page are per-page audits containing pageUrl, title, httpStatus and the full audit object. The single record with type of site-summary aggregates the whole crawl.

How do I find the most important problems to fix? Read topIssues in the summary record. It lists every distinct issue sorted by the number of pages affected, and each entry includes the URLs where the issue occurs.

What does crawlCoverage tell me? It breaks the crawl into indexable pages versus non-indexable ones, counting noindex pages, canonicalised pages, 4xx errors, 5xx errors and redirects. It is the fastest way to spot indexation problems.

Can I audit accessibility on its own? Yes. Set auditAccessibility to true and turn the other eight category toggles off. The run will be considerably faster and the output focused on language declaration, ARIA usage and skip links.

Does the auditor use a headless browser? No. It works from HTTP requests and HTML parsing, which is why crawls are fast. The trade-off is that content injected purely by client-side JavaScript is not visible to the audit.

Can I exclude parts of a site from the crawl? Yes, using excludeUrlPatterns, which accepts regex and wildcard patterns. This is essential for keeping faceted navigation and internal search URLs out of a crawl.

How do I compare two audits over time? Run the auditor with identical settings and compare the averageScore and categoryAverages values from each run's site-summary record. Saving the configuration as an Apify task keeps the settings consistent.

Can I schedule recurring audits? Yes. Apify's scheduler runs the Actor on any cron expression, and a webhook can post the summary into Slack or push it to your own reporting service after each run.

Will crawling slow down my website? It can if concurrency is set too high for your infrastructure. maxConcurrency defaults to 5; reduce it when auditing small or shared-hosting sites to keep the crawl gentle.

What export formats are available? JSON, CSV, XLSX, XML, RSS and HTML from the Apify dataset UI, plus programmatic access via the dataset API. JSON is recommended because of the nested audit and crawlCoverage structures.

---

🆘 Support & Feedback

If a crawl behaves unexpectedly or an audit result looks wrong, please open a ticket on the Issues tab of this Actor and include the run ID plus an example URL. Reports filed there are tracked and fixed fastest.

Need additional audit checks, custom scoring weights, sitemap-driven crawling, or audit results delivered straight into your own reporting stack? Email scraperhubapi@gmail.com with your requirements.

If the SEO & Accessibility Auditor Scraper is useful to your team, please leave a review on the Actor page. Ratings and written feedback shape which checks get added next.

---

⚖️ Disclaimer

This SEO auditor accesses only publicly available web pages and reads what any ordinary visitor or search-engine crawler would receive. It does not log in, bypass authentication, defeat access controls, or attempt to reach content behind a paywall or login wall.

You are responsible for how you use this Actor. Crawl only sites you own or have permission to audit, honour robots.txt directives, and set maxConcurrency responsibly so that auditing does not degrade the performance of the target site. Where audited pages contain personal data — author bylines, contact details, testimonials — and you retain that data, GDPR, UK GDPR, CCPA and similar privacy laws apply, so establish a lawful basis and appropriate retention limits.

Respect the terms of service of every site you crawl and comply with all applicable laws in your jurisdiction. Audit scores produced by this Actor are diagnostic signals, not guarantees of search ranking, legal accessibility compliance or conformance certification — formal WCAG conformance requires manual assessment alongside automated checks.

If you believe personal data relating to you has been collected through this Actor and you would like it removed, contact scraperhubapi@gmail.com and we will action the request.