SEO Auditor

🔍 SEO Auditor – Technical SEO Site Crawler, On-Page Audit & Issue Scoring

The SEO auditor crawls a website and produces a scored technical SEO audit covering meta tags, headings, images, links, structured data, content quality, accessibility, performance and technical health. Point this SEO auditor at a start URL, choose which audit modules to run, and it returns one record per crawled page with a full audit object, plus a summary record with average scores, a score distribution, crawl coverage and a ranked list of top issues.

It is built for the practical case where you need an on-page and technical SEO picture quickly and repeatably — before a site migration, during a client audit, as a pre-deploy regression check, or on a schedule to catch regressions that creep in with routine content publishing. Everything is configurable per module, so you can run a narrow meta-tags-only pass over hundreds of pages, or a full nine-module audit over a handful.

---

📊 What Data Can You Extract with This SEO Auditor?

The actor writes two record types: page records and a summary record. Together they cover five categories:

Category	Fields	What you get
🏷️ Record identity	type	Distinguishes a page record from the run summary, so you can split the dataset in one filter
📄 Page identity	pageUrl, title, httpStatus	The audited URL, its title tag, and the HTTP status code returned
🔬 Audit detail	audit	The full per-page audit object containing the overall score and per-category findings
📈 Run summary	pagesCrawled, averageScore, categoryAverages, distribution, crawlCoverage	How many pages were crawled, the average score, per-category averages, the score distribution and coverage statistics
🚨 Issues	topIssues, error	A ranked array of the most common problems across the crawl, and any per-page error message

The categoryAverages object on the summary record is the field that changes how an audit gets used. A single overall score tells you a site is at 52 out of 100 but not what to do about it. Per-category averages tell you whether the deficit sits in meta tags, headings, images, accessibility or schema — which is the difference between a report and a work plan.

---

🌟 Key Features of the SEO Auditor

Feature	Description
🧩 Nine independent audit modules	Meta tags, headings, images, links, content, schema, technical, performance and accessibility, each toggled separately
🕷️ Configurable crawler	crawlPages follows links from your start URLs, with maxPages capping the crawl and includeSubdomains widening it
🤖 robots.txt compliance	respectRobotsTxt defaults to true, so the crawl honours the site's own directives
🚫 URL exclusion patterns	excludeUrlPatterns keeps the crawl off admin paths, faceted navigation and other low-value URLs
📊 Scored output	Each page receives an overall score, and the run summary aggregates averages, distribution and per-category breakdowns
🥇 Ranked issue list	topIssues surfaces the highest-frequency problems across the crawl so remediation can be prioritised
⚡ Parallel processing	maxConcurrency controls how many pages are audited simultaneously
♿ Accessibility checks	Accessibility auditing is enabled by default alongside the SEO modules
🧾 Two-record output	Per-page records plus one summary record, distinguished by the type field

---

🚀 Why Choose This SEO Auditor?

Modular audits, not an all-or-nothing report. Every audit category is an independent boolean. Running a meta-tags-and-headings-only pass over 200 pages is a completely different job from a full nine-module audit over five, and both are one input change apart. That makes the actor usable both for broad sweeps and for targeted checks.

A summary record you can act on. Alongside per-page detail, the run produces averageScore, categoryAverages, distribution, crawlCoverage and topIssues. The distribution matters as much as the average: a site where every page scores 60 needs different work from one where half score 90 and half score 30.

Crawl control that respects the site. respectRobotsTxt defaults to true, includeSubdomains defaults to false, and maxPages defaults to a conservative 5. The defaults are deliberately polite, and every constraint is explicit rather than hidden.

Structured output built for automation. Because the dataset is JSON with a stable schema, the audit slots into CI pipelines, scheduled monitoring and reporting dashboards without manual export steps. Comparing averageScore and categoryAverages between runs gives you regression detection for free.

---

📥 Input

{
  "startUrls": ["https://example.com"],
  "crawlPages": true,
  "maxPages": 5,
  "maxConcurrency": 5,
  "includeSubdomains": false,
  "respectRobotsTxt": true,
  "excludeUrlPatterns": [],
  "auditMetaTags": true,
  "auditHeadings": true,
  "auditImages": true,
  "auditLinks": true,
  "auditContent": true,
  "auditSchema": true,
  "auditTechnical": true,
  "auditPerformance": true,
  "auditAccessibility": true
}

🔧 SEO Auditor Input Fields

Field	Type	Required	Default	Description
startUrls	array	No	["https://example.com"]	URLs to start the audit from
auditAccessibility	boolean	No	true	Enable accessibility auditing
auditContent	boolean	No	true	Enable content length and quality auditing
auditHeadings	boolean	No	true	Enable headings structure auditing
auditImages	boolean	No	true	Enable images and alt tags auditing
auditLinks	boolean	No	true	Enable internal and external links auditing
auditMetaTags	boolean	No	true	Enable meta tags auditing
auditPerformance	boolean	No	true	Enable basic performance metrics auditing
auditSchema	boolean	No	true	Enable structured data auditing
auditTechnical	boolean	No	true	Enable technical SEO auditing
crawlPages	boolean	No	true	Whether to crawl other pages linked from start URLs
includeSubdomains	boolean	No	false	Whether to include subdomains when crawling
respectRobotsTxt	boolean	No	true	Respect rules defined in robots.txt
maxPages	integer	No	5	Maximum number of pages to crawl
maxConcurrency	integer	No	5	Maximum number of pages to process concurrently
excludeUrlPatterns	array	No	[]	List of URL patterns to exclude from crawling

💡 Input Examples

Single-page deep audit, no crawling:

{
  "startUrls": ["https://example.com/pricing"],
  "crawlPages": false
}

Broad meta-and-headings sweep across a larger site:

{
  "startUrls": ["https://example.com"],
  "crawlPages": true,
  "maxPages": 250,
  "maxConcurrency": 10,
  "auditMetaTags": true,
  "auditHeadings": true,
  "auditImages": false,
  "auditLinks": false,
  "auditContent": false,
  "auditSchema": false,
  "auditTechnical": false,
  "auditPerformance": false,
  "auditAccessibility": false,
  "excludeUrlPatterns": ["/tag/", "/author/", "?utm_"]
}

Full audit including subdomains, ignoring robots directives for an internal staging site you own:

{
  "startUrls": ["https://staging.example.com"],
  "crawlPages": true,
  "includeSubdomains": true,
  "respectRobotsTxt": false,
  "maxPages": 100
}

---

📤 Output

{
  "type": "page",
  "pageUrl": "https://example.com",
  "title": "Example Domain",
  "httpStatus": 200,
  "audit": {
    "url": "https://example.com",
    "title": "Example Domain",
    "httpStatus": 200,
    "overallScore": 52,
    "categories": { "...": "per-category findings" }
  }
}

🧾 SEO Auditor Page Record Fields

Field	Type	Description
type	string | null	Record type — page for per-page audit records
pageUrl	string | null	URL of the audited page
title	string | null	Title of the page
httpStatus	integer | null	HTTP status code returned
audit	object | null	Full audit object for the page, including the overall score and per-category findings
error	string | null	Error message, if the page failed to process

🧾 SEO Auditor Summary Record Fields

Field	Type	Description
type	string | null	Record type identifying the summary record
pagesCrawled	integer | null	Number of pages crawled in the run
averageScore	integer | null	Average score across all audited pages
categoryAverages	object | null	Average score per audit category
distribution	object | null	Distribution of page scores across the crawl
crawlCoverage	object | null	Coverage statistics for the crawl
topIssues	array | null	Ranked list of the most common issues found

---

💻 How to Use the SEO Auditor (Step by Step)

Step 1: Choose your start URLs

startUrls takes an array. For a whole-site audit, one homepage URL is usually enough — the crawler follows links from there. For a targeted audit, list the specific pages you care about and set crawlPages to false so the actor audits exactly those and nothing else. Mixing approaches works too: several section landing pages as seeds gives better coverage of a large site than a single homepage.

Step 2: Decide crawl scope

crawlPages defaults to true, maxPages to 5 and includeSubdomains to false. Those defaults are for a quick smoke test, not a real audit. For a genuine site review, raise maxPages to match the size of the site section you are examining. Enable includeSubdomains only when subdomains are genuinely part of the same property — otherwise the crawl can wander into unrelated content.

Step 3: Exclude the URLs that waste crawl budget

excludeUrlPatterns is the input most people skip and most people should use. Tag archives, author pages, faceted navigation, print views, session parameters and paginated listings consume crawl budget and generate near-duplicate audit records. Excluding them concentrates the audit on pages that actually matter and makes averageScore more meaningful.

Step 4: Select the audit modules you need

All nine modules default to true. A full audit is the right starting point for an unfamiliar site. Once you know where the problems are, switching off irrelevant modules makes subsequent runs faster and the output easier to read — an images-and-accessibility-only pass, for example, is a focused way to work through alt text remediation.

Step 5: Set concurrency responsibly

maxConcurrency defaults to 5 pages in parallel. That is a reasonable balance for most sites. Raise it for large crawls on infrastructure that can handle the load; lower it for small or shared hosting where a burst of parallel requests could affect real users. Combined with respectRobotsTxt, this is how you stay a good citizen while auditing.

Step 6: Run the audit and read the summary first

Start the run, then filter the dataset on type to isolate the summary record. Read averageScore for the headline, categoryAverages to find which discipline is weakest, distribution to see whether problems are concentrated or systemic, and topIssues for the ranked remediation list. This takes a minute and tells you where to spend the next hour.

Step 7: Drill into page records and re-run after fixes

With priorities set, filter to page records and sort by the overall score inside audit to find the worst offenders. Check httpStatus for anything that is not 200 and error for pages that failed outright. After remediation, re-run with identical inputs and compare averageScore and categoryAverages — a like-for-like comparison is the cleanest evidence that the work landed.

---

🔌 API Access & Integrations

Run the SEO auditor and get the audit dataset back in one call:

curl -X POST "https://api.apify.com/v2/acts/scrapers-hub~seo-auditor/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": ["https://example.com"],
    "crawlPages": true,
    "maxPages": 25,
    "maxConcurrency": 5
  }'

Python, using the official client:

from apify_client import ApifyClient

client = ApifyClient("YOUR_TOKEN")

run = client.actor("scrapers-hub/seo-auditor").call(
    run_input={
        "startUrls": ["https://example.com"],
        "crawlPages": True,
        "maxPages": 50,
        "excludeUrlPatterns": ["/tag/", "/author/"],
    }
)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    if item.get("type") == "page":
        print(item["httpStatus"], item["pageUrl"], item["title"])
    else:
        print("Average score:", item.get("averageScore"))
        print("Top issues:", item.get("topIssues"))

Audit results can be routed into Zapier, Make, Google Sheets or Slack, or pushed to your own monitoring system with an Apify webhook triggered on run completion.

---

💡 Best Use Cases for SEO Audit Data

🏗️ Pre-migration and pre-launch checks

Before a replatform or redesign goes live, run the SEO auditor over the staging site and the production site with identical inputs. Comparing averageScore, categoryAverages and topIssues between the two exposes regressions — dropped meta descriptions, broken heading hierarchies, missing structured data — while they are still cheap to fix.

📋 Client audits and proposal evidence

Agencies pitching SEO work need concrete findings, not generalities. A run over a prospect's site produces topIssues as a ranked problem list and distribution as evidence of how widespread each problem is. That is a far more persuasive artefact than a checklist, and it is repeatable at the end of the engagement to demonstrate improvement.

🔁 Continuous regression monitoring

Schedule the auditor weekly against your key templates and alert on drops in averageScore or specific categoryAverages. Content teams routinely publish pages with missing meta descriptions or broken heading order; scheduled auditing catches that within days rather than at the next quarterly review.

♿ Accessibility compliance sweeps

With auditAccessibility enabled and the other modules switched off, the actor becomes a focused accessibility scanner. Combined with auditImages for alt-text coverage, it produces the evidence base for a WCAG remediation backlog, prioritised by the frequency data in topIssues.

🧬 Structured data validation at scale

auditSchema checks structured data across the crawl. For sites relying on rich results — product, article, FAQ, local business markup — running this across every template catches missing or malformed schema on pages nobody thought to check, which is usually where rich result eligibility quietly disappears.

🔗 Internal linking and link health review

auditLinks examines internal and external linking. Aggregating findings across a crawl reveals orphaned sections, excessive outbound linking on thin pages, and broken destinations. Cross-referencing with httpStatus on page records shows where the crawler itself hit non-200 responses.

📉 Content quality triage

auditContent scores content length and quality. On large content sites, sorting page records by content findings identifies thin pages that are candidates for consolidation, expansion or removal — the exact inventory decision that content pruning projects need and rarely have data for.

---

⚙️ Tips for Better SEO Auditing Results

• Raise maxPages before your first real audit. The default of 5 is a smoke test. Set it to match the section you are actually reviewing, or the summary statistics will not be representative.
• Use excludeUrlPatterns aggressively. Tag archives, faceted URLs and tracking parameters produce near-duplicate records that dilute averageScore and crowd out genuine findings.
• Disable modules you are not acting on. A focused two-module run is faster and produces output a human can read end to end.
• Keep respectRobotsTxt enabled on sites you do not own. Disable it only on your own staging environments where you control the directives.
• Read distribution alongside averageScore. An average hides whether problems are systemic across every page or concentrated in one bad template — and the fix is completely different in each case.
• Re-run with identical inputs after fixes. Changing maxPages or module flags between runs makes score comparisons meaningless. Keep the configuration constant to measure change.

---

🛠️ Troubleshooting

The audit only covered a few pages. maxPages defaults to 5. Raise it. Also confirm crawlPages is true — with it set to false, only the URLs in startUrls are audited, which is the intended behaviour for targeted checks.

Pages I expected are missing from the crawl. Check three things: whether respectRobotsTxt is excluding them via the site's own directives, whether they match an entry in excludeUrlPatterns, and whether they live on a subdomain while includeSubdomains is false.

Some records have a populated error field. That page failed to process — typically a timeout, a network failure, or a response the parser could not handle. Check httpStatus on the same record for a clue, and re-run those URLs individually with crawlPages set to false.

Scores look lower than I expected. The score reflects every enabled module. A page with excellent meta tags but no structured data and poor accessibility will score modestly overall. Read categoryAverages on the summary record to see which module is pulling the number down before concluding the page is bad.

The crawl is slow on a large site. Raise maxConcurrency, and reduce the number of enabled audit modules. Both directly cut runtime. On sites you do not own, raise concurrency cautiously so the crawl does not affect real visitors.

---

❓ Frequently Asked Questions About SEO Auditing

What does the SEO auditor check? Nine categories: meta tags, heading structure, images and alt tags, internal and external links, content length and quality, structured data, technical SEO, basic performance metrics and accessibility. Each is independently toggleable.

How many pages can it audit in one run? As many as maxPages allows. The default is 5, which is intended as a quick check rather than a full audit — raise it for real work.

Can I audit a single page without crawling? Yes. Put the page in startUrls and set crawlPages to false. Only the listed URLs will be audited.

Does it respect robots.txt? Yes, by default. respectRobotsTxt is true unless you explicitly disable it, which you should only do on sites you own.

What is the difference between the page records and the summary record? The type field distinguishes them. Page records carry pageUrl, title, httpStatus and the per-page audit object. The summary record carries pagesCrawled, averageScore, categoryAverages, distribution, crawlCoverage and topIssues.

How is the score calculated? Each page receives an overall score inside its audit object based on the enabled modules, and the summary aggregates these into averageScore and per-module categoryAverages.

Can I audit subdomains? Yes. Set includeSubdomains to true. It defaults to false so that crawls stay within the exact hostname you provided.

How do I stop the crawler visiting low-value URLs? Add patterns to excludeUrlPatterns — tag and author archives, faceted navigation parameters, print views and tracking query strings are the usual candidates.

Does the auditor render JavaScript? The actor works over HTTP requests with HTML parsing. Content that only exists after client-side rendering may not be visible to the audit, so heavily JavaScript-dependent sites should be interpreted with that in mind.

Can I schedule recurring audits? Yes. Apify Schedules can run the actor on any cron expression, and a webhook can push each run's summary into Slack or your own dashboard for trend tracking.

What does crawlCoverage tell me? It reports coverage statistics for the crawl, which is how you confirm the audit actually reached a representative portion of the site rather than stalling early.

How do I prioritise which issues to fix first? Start with topIssues on the summary record — it is already ranked by prevalence — then cross-reference categoryAverages to see which discipline offers the biggest aggregate gain.

Can I export the audit to a spreadsheet? Yes. Apify datasets export to CSV, Excel, JSON, XML and HTML. Note that audit, categoryAverages, distribution and crawlCoverage are nested objects, so flatten them before a spreadsheet export.

Why does httpStatus matter in an SEO audit? Non-200 responses on pages that should be reachable are among the highest-impact technical SEO problems there are. Filtering page records by httpStatus is the fastest way to surface them.

Does the SEO auditor need proxies? No. It runs without proxy configuration, so there is nothing to set up before your first audit.

---

🆘 Support & Feedback

If a crawl behaves unexpectedly, an audit module produces results you cannot explain, or a run fails, please open a report on the Issues tab with your input configuration and the run ID.

For custom work — additional audit checks, bespoke scoring weights, integration into an existing reporting pipeline, or a private build for your agency — email scraperhubapi@gmail.com.

If this SEO auditor is useful in your workflow, a review on the actor page is genuinely appreciated and helps decide which checks get added next.

---

⚖️ Disclaimer

This SEO auditor crawls only publicly accessible web pages and, by default, honours the directives in each site's robots.txt. It does not log into websites, bypass authentication, or access content behind a paywall or membership gate.

Audited pages may incidentally contain personal data — author names, contact details, testimonials. Where they do, GDPR, UK GDPR, CCPA and comparable privacy regimes apply to your processing of the audit output, and you are responsible for establishing a lawful basis and honouring erasure requests. You are also responsible for ensuring you have permission to crawl any site you audit, for respecting its terms of service, and for setting maxConcurrency at a level that does not degrade service for real users.

To request removal of specific data collected by this actor, email scraperhubapi@gmail.com with the details.