# Website Tech Audit Scraper: robots.txt, Sitemaps & Meta (`arman-bd/website-tech-audit-scraper`) Actor

Audit any site's public technical files: robots.txt rules, sitemap discovery and URL counts, ads.txt sellers, security.txt contacts, meta tags, structured data and the detected tech stack with evidence.

- **URL**: https://apify.com/arman-bd/website-tech-audit-scraper.md
- **Developed by:** [Arman Hossain](https://apify.com/arman-bd) (community)
- **Categories:** Developer tools, SEO tools, MCP servers
- **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 audit scrapeds

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Website Tech Audit Scraper: robots.txt, Sitemaps & Meta

![Website Tech Audit: What a site runs on, and the evidence for every call: platform, generator, ads, sitemaps](https://api.apify.com/v2/key-value-stores/ZQOcNAOHrIgTacAmy/records/website-tech-audit-scraper.jpg)

**Website Tech Audit Scraper** reads the public technical files every site publishes for machines, `robots.txt`, sitemaps, `ads.txt`, `security.txt`, plus the homepage's meta tags, structured data and technology fingerprints, and returns one structured record per site.

Every file it reads is explicitly published for automated consumption. **No browser, no proxies, no login.** One request per enabled check, and `meta`, `schema` and `tech` share a single homepage fetch.

**Agent skill: [SKILL.md](https://api.apify.com/v2/key-value-stores/t7YoTxpZEJOWvw4Ug/records/website-tech-audit-scraper.md)**

```
https://api.apify.com/v2/key-value-stores/t7YoTxpZEJOWvw4Ug/records/website-tech-audit-scraper.md
```

### Every detection names its evidence

Most stack detectors hand you a list of technologies and ask you to trust it. This one ships the proof:

```json
{
 "technology": "WooCommerce",
 "category": "E-commerce",
 "confidence": "strong",
 "evidence": ["generator: WooCommerce", "html: woocommerce-inline-inline-css"]
}
```

Detection runs from an explicit signal table, each entry is a technology, a category and one matcher against a named haystack (response headers, `Set-Cookie`, `<meta name="generator">`, or the raw HTML). Whatever text the matcher hit is copied into `evidence`, so you can audit any claim without re-running the audit.

Signals that a page can trip **without running the technology** are marked weak and never decide `detectedPlatform` on their own. The real case that motivated this: `nytimes.com` embeds a single Wirecutter logo from a `/wp-content/` path. A naive detector calls the New York Times a WordPress site. This one reports WordPress as a `weak` signal with the image URL as evidence, and leaves `detectedPlatform` null.

### What you get

| Output field | Meaning |
|---|---|
| `site` | Origin audited |
| `robotsFound` | Whether a real `robots.txt` was served |
| `robotsRules` | Every user-agent group with its `allow`, `disallow` and `crawlDelay` |
| `crawlDelay`, `disallowedPaths` | The wildcard (`User-agent: *`) group, surfaced separately |
| `sitemapUrls` | Sitemaps declared in `robots.txt`, plus children found in sitemap indexes |
| `sitemapUrlCount` | URLs counted inside those sitemaps (when `expandSitemaps` is on) |
| `adsFound`, `adsSellerCount` | `ads.txt` presence and the number of authorised seller lines |
| `securityTxtFound`, `securityContacts` | `.well-known/security.txt` presence and its `Contact:` values |
| `metaTitle`, `metaDescription`, `ogTags` | Homepage `<title>`, meta description and every `og:*` tag |
| `schemaTypes` | Unique JSON-LD `@type` values found on the homepage |
| `detectedPlatform` | The single best-supported CMS / e-commerce / builder detection |
| `generator` | Raw `<meta name="generator">` content, joined |
| `technologies` | Full detection list with category, confidence and evidence |
| `scrapedAt` | Run timestamp |

A `RUN_SUMMARY` record in the key-value store holds per-run counts, the checks used, sites that failed, and sites that produced a **partial** audit.

### Common use cases

- **Bulk technical SEO audits.** Crawlability, sitemap coverage and meta hygiene across a client portfolio.
- **Pre-migration checks.** Confirm what a site declares before and after a replatform.
- **Lead qualification.** Filter a prospect list down to the sites running a platform you integrate with.
- **Attack-surface and OSINT work.** Collect security contacts and infrastructure signals passively.

### Quick start

Everything, on a handful of sites:

```json
{
 "sites": ["nytimes.com", "woocommerce.com", "vercel.com"]
}
```

Stack detection only, one request per site:

```json
{
 "sites": ["store-a.com", "store-b.com", "store-c.com"],
 "checks": ["tech", "meta"]
}
```

Full crawlability audit with sitemap URL counts:

```json
{
 "sites": ["example.com"],
 "checks": ["robots", "sitemap"],
 "expandSitemaps": true,
 "maxUrlsPerSitemap": 5000
}
```

### Input

| Field | Type | Default | Notes |
|---|---|---|---|
| `sites` | array | - | **Required.** Domains or URLs. Paths are stripped, every file audited lives at the origin. |
| `checks` | array | all seven | `robots`, `sitemap`, `ads`, `security`, `meta`, `schema`, `tech`. Fewer checks = fewer requests. |
| `expandSitemaps` | boolean | `false` | Follow sitemaps, walk indexes into children, and count URLs. |
| `maxUrlsPerSitemap` | integer | `1000` | Stop counting a site's sitemap URLs at this number. |

`meta`, `schema` and `tech` are served by one homepage fetch, so enabling all three costs the same as enabling one.

### Output example

```json
{
 "site": "https://woocommerce.com",
 "robotsFound": true,
 "robotsRules": [
 { "userAgents": ["*"], "allow": [], "disallow": ["/wp-content/uploads/wc-logs/"], "crawlDelay": null }
 ],
 "crawlDelay": null,
 "disallowedPaths": ["/wp-content/uploads/wc-logs/", "/*?add-to-cart="],
 "sitemapUrls": ["https://woocommerce.com/sitemap.xml", "https://woocommerce.com/post-sitemap.xml"],
 "sitemapUrlCount": 200,
 "adsFound": false,
 "adsSellerCount": null,
 "securityTxtFound": false,
 "securityContacts": [],
 "metaTitle": "WooCommerce",
 "metaDescription": "WooCommerce is a customizable, open-source ecommerce platform built on WordPress.",
 "ogTags": { "og:type": "website", "og:site_name": "WooCommerce" },
 "schemaTypes": ["BreadcrumbList", "Organization", "SearchAction"],
 "detectedPlatform": "WooCommerce",
 "generator": "WordPress 7.0.2; WooCommerce 11.1.0",
 "technologies": [
 { "technology": "WordPress", "category": "CMS", "confidence": "strong", "evidence": ["generator: WordPress 7.0.2"] },
 { "technology": "WooCommerce", "category": "E-commerce", "confidence": "strong", "evidence": ["generator: WooCommerce"] },
 { "technology": "WordPress VIP", "category": "Hosting", "confidence": "strong", "evidence": ["header: x-powered-by: WordPress VIP"] },
 { "technology": "Nginx", "category": "Web server", "confidence": "strong", "evidence": ["header: server: nginx"] }
 ],
 "scrapedAt": "2026-08-06T12:00:00.000Z"
}
```

### What it can detect

Around 60 technologies across CMS, e-commerce, site builders, static site generators, JS frameworks, hosting and CDN, web servers and runtimes, analytics, monitoring, support and payments, including WordPress, WooCommerce, Shopify, Magento, Salesforce Commerce Cloud, Drupal, Ghost, Wix, Squarespace, Webflow, Framer, Next.js, Nuxt, SvelteKit, Astro, Hugo, Vercel, Netlify, Cloudflare, Fastly, Akamai and more.

`detectedPlatform` picks the most specific platform category first, a WooCommerce store is also a WordPress site, and "WooCommerce" is the more useful answer.

### API example

```bash
curl -X POST "https://api.apify.com/v2/acts/arman-bd~website-tech-audit-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{
 "sites": ["nytimes.com", "woocommerce.com"],
 "checks": ["robots", "tech", "meta"]
 }'
```

### JavaScript example

```js
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_TOKEN' });
const run = await client.actor('arman-bd/website-tech-audit-scraper').call({
 sites: ['nytimes.com', 'woocommerce.com', 'vercel.com'],
 checks: ['tech'],
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const site of items) {
 console.log(`${site.site} → ${site.detectedPlatform ?? 'unknown'}`);
 for (const t of site.technologies) console.log(` ${t.technology} (${t.confidence}), ${t.evidence[0]}`);
}
```

### Limits and behaviour

- **HTML impostors are rejected.** A WordPress site answers `/ads.txt` with its themed 404 page, and some hosts serve that at HTTP 200. Treating "we got bytes" as "the file exists" would report a 234 KB HTML document as an ads.txt with hundreds of sellers, so both the status and the shape of the body are checked before a file counts as found.
- **`ads.txt` variables are not sellers.** Lines such as `OWNERDOMAIN=example.com` are declarations, not authorised sellers, and are excluded from `adsSellerCount`. A site can legitimately publish an ads.txt with zero sellers to declare that nobody may sell its inventory.
- **`robots.txt` is the source of truth for sitemaps.** `/sitemap.xml` is only probed when `robots.txt` declares none.
- **Gzipped sitemaps are listed, not expanded.** `.xml.gz` files are served as raw gzip with no `content-encoding` header, so their bytes cannot be read as text. They appear in `sitemapUrls` and are skipped by the URL counter.
- **A blocked homepage produces a partial audit, not a failure.** If the homepage returns 403 behind a bot wall, the `robots`/`ads`/`security` findings are still saved, `meta`/`schema`/`tech` are left empty, and the site is listed in `RUN_SUMMARY.partialAudits`.
- **Bad sites don't kill the run.** DNS failures and unreachable hosts are recorded in `RUN_SUMMARY.failures`; the Actor only errors out if *every* site fails.
- **Transient errors are retried.** 429, 5xx and network faults get three attempts with exponential backoff and a 30-second timeout.
- **Passive collection only.** Nothing is probed beyond documented well-known paths and the homepage. No authentication, no vulnerability scanning, no access-control bypass.

### FAQ

**Do I need a proxy?** No. Proxy configuration is not required to run this Actor.

**Why is `detectedPlatform` null when `technologies` lists a CMS?** Because only a weak signal fired, something a page can trip by embedding a third-party asset. Check the `evidence` field: if it is a single CDN or image URL, the site probably just links to that platform rather than running it.

**Why did a site return no technologies at all?** Some sites serve a JavaScript challenge or a minimal shell to non-browser clients. There is nothing in that response to match, and inventing a detection would defeat the point of the evidence field.

**Can I add my own signals?** The signal table is a single readable array at the top of `src/main.js`, one line per matcher. Fork and add a row.

**How many sites can I pass at once?** There is no hard cap. Sites are audited sequentially at up to five requests each, so hundreds per run is normal.

**Can I schedule it?** Yes, it is designed for scheduled runs. Diff on `detectedPlatform` or `robotsRules` to detect replatforms and crawlability regressions.

**Can I integrate it with something else?** Yes, Apify API, client libraries, webhooks, scheduled runs, dataset exports (JSON/CSV/Excel) or MCP. Output is structured JSON.

# Actor input Schema

## `sites` (type: `array`):

Domains or URLs. A bare domain ('nytimes.com'), a hostname with www, or a full page URL are all accepted. the path is stripped because every file audited here lives at the site root.

## `checks` (type: `array`):

Which parts of the audit to perform. Fewer checks means fewer requests per site. 'meta', 'schema' and 'tech' share a single homepage fetch.

## `expandSitemaps` (type: `boolean`):

Follow every discovered sitemap, walk sitemap indexes into their children, and count the URLs inside. Adds one request per sitemap file, so it is off by default.

## `maxUrlsPerSitemap` (type: `integer`):

Stop counting sitemap URLs for a site once this many have been seen. Guards against news publishers with millions of URLs. Only applies when 'Expand sitemaps' is on.

## Actor input object example

```json
{
  "sites": [
    "nytimes.com",
    "https://www.shopify.com/"
  ],
  "checks": [
    "robots",
    "sitemap",
    "ads",
    "security",
    "meta",
    "schema",
    "tech"
  ],
  "expandSitemaps": false,
  "maxUrlsPerSitemap": 1000
}
```

# Actor output Schema

## `items` (type: `string`):

Every record the run produced.

## `runsummary` (type: `string`):

The RUN\_SUMMARY record from the run's key-value store.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sites": [
        "nytimes.com",
        "woocommerce.com"
    ],
    "checks": [
        "robots",
        "sitemap",
        "ads",
        "security",
        "meta",
        "schema",
        "tech"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("arman-bd/website-tech-audit-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "sites": [
        "nytimes.com",
        "woocommerce.com",
    ],
    "checks": [
        "robots",
        "sitemap",
        "ads",
        "security",
        "meta",
        "schema",
        "tech",
    ],
}

# Run the Actor and wait for it to finish
run = client.actor("arman-bd/website-tech-audit-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sites": [
    "nytimes.com",
    "woocommerce.com"
  ],
  "checks": [
    "robots",
    "sitemap",
    "ads",
    "security",
    "meta",
    "schema",
    "tech"
  ]
}' |
apify call arman-bd/website-tech-audit-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,arman-bd/website-tech-audit-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/9zzXpGgpIUfXLT33i/builds/KQpz6uIjw0OChN2Wf/openapi.json