# Website Tech Stack & Security Audit - SSL, Headers, Pixels (`optimistical_lectern/website-security-compliance-auditor`) Actor

Audit any list of websites in one run: CMS and tech stack, analytics and ad pixels, cookie consent tool, CDN/WAF, SSL certificate expiry and TLS version, HTTP security headers with an A+ to F grade, security.txt, and which AI crawlers robots.txt blocks. No browser, no proxies.

- **URL**: https://apify.com/optimistical_lectern/website-security-compliance-auditor.md
- **Developed by:** [Ivan Honer](https://apify.com/optimistical_lectern) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.25 / 1,000 websites

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## 🛡️ Website Tech Stack & Security Audit: SSL, Headers, Pixels

Audit any list of websites in one run and get one row per site with its **technology stack** (CMS, framework, CDN/WAF, analytics, ad pixels, chat, payments, cookie consent tool), its **SSL certificate and TLS setup**, its **HTTP security headers graded A+ to F**, its **security.txt**, and which **AI crawlers** its robots.txt blocks.

- ✅ **Tech stack from the page code**: script and asset URLs, meta tags and response headers, never words on the page, so a site that mentions Shopify is not called a Shopify store
- ✅ **SSL certificate details**: issuer, expiry and days left, TLS version, cipher, SANs, fingerprint, and whether the certificate is trusted
- ✅ **Security headers grade** with the missing headers and plain-English findings
- ✅ **AI crawler rules** read properly from robots.txt, including grouped rules
- ✅ **Fast and cheap**: plain HTTP and TLS, no browser, no proxies
- ✅ **Unreachable sites and invalid inputs are free**

### What does it do?

For every website it:

1. Connects over TLS and reads the certificate: who issued it, when it expires, whether it is trusted, the TLS version and cipher.
2. Loads the home page, following redirects, and checks whether plain `http://` redirects to `https://`.
3. Grades the HTTP security headers (HSTS, CSP, X-Frame-Options, X-Content-Type-Options, Referrer-Policy, Permissions-Policy, cross-origin policies) and flags version leaks and wildcard CORS.
4. Identifies the technology from the page code and headers.
5. Reads `/.well-known/security.txt`, `robots.txt` and `llms.txt`.
6. Finds the security or trust page and the privacy, subprocessor and DPA links, and lists the compliance frameworks those pages mention.

### Use cases

- 🎯 **Agency and freelancer prospecting.** Find sites with an expiring certificate, a failing security grade or an outdated stack, and lead with the fix.
- 🔎 **Technographic lead lists.** Every Shopify store, WordPress site or Next.js app in a list of domains, with their analytics and ad tools.
- 🛡️ **Vendor and portfolio security reviews.** A quick external hygiene check across hundreds of vendors or your own domains.
- ⏰ **Certificate monitoring.** Schedule it daily and alert when `sslDaysRemaining` drops below 14.
- 🤖 **AI crawler research.** Which publishers block GPTBot, ClaudeBot, PerplexityBot or Google-Extended.

### What data can you get?

| Group | Fields |
|---|---|
| Technology | `cmsOrFramework`, `wafOrCdn`, `allDetectedTechnologies`, `techCategories` (CMS and frameworks, analytics, ad pixels, consent tools, chat, payments), `cookieConsentBanner`, `gtmContainerIds`, `ga4MeasurementIds` |
| SSL / TLS | `sslValid`, `sslIssuer`, `sslSubject`, `sslValidFrom`, `sslValidTo`, `sslDaysRemaining`, `sslExpired`, `sslSelfSigned`, `tlsVersion`, `cipherSuite`, `sslSanDomains`, `sslSha256Fingerprint`, `sslError` |
| Security headers | `securityGrade` (A+ to F), `securityScore` (0–100), `securityHeaders`, `missingSecurityHeaders`, `headerWarnings`, `serverHeader`, `xPoweredBy`, `cookiesSetCount`, `cookiesMissingSecureOrHttpOnly`, `httpRedirectsToHttps` |
| security.txt | `hasSecurityTxt`, `securityTxtUrl`, `securityTxtContacts`, `securityTxtExpires`, `securityTxtExpired`, `securityTxtPolicy` |
| Crawlers | `hasRobotsTxt`, `hasLlmsTxt`, `blockedAiCrawlers` |
| Compliance pages | `trustCenterUrl`, `trustCenterPlatform`, `privacyPolicyUrl`, `termsOfServiceUrl`, `subprocessorsUrl`, `dpaUrl`, `complianceMentions` |
| Summary | `riskFlags`: the findings worth acting on, in plain English |
| Basics | `domain`, `finalUrl`, `httpStatusCode`, `title`, `redirectChain`, `status` |

### How the grade works

The score starts at 0 and adds points for a valid certificate on TLS 1.2 or 1.3 (25), HSTS (20), a Content-Security-Policy (20), framing protection (10), `nosniff` (10), a Referrer-Policy (7), a Permissions-Policy (5) and a cross-origin policy (3). It takes points off for HSTS with `max-age=0`, `'unsafe-inline'` in the CSP, and server or framework version leaks. A+ is 90 or more, A 80, B 70, C 55, D 40, and below that F.

### How to use it

1. Click **Try for free** (or **Start**) at the top of this page.
2. Paste websites into **Websites or domains**, one per line.
3. Leave the checks switched on, or turn off any you do not need for a faster run.
4. Press **Start**, then download the results from the **Output** tab as Excel, CSV or JSON.

### Input example

```json
{
  "websites": ["stripe.com", "wordpress.org", "shopify.com"],
  "checkSecurityTxt": true,
  "discoverCompliancePages": true,
  "checkRobotsAndAiBots": true
}
```

### Output example

Shortened; real values from an audit of wordpress.org in October 2026:

```json
{
  "domain": "wordpress.org",
  "status": "ok",
  "securityGrade": "C",
  "securityScore": 55,
  "cmsOrFramework": "WordPress",
  "allDetectedTechnologies": ["WordPress", "Google Tag Manager (GTM)"],
  "sslValid": true,
  "sslDaysRemaining": 68,
  "tlsVersion": "TLSv1.3",
  "httpRedirectsToHttps": true,
  "hasSecurityTxt": true,
  "trustCenterUrl": "https://wordpress.org/about/security/",
  "missingSecurityHeaders": ["Content-Security-Policy (CSP)", "..."],
  "riskFlags": ["Missing Content-Security-Policy (CSP) header"]
}
```

A news site that blocks AI crawlers returns, for example, `"blockedAiCrawlers": ["GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot", "Claude-SearchBot", "PerplexityBot", "Google-Extended", "CCBot", "Bytespider", "Applebot-Extended"]`.

### Statuses and errors

| `status` | Meaning | Charged? |
|---|---|---|
| `ok` | The site was audited. Individual checks that could not run show `null` | Yes |
| `unreachable` | The site could not be loaded over HTTPS or HTTP. `error` says why | No |
| `blocked` | The site's bot protection or rate limiting refused the request (for example Cloudflare's "Too many requests"). The certificate details are still included, but headers and technology are not graded | No |
| `invalid_input` | Not a domain or URL, or it points at a private or internal address | No |

### How much does it cost?

You pay per website audited. Unreachable and invalid inputs are free, and platform usage is included.

| Websites | Price on the Free and Bronze tiers |
|---|---|
| 100 | $0.30 |
| 1,000 | $3.00 |
| 10,000 | $30.00 |

### FAQ

#### Is this a penetration test or vulnerability scan?

No. It only reads what any visitor's browser receives: the certificate, response headers, the home page and a few public files. It sends no attack traffic, so it is safe to run on any public site, and it is not a substitute for a real security assessment.

#### Why was a site `blocked`?

Some large sites behind bot protection (Cloudflare, Akamai and others) refuse automated requests from cloud servers, or rate-limit them. The tool retries once, and if the site still refuses it returns `blocked` with the certificate details it could read, and does not charge for it. In testing, most business websites answered normally; a few very large consumer brands and news sites did not.

#### Why did it miss a technology I know the site uses?

It reads the HTML the server sends. Tools added later by JavaScript or a tag manager, and anything behind a login, may not be visible. That is also why the cookie-consent finding says a consent tool "may still load through a tag manager".

#### What counts as an AI crawler being blocked?

A robots.txt group that names the crawler and disallows the whole site (`Disallow: /`). Rules that only block some paths, or a general `User-agent: *` rule, are not counted.

#### Can I use it through the API or from an AI agent?

Yes:

```bash
curl -X POST "https://api.apify.com/v2/acts/optimistical_lectern~website-security-compliance-auditor/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"websites": ["stripe.com"]}'
```

From Python: `ApifyClient("YOUR_APIFY_TOKEN").actor("optimistical_lectern/website-security-compliance-auditor").call(run_input={"websites": ["stripe.com"]})`. AI agents can use it through Apify's MCP server at `mcp.apify.com`. It runs with limited permissions.

#### Something looks wrong?

Open an issue on the **Issues** tab with the website. Every report is read.

### Related tools

- [🌐 Domain WHOIS, DNS & Email Security](https://apify.com/optimistical_lectern/domain-whois-dns-lookup): registration, mail provider, SPF and DMARC for the same domains
- [📇 Website Contact Extractor](https://apify.com/optimistical_lectern/website-contact-extractor): who to contact at each site

# Actor input Schema

## `websites` (type: `array`):

One per line. Domains (stripe.com), URLs (https://stripe.com) and email addresses all work.

## `checkSecurityTxt` (type: `boolean`):

Probes /.well-known/security.txt and /security.txt (RFC 9116) for vulnerability disclosure contacts, encryption keys, and policy links.

## `discoverCompliancePages` (type: `boolean`):

Looks for a security or trust page and privacy, subprocessor and DPA links, and lists compliance frameworks the pages mention (SOC 2, ISO 27001, HIPAA, GDPR...).

## `checkRobotsAndAiBots` (type: `boolean`):

Checks /robots.txt and /llms.txt to see if GPTBot, ClaudeBot, PerplexityBot, or Google-Extended are blocked.

## `maxConcurrency` (type: `integer`):

Number of domains to audit in parallel (1 to 50).

## `timeoutSecs` (type: `integer`):

Per-request HTTP and TLS handshake timeout in seconds.

## Actor input object example

```json
{
  "websites": [
    "stripe.com",
    "wordpress.org",
    "shopify.com"
  ],
  "checkSecurityTxt": true,
  "discoverCompliancePages": true,
  "checkRobotsAndAiBots": true,
  "maxConcurrency": 15,
  "timeoutSecs": 12
}
```

# Actor output Schema

## `results` (type: `string`):

One item per website.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "websites": [
        "stripe.com",
        "wordpress.org",
        "shopify.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("optimistical_lectern/website-security-compliance-auditor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "websites": [
        "stripe.com",
        "wordpress.org",
        "shopify.com",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("optimistical_lectern/website-security-compliance-auditor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "websites": [
    "stripe.com",
    "wordpress.org",
    "shopify.com"
  ]
}' |
apify call optimistical_lectern/website-security-compliance-auditor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,optimistical_lectern/website-security-compliance-auditor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/0SFqRVvZOcXwmlZNG/builds/3gapcNngds1p4VJxe/openapi.json