# Bulk Tech Stack Lookup (Wappalyzer Alternative) (`adamhaleworks/bulk-tech-stack-lookup`) Actor

Bulk tech stack lookup for any list of domains, a Wappalyzer and BuiltWith alternative: CMS, ecommerce platform, frameworks, analytics, CDN and hosting, plus email provider, SPF/DMARC grade, TLS certificate expiry, security headers grade, AI crawler policy and llms.txt. No browser, fast and cheap.

- **URL**: https://apify.com/adamhaleworks/bulk-tech-stack-lookup.md
- **Developed by:** [Adam Hale](https://apify.com/adamhaleworks) (community)
- **Categories:** Lead generation, Developer tools, SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Bulk Tech Stack Lookup: a Wappalyzer and BuiltWith alternative with DNS, email security and TLS checks

Paste a list of domains and get one clean row per domain:

- **Tech stack.** CMS, ecommerce platform, JavaScript framework, analytics, tag manager, CDN, hosting, marketing tools and 7,000+ other technologies, with versions.
- **Email setup.** Email provider (Google Workspace, Microsoft 365, Proofpoint and others), sending services found in SPF (SendGrid, Mailgun, HubSpot and others), and an SPF/DMARC grade with the DMARC reporting vendor.
- **SaaS tools from DNS verification records.** For example Atlassian, OpenAI, DocuSign or Stripe.
- **TLS certificate.** Issuer, expiry date and days left.
- **Security headers.** Score and A–F grade.
- **AI crawler policy.** Which AI bots robots.txt blocks, and whether llms.txt exists.
- **security.txt and ads.txt**, plus the sitemaps declared in robots.txt.

No browser is used, so it is fast and cheap. **Free while in launch; planned pay-per-event price: $3 per 1,000 domains**, everything included, charged only for successful lookups (dead, unreachable and blocked sites will stay free).

### What can I use it for?

- **Lead qualification and segmentation.** Find Shopify stores using Klaviyo, WordPress sites without a CDN, or companies on Microsoft 365 with a weak DMARC policy.
- **Agency prospecting.** Look for expiring TLS certificates, F-grade security headers, missing DMARC or an outdated CMS version.
- **Competitive and market research.** Measure technology adoption across a list of sites, such as Next.js versus WordPress, or which CDN is most common in an industry.
- **Email deliverability audits.** Check SPF lookup counts over the limit of 10, DMARC `p=none`, missing MTA-STS or BIMI.
- **AI and SEO policy tracking.** See who blocks GPTBot, ClaudeBot or Google-Extended, who publishes `llms.txt`, and who uses Content-Signal directives.
- **Domain list hygiene.** Flag dead domains (NXDOMAIN), unreachable sites and redirects to other domains. Dead and unreachable rows are not charged, now or under the planned pricing.

### What data does it return?

Every row has flat summary columns that are easy to use in a spreadsheet, plus nested detail objects.

| Field | Example |
|---|---|
| `domain`, `finalUrl`, `status` | `wordpress.org`, `https://wordpress.org/`, `ok` |
| `cms`, `cdn`, `hosting` | `WordPress`, `Cloudflare`, `Vercel` |
| `technologyNames` | `["WordPress", "PHP", "Nginx", "Google Tag Manager", ...]` |
| `technologies[]` | name, version, confidence, categories, website, `detectedBy`, CPE |
| `emailProvider`, `dnsProvider` | `Google Workspace`, `Amazon Route 53` |
| `emailSecurityGrade` | A–F, based on SPF policy and lookups, DMARC policy, MTA-STS and BIMI |
| `dns` | A/AAAA/NS/MX, SPF (policy, includes, lookup count), DMARC (policy, pct, reporting vendor), CAA issuers, SaaS verifications |
| `tls` | valid, issuer, notBefore/notAfter, `daysUntilExpiry`, lifetime, SANs, protocol, key type |
| `securityHeaders` | score 0–100, grade, present/missing headers, HSTS and CSP details, version leaks |
| `aiCrawlers` | blocked and allowed AI bots, bots named in robots.txt, `blocksAllAiTraining`, Content-Signal, llms.txt / llms-full.txt |
| `wellKnown` | security.txt (found, expires, signed), ads.txt (seller records, top ad systems), sitemap URLs |
| `page` | title, meta description, language, generator, canonical URL, Open Graph, JSON-LD types |
| `http` | status code, HTTP version, redirect chain, response time, server |

`status` is `ok`, `unreachable` (no HTTP answer over HTTPS or HTTP), `dns_error` (the domain does not exist), `blocked_by_site` (the site answered with 403, 429 or a bot-protection page), `disallowed_by_robots_txt` (robots.txt does not allow this actor to fetch the homepage) or `invalid_input`; `error` explains anything other than `ok`.

#### Example output (one row, shortened)

This row is from a real run on 2026-10-03:

```json
{
  "domain": "wordpress.org",
  "status": "ok",
  "finalUrl": "https://wordpress.org/",
  "title": "Blog Tool, Publishing Platform, and CMS – WordPress.org",
  "cms": "WordPress",
  "technologyNames": ["WordPress", "WordPress Block Editor", "Gutenberg", "MySQL", "PHP", "Nginx",
                      "Google Font API", "Google Tag Manager", "HSTS", "Let's Encrypt", "Open Graph", "RSS"],
  "emailSecurityGrade": "A",
  "securityHeadersGrade": "E",
  "tlsDaysUntilExpiry": 79,
  "blocksAiTraining": false,
  "hasLlmsTxt": true,
  "technologies": [
    {"name": "WordPress", "version": "7.2", "confidence": 100, "categories": ["CMS", "Blogs"],
     "detectedBy": ["dom", "headers", "html", "meta", "scriptSrc", "scripts"],
     "cpe": "cpe:2.3:a:wordpress:wordpress:*:*:*:*:*:*:*:*"},
    {"name": "Gutenberg", "version": "24.1.0", "confidence": 100, "categories": ["WordPress plugins", "Editors"],
     "detectedBy": ["dom", "scriptSrc"]}
  ],
  "dns": {
    "mx": ["smtp1-dca.wordpress.org", "smtp2-dca.wordpress.org"],
    "spf": {"policy": "fail", "includes": ["helpscoutemail.com"], "dnsLookupCount": 1, "exceedsLookupLimit": false},
    "dmarc": {"policy": "reject", "pct": 100, "reportingDomains": ["ag.dmarcian.com"], "reportingVendors": ["dmarcian"]},
    "sendingServices": ["Help Scout"]
  },
  "tls": {"valid": true, "issuer": "Let's Encrypt", "notAfter": "2026-12-22T19:43:57+00:00", "daysUntilExpiry": 79,
          "lifetimeDays": 90, "protocol": "TLSv1.3", "keyType": "EC secp256r1"},
  "securityHeaders": {"score": 30, "grade": "E", "present": ["Strict-Transport-Security", "X-Frame-Options"],
                      "missing": ["Content-Security-Policy", "X-Content-Type-Options", "Referrer-Policy", "..."],
                      "hsts": {"maxAge": 3600, "includeSubDomains": false, "preload": false}},
  "aiCrawlers": {"robotsTxtFound": true, "blockedBots": [], "namedBots": ["GPTBot", "ClaudeBot", "Google-Extended", "..."],
                 "blocksAllAiTraining": false, "llmsTxt": {"found": true, "title": "WordPress", "linkCount": 36}},
  "wellKnown": {"securityTxt": {"found": true, "expires": "2027-06-30T15:00:00.000Z", "hasPolicy": true},
                "sitemaps": ["https://wordpress.org/sitemap.xml", "https://wordpress.org/news-sitemap.xml", "..."]}
}
```

The Output tab offers four ready-made views: **Overview**, **Email & DNS security**, **TLS & headers** and **AI crawler policy**. You can export any of them as CSV, Excel, JSON or XML, or fetch them through the Apify API.

### How much does it cost?

**Free while in launch.** The actor charges no fee of its own right now; you pay only the Apify platform usage of your runs (the free Apify plan includes $5 of usage a month). The pay-per-event price below is planned and applies only once it is switched on; Apify notifies existing users before a price change takes effect.

Planned pricing, pay per event, only for successful lookups:

| Event | Planned price |
|---|---|
| Domain analyzed (successful lookup, all checks included) | **$0.003** (= $3 per 1,000 domains) |

- **Charged:** only rows with `status: ok`, where the homepage answered and was analysed.
- **Not charged:** everything else. That covers dead domains (`dns_error`), sites that gave no HTTP answer (`unreachable`), sites that refused our request (`blocked_by_site`), sites whose robots.txt does not allow the homepage fetch (`disallowed_by_robots_txt`), invalid inputs (for example `10.0.0.1` or `localhost`), timeouts and internal errors. These rows are still written so you can see what happened, with whatever DNS, email, TLS and robots.txt results could be collected, so list cleaning is free.
- **Cost cap:** once pricing is on, set **Maximum cost per run** and the actor stops cleanly when it reaches it. Uncharged rows don't count toward it.

For comparison (prices checked 2026-10-03): Wappalyzer's API starts at $250/month, BuiltWith's plans start at $295/month, and the most-used tech-stack actor on the Apify Store charges $0.10 per domain.

### How to use it

1. Paste domains, URLs or email addresses into **Domains or URLs**. Only the domain part of an email address is used, and duplicates are removed.
2. Optionally switch off checks you don't need. This makes runs faster but not cheaper.
3. Click **Start**. With the defaults, about 20 domains are checked in parallel.

#### Input example

```json
{
  "domains": ["apify.com", "https://www.shopify.com", "ops@stripe.com", "wordpress.org"],
  "maxConcurrency": 20,
  "minConfidence": 50
}
```

#### API example

```bash
curl -X POST "https://api.apify.com/v2/acts/adamhaleworks~bulk-tech-stack-lookup/run-sync-get-dataset-items?token=<TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"domains": ["apify.com", "wordpress.org"]}'
```

### How it works

- **Homepage fetch.** robots.txt is read first. HTTPS is tried first, then HTTP if HTTPS fails. Redirects are followed one hop at a time, and each new site's robots.txt is checked before the hop. If the homepage is readable, the actor also reads llms.txt, llms-full.txt, security.txt and ads.txt from the same origin, skipping any that robots.txt disallows.
- **Polite by design.** Every request identifies the actor as `bulk-tech-stack-lookup/1.0 (+<link to this Store page>)`; there is no browser disguise, no proxy and no retry with a different identity. It follows robots.txt rules for `bulk-tech-stack-lookup` (RFC 9309: a missing robots.txt allows everything, an unreadable one allows nothing). If the site answers with 403, keeps answering 429 or serves a bot-protection page (Cloudflare, DataDome and similar), the row is `blocked_by_site` and you are not charged.
- **Technology detection.** It matches the homepage HTML, scripts, meta tags, headers, cookies, DNS records and certificate issuer against the open [webappanalyzer](https://github.com/enthec/webappanalyzer) fingerprint set (GPL-3.0, 7,000+ technologies, version pinned for reproducible results). An Aho-Corasick prefilter makes detection take about 60 ms per page.
- **DNS checks.** These go through public resolvers (1.1.1.1 and 8.8.8.8 by default). SPF includes are followed recursively to count DNS lookups.
- **TLS check.** The certificate is read with a direct handshake. Invalid certificates (expired, self-signed or wrong host) are still parsed and reported with the reason.
- **Reliability.** Network errors, HTTP 429 and 5xx get retries with exponential backoff, and each domain has a time limit. Progress is saved, so a migrated run continues where it stopped.

### Limitations

- Only the homepage is analysed, and no JavaScript is executed. Technologies that appear only after client-side rendering or on inner pages can be missed. Wappalyzer's `js` and `xhr` rules are skipped for the same reason.
- `detectedBy: ["dns:TXT"]` means the company has verified the domain with that SaaS tool. It does not mean the tool runs on the website.
- Sites behind strict bot protection refuse automated clients. Those rows are `blocked_by_site`, with the DNS, email and TLS results but no tech stack, and they are not charged. The actor does not try to get around bot protection and has no proxy option.

### Is it legal? What about personal data?

The actor reads only public, technical, non-personal data: HTTP headers, HTML markup, public DNS records and TLS certificates. These are the same data a browser or `dig` sees. It does **not** collect names, email addresses or phone numbers. For example, security.txt is reported as present or absent, without storing contact values, and only the domains in DMARC reporting addresses are kept. Each domain gets only a handful of light requests: the homepage plus robots.txt, llms.txt, llms-full.txt, security.txt and ads.txt, and a retry only when one fails. They identify the actor by name, follow robots.txt, and parallelism is capped by the concurrency setting. You are responsible for using the results in line with the laws that apply to you.

This actor is not affiliated with or endorsed by Wappalyzer or BuiltWith. Their names are used only to describe what it can replace.

### FAQ

**Why do technology counts differ from Wappalyzer's browser extension?** The extension runs JavaScript in your browser and sees inner pages. This actor checks the raw homepage in bulk, which is far cheaper, and adds DNS, TLS, email and AI-policy data that the extension doesn't provide.

**Can I monitor domains over time?** Yes. Schedule the actor (for example weekly) and compare `tlsDaysUntilExpiry`, `technologyNames` or `emailSecurityGrade` between runs.

**Does it work for subdomains?** Yes. `shop.example.com` is fetched as given. Email and DNS records are read from the registrable domain (`example.com`).

**Why is a site `blocked_by_site` or `disallowed_by_robots_txt`?** The site refused automated clients, or its robots.txt does not allow the homepage to be fetched (for example `User-agent: *` with `Disallow: /`). The actor accepts both, and you are not charged for those rows.

**I run a website. How do I control this actor?** It sends `bulk-tech-stack-lookup` in its User-Agent. Add a robots.txt group for that name (`User-agent: bulk-tech-stack-lookup` with `Disallow: /`), or answer its requests with 403. Either way it records that and fetches nothing else from your site in that run. The TLS certificate check still runs, because it is a handshake rather than a page request.

**Something is wrong or missing?** Open an issue on the actor's Issues tab and include the domain. Fingerprint fixes are easy to add.

***

Author: Adam Hale

# Actor input Schema

## `domains` (type: `array`):

Domains, URLs or even email addresses (only the domain part is used). Duplicates are removed. Example: apify.com, https://www.shopify.com, ops@stripe.com.

## `detectTechnologies` (type: `boolean`):

Detect CMS, ecommerce platform, frameworks, analytics, CDN, hosting, marketing tools and 7,000+ other technologies from the homepage, headers, cookies, DNS and certificate.

## `checkDns` (type: `boolean`):

Email provider (MX), DNS provider (NS), sending services from SPF, SPF lookup count, DMARC policy and reporting vendor, MTA-STS, BIMI, CAA and SaaS domain-verification records.

## `checkTls` (type: `boolean`):

Certificate issuer, expiry date, days until expiry, validity, hostname match, protocol and key type.

## `checkSecurityHeaders` (type: `boolean`):

Score and grade for HSTS, CSP, X-Frame-Options, X-Content-Type-Options, Referrer-Policy, Permissions-Policy and COOP, plus server version leaks.

## `checkAiPolicy` (type: `boolean`):

Which AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and more) robots.txt blocks, Content-Signal directives, and whether llms.txt / llms-full.txt exist.

## `checkWellKnown` (type: `boolean`):

Presence and expiry of /.well-known/security.txt (contact values are not stored), ads.txt seller counts and ad systems, and sitemap URLs declared in robots.txt.

## `minConfidence` (type: `integer`):

Only report technologies detected with at least this confidence (0-100).

## `useOpenFingerprints` (type: `boolean`):

Load the community-maintained enthec/webappanalyzer fingerprints (7,000+ technologies) at the start of the run. If off, or if the download fails, a built-in set of about 60 common technologies is used.

## `fingerprintsVersion` (type: `string`):

Git commit, tag or branch of enthec/webappanalyzer to load. Pinned by default for reproducible results; use "main" for the newest rules.

## `maxConcurrency` (type: `integer`):

How many domains are checked in parallel.

## `requestTimeoutSecs` (type: `integer`):

Timeout for each HTTP request, TLS handshake and DNS lookup.

## `maxRetries` (type: `integer`):

Retries for network errors, HTTP 429 and 5xx responses (with exponential backoff).

## `dnsResolvers` (type: `array`):

Public resolvers used for DNS checks. Leave empty to use the system resolver.

## `includeRawHeaders` (type: `boolean`):

Add the homepage's HTTP response headers (without Set-Cookie) to each result.

## Actor input object example

```json
{
  "domains": [
    "apify.com",
    "wordpress.org",
    "shopify.com"
  ],
  "detectTechnologies": true,
  "checkDns": true,
  "checkTls": true,
  "checkSecurityHeaders": true,
  "checkAiPolicy": true,
  "checkWellKnown": true,
  "minConfidence": 50,
  "useOpenFingerprints": true,
  "fingerprintsVersion": "eea872af449e207e055398f7369d11ee48c8ea03",
  "maxConcurrency": 20,
  "requestTimeoutSecs": 15,
  "maxRetries": 2,
  "dnsResolvers": [
    "1.1.1.1",
    "8.8.8.8",
    "1.0.0.1",
    "8.8.4.4"
  ],
  "includeRawHeaders": false
}
```

# Actor output Schema

## `results` (type: `string`):

Dataset items, one per domain. Views: overview, email & DNS security, TLS & headers, AI crawler policy.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "apify.com",
        "wordpress.org",
        "shopify.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("adamhaleworks/bulk-tech-stack-lookup").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "domains": [
        "apify.com",
        "wordpress.org",
        "shopify.com",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("adamhaleworks/bulk-tech-stack-lookup").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "apify.com",
    "wordpress.org",
    "shopify.com"
  ]
}' |
apify call adamhaleworks/bulk-tech-stack-lookup --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,adamhaleworks/bulk-tech-stack-lookup"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/5bwaD9wix3kCZOYAQ/builds/CAF4saoHmHaCVwxeP/openapi.json