# Website Technology Detector | CMS, Frameworks & DNS (`peerless_columbine/independent-technology-evidence`) Actor

Identify website technologies from public HTTP evidence and DNS. Batch URLs, export structured results, and inspect match evidence, errors and partial scans. Pay per unique analyzed website.

- **URL**: https://apify.com/peerless_columbine/independent-technology-evidence.md
- **Developed by:** [tingyou333 zhuang](https://apify.com/peerless_columbine) (community)
- **Categories:** Developer tools, Lead generation, SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$20.00 / 1,000 analyzed sites

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website Technology Evidence

Analyze a list of public websites to qualify leads by CMS, ecommerce platform,
framework, analytics, hosting and other visible technology signals. Results
include technology names, categories, versions when exposed, and optional
matched-signal evidence. Public DNS adds email and infrastructure signals.

### Input

```json
{"urls":["https://www.shopify.com/"],"detectViaDns":true,"includeEvidence":true}
```

| Input | Default | Meaning |
|---|---|---|
| `urls` | required | HTTP(S) webpages, optional scheme; exact normalized duplicates are removed |
| `detectViaDns` | true | TXT, MX, NS, SOA and CNAME evidence |
| `includeDescriptions` | false | Add vendor descriptions from the catalogue |
| `includeEvidence` | false | Include matched signal names for each technology |
| `maxConcurrency` | 10 | Concurrent websites, 1–50 |
| `timeoutSecs` | 30 | Website resolution/download/redirect deadline, 5–120 seconds per attempt |
| `maxRetries` | 1 | Transient source retries, 0–5; stable access denials are not retried |
| `proxyConfiguration` | disabled | Optional Apify Proxy or custom HTTP(S) gateway; requires `useApifyProxy:true` |

Proxy access requires explicit `useApifyProxy:true`, including custom `proxyUrls`.
Custom HTTP(S) gateways retain the same public-address, redirect, Host, SNI and
TLS checks and cannot be combined with Apify country or group selectors.
`countryCode` and `apifyProxyCountry` must agree and require enabled Apify Proxy.
Unavailable initialization or an invalid selected gateway fails before source
collection; there is no direct fallback. Tiered proxy configuration is rejected.
Before each website attempt, the Actor checks the gateway through Apify's
browser-info endpoint and records the observed public exit IP and country.
An unknown or mismatched requested country stops that website without a
source request or event fee. The check, page and redirects share one private
proxy session. Providers can replace a session's IP; the result describes the
observed check rather than guaranteeing permanent IP persistence. Custom
gateways must also pass the exit check. `checkAccess:false` does not disable it.
DNS fingerprint evidence comes from the Actor's resolver, separately from the
page's proxy route.

### Output and completion

One result per analyzed website includes `url`, `finalUrl`, `success`, `partial`,
`technologyCount`, `technologyNames`, `technologies`, `byCategory`, `byGroup`,
`server`, `responseTimeMs`, `fetchedAt`, `dnsEvidence`, `detectionDiagnostics`,
and `sourceEvidence`. Proxy results additionally include the exit check's
country, public IP, timestamp and response hash, without gateway credentials.
`responseTimeMs` includes pacing, retries, proxy verification, page transfer,
DNS evidence and detection; it is not just matching time.

DNS outcomes distinguish observed records, no-data responses and unknown
timeouts/errors. A missing record is not silently treated as a negative signal.
Truncation and detector limits mark the row partial. The SUMMARY record gives
delivered, failed, partial, duplicate and undelivered counts plus the stop reason.
The overview includes categories, server, HTTP status and source errors. The
technology view expands each detected technology into its own row for exports.
An all-failed source run fails; it does not report a successful empty scrape.

Failures are separate free diagnostic rows with `success:false`, `errorType`
and a concise error. Already saved output remains available after interruption.
The catalogue uses 7,628 pinned GPL entries, including entries whose browser-only
signals cannot be observed. This HTTP-only Actor executes no page JavaScript.
It does not promise browser-extension coverage or independently confirmed
technology accuracy. Match confidence is a rule score, not a probability.

### Charging

The charging unit is one unique, delivered, analyzed website, using the
`site-analyzed` event price displayed in Apify Store.
Zero-detection HTML may still be an analyzed result. Partial useful results are
explicitly marked. Invalid, blocked, failed, duplicate and skipped websites have
no Actor event fee. There is no additional Actor startup or per-technology fee.

Data is saved before its event is charged. A lost charge acknowledgement stops
the run and retains its delivery journal; it is never retried automatically.
Resuming the same persisted run skips acknowledged deliveries. Incomplete
journals, changed input, and legacy storage without a run manifest stop for
reconciliation; they are never overwritten to make a fresh charge attempt.
The run checks the event budget before starting or delivering further work.
Already in-flight source requests may finish after the last affordable result.
The standard pay-per-event price includes run compute, storage and optional
built-in Apify Proxy usage. Fees charged by external custom proxy providers
are not included; check your provider's pricing. The event limit bounds Actor
event fees; already running requests can still incur underlying platform usage.
Separate post-run storage or Dataset/API operations may incur Apify usage charges.

### Access and limits

Only public HTTP(S) destinations are accepted. Each redirect is checked again;
the connection is pinned to a validated public IP and retains the original Host,
TLS server name and certificate verification. If local DNS returns only a proxy's
198.18/15 fake-IP range, a fixed Google public DNS HTTPS lookup may supply a
verified public answer. Other private destinations remain blocked.

On Apify, the SDK's built-in proxy gateway may use the platform's internal
network when its endpoint matches the cloud-provided proxy settings. This
exception applies only to the gateway; website targets, redirects and custom
proxy gateways must still resolve to public addresses. Failed exit checks
include a safe stage/reason and, when received, an HTTP status; credentials and
upstream error text are never included.

Downloads are capped at 8 MiB, with at most 2 million HTML characters collected.
Matching, selector allocation and regex execution have additional published
limits in `src/engine.js`; diagnostics identify incomplete scans. DNS queries
have a four-second deadline. Use only pages you are permitted to access and
respect source terms and applicable data use requirements.

### Memory and partial results

**4096 MB** is the default and recommended memory setting. **1024 MB** is the
minimum and **4096 MB** the maximum. Real-website collection and US proxy
verification have been tested in the cloud at 4096 MB. The 1024 MB setting has
local test coverage only; its cloud behaviour has not been validated.

Low CPU availability can interrupt a scan even on a small page. The detector
keeps a 40 ms wall-clock slice and a five-second detection deadline. Successful
page retrieval can therefore still produce `partial:true`, with the specific
limits recorded in `detectionDiagnostics`. Unknown DNS, truncated content and
collection limits also remain partial. Use the technologies as observed evidence;
missing technologies in a partial result are unresolved, not confirmed absent.
More memory does not reveal browser-only signals or guarantee complete detection.

### Development and testing

`npm ci --ignore-scripts` installs the pinned Node dependency graph. The Python
requirements files describe only the standard-library schema-build stage;
they do not replace the Node runtime lock. `npm test` checks
the engine, collection, destination safety, cost caps and persistence/charge
failures without new source traffic. Saved real-source differential tests are
limited fixtures, not a claim of overall superiority. Native CLI validation is a
separate build stage and must actually succeed before release.

GPL-3.0 source and fingerprint notices are in LICENSE and NOTICE. The detector
implementation is independent; the community fingerprint data is licensed reuse.

# Actor input Schema

## `urls` (type: `array`):

Public HTTP(S) webpages; optional scheme. Exact normalized duplicates are fetched once.

## `detectViaDns` (type: `boolean`):

TXT, MX, NS, SOA and CNAME signals. Unknown and no-data results are distinguished.

## `includeDescriptions` (type: `boolean`):

Include catalogue descriptions for detected technologies.

## `includeEvidence` (type: `boolean`):

Include the passive signals that matched each detected technology.

## `maxConcurrency` (type: `integer`):

Maximum number of websites processed concurrently.

## `timeoutSecs` (type: `integer`):

Time limit in seconds for each website request.

## `maxRetries` (type: `integer`):

Maximum retries after a website collection attempt fails.

## `proxyConfiguration` (type: `object`):

Optional built-in Apify Proxy or custom HTTP(S) gateway. Built-in Apify Proxy usage during standard pay-per-event runs is included in the event price. Custom providers may charge separately; check your provider's pricing. Separate post-run storage or Dataset/API charges may also apply.

## Actor input object example

```json
{
  "urls": [
    "https://www.shopify.com/"
  ],
  "detectViaDns": true,
  "includeDescriptions": false,
  "includeEvidence": false,
  "maxConcurrency": 10,
  "timeoutSecs": 30,
  "maxRetries": 1,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://www.shopify.com/"
    ],
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("peerless_columbine/independent-technology-evidence").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "urls": ["https://www.shopify.com/"],
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("peerless_columbine/independent-technology-evidence").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://www.shopify.com/"
  ],
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call peerless_columbine/independent-technology-evidence --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,peerless_columbine/independent-technology-evidence"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/G8NOjiC8hPYtayaff/builds/YTCXZVhD86hqHB0pb/openapi.json