Website Technology Evidence

Analyze a list of public websites to qualify leads by CMS, ecommerce platform, framework, analytics, hosting and other visible technology signals. Results include technology names, categories, versions when exposed, and optional matched-signal evidence. Public DNS adds email and infrastructure signals.

Input

{"urls":["https://www.shopify.com/"],"detectViaDns":true,"includeEvidence":true}

Input	Default	Meaning
urls	required	HTTP(S) webpages, optional scheme; exact normalized duplicates are removed
detectViaDns	true	TXT, MX, NS, SOA and CNAME evidence
includeDescriptions	false	Add vendor descriptions from the catalogue
includeEvidence	false	Include matched signal names for each technology
maxConcurrency	10	Concurrent websites, 1–50
timeoutSecs	30	Website resolution/download/redirect deadline, 5–120 seconds per attempt
maxRetries	1	Transient source retries, 0–5; stable access denials are not retried
proxyConfiguration	disabled	Optional Apify Proxy or custom HTTP(S) gateway; requires useApifyProxy:true

Proxy access requires explicit useApifyProxy:true, including custom proxyUrls. Custom HTTP(S) gateways retain the same public-address, redirect, Host, SNI and TLS checks and cannot be combined with Apify country or group selectors. countryCode and apifyProxyCountry must agree and require enabled Apify Proxy. Unavailable initialization or an invalid selected gateway fails before source collection; there is no direct fallback. Tiered proxy configuration is rejected. Before each website attempt, the Actor checks the gateway through Apify's browser-info endpoint and records the observed public exit IP and country. An unknown or mismatched requested country stops that website without a source request or event fee. The check, page and redirects share one private proxy session. Providers can replace a session's IP; the result describes the observed check rather than guaranteeing permanent IP persistence. Custom gateways must also pass the exit check. checkAccess:false does not disable it. DNS fingerprint evidence comes from the Actor's resolver, separately from the page's proxy route.

Output and completion

One result per analyzed website includes url, finalUrl, success, partial, technologyCount, technologyNames, technologies, byCategory, byGroup, server, responseTimeMs, fetchedAt, dnsEvidence, detectionDiagnostics, and sourceEvidence. Proxy results additionally include the exit check's country, public IP, timestamp and response hash, without gateway credentials. responseTimeMs includes pacing, retries, proxy verification, page transfer, DNS evidence and detection; it is not just matching time.

DNS outcomes distinguish observed records, no-data responses and unknown timeouts/errors. A missing record is not silently treated as a negative signal. Truncation and detector limits mark the row partial. The SUMMARY record gives delivered, failed, partial, duplicate and undelivered counts plus the stop reason. The overview includes categories, server, HTTP status and source errors. The technology view expands each detected technology into its own row for exports. An all-failed source run fails; it does not report a successful empty scrape.

Failures are separate free diagnostic rows with success:false, errorType and a concise error. Already saved output remains available after interruption. The catalogue uses 7,628 pinned GPL entries, including entries whose browser-only signals cannot be observed. This HTTP-only Actor executes no page JavaScript. It does not promise browser-extension coverage or independently confirmed technology accuracy. Match confidence is a rule score, not a probability.

Charging

The charging unit is one unique, delivered, analyzed website, using the site-analyzed event price displayed in Apify Store. Zero-detection HTML may still be an analyzed result. Partial useful results are explicitly marked. Invalid, blocked, failed, duplicate and skipped websites have no Actor event fee. There is no additional Actor startup or per-technology fee.

Data is saved before its event is charged. A lost charge acknowledgement stops the run and retains its delivery journal; it is never retried automatically. Resuming the same persisted run skips acknowledged deliveries. Incomplete journals, changed input, and legacy storage without a run manifest stop for reconciliation; they are never overwritten to make a fresh charge attempt. The run checks the event budget before starting or delivering further work. Already in-flight source requests may finish after the last affordable result. The standard pay-per-event price includes run compute, storage and optional built-in Apify Proxy usage. Fees charged by external custom proxy providers are not included; check your provider's pricing. The event limit bounds Actor event fees; already running requests can still incur underlying platform usage. Separate post-run storage or Dataset/API operations may incur Apify usage charges.

Access and limits

Only public HTTP(S) destinations are accepted. Each redirect is checked again; the connection is pinned to a validated public IP and retains the original Host, TLS server name and certificate verification. If local DNS returns only a proxy's 198.18/15 fake-IP range, a fixed Google public DNS HTTPS lookup may supply a verified public answer. Other private destinations remain blocked.

On Apify, the SDK's built-in proxy gateway may use the platform's internal network when its endpoint matches the cloud-provided proxy settings. This exception applies only to the gateway; website targets, redirects and custom proxy gateways must still resolve to public addresses. Failed exit checks include a safe stage/reason and, when received, an HTTP status; credentials and upstream error text are never included.

Downloads are capped at 8 MiB, with at most 2 million HTML characters collected. Matching, selector allocation and regex execution have additional published limits in src/engine.js; diagnostics identify incomplete scans. DNS queries have a four-second deadline. Use only pages you are permitted to access and respect source terms and applicable data use requirements.

Memory and partial results

4096 MB is the default and recommended memory setting. 1024 MB is the minimum and 4096 MB the maximum. Real-website collection and US proxy verification have been tested in the cloud at 4096 MB. The 1024 MB setting has local test coverage only; its cloud behaviour has not been validated.

Low CPU availability can interrupt a scan even on a small page. The detector keeps a 40 ms wall-clock slice and a five-second detection deadline. Successful page retrieval can therefore still produce partial:true, with the specific limits recorded in detectionDiagnostics. Unknown DNS, truncated content and collection limits also remain partial. Use the technologies as observed evidence; missing technologies in a partial result are unresolved, not confirmed absent. More memory does not reveal browser-only signals or guarantee complete detection.

Development and testing

npm ci --ignore-scripts installs the pinned Node dependency graph. The Python requirements files describe only the standard-library schema-build stage; they do not replace the Node runtime lock. npm test checks the engine, collection, destination safety, cost caps and persistence/charge failures without new source traffic. Saved real-source differential tests are limited fixtures, not a claim of overall superiority. Native CLI validation is a separate build stage and must actually succeed before release.

GPL-3.0 source and fingerprint notices are in LICENSE and NOTICE. The detector implementation is independent; the community fingerprint data is licensed reuse.