Website Intelligence API

What does Website Intelligence API do?

Website Intelligence API creates a structured technical and SEO report for any public website. It identifies website technologies and comment platforms, extracts Schema.org data, checks structured-data quality, evaluates robots.txt, and inspects XML sitemaps.

Use the Actor in Apify Console or connect it to your workflow through the Apify API. Results can be scheduled, monitored, exported, sent through webhooks, and integrated with other applications. To try it, enter a public page such as apify.com.

Why use Website Intelligence API?

Website Intelligence API brings several common website checks into one consistent report. It is useful for:

• Technical SEO audits covering robots.txt, sitemaps, and structured data.
• Technology research for CMS, analytics, e-commerce, frontend, and infrastructure discovery.
• Lead enrichment with factual information about a prospect's website.
• Competitive research across lists of company or product websites.
• Migration planning when reviewing technologies used by existing sites.
• Quality assurance for Schema.org markup and public crawl configuration.

Reports distinguish processing errors from website findings, making them suitable for dashboards, alerts, and repeatable audits.

How to analyze a website

1. Open Website Intelligence API in Apify Console.
2. Add one or more public pages under Start URLs.
3. Select the checks and output details you need.
4. Adjust limits or proxy settings when required.
5. Click Start.
6. Review the results in the Dataset or download them in your preferred format.

No coding or selector configuration is required.

Input

The most commonly used input fields are:

Field	Description
startUrls	Public HTTP or HTTPS pages to analyze.
analyzeComments	Detect comment platforms and comment forms.
analyzeTechnologies	Detect website technologies and product versions.
analyzeStructuredData	Extract and validate JSON-LD, Microdata, and RDFa.
commentsMaxDepth	Link depth used to find pages containing comments.
schemaMaxDepth	Link depth used for Schema.org extraction and validation.
brokenDataMaxDepth	Link depth used for broken structured-data checks.
maxDiscoveryPages	Maximum additional pages followed during a run.
maxLinksPerPage	Maximum eligible links considered on each page.
analyzeRobots	Evaluate robots.txt for the selected user agent.
analyzeSitemap	Discover and inspect XML sitemaps.
maxConcurrency	Maximum number of pages processed concurrently.
timeoutSecs	Request timeout in seconds.
proxyConfiguration	Optional Apify Proxy or custom proxy settings.

Depth 0 analyzes only the supplied pages. Higher values follow links on the same hostname. Links to other hostnames and subdomains are not followed; add those sites explicitly under Start URLs when needed.

Additional controls in the Input tab let you configure retries, evidence, sitemap limits, version detection, raw structured-data fragments, and sitemap-index traversal.

Example input:

{
    "startUrls": [{ "url": "https://apify.com/" }, { "url": "https://apify.com/store" }],
    "analyzeComments": true,
    "analyzeTechnologies": true,
    "analyzeStructuredData": true,
    "commentsMaxDepth": 1,
    "schemaMaxDepth": 0,
    "brokenDataMaxDepth": 0,
    "maxDiscoveryPages": 100,
    "maxLinksPerPage": 50,
    "analyzeRobots": true,
    "analyzeSitemap": true,
    "maxConcurrency": 3,
    "proxyConfiguration": { "useApifyProxy": false }
}

Output

The Actor produces a report for every submitted URL. This simplified example shows the main result groups:

{
    "inputUrl": "https://apify.com/",
    "url": "https://apify.com/",
    "finalUrl": "https://apify.com/",
    "status": "success",
    "page": {
        "status": "success",
        "data": {
            "statusCode": 200,
            "contentType": "text/html",
            "title": "Apify"
        }
    },
    "technologies": {
        "status": "success",
        "data": {
            "items": [],
            "summary": { "technologyCount": 0, "categories": [] }
        }
    },
    "comments": {
        "status": "success",
        "data": {
            "commentSystem": { "detected": false },
            "foundAtUrl": null,
            "depth": null,
            "pagesAnalyzed": 1
        }
    },
    "structuredData": {
        "status": "success",
        "data": {
            "detected": false,
            "formats": [],
            "types": [],
            "pagesAnalyzed": 1,
            "items": []
        }
    },
    "robots": {
        "status": "success",
        "data": { "exists": false, "valid": false }
    },
    "sitemap": {
        "status": "success",
        "data": { "found": false, "sitemapCount": 0, "urlCount": 0 }
    },
    "issues": [],
    "errors": [],
    "discovery": {
        "pagesCrawled": 1,
        "maxDepthReached": 0,
        "truncated": false
    }
}

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel.

Data table

Field	Description
status	Overall processing status: success, partial, or failed.
page	HTTP response details, final page title, and content type.
technologies	Detected technologies, categories, versions, and optional evidence.
comments	Detected comment provider, state, confidence, and evidence.
structuredData	Structured-data items, formats, types, and validation summary.
robots	robots.txt status, applicable rules, crawl delay, and sitemap declarations.
sitemap	Sitemap files, URL counts, samples, dates, and validation statistics.
issues	Structured-data, robots.txt, and sitemap findings.
errors	Processing errors associated with a specific report section.
discovery	Number of pages checked, maximum reached depth, and limit status.

How much does website analysis cost?

Run cost depends on the number of pages, page loading time, sitemap size, enabled checks, proxy usage, and selected Actor memory. Start with a small URL list and default limits to estimate consumption for your targets. You can reduce cost by disabling checks you do not need and limiting the number of sitemap files or URLs collected.

Apify plan credits and free-tier availability can change, so consult the current billing information in Apify Console for exact limits.

Tips and advanced options

• Keep concurrency modest when analyzing many pages from the same domain.
• Start comment discovery at depth 1; article pages commonly contain the comment area.
• Keep Schema.org and broken-data depth at 0 for targeted URL audits, then increase gradually for wider checks.
• Use maxDiscoveryPages and maxLinksPerPage to keep runtime and cost predictable.
• Use a proxy when a target applies geographic restrictions or rate limits.
• Reduce sitemap limits when predictable runtime is more important than complete URL collection.
• Enable evidence when reviewing technology detections manually.
• Enable raw structured-data fragments only when they are needed for debugging or downstream processing.
• Treat entries under issues as website findings; check errors for processing failures.

FAQ, disclaimers, and support

Can it analyze JavaScript-powered websites?

Yes. Website Intelligence API supports pages whose visible content or technical signals depend on JavaScript.

What happens if one check fails?

Available findings are still included. The affected report section and overall result indicate whether processing was partial or failed.

Is website analysis legal?

You are responsible for complying with target-site terms, robots policies, copyright, privacy requirements, and applicable laws. Use reasonable concurrency and do not collect personal or sensitive information without a lawful basis.

Authenticated pages, anti-bot systems, malformed markup, inaccessible resources, and exceptionally large sitemaps can affect results. Use the Actor's Issues tab to report a problem or request an enhancement. Custom website-intelligence solutions and private integrations are also available on request.