# Website Intelligence API (`ib4ngz/website-intelligence-api`) Actor

Analyze website technologies, comments, structured data, robots.txt, and sitemaps in one run.

- **URL**: https://apify.com/ib4ngz/website-intelligence-api.md
- **Developed by:** [Iqbal R](https://apify.com/ib4ngz) (community)
- **Categories:**
- **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $4.00 / 1,000 reports

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

### What does Website Intelligence API do?

**Website Intelligence API creates a structured technical and SEO report for any public website.** It identifies website technologies and comment platforms, extracts Schema.org data, checks structured-data quality, evaluates robots.txt, and inspects XML sitemaps.

Use the Actor in Apify Console or connect it to your workflow through the Apify API. Results can be scheduled, monitored, exported, sent through webhooks, and integrated with other applications. To try it, enter a public page such as [apify.com](https://apify.com/).

### Why use Website Intelligence API?

Website Intelligence API brings several common website checks into one consistent report. It is useful for:

- **Technical SEO audits** covering robots.txt, sitemaps, and structured data.
- **Technology research** for CMS, analytics, e-commerce, frontend, and infrastructure discovery.
- **Lead enrichment** with factual information about a prospect's website.
- **Competitive research** across lists of company or product websites.
- **Migration planning** when reviewing technologies used by existing sites.
- **Quality assurance** for Schema.org markup and public crawl configuration.

Reports distinguish processing errors from website findings, making them suitable for dashboards, alerts, and repeatable audits.

### How to analyze a website

1. Open Website Intelligence API in Apify Console.
2. Add one or more public pages under **Start URLs**.
3. Select the checks and output details you need.
4. Adjust limits or proxy settings when required.
5. Click **Start**.
6. Review the results in the Dataset or download them in your preferred format.

No coding or selector configuration is required.

### Input

The most commonly used input fields are:

| Field                   | Description                                               |
| ----------------------- | --------------------------------------------------------- |
| `startUrls`             | Public HTTP or HTTPS pages to analyze.                    |
| `analyzeComments`       | Detect comment platforms and comment forms.               |
| `analyzeTechnologies`   | Detect website technologies and product versions.         |
| `analyzeStructuredData` | Extract and validate JSON-LD, Microdata, and RDFa.        |
| `commentsMaxDepth`      | Link depth used to find pages containing comments.        |
| `schemaMaxDepth`        | Link depth used for Schema.org extraction and validation. |
| `brokenDataMaxDepth`    | Link depth used for broken structured-data checks.        |
| `maxDiscoveryPages`     | Maximum additional pages followed during a run.           |
| `maxLinksPerPage`       | Maximum eligible links considered on each page.           |
| `analyzeRobots`         | Evaluate robots.txt for the selected user agent.          |
| `analyzeSitemap`        | Discover and inspect XML sitemaps.                        |
| `maxConcurrency`        | Maximum number of pages processed concurrently.           |
| `timeoutSecs`           | Request timeout in seconds.                               |
| `proxyConfiguration`    | Optional Apify Proxy or custom proxy settings.            |

Depth `0` analyzes only the supplied pages. Higher values follow links on the same hostname. Links to other hostnames and subdomains are not followed; add those sites explicitly under **Start URLs** when needed.

Additional controls in the Input tab let you configure retries, evidence, sitemap limits, version detection, raw structured-data fragments, and sitemap-index traversal.

Example input:

```json
{
    "startUrls": [{ "url": "https://apify.com/" }, { "url": "https://apify.com/store" }],
    "analyzeComments": true,
    "analyzeTechnologies": true,
    "analyzeStructuredData": true,
    "commentsMaxDepth": 1,
    "schemaMaxDepth": 0,
    "brokenDataMaxDepth": 0,
    "maxDiscoveryPages": 100,
    "maxLinksPerPage": 50,
    "analyzeRobots": true,
    "analyzeSitemap": true,
    "maxConcurrency": 3,
    "proxyConfiguration": { "useApifyProxy": false }
}
```

### Output

The Actor produces a report for every submitted URL. This simplified example shows the main result groups:

```json
{
    "inputUrl": "https://apify.com/",
    "url": "https://apify.com/",
    "finalUrl": "https://apify.com/",
    "status": "success",
    "page": {
        "status": "success",
        "data": {
            "statusCode": 200,
            "contentType": "text/html",
            "title": "Apify"
        }
    },
    "technologies": {
        "status": "success",
        "data": {
            "items": [],
            "summary": { "technologyCount": 0, "categories": [] }
        }
    },
    "comments": {
        "status": "success",
        "data": {
            "commentSystem": { "detected": false },
            "foundAtUrl": null,
            "depth": null,
            "pagesAnalyzed": 1
        }
    },
    "structuredData": {
        "status": "success",
        "data": {
            "detected": false,
            "formats": [],
            "types": [],
            "pagesAnalyzed": 1,
            "items": []
        }
    },
    "robots": {
        "status": "success",
        "data": { "exists": false, "valid": false }
    },
    "sitemap": {
        "status": "success",
        "data": { "found": false, "sitemapCount": 0, "urlCount": 0 }
    },
    "issues": [],
    "errors": [],
    "discovery": {
        "pagesCrawled": 1,
        "maxDepthReached": 0,
        "truncated": false
    }
}
```

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel.

### Data table

| Field            | Description                                                                 |
| ---------------- | --------------------------------------------------------------------------- |
| `status`         | Overall processing status: `success`, `partial`, or `failed`.               |
| `page`           | HTTP response details, final page title, and content type.                  |
| `technologies`   | Detected technologies, categories, versions, and optional evidence.         |
| `comments`       | Detected comment provider, state, confidence, and evidence.                 |
| `structuredData` | Structured-data items, formats, types, and validation summary.              |
| `robots`         | robots.txt status, applicable rules, crawl delay, and sitemap declarations. |
| `sitemap`        | Sitemap files, URL counts, samples, dates, and validation statistics.       |
| `issues`         | Structured-data, robots.txt, and sitemap findings.                          |
| `errors`         | Processing errors associated with a specific report section.                |
| `discovery`      | Number of pages checked, maximum reached depth, and limit status.           |

### How much does website analysis cost?

Run cost depends on the number of pages, page loading time, sitemap size, enabled checks, proxy usage, and selected Actor memory. Start with a small URL list and default limits to estimate consumption for your targets. You can reduce cost by disabling checks you do not need and limiting the number of sitemap files or URLs collected.

Apify plan credits and free-tier availability can change, so consult the current billing information in Apify Console for exact limits.

### Tips and advanced options

- Keep concurrency modest when analyzing many pages from the same domain.
- Start comment discovery at depth 1; article pages commonly contain the comment area.
- Keep Schema.org and broken-data depth at 0 for targeted URL audits, then increase gradually for wider checks.
- Use `maxDiscoveryPages` and `maxLinksPerPage` to keep runtime and cost predictable.
- Use a proxy when a target applies geographic restrictions or rate limits.
- Reduce sitemap limits when predictable runtime is more important than complete URL collection.
- Enable evidence when reviewing technology detections manually.
- Enable raw structured-data fragments only when they are needed for debugging or downstream processing.
- Treat entries under `issues` as website findings; check `errors` for processing failures.

### FAQ, disclaimers, and support

#### Can it analyze JavaScript-powered websites?

Yes. Website Intelligence API supports pages whose visible content or technical signals depend on JavaScript.

#### What happens if one check fails?

Available findings are still included. The affected report section and overall result indicate whether processing was partial or failed.

#### Is website analysis legal?

You are responsible for complying with target-site terms, robots policies, copyright, privacy requirements, and applicable laws. Use reasonable concurrency and do not collect personal or sensitive information without a lawful basis.

Authenticated pages, anti-bot systems, malformed markup, inaccessible resources, and exceptionally large sitemaps can affect results. Use the Actor's **Issues** tab to report a problem or request an enhancement. Custom website-intelligence solutions and private integrations are also available on request.

# Actor input Schema

## `startUrls` (type: `array`):

One or more HTTP(S) pages to analyze. Duplicate pages are crawled once while output cardinality is preserved.

## `maxConcurrency` (type: `integer`):

Maximum number of pages analyzed concurrently.

## `maxRequestRetries` (type: `integer`):

Retries after a failed browser navigation.

## `timeoutSecs` (type: `integer`):

Timeout in seconds for browser navigation and site-resource HTTP requests.

## `commentsMaxDepth` (type: `integer`):

Internal-link levels searched for comment systems. Use 0 to check only supplied pages.

## `schemaMaxDepth` (type: `integer`):

Internal-link levels searched for Schema.org extraction and validation. Use 0 for supplied pages only.

## `brokenDataMaxDepth` (type: `integer`):

Internal-link levels scanned for broken structured-data findings. Use 0 for supplied pages only.

## `maxDiscoveryPages` (type: `integer`):

Maximum additional pages followed across the run, excluding supplied Start URLs.

## `maxLinksPerPage` (type: `integer`):

Maximum eligible same-hostname links considered from each analyzed page.

## `analyzeComments` (type: `boolean`):

Detect embedded and native comment systems, including lazy-loaded widgets.

## `analyzeTechnologies` (type: `boolean`):

Detect CMS, analytics, commerce, frontend, and infrastructure technologies.

## `detectVersions` (type: `boolean`):

Attempt to extract version numbers from available technology evidence.

## `includeEvidence` (type: `boolean`):

Include bounded evidence explaining each technology detection.

## `analyzeStructuredData` (type: `boolean`):

Extract and validate JSON-LD, Microdata, and RDFa from each page.

## `includeRawStructuredData` (type: `boolean`):

Includes bounded source fragments, never the full page HTML.

## `analyzeRobots` (type: `boolean`):

Fetch and evaluate robots.txt once for each unique website origin.

## `analyzeSitemap` (type: `boolean`):

Discover and inspect sitemap files once for each unique website origin.

## `userAgent` (type: `string`):

User-agent token used when evaluating robots.txt rules.

## `maxSitemaps` (type: `integer`):

Maximum number of sitemap and sitemap-index files processed per origin.

## `maxSitemapUrls` (type: `integer`):

Maximum number of unique URLs collected from sitemaps per origin.

## `followSitemapIndex` (type: `boolean`):

Follow child sitemap references found in sitemap index files.

## `proxyConfiguration` (type: `object`):

Optional Apify Proxy or custom proxy settings used by browser and site-resource requests.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://apify.com"
    }
  ],
  "maxConcurrency": 3,
  "maxRequestRetries": 2,
  "timeoutSecs": 45,
  "commentsMaxDepth": 1,
  "schemaMaxDepth": 0,
  "brokenDataMaxDepth": 0,
  "maxDiscoveryPages": 100,
  "maxLinksPerPage": 50,
  "analyzeComments": true,
  "analyzeTechnologies": true,
  "detectVersions": true,
  "includeEvidence": true,
  "analyzeStructuredData": true,
  "includeRawStructuredData": false,
  "analyzeRobots": true,
  "analyzeSitemap": true,
  "userAgent": "*",
  "maxSitemaps": 50,
  "maxSitemapUrls": 10000,
  "followSitemapIndex": true,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://apify.com"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("ib4ngz/website-intelligence-api").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://apify.com" }] }

# Run the Actor and wait for it to finish
run = client.actor("ib4ngz/website-intelligence-api").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://apify.com"
    }
  ]
}' |
apify call ib4ngz/website-intelligence-api --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,ib4ngz/website-intelligence-api"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/JfSg0fF6yVmeasHza/builds/XsJdThgjo7gzgbjQu/openapi.json