# Company Website Profiler - Enrichment & Tech Stack (`abusch500/site-intel-company-profile`) Actor

Turn domains into structured company profiles: name, legal name, public email, address, social links, tech stack, sitemap size, SEO and security data. Reads only what sites publish about themselves and obeys robots.txt. Never charges for a domain it could not read.

- **URL**: https://apify.com/abusch500/site-intel-company-profile.md
- **Developed by:** [Alex Busch](https://apify.com/abusch500) (community)
- **Categories:** Lead generation, Automation, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$4.00 / 1,000 domain profileds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Company Website Profiler - Enrichment & Tech Stack

**Company profile from a domain.** Paste a list of company websites and get one
clean row per company: name, legal name, logo, description, public contact
details, social links, technology stack, sitemap size, SEO and security posture

- 51 columns plus an 18-field SEO block, each labelled with where it came from.

**$4 per 1,000 domains, all-in.** Nothing extra for compute. **$0 for any
domain that could not be read** - blocked, parked, dead or robots-disallowed
rows come back marked `ok: false` with a plain-English reason, and are never
billed.

```
input   ["stripe.com", "apify.com", "joespizza.example"]
output  one row per domain, every value traceable to its source
```

***

### What you get for each company website

Fill rates are measured, not estimated. Two runs on 2026-09-02, after the
latest fixes: **52 large brands and institutions** and **18 smaller
businesses, agencies and retailers**. The raw run is in
`tests/scale-test-2026-09-02b.txt` in the source.

| Field | What it is | Filled: big brands | Filled: smaller businesses |
|---|---|---|---|
| `fieldSources` | where every value above came from | 100% | 100% |
| `completeness` | 0-100 score for the whole row | 100% | 100% |
| `companyName` | the company's name | 100% | 100% |
| `legalName` | registered entity, when published | 15% | 7% |
| `description` | the company's own description of itself | 89% | 100% |
| `logo`, `favicon` | absolute image URLs | 91% | 93% |
| `email`, `emails` | public business addresses, role desks first | 15% | 29% |
| `phone`, `phones` | published numbers, de-duplicated | 15% | 36% |
| `address`, `country` | postal address from structured data | 19% | 21% |
| `socials` | LinkedIn, X, GitHub, YouTube, Facebook, Instagram, TikTok, Discord, Reddit, Mastodon, Bluesky, Crunchbase | 81% | 93% |
| `techHints` | CMS, framework, CDN, analytics, chat, payments | 87% | 93% |
| `sitemapUrlCount` | pages the site publishes, with `sitemapChildCount`, `sitemapLastModified` and `sitemapCountCapped` | 70% | 79% |
| `seo` | title, meta description, canonical, OpenGraph, structured-data types, hreflang, heading counts, images missing alt text, internal vs external links | 100% | 100% |
| structured data | schema.org types found on the page | 62% | 79% |
| `securityTxt`, `securityContact` | RFC 9116 contact | 60% | 14% |
| `llmsTxt` | the emerging AI-readers convention | 51% | 29% |
| `feeds`, `feedCount` | RSS or Atom feeds that actually resolve | 19% | 21% |
| `https`, `securityHeaders`, `securityHeaderScore` | transport and six security headers scored | 100% | 100% |
| `language`, `organizationType`, `foundingDate` | from `html lang` and schema.org | varies | varies |
| `redirectedTo` | set when the domain redirects to another host | - | - |
| `performance` | Core Web Vitals and Lighthouse (optional, free Google key) | off by default | off by default |

`emails` and `phones` keep every value found; `email` and `phone` are the best
single pick. Rows that could not be read still carry all 51 columns, so your
spreadsheet never changes shape.

### Why the source of every field matters

A company name read from a `schema.org` Organization block is a fact the
company published. A company name guessed from a page title is a guess. Most
tools hand you both and let you find out the hard way. This one labels them:

```json
"companyName": "Apify",
"fieldSources": {
  "companyName": "json-ld",
  "description": "json-ld",
  "email": "json-ld",
  "address": "json-ld",
  "country": "json-ld",
  "language": "html lang",
  "socials": "json-ld sameAs + site links",
  "techHints": "markup + headers",
  "securityContact": "security.txt"
}
```

Sort by `fieldSources.companyName` and every `json-ld` row is ready to use.
The rows labelled `title (least reliable)` are the ones worth a human glance.

**You are never charged for an empty row.** A domain that is parked, dead,
blocked or asks robots not to be read still comes back as a row so your list
keeps its shape, with `ok: false` and a reason. It costs nothing. The run
summary prints how many rows were actually charged for.

### How to profile a list of domains

1. Click **Try for free** at the top of this page.
2. Paste your domains into **Domains or URLs**, one per line. `stripe.com` and
   `https://stripe.com/pricing` both work; duplicates are removed.
3. Press **Start**.
4. Open the **Output** tab and download CSV, JSON or Excel, or pull the data
   from the API (below).

A 50-domain list finishes in well under a minute at the default settings:
the measured run above did 52 domains in 41 seconds at concurrency 8, roughly
3.8 seconds per domain.

### Pricing: $4 per 1,000 domains, nothing for failures

**$0.004 per domain profiled.** Pay per event.

| Domains profiled | Cost |
|---|---|
| 1,000 | $4 |
| 10,000 | $40 |
| 100,000 | $400 |

- **Only `ok: true` rows are billed.** Blocked, dead, parked and
  robots-disallowed domains are free.
- **No separate compute bill.** You are not invoiced for the platform usage
  this run consumes on top of the price above.
- The run log ends with a `charged:` line so you can check the number yourself.

### Example output

A real row, from a run on 2026-09-02 (trimmed for length - the full row has 51
columns):

```json
{
  "domain": "apify.com",
  "url": "https://apify.com/",
  "ok": true,
  "companyName": "Apify",
  "legalName": "Apify Technologies s.r.o.",
  "description": "Thousands of tools to automate your business...",
  "email": "support@apify.com",
  "address": { "locality": "Prague", "postalCode": "11000", "country": "CZ" },
  "country": "CZ",
  "socials": {
    "github": "https://github.com/apify",
    "linkedin": "https://www.linkedin.com/company/apify",
    "twitter": "https://x.com/apify",
    "tiktok": "https://www.tiktok.com/@apifytech",
    "discord": "https://discord.com/invite/jyEM2PRvMU"
  },
  "socialCount": 5,
  "techHints": ["Next.js", "HubSpot", "Google Tag Manager", "Intercom", "AWS CloudFront"],
  "sitemapFound": true,
  "sitemapUrlCount": 200000,
  "sitemapChildCount": 13,
  "sitemapCountCapped": true,
  "securityTxt": true,
  "llmsTxt": true,
  "https": true,
  "securityHeaderScore": 4,
  "fieldSources": { "companyName": "json-ld", "email": "json-ld", "address": "json-ld" },
  "completeness": 92
}
```

Every row carries `completeness`, 0 to 100, so you can sort a delivery instead
of eyeballing it. Real scores from the same run:

| Domain | Completeness | What was found |
|---|---|---|
| apify.com | 92% | legal name, Prague address, `support@` desk, 5 socials, 5 technologies |
| kingarthurbaking.com | 83% | name from schema.org, toll-free number, 5 socials, 7,591-page sitemap |
| zingermans.com | 79% | `service@` desk chosen over `bizgifts@`, phone, 2 socials |
| example.com | 29% | a stub page, and the score says so |

### Company enrichment use cases

- **Enriching a lead list.** Domains from a form, a conference or a CRM export
  become names, descriptions, technology reads and - where the business
  publishes them - a contact desk.
- **Qualifying prospects.** The technology list tells you whether they are a
  fit, and `completeness` tells you which rows need a human.
- **Site audit and portfolio hygiene.** Run your own domains and see which are
  missing structured data, security headers, HTTPS or a sitemap.
- **Feeding an AI agent.** One clean JSON row per company, with sources, so a
  model is not guessing from raw HTML.

### Run it from code or your automation tool

**JavaScript** (`npm install apify-client`):

```js
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });
const run = await client.actor('abusch500/site-intel-company-profile').call({
    domains: ['stripe.com', 'apify.com'],
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);
```

**Python** (`pip install apify-client`):

```python
from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("abusch500/site-intel-company-profile").call(
    run_input={"domains": ["stripe.com", "apify.com"]}
)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)
```

**cURL** - run and get the rows back in one call:

```bash
curl -X POST "https://api.apify.com/v2/acts/abusch500~site-intel-company-profile/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"domains": ["stripe.com", "apify.com"]}'
```

This actor is a standard Apify Actor, so the platform's own integrations are
available on the tabs above: **n8n**, **Make**, **Zapier**, **MCP** for AI
agents, webhooks, and **scheduling** for a recurring refresh. See the
**Integrations** and **API** tabs of this actor for the exact wiring.

### Input

| Field | Type | Default | Notes |
|---|---|---|---|
| `domains` | array | required | `"stripe.com"` or `"https://stripe.com/"` both work |
| `maxConcurrency` | integer | 5 | 1 to 20 |
| `requestTimeoutSecs` | integer | 20 | per request |
| `respectRobotsTxt` | boolean | true | leave it on unless you own the domains |
| `includePerformance` | boolean | false | adds Core Web Vitals |
| `pageSpeedApiKey` | string | empty | free Google key, no billing details needed |

```json
{
  "domains": ["stripe.com", "apify.com"],
  "maxConcurrency": 5,
  "includePerformance": false
}
```

### Where the data comes from

Only from places built to be read by machines:

| Source | What it is |
|---|---|
| `schema.org` JSON-LD | structured data published for machines |
| OpenGraph and Twitter cards | published so other sites can display a preview |
| standard `<meta>` tags | the page's own description of itself |
| `/robots.txt` | the site's crawling policy, read first and obeyed |
| `/sitemap.xml` | published so crawlers can find pages |
| `/.well-known/security.txt` | RFC 9116 public security contact |
| `/llms.txt`, `/humans.txt`, `/ads.txt` | published conventions |
| RSS and Atom feeds | published for syndication |
| HTTP response headers | sent to every visitor |
| Google PageSpeed Insights | Google's own free public API |

Nothing behind a login. Nothing behind a paywall.

**On robots.txt.** It is fetched before anything else. `Allow` and `Disallow`
rules are obeyed, including a group naming `SiteIntelBot`, and a site with a
blanket disallow is skipped and not charged for. `Crawl-delay` is read and
reported in the row as `robotsTxt.crawlDelay`, but not yet applied as a wait.
The `respectRobotsTxt` switch exists for running this against domains you own.

**On personal data.** This reads only contact routes a business publishes on
its own homepage - a `mailto:` link, a `tel:` link, a schema.org `email`. There
is no people search, no email-pattern guessing and no personal profiles:
`linkedin.com/in/<person>` URLs are deliberately rejected, and only
`/company/` and `/school/` pages are recorded. Where a company publishes a
role desk (`info@`, `support@`, `sales@`) it is preferred over anything else
found on the page.

### For site owners

This actor identifies itself as **`SiteIntelBot`**. To opt out, add this to
your `robots.txt`:

```
User-agent: SiteIntelBot
Disallow: /
```

It reads your homepage once, plus the standard published files (robots.txt,
sitemap.xml and its children, security.txt, llms.txt, humans.txt, ads.txt and
any declared feeds). Measured on 2026-09-02: **18 to 34 requests per domain
per run**, and never more than about 40 even for a large sitemap index.
Requests carry a contactable User-Agent pointing at this page.

### FAQ

**Why is email or phone empty for many domains?**
Because most companies do not publish one on the homepage. Measured: 15% of
large brands and 29% of smaller businesses publish an email; 15% and 36% a
phone. This is a live read of what the site actually says, not a database of
guessed addresses, so an empty field means the site did not publish it.

**Why was a domain skipped, and was I charged?**
You were not charged. A skipped row has `ok: false` and an `error` saying why:
`HTTP 403`, `fetch failed`, `robots.txt disallows automated access`. Only
`ok: true` rows are billed.

**How many domains get skipped?**
In the two measured runs: 5 of 52 large brands (3 of those blocked with
HTTP 403 or 429) and 4 of 18 smaller businesses (3 blocked). Roughly one
commercial domain in ten refuses datacenter traffic outright, whatever is
asking. Those rows are free.

**Are there compute or platform charges on top of the price?**
No. The price per profiled domain is the whole cost of the run to you.

**Is this legal? Is it GDPR-safe?**
It reads only what a site publishes for machines to read - structured data,
meta tags and standard files at fixed public paths - and it obeys robots.txt.
Nothing behind a login or a paywall. It does not collect personal profiles or
guess at individuals' addresses. You are still responsible for how you use
business contact details in your own jurisdiction.

**How is this different from a contact finder?**
A contact finder crawls many pages of a site hunting for addresses. This reads
one page and the standard files, and returns a whole profile - identity, tech,
structure, SEO, security - with a source label on each value. If contacts are
the only thing you need, a dedicated contact finder will find more of them.

**How is it different from an enrichment database?**
A database returns what it stored, whenever it stored it. This fetches the site
now, and tells you which field came from where. Nothing is cached between runs.

**How fresh is the data?**
It is fetched during your run. `fetchedAt` on every row is the timestamp.

**How fast is it, and how big a list can I send?**
About 4 seconds per domain, run in parallel. 52 domains took 41 seconds at
concurrency 8. Lists are limited only by your own run timeout; memory stays
flat regardless of list size.

**What does `completeness` mean?**
A weighted 0-100 score of how much of the profile actually filled in. Rows in
the nineties are ready to use, rows in the thirties need a human. Sort by it.

**What does `sitemapCountCapped` mean?**
A sitemap index is followed one level, up to 25 child sitemaps, and counting
stops at 200,000 URLs. When either limit is reached the flag is `true` and the
count is a floor, not an exact figure. `sitemapChildCount` tells you how many
child sitemaps the index actually declared.

**Why is `performance` null?**
Core Web Vitals are off by default because they need a free Google PageSpeed
API key and add 10-25 seconds per domain. Turn on `includePerformance` and
paste a key to fill it.

**Can I turn robots.txt off for my own sites?**
Yes - `respectRobotsTxt: false`. It is there for domains you own and control.
Leave it on for everything else.

**How do I block this bot?**
See **For site owners** above. `User-agent: SiteIntelBot` / `Disallow: /`.

**Does it work on non-English sites?**
Yes. The name and description are whatever the site publishes, in whatever
language it publishes them, and `language` reports the site's own declared
language.

### Notes and limits

- **Roughly one commercial domain in ten answers HTTP 403 to any datacenter
  IP.** Measured: 3 of 52 large brands, 3 of 18 smaller businesses. It is a
  web-application firewall rule, not robots.txt, and it is not something this
  actor tries to defeat. Those rows come back marked and unbilled.
- **Technology detection is hints, not certainty.** It reads what the page
  actually loads - script, stylesheet, image and iframe URLs, inline scripts,
  the generator tag and response headers. A site behind a proxy may hide what
  it runs. A page that merely mentions a platform in its copy is deliberately
  not counted.
- **Sitemap counts on very large sites are a floor.** See
  `sitemapCountCapped` above.
- **Performance data needs a free key** from the Google Cloud Console, no
  billing details. Without it every other field still works.
- **Field data in the performance block is often absent.** Google only reports
  real-user measurements for origins above a traffic threshold. The row says
  so rather than leaving you guessing.

### Support and changelog

Found a wrong value? Open an issue on the **Issues** tab with the domain.
Wrong data gets fixed, and the fix ships to everyone. Issues are read daily.

**0.2 - 2026-09-02**

- Company names cleaned up: `Welcome to ...` greetings, `Home - Brand` titles
  and `Brand -- Tagline` separators are handled; PWA labels are no longer read
  as company names.
- Technology detection now tests only what a page actually loads, so an agency
  that writes about Squarespace and Webflow is no longer reported as running
  them. Duplicate spellings (`Joomla!` / `Joomla`) merged.
- Social links: platform utility pages (`/login`, `/features`, `/policies`)
  rejected, and personal `linkedin.com/in/` profiles are no longer recorded.
- `email` now prefers a published role desk over a personal address.
- Duplicate phone numbers written two ways are merged.
- Sitemap indexes are followed 25 children deep instead of 5, fetched in
  parallel, with relative URLs resolved; `sitemapChildCount` and
  `sitemapLastModified` added.
- Every row - including failures - now carries all 51 columns, so CSV columns
  never move.
- `fieldSources` extended to `legalName`, `foundingDate`, `organizationType`,
  `country`, `language`, `techHints`, `securityContact`, `server`, `poweredBy`.

**0.1 - 2026-09-02** - first public release.

# Actor input Schema

## `domains` (type: `array`):

One per line. Bare domains (stripe.com) or full URLs (https://stripe.com/) both work.

## `includePerformance` (type: `boolean`):

Uses Google's free PageSpeed Insights API. Needs a free API key below. Adds roughly 10-25 seconds per domain.

## `pageSpeedApiKey` (type: `string`):

Free from Google Cloud Console, no billing details required. Only used when performance is turned on. Leave empty to skip performance data.

## `maxConcurrency` (type: `integer`):

How many domains to profile at once.

## `requestTimeoutSecs` (type: `integer`):

How long to wait for each request before giving up. Raise it for slow sites, lower it to move faster through a big list.

## `respectRobotsTxt` (type: `boolean`):

Leave this on. It is the reason this actor is safe to run at scale.

## Actor input object example

```json
{
  "domains": [
    "stripe.com"
  ],
  "includePerformance": false,
  "maxConcurrency": 5,
  "requestTimeoutSecs": 20,
  "respectRobotsTxt": true
}
```

# Actor output Schema

## `profiles` (type: `string`):

One row per domain. Every field carries the source it came from, plus a completeness score from 0 to 100 so a delivery can be sorted rather than eyeballed.

## `profilesCsv` (type: `string`):

The same rows as a spreadsheet, for dropping straight into a CRM or a lead list.

## `summary` (type: `string`):

How many domains were profiled, how many were skipped, and how many were charged for. Skipped domains are never charged.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "stripe.com",
        "apify.com",
        "example.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("abusch500/site-intel-company-profile").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "domains": [
        "stripe.com",
        "apify.com",
        "example.com",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("abusch500/site-intel-company-profile").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "stripe.com",
    "apify.com",
    "example.com"
  ]
}' |
apify call abusch500/site-intel-company-profile --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,abusch500/site-intel-company-profile"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/xiWDxkiySYadZP3CY/builds/PeSAUqSfOF4czneet/openapi.json