# AI Website Categorization: IAB, Industry & Brand Safety (`messnomore/website-categorization`) Actor Classify domains with AI: IAB Content Taxonomy 3.1 category (Tier 1 and 2 IDs), industry, business model, language and brand safety flags for each website, plus parked, dead and redirected domain detection. Reads only public pages and respects robots.txt. - **URL**: https://apify.com/messnomore/website-categorization.md - **Developed by:** [Wong Xian Qi](https://apify.com/messnomore) (community) - **Categories:** AI, SEO tools, Lead generation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $4.20 / 1,000 classified domains This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ### What does AI Website Categorization do? **AI Website Categorization** classifies any list of domains into the **IAB Content Taxonomy 3.1** (Tier 1 and Tier 2 category with official IAB IDs) and adds the **industry**, **business model**, **language**, **brand safety flags** and a one-sentence **description** of each website. It also tells you which domains are **parked, for sale, dead, redirected, blocked or disallowed by robots.txt**, so you can clean a domain list in one pass. Paste domains in Apify Console with no code, or use it as a **website categorization API** / **domain classification API** from your own scripts. Each domain is classified from its own homepage and About page, read politely and only when robots.txt allows it; sites that block automated visitors can be classified from the AI model's own knowledge. ### Why use AI Website Categorization? Labelling thousands of websites by hand, or stitching together a crawler, an IAB lookup table and an AI model, takes days. This Actor does it for you: - 🏷️ **IAB Content Taxonomy 3.1 with IDs**: Tier 1 and Tier 2 categories (`{"id": "52", "name": "Business and Finance"}`) ready for ad servers, DSPs, SSPs and contextual targeting. - 🛡️ **Brand safety flags**: the 11 IAB Sensitive Topics (adult, arms, crime, hate speech, piracy, illegal drugs, tobacco and alcohol, terrorism and more) plus gambling, with an overall risk level (none, low, medium, high). - 🏭 **Industry and business model**: "Payments / FinTech", "B2B SaaS", "E-commerce / online store", "Media / publisher" and more, for lead scoring and CRM enrichment. - 🅿️ **Parked and dead domain detection**: parking nameservers, domain marketplaces (Sedo, Dan, Afternic, HugeDomains...), for-sale pages, default server pages, DNS failures and HTTP errors. - 🧩 **Your own labels**: add up to 100 custom categories and get the best match for every site, at no extra cost. - 🤖 **Respectful by design**: robots.txt is checked before every request, at most 3 pages per domain, and bot protection is never bypassed. - 🔌 **Apify platform features**: run on a schedule, call it by API, export to CSV or Excel, or send results to Google Sheets, Slack, Make or Zapier with [integrations](https://docs.apify.com/platform/integrations). Who uses it: **ad-tech and programmatic teams** (contextual targeting, inventory and publisher vetting), **brand safety and trust & safety teams**, **sales and marketing teams** (lead enrichment, ICP filtering) and **domain investors** (parked domain checks). ### What data does AI Website Categorization return? | Field | Description | | ----------------------- | ---------------------------------------------------------------------------- | | `domain` | The domain from your input. | | `status` | `ok`, `redirect`, `parked`, `unreachable`, `robots_disallowed` or `blocked`. | | `statusReason` | Why, e.g. "Domain uses parking nameservers (afternic.com)". | | `iabTier1` / `iabTier2` | IAB Content Taxonomy 3.1 category: `{ "id", "name" }`. | | `industry` | Industry as "Sector / Subsector", e.g. "Payments / FinTech". | | `businessModel` | One of 19 business models, e.g. "B2B SaaS", "Marketplace / platform". | | `language` | Main content language (ISO 639-1, e.g. `en`, `de`, `ja`). | | `brandSafetyRisk` | `none`, `low`, `medium` or `high`. | | `brandSafetyFlags` | Sensitive topics found, each with its IAB ID. | | `description` | One neutral sentence about what the website is and does. | | `confidence` | 0–1 confidence of the IAB category. | | `customLabel` | Your best matching custom label (custom taxonomy only). | | `classificationSource` | `website` (from its pages) or `ai_knowledge` (site could not be read). | Also included: `finalUrl`, `httpStatus`, page `title`, `pagesAnalyzed`, `customLabelConfidence`, `taxonomyVersion` and `checkedAt`. ### How to categorize websites with AI Website Categorization 1. Click **Try for free** and open the **Input** tab. 2. Paste your domains into **Domains**, one per line (URLs work too; only the host name is used). 3. Optionally choose **IAB + my own labels** and add your custom labels. 4. Keep **Pages per domain** at 2 (homepage + About page) and click **Start**. 5. Watch the results appear in the **Output** tab (40 domains are checked in parallel, so 1,000 domains usually take a few minutes), then export them as CSV, Excel or JSON. ### How much does it cost to classify websites? AI Website Categorization uses **pay-per-event pricing**, and you only pay full price for domains that get a category: - **$6.00 per 1,000 classified domains** ($0.006 each), whether classified from the website or from AI knowledge. - **$1.00 per 1,000 domains that cannot be classified** ($0.001 each): parked, non-existent, or blocked or unreachable and unknown to the AI. - **Free**: invalid or duplicate entries, reachable sites where the AI call itself failed, and internal errors. - A tiny start fee per run (a fraction of a cent). There is no charge for compute time or proxies. A list of 10,000 domains where 80% can be classified costs about **$50**. Apify stops the run when it reaches the maximum cost you set, so a run never costs more than your limit. Apify's free plan includes monthly platform credit, enough to try the Actor on hundreds of domains. ### Input See the **Input** tab for every option. Example input: ```json { "domains": ["stripe.com", "bbc.co.uk", "etsy.com", "bluewidget.com"], "taxonomy": "iab_content_3", "includeBrandSafety": true, "aiKnowledgeFallback": true, "maxPagesPerDomain": 2 } ``` - **taxonomy**: `iab_content_3` returns the IAB category; `custom` also picks one of your `customLabels` (2–100 labels). - **aiKnowledgeFallback**: when a site blocks automated visitors, disallows them in robots.txt, or drops the connection, classify it from the AI model's own knowledge of that domain instead of leaving it empty. Turn it off if you only want classifications based on the live website. - **maxPagesPerDomain**: 1 = homepage only, 2 = homepage + About page, 3 = also one products or services page. ### Output You can download the dataset in various formats such as **JSON, HTML, CSV or Excel**. Example results: ```json [ { "domain": "stripe.com", "status": "ok", "statusReason": null, "finalUrl": "https://stripe.com/", "iabTier1": { "id": "596", "name": "Technology & Computing" }, "iabTier2": { "id": "599", "name": "Computing" }, "industry": "Financial Services / Payments", "businessModel": "B2B SaaS", "language": "en", "brandSafetyRisk": "none", "brandSafetyFlags": [], "description": "Stripe is a financial services platform that helps businesses accept payments and move money through APIs and tooling.", "confidence": 0.8, "classificationSource": "website", "pagesAnalyzed": ["https://stripe.com/", "https://stripe.com/about"] }, { "domain": "glock.com", "status": "ok", "finalUrl": "https://us.glock.com/en", "iabTier1": { "id": "v9i3On", "name": "Sensitive Topics" }, "iabTier2": { "id": "avbNf2", "name": "Arms & Ammunition" }, "industry": "Manufacturing / Firearms", "businessModel": "Manufacturing", "brandSafetyRisk": "medium", "brandSafetyFlags": [{ "code": "arms", "name": "Arms & Ammunition", "iabId": "avbNf2" }], "confidence": 0.9, "classificationSource": "website" }, { "domain": "etsy.com", "status": "blocked", "statusReason": "Bot protection challenge (DataDome)", "iabTier1": { "id": "473", "name": "Shopping" }, "iabTier2": null, "industry": "Retail / Marketplace", "businessModel": "Marketplace / platform", "description": "Etsy operates an online marketplace for handmade, vintage goods and craft supplies.", "confidence": 0.8, "classificationSource": "ai_knowledge" }, { "domain": "bluewidget.com", "status": "parked", "statusReason": "Domain uses parking nameservers (afternic.com)", "iabTier1": null, "classificationSource": null } ] ``` A `SUMMARY` record in the run's key-value store counts domains per status, classified domains and charged events. ### Use AI Website Categorization as an API ```bash curl -X POST "https://api.apify.com/v2/acts/messnomore~website-categorization/run-sync-get-dataset-items" \ -H "Authorization: Bearer YOUR_APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{"domains": ["stripe.com", "bbc.co.uk"]}' ``` For large lists, start the run asynchronously and read the dataset when it finishes. See the **API** tab for clients in JavaScript and Python. You can also run it from AI assistants through the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp). ### Tips for better results - Use **Pages per domain = 1** for the fastest runs on very large lists; 2 gives better results for company websites. - Filter on `status = ok` and `confidence ≥ 0.7` when you need only high-certainty categories. - Use **custom labels** for your own segments (for example your ICP tiers) and keep the IAB category for ad-tech. - Domains are de-duplicated by host name: `www.example.com` and `example.com` are checked separately on purpose, because they can be different sites. ### Related guides - [How do I categorize a list of websites by IAB category, brand safety and parked status with an API?](https://wong13977-oss.github.io/website-categorization-api/) ### FAQ #### How does the website classification work? For each domain the Actor checks robots.txt, reads the homepage (following redirects) and, when linked, the About page. It extracts the title, meta description, headings, structured data types and visible text, and sends that text to an AI model (OpenAI) together with the full IAB Content Taxonomy 3.1. The model picks the category by name (it cannot answer with a category that does not exist), plus the industry, business model, language, brand safety topics with their evidence, and a confidence score. The Actor maps the category to its official IAB ID and validates every value before saving it. #### Which IAB taxonomy version is used? IAB Tech Lab Content Taxonomy 3.1, Tier 1 and Tier 2 (37 + 323 categories). Version 3.1 has no general "News" category, so general news publishers and government portals are classified as **Politics** unless another topic clearly dominates; the `industry` field still says "News / Broadcasting". #### How accurate is it? In our tests on more than 150 well-known websites, about 97% got a sensible IAB category (the misses were close calls, such as an electronics maker filed under Business) and the language was always right. On a brand safety set run twice (casinos, betting, spirits, beer, tobacco, vaping, guns, adult, plus soft-drink and sports controls) every flag was correct in both runs, and ordinary sites got no flags. Some brands fit two neighbouring categories (a sportswear brand can be Shopping or Style & Fashion). Use `confidence` to filter uncertain results. #### What do the statuses mean? - `ok`: classified from its own pages. `redirect`: the domain forwards to another domain, and the target site is classified. - `parked`: parking nameservers, a domain marketplace, a for-sale page, a default server page or a "coming soon" placeholder. - `unreachable`: no DNS record, connection or TLS failure, an HTTP error, or a page with no readable text (for example a site that only renders with JavaScript). - `robots_disallowed`: the site's robots.txt does not allow our crawler. `blocked`: the site answered with bot protection or "access denied". #### What is the AI knowledge fallback? Many large websites block automated visitors, and some simply drop or stall the connection. Instead of returning nothing, the Actor can ask the AI model what it already knows about that exact domain, without visiting the site again. Domains that do not exist (no DNS record) and parked domains are never classified this way. These results are marked `classificationSource: "ai_knowledge"` and their confidence is capped at 0.8. If the model does not know the domain (or the AI call fails), the item stays unclassified and costs only $0.001. #### Does it respect robots.txt and website terms? Yes. robots.txt is checked (for our `WebsiteCategorizationBot` user agent and the `*` group) before every page, and disallowed pages are skipped and reported. The Actor identifies itself with its own user agent, reads at most 3 public pages per domain, never logs in and never solves CAPTCHAs or works around bot protection. #### Is my input stored or shared? The domains and the text of their public pages are sent to the OpenAI API only to classify them; OpenAI does not use API data to train its models. Like every Apify run, your input and results stay in your own run storage, which you can delete at any time. #### Is it legal to categorize websites? The Actor only reads public homepages and About pages and stores derived labels plus a one-sentence description, not the page content. Our Actors are ethical and do not extract any private user data, such as email addresses, gender, or location. They only extract what the user has chosen to share publicly. We therefore believe that our Actors, when used for ethical purposes by Apify users, are safe. However, you should be aware that your results could contain personal data. Personal data is protected by the GDPR in the European Union and by other regulations around the world. You should not scrape personal data unless you have a legitimate reason to do so. If you're unsure whether your reason is legitimate, consult your lawyers. #### Is this an official IAB product? No. This Actor is not affiliated with or endorsed by IAB Tech Lab. It uses the [IAB Tech Lab Content Taxonomy 3.1](https://github.com/InteractiveAdvertisingBureau/Taxonomies), which IAB Tech Lab publishes under the [Creative Commons Attribution 3.0 License](https://creativecommons.org/licenses/by/3.0/). #### I found a wrong category or status. What can I do? Open the **Issues** tab and include the domain. Feedback and feature requests are welcome there too. # Actor input Schema ## `domains` (type: `array`): One domain or URL per entry, e.g. "stripe.com" or "https://www.bbc.co.uk/news" (up to 10,000 per run). Only the host name is used. Duplicates and invalid entries are skipped and never charged. ## `taxonomy` (type: `string`): "IAB Content Taxonomy 3.1" returns the IAB Tier 1 and Tier 2 category (official IDs). "IAB + my own labels" also picks the best of your custom labels for each site (same price). ## `customLabels` (type: `array`): Used only with "IAB + my own labels": 2 to 100 labels (up to 80 characters each), e.g. "SaaS", "E-commerce", "Agency", "News media". Each site gets the best fitting label, or none when nothing fits. ## `includeBrandSafety` (type: `boolean`): Flag sensitive topics (adult, gambling, alcohol and tobacco, weapons, hate speech, piracy, spam and malware, and the other IAB Sensitive Topics) with an overall risk level (none, low, medium, high). ## `aiKnowledgeFallback` (type: `boolean`): When a site blocks automated visitors, disallows them in robots.txt, or drops or stalls the connection (common on large sites), classify it from the AI model's own knowledge of that domain (the site is not visited again). Results are marked classificationSource = "ai_knowledge". Domains the model does not know stay unclassified. ## `maxPagesPerDomain` (type: `integer`): 1 = homepage only (fastest). 2 = homepage + About page (recommended). 3 = also one products or services page. The price per domain is the same. ## Actor input object example ```json { "domains": [ "stripe.com", "bbc.co.uk", "shopify.com" ], "taxonomy": "iab_content_3", "customLabels": [], "includeBrandSafety": true, "aiKnowledgeFallback": true, "maxPagesPerDomain": 2 } ``` # Actor output Schema ## `results` (type: `string`): No description ## `summary` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "domains": [ "stripe.com", "bbc.co.uk", "shopify.com" ] }; // Run the Actor and wait for it to finish const run = await client.actor("messnomore/website-categorization").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "domains": [ "stripe.com", "bbc.co.uk", "shopify.com", ] } # Run the Actor and wait for it to finish run = client.actor("messnomore/website-categorization").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "domains": [ "stripe.com", "bbc.co.uk", "shopify.com" ] }' | apify call messnomore/website-categorization --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,messnomore/website-categorization" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/DqcIOdhzhC3gDKRk8/builds/Sa0Wm9xV41rzbycoF/openapi.json