AI Website Categorization: IAB, Industry & Brand Safety

What does AI Website Categorization do?

AI Website Categorization classifies any list of domains into the IAB Content Taxonomy 3.1 (Tier 1 and Tier 2 category with official IAB IDs) and adds the industry, business model, language, brand safety flags and a one-sentence description of each website. It also tells you which domains are parked, for sale, dead, redirected, blocked or disallowed by robots.txt, so you can clean a domain list in one pass.

Paste domains in Apify Console with no code, or use it as a website categorization API / domain classification API from your own scripts. Each domain is classified from its own homepage and About page, read politely and only when robots.txt allows it; sites that block automated visitors can be classified from the AI model's own knowledge.

Why use AI Website Categorization?

Labelling thousands of websites by hand, or stitching together a crawler, an IAB lookup table and an AI model, takes days. This Actor does it for you:

• 🏷️ IAB Content Taxonomy 3.1 with IDs: Tier 1 and Tier 2 categories (

  {"id": "52", "name": "Business and Finance"}
  ) ready for ad servers, DSPs, SSPs and contextual targeting.
• 🛡️ Brand safety flags: the 11 IAB Sensitive Topics (adult, arms, crime, hate speech, piracy, illegal drugs, tobacco and alcohol, terrorism and more) plus gambling, with an overall risk level (none, low, medium, high).
• 🏭 Industry and business model: "Payments / FinTech", "B2B SaaS", "E-commerce / online store", "Media / publisher" and more, for lead scoring and CRM enrichment.
• 🅿️ Parked and dead domain detection: parking nameservers, domain marketplaces (Sedo, Dan, Afternic, HugeDomains...), for-sale pages, default server pages, DNS failures and HTTP errors.
• 🧩 Your own labels: add up to 100 custom categories and get the best match for every site, at no extra cost.
• 🤖 Respectful by design: robots.txt is checked before every request, at most 3 pages per domain, and bot protection is never bypassed.
• 🔌 Apify platform features: run on a schedule, call it by API, export to CSV or Excel, or send results to Google Sheets, Slack, Make or Zapier with integrations.

Who uses it: ad-tech and programmatic teams (contextual targeting, inventory and publisher vetting), brand safety and trust & safety teams, sales and marketing teams (lead enrichment, ICP filtering) and domain investors (parked domain checks).

What data does AI Website Categorization return?

Field	Description
domain	The domain from your input.
status	ok, redirect, parked, unreachable, robots_disallowed or blocked.
statusReason	Why, e.g. "Domain uses parking nameservers (afternic.com)".
iabTier1 / iabTier2	IAB Content Taxonomy 3.1 category: { "id", "name" }.
industry	Industry as "Sector / Subsector", e.g. "Payments / FinTech".
businessModel	One of 19 business models, e.g. "B2B SaaS", "Marketplace / platform".
language	Main content language (ISO 639-1, e.g. en, de, ja).
brandSafetyRisk	none, low, medium or high.
brandSafetyFlags	Sensitive topics found, each with its IAB ID.
description	One neutral sentence about what the website is and does.
confidence	0–1 confidence of the IAB category.
customLabel	Your best matching custom label (custom taxonomy only).
classificationSource	website (from its pages) or ai_knowledge (site could not be read).

Also included: finalUrl, httpStatus, page title, pagesAnalyzed, customLabelConfidence, taxonomyVersion and checkedAt.

How to categorize websites with AI Website Categorization

1. Click Try for free and open the Input tab.
2. Paste your domains into Domains, one per line (URLs work too; only the host name is used).
3. Optionally choose IAB + my own labels and add your custom labels.
4. Keep Pages per domain at 2 (homepage + About page) and click Start.
5. Watch the results appear in the Output tab (40 domains are checked in parallel, so 1,000 domains usually take a few minutes), then export them as CSV, Excel or JSON.

How much does it cost to classify websites?

AI Website Categorization uses pay-per-event pricing, and you only pay full price for domains that get a category:

• $6.00 per 1,000 classified domains ($0.006 each), whether classified from the website or from AI knowledge.
• $1.00 per 1,000 domains that cannot be classified ($0.001 each): parked, non-existent, or blocked or unreachable and unknown to the AI.
• Free: invalid or duplicate entries, reachable sites where the AI call itself failed, and internal errors.
• A tiny start fee per run (a fraction of a cent). There is no charge for compute time or proxies.

A list of 10,000 domains where 80% can be classified costs about $50. Apify stops the run when it reaches the maximum cost you set, so a run never costs more than your limit. Apify's free plan includes monthly platform credit, enough to try the Actor on hundreds of domains.

Input

See the Input tab for every option. Example input:

{
    "domains": ["stripe.com", "bbc.co.uk", "etsy.com", "bluewidget.com"],
    "taxonomy": "iab_content_3",
    "includeBrandSafety": true,
    "aiKnowledgeFallback": true,
    "maxPagesPerDomain": 2
}

• taxonomy: iab_content_3 returns the IAB category; custom also picks one of your customLabels (2–100 labels).
• aiKnowledgeFallback: when a site blocks automated visitors, disallows them in robots.txt, or drops the connection, classify it from the AI model's own knowledge of that domain instead of leaving it empty. Turn it off if you only want classifications based on the live website.
• maxPagesPerDomain: 1 = homepage only, 2 = homepage + About page, 3 = also one products or services page.

Output

You can download the dataset in various formats such as JSON, HTML, CSV or Excel. Example results:

[
    {
        "domain": "stripe.com",
        "status": "ok",
        "statusReason": null,
        "finalUrl": "https://stripe.com/",
        "iabTier1": { "id": "596", "name": "Technology & Computing" },
        "iabTier2": { "id": "599", "name": "Computing" },
        "industry": "Financial Services / Payments",
        "businessModel": "B2B SaaS",
        "language": "en",
        "brandSafetyRisk": "none",
        "brandSafetyFlags": [],
        "description": "Stripe is a financial services platform that helps businesses accept payments and move money through APIs and tooling.",
        "confidence": 0.8,
        "classificationSource": "website",
        "pagesAnalyzed": ["https://stripe.com/", "https://stripe.com/about"]
    },
    {
        "domain": "glock.com",
        "status": "ok",
        "finalUrl": "https://us.glock.com/en",
        "iabTier1": { "id": "v9i3On", "name": "Sensitive Topics" },
        "iabTier2": { "id": "avbNf2", "name": "Arms & Ammunition" },
        "industry": "Manufacturing / Firearms",
        "businessModel": "Manufacturing",
        "brandSafetyRisk": "medium",
        "brandSafetyFlags": [{ "code": "arms", "name": "Arms & Ammunition", "iabId": "avbNf2" }],
        "confidence": 0.9,
        "classificationSource": "website"
    },
    {
        "domain": "etsy.com",
        "status": "blocked",
        "statusReason": "Bot protection challenge (DataDome)",
        "iabTier1": { "id": "473", "name": "Shopping" },
        "iabTier2": null,
        "industry": "Retail / Marketplace",
        "businessModel": "Marketplace / platform",
        "description": "Etsy operates an online marketplace for handmade, vintage goods and craft supplies.",
        "confidence": 0.8,
        "classificationSource": "ai_knowledge"
    },
    {
        "domain": "bluewidget.com",
        "status": "parked",
        "statusReason": "Domain uses parking nameservers (afternic.com)",
        "iabTier1": null,
        "classificationSource": null
    }
]

A SUMMARY record in the run's key-value store counts domains per status, classified domains and charged events.

Use AI Website Categorization as an API

curl -X POST "https://api.apify.com/v2/acts/messnomore~website-categorization/run-sync-get-dataset-items" \
  -H "Authorization: Bearer YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"domains": ["stripe.com", "bbc.co.uk"]}'

For large lists, start the run asynchronously and read the dataset when it finishes. See the API tab for clients in JavaScript and Python. You can also run it from AI assistants through the Apify MCP server.

Tips for better results

• Use Pages per domain = 1 for the fastest runs on very large lists; 2 gives better results for company websites.
• Filter on status = ok and confidence ≥ 0.7 when you need only high-certainty categories.
• Use custom labels for your own segments (for example your ICP tiers) and keep the IAB category for ad-tech.
• Domains are de-duplicated by host name: www.example.com and example.com are checked separately on purpose, because they can be different sites.

Related guides

• How do I categorize a list of websites by IAB category, brand safety and parked status with an API?

FAQ

How does the website classification work?

For each domain the Actor checks robots.txt, reads the homepage (following redirects) and, when linked, the About page. It extracts the title, meta description, headings, structured data types and visible text, and sends that text to an AI model (OpenAI) together with the full IAB Content Taxonomy 3.1. The model picks the category by name (it cannot answer with a category that does not exist), plus the industry, business model, language, brand safety topics with their evidence, and a confidence score. The Actor maps the category to its official IAB ID and validates every value before saving it.

Which IAB taxonomy version is used?

IAB Tech Lab Content Taxonomy 3.1, Tier 1 and Tier 2 (37 + 323 categories). Version 3.1 has no general "News" category, so general news publishers and government portals are classified as Politics unless another topic clearly dominates; the industry field still says "News / Broadcasting".

How accurate is it?

In our tests on more than 150 well-known websites, about 97% got a sensible IAB category (the misses were close calls, such as an electronics maker filed under Business) and the language was always right. On a brand safety set run twice (casinos, betting, spirits, beer, tobacco, vaping, guns, adult, plus soft-drink and sports controls) every flag was correct in both runs, and ordinary sites got no flags. Some brands fit two neighbouring categories (a sportswear brand can be Shopping or Style & Fashion). Use confidence to filter uncertain results.

What do the statuses mean?

• ok: classified from its own pages. redirect: the domain forwards to another domain, and the target site is classified.
• parked: parking nameservers, a domain marketplace, a for-sale page, a default server page or a "coming soon" placeholder.
• unreachable: no DNS record, connection or TLS failure, an HTTP error, or a page with no readable text (for example a site that only renders with JavaScript).
• robots_disallowed: the site's robots.txt does not allow our crawler. blocked: the site answered with bot protection or "access denied".

What is the AI knowledge fallback?

Many large websites block automated visitors, and some simply drop or stall the connection. Instead of returning nothing, the Actor can ask the AI model what it already knows about that exact domain, without visiting the site again. Domains that do not exist (no DNS record) and parked domains are never classified this way. These results are marked classificationSource: "ai_knowledge" and their confidence is capped at 0.8. If the model does not know the domain (or the AI call fails), the item stays unclassified and costs only $0.001.

Does it respect robots.txt and website terms?

Yes. robots.txt is checked (for our WebsiteCategorizationBot user agent and the * group) before every page, and disallowed pages are skipped and reported. The Actor identifies itself with its own user agent, reads at most 3 public pages per domain, never logs in and never solves CAPTCHAs or works around bot protection.

Is my input stored or shared?

The domains and the text of their public pages are sent to the OpenAI API only to classify them; OpenAI does not use API data to train its models. Like every Apify run, your input and results stay in your own run storage, which you can delete at any time.

Is it legal to categorize websites?

The Actor only reads public homepages and About pages and stores derived labels plus a one-sentence description, not the page content. Our Actors are ethical and do not extract any private user data, such as email addresses, gender, or location. They only extract what the user has chosen to share publicly. We therefore believe that our Actors, when used for ethical purposes by Apify users, are safe. However, you should be aware that your results could contain personal data. Personal data is protected by the GDPR in the European Union and by other regulations around the world. You should not scrape personal data unless you have a legitimate reason to do so. If you're unsure whether your reason is legitimate, consult your lawyers.

Is this an official IAB product?

No. This Actor is not affiliated with or endorsed by IAB Tech Lab. It uses the IAB Tech Lab Content Taxonomy 3.1, which IAB Tech Lab publishes under the Creative Commons Attribution 3.0 License.

I found a wrong category or status. What can I do?

Open the Issues tab and include the domain. Feedback and feature requests are welcome there too.