# Topical Cluster Url Mapper (`junipr/topical-cluster-url-mapper`) Actor

Map URLs into topical clusters using page titles, headings, breadcrumbs, internal links, tags, and supplied taxonomy rules to reveal coverage gaps and...

- **URL**: https://apify.com/junipr/topical-cluster-url-mapper.md
- **Developed by:** [junipr](https://apify.com/junipr) (community)
- **Categories:** SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $11.50 / 1,000 url analyzeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Topical Cluster Url Mapper

Map public site pages into transparent topical clusters using page titles, meta descriptions, headings, breadcrumbs, category tags, and internal-link anchor text. You provide the taxonomy labels or matching rules; the Actor shows which source signals supported each assignment and leaves weak matches `unclustered`.

Use the output to inventory content hubs, group URLs for a migration, inspect thin clusters, or find pages whose subject is unclear. Cluster names are not inferred from an opaque model. They come from explicit `taxonomySeeds` and `clusterRules`, or from visible page terms when no rules match.

### How classification works

For each accepted URL, the Actor:

1. extracts title, meta description, H1–H3 headings, breadcrumbs, category/tag labels, and same-domain link anchors;
2. builds a frequency-ranked list of visible topic words;
3. scores each configured cluster from exact label phrases, terms, and valid regular-expression patterns;
4. compares the winning score with `minClusterConfidence`;
5. emits the configured cluster, a visible-term cluster, or `unclustered`;
6. calculates overlap and sparse-coverage hints across the supplied URL set.

Invalid optional regular expressions are ignored. Empty, blocked, and human-verification pages produce explicit diagnostic rows.

### Input

Supply URLs in `startUrls`, an XML sitemap in `sitemapUrls`, or captured records in `htmlInputs`. Captured records require `sourceUrl` plus `html` or `currentHtml`.

| Field | Purpose | Default / cap |
| --- | --- | --- |
| `startUrls` | Public pages to classify | empty |
| `sitemapUrls` | Public XML sitemaps whose `<loc>` pages are accepted | empty |
| `htmlInputs` | Captured HTML records for deterministic runs | empty |
| `taxonomySeeds` | Cluster labels whose meaningful words become terms | empty |
| `clusterRules` | Objects with `name`, optional `terms`, and optional `patterns` | empty |
| `includeInternalLinks` | Include same-domain link anchors and link counts | `true` |
| `includeBreadcrumbs` | Include visible breadcrumb text | `true` |
| `minClusterConfidence` | Minimum score ratio before a configured assignment is accepted | 0.3; range 0–1 |
| `dedupeNearDuplicates` | Collapse repeated page fingerprints | `true` |
| `maxUrls` | Maximum accepted page URLs | 25 |
| `maxClusters` | Maximum rule and seed definitions | 25 by default, maximum 100 |
| `allowedDomains` | Optional hostname allowlist | empty |
| `includeEvidence` | Keep evidence excerpts in rows | `true` |
| `includeRawSnapshots` | Store raw HTML snapshots | `false` |
| `maxChargeUsd` | Stop before the next event exceeds the budget | 35 USD maximum |

Example:

```json
{
  "startUrls": [
    "https://docs.example/project-planning",
    "https://docs.example/resource-planning"
  ],
  "taxonomySeeds": ["Project planning", "Resource planning"],
  "clusterRules": [
    {
      "name": "Portfolio reporting",
      "terms": ["portfolio", "reporting"],
      "patterns": ["delivery dashboards?"]
    }
  ],
  "minClusterConfidence": 0.25,
  "maxUrls": 10,
  "maxPages": 10,
  "allowedDomains": ["docs.example"],
  "maxChargeUsd": 5
}
```

The Store default supplies no pages and returns a nonempty `source_required` diagnostic. It does not fetch an arbitrary broad site.

### Dataset output

Every row includes common provenance and these mapping fields:

- `url` and `canonicalUrl` identify the accepted page;
- `clusterName` is the winning label, a visible-term label, or `unclustered`;
- `clusterConfidence` is the bounded score ratio used for the threshold decision;
- `primaryTopic` and `secondaryTopics` expose the highest-frequency visible terms;
- `breadcrumbPath` preserves detected breadcrumb text when enabled;
- `internalLinkCount` counts accepted same-domain links when enabled;
- `overlapGroupId` marks repeated page fingerprints when duplicates are retained;
- `coverageGapHint` identifies unclustered pages or clusters represented by only one supplied URL;
- `issueCode`, `severity`, `evidenceSnippet`, and `recommendation` explain the classification.

Near-duplicate deduplication is based on cluster name, page title, and primary topic. Set `dedupeNearDuplicates` to `false` when you need every URL preserved and want overlap identifiers on repeated fingerprints.

### Reports and artifacts

With `includeReport: true`, key-value storage receives a Markdown summary, a complete CSV, issue rows in JSON, bounded evidence excerpts, and run metadata. Raw HTML snapshots are written only when `includeRawSnapshots` is explicitly enabled.

These artifacts make assignments auditable: analysts can trace a cluster back to titles, headings, breadcrumbs, tags, and link anchors rather than accepting a label without evidence.

### Pay-per-event pricing

| Event | Price (USD) | When charged |
| --- | ---: | --- |
| `actor-start` | 0.035 | after the run begins successfully |
| `url-analyzed` | 0.0115 | once per accepted source unit |
| `topic-signal-extracted` | 0.0115 | before a row backed by topic signals |
| `cluster-generated` | 0.0115 | before each paid cluster row |
| `executive-report-generated` | 0.4000 | before report artifacts are stored |

Apify platform usage pass-through is off. `maxChargeUsd` is checked before every paid event and output write; rejected or partial charges stop the run cleanly.

### Safety and interpretation

Only public HTTP and HTTPS pages are fetched. Loopback and private IPv4 targets are rejected. The optional domain allowlist can restrict accepted hosts, page counts are capped, and discovered page links are never added to the fetch queue. Internal links are analyzed only as on-page evidence.

Cluster assignments describe the supplied evidence set. They do not prove topical authority, search performance, or a recommended information architecture on their own. Review sparse clusters, ambiguous names, redirects, canonicalization, and business requirements before changing navigation or migrating URLs.

# Actor input Schema

## `startUrls` (type: `array`):

Public page URLs to classify into topical clusters.

## `sitemapUrls` (type: `array`):

Optional public XML sitemap URLs. Accepted page URLs remain bounded by maxUrls.

## `htmlInputs` (type: `array`):

Optional records with sourceUrl and html or currentHtml for deterministic analysis.

## `taxonomySeeds` (type: `array`):

Explicit cluster labels whose words become transparent matching terms.

## `clusterRules` (type: `array`):

Rules with name plus optional terms and regular-expression patterns.

## `includeInternalLinks` (type: `boolean`):

Include visible internal links as clustering evidence.

## `includeBreadcrumbs` (type: `boolean`):

Include breadcrumb text as clustering evidence.

## `minClusterConfidence` (type: `number`):

Scores below this threshold are labeled unclustered.

## `dedupeNearDuplicates` (type: `boolean`):

Merge near-duplicate URLs before cluster assignment.

## `maxUrls` (type: `integer`):

Maximum URLs to map in one run.

## `maxClusters` (type: `integer`):

Maximum topic clusters to emit in one run.

## `allowedDomains` (type: `array`):

Optional hostname allowlist for fetched pages.

## `maxPages` (type: `integer`):

Maximum pages to fetch in one run.

## `maxDepth` (type: `integer`):

General link discovery is disabled.

## `includeEvidence` (type: `boolean`):

Include short source evidence snippets in output rows.

## `includeRawSnapshots` (type: `boolean`):

Opt in to raw HTML artifacts in key-value storage.

## `requestDelayMs` (type: `integer`):

Delay in milliseconds between outbound page requests.

## `timeoutMs` (type: `integer`):

Maximum time in milliseconds to wait for a page request.

## `userAgentMode` (type: `string`):

User agent profile to use for public page requests.

## `maxChargeUsd` (type: `number`):

Maximum estimated PPE spend before the actor exits gracefully.

## `includeReport` (type: `boolean`):

Write the markdown summary report to the run key-value store.

## Actor input object example

```json
{
  "startUrls": [],
  "sitemapUrls": [],
  "htmlInputs": [],
  "taxonomySeeds": [],
  "clusterRules": [],
  "includeInternalLinks": true,
  "includeBreadcrumbs": true,
  "minClusterConfidence": 0.3,
  "dedupeNearDuplicates": true,
  "maxUrls": 25,
  "maxClusters": 25,
  "allowedDomains": [],
  "maxPages": 1,
  "maxDepth": 0,
  "includeEvidence": true,
  "includeRawSnapshots": false,
  "requestDelayMs": 250,
  "timeoutMs": 15000,
  "userAgentMode": "standard",
  "maxChargeUsd": 35,
  "includeReport": true
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `report` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("junipr/topical-cluster-url-mapper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("junipr/topical-cluster-url-mapper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call junipr/topical-cluster-url-mapper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,junipr/topical-cluster-url-mapper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/W51WTQ6JNff0HX28V/builds/LudgReMgjmQDopXBE/openapi.json