Topical Cluster Url Mapper

Map public site pages into transparent topical clusters using page titles, meta descriptions, headings, breadcrumbs, category tags, and internal-link anchor text. You provide the taxonomy labels or matching rules; the Actor shows which source signals supported each assignment and leaves weak matches unclustered.

Use the output to inventory content hubs, group URLs for a migration, inspect thin clusters, or find pages whose subject is unclear. Cluster names are not inferred from an opaque model. They come from explicit taxonomySeeds and clusterRules, or from visible page terms when no rules match.

How classification works

For each accepted URL, the Actor:

1. extracts title, meta description, H1–H3 headings, breadcrumbs, category/tag labels, and same-domain link anchors;
2. builds a frequency-ranked list of visible topic words;
3. scores each configured cluster from exact label phrases, terms, and valid regular-expression patterns;
4. compares the winning score with minClusterConfidence;
5. emits the configured cluster, a visible-term cluster, or unclustered;
6. calculates overlap and sparse-coverage hints across the supplied URL set.

Invalid optional regular expressions are ignored. Empty, blocked, and human-verification pages produce explicit diagnostic rows.

Input

Supply URLs in startUrls, an XML sitemap in sitemapUrls, or captured records in htmlInputs. Captured records require sourceUrl plus html or currentHtml.

Field	Purpose	Default / cap
startUrls	Public pages to classify	empty
sitemapUrls	Public XML sitemaps whose <loc> pages are accepted	empty
htmlInputs	Captured HTML records for deterministic runs	empty
taxonomySeeds	Cluster labels whose meaningful words become terms	empty
clusterRules	Objects with name, optional terms, and optional patterns	empty
includeInternalLinks	Include same-domain link anchors and link counts	true
includeBreadcrumbs	Include visible breadcrumb text	true
minClusterConfidence	Minimum score ratio before a configured assignment is accepted	0.3; range 0–1
dedupeNearDuplicates	Collapse repeated page fingerprints	true
maxUrls	Maximum accepted page URLs	25
maxClusters	Maximum rule and seed definitions	25 by default, maximum 100
allowedDomains	Optional hostname allowlist	empty
includeEvidence	Keep evidence excerpts in rows	true
includeRawSnapshots	Store raw HTML snapshots	false
maxChargeUsd	Stop before the next event exceeds the budget	35 USD maximum

Example:

{
  "startUrls": [
    "https://docs.example/project-planning",
    "https://docs.example/resource-planning"
  ],
  "taxonomySeeds": ["Project planning", "Resource planning"],
  "clusterRules": [
    {
      "name": "Portfolio reporting",
      "terms": ["portfolio", "reporting"],
      "patterns": ["delivery dashboards?"]
    }
  ],
  "minClusterConfidence": 0.25,
  "maxUrls": 10,
  "maxPages": 10,
  "allowedDomains": ["docs.example"],
  "maxChargeUsd": 5
}

The Store default supplies no pages and returns a nonempty source_required diagnostic. It does not fetch an arbitrary broad site.

Dataset output

Every row includes common provenance and these mapping fields:

• url and canonicalUrl identify the accepted page;
• clusterName is the winning label, a visible-term label, or unclustered;
• clusterConfidence is the bounded score ratio used for the threshold decision;
• primaryTopic and secondaryTopics expose the highest-frequency visible terms;
• breadcrumbPath preserves detected breadcrumb text when enabled;
• internalLinkCount counts accepted same-domain links when enabled;
• overlapGroupId marks repeated page fingerprints when duplicates are retained;
• coverageGapHint identifies unclustered pages or clusters represented by only one supplied URL;
• issueCode, severity, evidenceSnippet, and recommendation explain the classification.

Near-duplicate deduplication is based on cluster name, page title, and primary topic. Set dedupeNearDuplicates to false when you need every URL preserved and want overlap identifiers on repeated fingerprints.

Reports and artifacts

With includeReport: true, key-value storage receives a Markdown summary, a complete CSV, issue rows in JSON, bounded evidence excerpts, and run metadata. Raw HTML snapshots are written only when includeRawSnapshots is explicitly enabled.

These artifacts make assignments auditable: analysts can trace a cluster back to titles, headings, breadcrumbs, tags, and link anchors rather than accepting a label without evidence.

Pay-per-event pricing

Event	Price (USD)	When charged
actor-start	0.035	after the run begins successfully
url-analyzed	0.0115	once per accepted source unit
topic-signal-extracted	0.0115	before a row backed by topic signals
cluster-generated	0.0115	before each paid cluster row
executive-report-generated	0.4000	before report artifacts are stored

Apify platform usage pass-through is off. maxChargeUsd is checked before every paid event and output write; rejected or partial charges stop the run cleanly.

Safety and interpretation

Only public HTTP and HTTPS pages are fetched. Loopback and private IPv4 targets are rejected. The optional domain allowlist can restrict accepted hosts, page counts are capped, and discovered page links are never added to the fetch queue. Internal links are analyzed only as on-page evidence.

Cluster assignments describe the supplied evidence set. They do not prove topical authority, search performance, or a recommended information architecture on their own. Review sparse clusters, ambiguous names, redirects, canonicalization, and business requirements before changing navigation or migrating URLs.