Blog Archive Taxonomy Extractor

Extract an article inventory from public blog index, category, tag, author, and archive pages. The Actor links visible cards to article URLs and captures titles, authors, publish and update dates, categories, tags, pagination position, duplicate URLs, and missing archive evidence.

This is designed for editorial migrations, archive quality audits, taxonomy cleanup, and content operations. It works from visible <article> cards first and can fall back to parseable Article or BlogPosting JSON-LD when a page does not expose article cards.

What is extracted

• absolute article URLs resolved against the archive page;
• visible article titles from headings or card links;
• author names from author-labelled elements;
• datetime, data-published, and data-updated values;
• category and tag labels from common class and rel=tag patterns;
• Article and BlogPosting headline, URL, author, dates, article section, and comma-separated keywords;
• pagination page number for each archive source;
• repeated article URLs across supplied archive pages;
• missing URL, title, author, and publish-date evidence.

The Actor never fills missing authors or dates from guesses. A record with incomplete evidence is preserved and marked with an archive issue so the inventory remains useful.

Input

Use archiveUrls for public archive pages, sitemapUrls for a bounded set from XML sitemaps, or htmlInputs for captured archive HTML.

Field	Purpose	Default / cap
archiveUrls	Public archive, category, tag, or author-index URLs	empty
startUrls	Additional public archive URLs	empty
sitemapUrls	Public XML sitemaps	empty
htmlInputs	Captured records with sourceUrl and HTML	empty
followPagination	Follow a same-domain rel=next link	false
maxDepth	Maximum rel=next depth	0 by default, maximum 1
maxArchivePages	Maximum accepted archive pages	1 by default, maximum 25
maxArticles	Maximum article records per archive page	100, maximum 1,000
extractAuthors	Extract visible or structured author names	true
extractTags	Extract categories and tags	true
extractDates	Extract publication and update dates	true
allowedDomains	Optional hostname allowlist	empty
includeEvidence	Keep evidence excerpts in rows	true
includeRawSnapshots	Store raw accepted archive HTML	false
maxChargeUsd	Stop before the next event exceeds this amount	15 USD maximum

Example:

{
  "archiveUrls": ["https://blog.example/archive"],
  "followPagination": true,
  "maxDepth": 1,
  "maxArchivePages": 2,
  "maxArticles": 100,
  "extractAuthors": true,
  "extractTags": true,
  "extractDates": true,
  "allowedDomains": ["blog.example"],
  "maxChargeUsd": 3
}

When pagination is enabled, only the first same-domain rel=next URL on an accepted page can enter the bounded queue. Other discovered links are not crawled. The Store default has no archive source and returns a nonempty source_required diagnostic.

Dataset output

Each dataset item has stable provenance plus archive fields:

• archiveUrl identifies the page where the article record appeared;
• articleUrl and articleTitle identify the linked content;
• authorName, publishedDate, and updatedDate preserve visible or structured values;
• categories and tags are arrays suitable for spreadsheet or database import;
• paginationPage is an integer beginning at 1;
• duplicateArticleUrl is set when the same resolved URL appeared previously;
• archiveHealthIssue records missing evidence, duplicates, blocked sources, or source diagnostics;
• issueCode, severity, evidenceSnippet, and recommendation explain what needs attention;
• sourceUrl, canonicalUrl, httpStatus, and scannedAt preserve source context.

Turning an extraction toggle off returns the affected fields as null or empty arrays and prevents those omissions from being flagged. maxArticles is applied before dataset rows are produced.

Reports

With reports enabled, key-value storage receives:

• a Markdown run summary;
• a CSV with every article and diagnostic row;
• a JSON issue list;
• bounded evidence excerpts;
• metadata with page counts, row counts, issue codes, pricing mode, and consumed input fields.

Raw archive HTML is opt-in and stored separately from ordinary reports.

Pay-per-event pricing

Event	Price (USD)	When charged
actor-start	0.0100	after successful startup
archive-page-scanned	0.0065	once for each accepted archive source unit
taxonomy-record-extracted	0.00295	before a paid article/taxonomy row
archive-issue-emitted	0.00295	before a paid archive diagnostic or duplicate row
audit-report-generated	0.0800	before report artifacts are stored

Apify platform usage pass-through is off. Before an output row or report is written, the Actor verifies the next event fits within maxChargeUsd and that the charge was fully accepted.

Safety and limits

The fetcher accepts only public HTTP and HTTPS URLs. Loopback and private IPv4 targets are rejected. Domain allowlists, page caps, request timeouts, and bounded delays are applied before fetching. It does not bypass access controls or human-verification challenges; those sources produce diagnostics.

Archive markup varies. Client-rendered cards may require captured HTML from an authorized browser workflow. A missing field means it was not observed in the accepted source, not that the underlying article lacks it. Validate dates, author identity, canonical URLs, and taxonomy policy before a migration or bulk editorial change.