Trade Journals Scraper — AI-Ready Markdown

Scrapes 21 niche industry trade journals into one unified schema built for LLM ingestion. Every article comes back as clean Markdown with the ads, share bars, newsletter CTAs, related-article rails and event promos already stripped out — ready to chunk into a RAG pipeline or a fine-tuning set without a second cleaning pass.

No login, no cookies, no browser. HTTP only.

Publications covered

Industry	Publications
Pharma / Biotech	FiercePharma, FierceBiotech, BioPharma Dive, pharmaphorum
Healthcare	FierceHealthcare, Healthcare Dive, MedCity News
Freight & Logistics	FreightWaves, Transport Topics, FleetOwner, Logistics Management, SupplyChainBrain, Supply Chain Dive
Energy & Construction	Utility Dive, Construction Dive
Retail & Consumer	Retail Dive, Food Dive, Marketing Dive
Enterprise & Finance	CIO Dive, Banking Dive, HR Dive

Not on the list? Paste any publication or article URL. Unlisted sites run through a generic extractor that finds the article body by prose density rather than by a hard-coded selector — it was verified against six unrelated CMS platforms (WordPress, Drupal 9, Drupal 11, Nuxt, Industry Dive's custom CMS, and Endeavor's).

What you can feed it

• Publications — pick from the list, or give a site URL
• Category archive URLs — e.g. https://www.fiercepharma.com/manufacturing crawls just that section
• Article URLs — exact articles, discovery skipped
• Industry topics — keep only articles matching Biotech, Supply Chain, M&A, …
• Date window — publishedAfter / publishedBefore, ideal for incremental corpus refreshes

All of these can be combined in a single run, and they all land in one dataset.

Output

Three record types share the dataset, told apart by recordType.

ARTICLE — the unified AI-ready row

Field	Notes
articleId	Stable 16-char hash of the canonical URL — comparable across publications, unlike per-CMS ids
source	"FiercePharma", "FreightWaves", …
industryTopic	The publication's own section/tag, falling back to this actor's industry label
title, author
publishedDate	Always ISO 8601 UTC with a Z — normalised from RFC-2822, naive-local and offset-ISO sources
url	Final URL after redirects
contentMarkdown	The cleaned article body

Plus description, modifiedDate, keywords, imageUrl, publisher, publicationDomain, isAccessibleForFree, wordCount, bodySelector (which rule found the body) and isRegisteredPublication. Turn on Also include raw body HTML to add contentHtml.

DISCOVERY_SUMMARY — one per publication

How the articles were found (feed / sitemap / S1-wp / S2-drupal / archive-links), how many were discovered vs. attempted vs. succeeded, and how many your filters removed — so a small result set is explainable rather than mysterious.

ERROR — one per input that failed

Every input maps to at least one row. Nothing disappears silently.

_error	Meaning
not_found	HTTP 404/410 — article de-listed
blocked_by_waf	Challenged on every TLS profile and exit IP
extraction_failed	Fetched fine, but the cleaned body came out under minBodyChars — usually a paywall teaser
discovery_failed	No feed, sitemap or recognisable article links
fetch_failed	Transient failure that survived the retry ladder
invalid_input	Refused before any request, with the reason

How the body is extracted

Metadata and body come from different layers, because on every publication tested the JSON-LD NewsArticle block carries a headline, author and date but an empty articleBody. So:

• Metadata — JSON-LD → OpenGraph/meta tags → the RSS entry → URL path
• Body — one of three, chosen by domain:
  • S1-wp WordPress REST (content.rendered, with author and taxonomy embedded)
  • S2-drupal Drupal JSON:API
  • S3-dom the article page, body located by selector or prose density

The _source field on every row names the pair that produced it, so you can tell a full-fidelity API row from a DOM-salvaged one without re-fetching.

Known limits

• Filtering by topic is done by this actor, not by the publisher. Only 2 of the 21 publications expose a server-side category endpoint, so filtering client-side is the only behaviour that is identical everywhere. The summary row always reports how many articles the filter removed.
• Paywalled articles return extraction_failed rather than a teaser row. That is deliberate — a 200-character teaser stored as if it were an article is worse than an honest error.
• Author is ~88% complete across a full run; some publications genuinely publish articles with no byline.
• Feed-based discovery reaches roughly the most recent 10–40 articles per publication. For deeper history use publishedAfter/publishedBefore with a category archive URL, or the sitemap-backed publications.
• Publications whose robots.txt disallows Claude/anthropic agents sitewide are refused with an explanation, not silently skipped.

Proxy

Residential is the default and is strongly recommended: most of these sites sit behind Cloudflare, which fingerprints datacenter ASNs before it looks at the TLS handshake. The actor rotates through four browser TLS fingerprints automatically — one publication in the registry answers 403 to Chrome and 200 to Safari on the same IP in the same second.