# Trade Journals Scraper — AI-Ready Markdown (`scrapyx/trade-journals-scraper`) Actor

Scrapes 21 niche industry trade journals (FiercePharma, FreightWaves, the Industry Dive network, Transport Topics and more) into one LLM-ready schema. Every article arrives as clean Markdown with ads, share bars and newsletter CTAs stripped — ready for RAG. Unlisted publications work too.

- **URL**: https://apify.com/scrapyx/trade-journals-scraper.md
- **Developed by:** [Ibnu Adzim](https://apify.com/scrapyx) (community)
- **Categories:** News, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.10 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Trade Journals Scraper — AI-Ready Markdown

Scrapes **21 niche industry trade journals** into one unified schema built
for LLM ingestion. Every article comes back as clean Markdown with the ads,
share bars, newsletter CTAs, related-article rails and event promos already
stripped out — ready to chunk into a RAG pipeline or a fine-tuning set
without a second cleaning pass.

No login, no cookies, no browser. HTTP only.

### Publications covered

| Industry | Publications |
| --- | --- |
| Pharma / Biotech | FiercePharma, FierceBiotech, BioPharma Dive, pharmaphorum |
| Healthcare | FierceHealthcare, Healthcare Dive, MedCity News |
| Freight & Logistics | FreightWaves, Transport Topics, FleetOwner, Logistics Management, SupplyChainBrain, Supply Chain Dive |
| Energy & Construction | Utility Dive, Construction Dive |
| Retail & Consumer | Retail Dive, Food Dive, Marketing Dive |
| Enterprise & Finance | CIO Dive, Banking Dive, HR Dive |

**Not on the list?** Paste any publication or article URL. Unlisted sites run
through a generic extractor that finds the article body by prose density
rather than by a hard-coded selector — it was verified against six unrelated
CMS platforms (WordPress, Drupal 9, Drupal 11, Nuxt, Industry Dive's custom
CMS, and Endeavor's).

### What you can feed it

- **Publications** — pick from the list, or give a site URL
- **Category archive URLs** — e.g. `https://www.fiercepharma.com/manufacturing`
  crawls just that section
- **Article URLs** — exact articles, discovery skipped
- **Industry topics** — keep only articles matching `Biotech`, `Supply Chain`, `M&A`, …
- **Date window** — `publishedAfter` / `publishedBefore`, ideal for incremental
  corpus refreshes

All of these can be combined in a single run, and they all land in one dataset.

### Output

Three record types share the dataset, told apart by `recordType`.

#### `ARTICLE` — the unified AI-ready row

| Field | Notes |
| --- | --- |
| `articleId` | Stable 16-char hash of the canonical URL — comparable across publications, unlike per-CMS ids |
| `source` | `"FiercePharma"`, `"FreightWaves"`, … |
| `industryTopic` | The publication's own section/tag, falling back to this actor's industry label |
| `title`, `author` | |
| `publishedDate` | **Always ISO 8601 UTC with a `Z`** — normalised from RFC-2822, naive-local and offset-ISO sources |
| `url` | Final URL after redirects |
| `contentMarkdown` | The cleaned article body |

Plus `description`, `modifiedDate`, `keywords`, `imageUrl`, `publisher`,
`publicationDomain`, `isAccessibleForFree`, `wordCount`, `bodySelector`
(which rule found the body) and `isRegisteredPublication`. Turn on
**Also include raw body HTML** to add `contentHtml`.

#### `DISCOVERY_SUMMARY` — one per publication

How the articles were found (`feed` / `sitemap` / `S1-wp` / `S2-drupal` /
`archive-links`), how many were discovered vs. attempted vs. succeeded, and
**how many your filters removed** — so a small result set is explainable
rather than mysterious.

#### `ERROR` — one per input that failed

Every input maps to at least one row. Nothing disappears silently.

| `_error` | Meaning |
| --- | --- |
| `not_found` | HTTP 404/410 — article de-listed |
| `blocked_by_waf` | Challenged on every TLS profile and exit IP |
| `extraction_failed` | Fetched fine, but the cleaned body came out under `minBodyChars` — usually a paywall teaser |
| `discovery_failed` | No feed, sitemap or recognisable article links |
| `fetch_failed` | Transient failure that survived the retry ladder |
| `invalid_input` | Refused before any request, with the reason |

### How the body is extracted

Metadata and body come from **different layers**, because on every
publication tested the JSON-LD `NewsArticle` block carries a headline, author
and date but an *empty* `articleBody`. So:

- **Metadata** — JSON-LD → OpenGraph/meta tags → the RSS entry → URL path
- **Body** — one of three, chosen by domain:
  - `S1-wp` WordPress REST (`content.rendered`, with author and taxonomy embedded)
  - `S2-drupal` Drupal JSON:API
  - `S3-dom` the article page, body located by selector or prose density

The `_source` field on every row names the pair that produced it, so you can
tell a full-fidelity API row from a DOM-salvaged one without re-fetching.

### Known limits

- **Filtering by topic is done by this actor, not by the publisher.** Only 2
  of the 21 publications expose a server-side category endpoint, so filtering
  client-side is the only behaviour that is identical everywhere. The summary
  row always reports how many articles the filter removed.
- **Paywalled articles** return `extraction_failed` rather than a teaser row.
  That is deliberate — a 200-character teaser stored as if it were an article
  is worse than an honest error.
- **Author is ~88% complete** across a full run; some publications genuinely
  publish articles with no byline.
- **Feed-based discovery reaches roughly the most recent 10–40 articles** per
  publication. For deeper history use `publishedAfter`/`publishedBefore` with
  a category archive URL, or the sitemap-backed publications.
- Publications whose robots.txt disallows Claude/anthropic agents sitewide
  are **refused with an explanation**, not silently skipped.

### Proxy

Residential is the default and is strongly recommended: most of these sites
sit behind Cloudflare, which fingerprints datacenter ASNs before it looks at
the TLS handshake. The actor rotates through four browser TLS fingerprints
automatically — one publication in the registry answers 403 to Chrome and 200
to Safari on the same IP in the same second.

# Actor input Schema

## `publications` (type: `array`):

Which trade journals to crawl. Built-in names: fiercepharma, fiercebiotech, fiercehealthcare, freightwaves, supplychaindive, biopharmadive, healthcaredive, constructiondive, utilitydive, ciodive, retaildive, marketingdive, hrdive, bankingdive, fooddive, transporttopics, supplychainbrain, medcitynews, pharmaphorum, fleetowner, logisticsmgmt. You can also type any other publication's URL (e.g. https://www.example-journal.com/) — unlisted sites fall back to a generic feed/sitemap + prose-density article extractor verified against six different CMS platforms. Paste a category archive URL (e.g. https://www.fiercepharma.com/manufacturing) to crawl just that section.

## `articleUrls` (type: `array`):

Exact article URLs to scrape directly, skipping discovery. Works on any publication, listed or not. Combine freely with 'Publications' in the same run.

## `categories` (type: `array`):

Keep only articles whose topic, tags, headline or URL mention one of these (e.g. Biotech, Supply Chain, M\&A). Case-insensitive, any-match. Applied by this actor after discovery — only 2 of the 21 built-in publications expose a server-side category filter, so filtering here keeps the behaviour identical everywhere. The DISCOVERY\_SUMMARY row reports how many articles the filter removed. Leave empty for everything.

## `maxArticlesPerPublication` (type: `integer`):

Upper bound on articles returned for each entry in 'Publications'. Article URLs are always fetched regardless of this cap.

## `publishedAfter` (type: `string`):

Keep only articles published on or after this date/time (e.g. 2026-08-01 or 2026-08-01T00:00:00Z). Articles whose date could not be determined are kept, never silently dropped.

## `publishedBefore` (type: `string`):

Keep only articles published on or before this date/time (e.g. 2026-08-30). Articles whose date could not be determined are kept, never silently dropped.

## `includeRawHtml` (type: `boolean`):

Add a 'contentHtml' field holding the cleaned article HTML alongside the markdown. Useful for debugging extraction; roughly triples dataset size.

## `minBodyChars` (type: `integer`):

Articles whose cleaned markdown comes out shorter than this are reported as an extraction failure rather than pushed as a row. Guards against paywall teasers being stored as if they were full articles. Lower it only if you deliberately want short posts.

## `maxConcurrency` (type: `integer`):

Parallel article fetches. Once the request-interval cap below binds, raising this buys nothing.

## `minRequestInterval` (type: `integer`):

Politeness pacing, applied to request STARTS across all workers. This, not concurrency, is the honest speed control. 0 disables pacing.

## `tlsProfile` (type: `string`):

Which browser TLS fingerprint to try first. The actor rotates through the rest of the pool automatically on a block, so this only sets the starting rung — change it only if you are debugging a specific publication.

## `proxyConfiguration` (type: `object`):

Apify Proxy on the shared datacenter pool. This is the default because it is included in your plan at no extra cost and this target works through it. If you start seeing blocks, challenges or empty results, switch the group here to Residential -- it uses real consumer IPs and gets through more, but Apify bills residential traffic per gigabyte, so leave it off unless you need it.

## Actor input object example

```json
{
  "publications": [
    "fiercepharma",
    "freightwaves",
    "supplychaindive"
  ],
  "categories": [],
  "maxArticlesPerPublication": 25,
  "includeRawHtml": false,
  "minBodyChars": 400,
  "maxConcurrency": 5,
  "minRequestInterval": 0,
  "tlsProfile": "chrome124",
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `items` (type: `string`):

One row per scraped record. See the dataset's default view for field definitions.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "fiercepharma",
        "freightwaves",
        "supplychaindive"
    ],
    "categories": []
};

// Run the Actor and wait for it to finish
const run = await client.actor("scrapyx/trade-journals-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "publications": [
        "fiercepharma",
        "freightwaves",
        "supplychaindive",
    ],
    "categories": [],
}

# Run the Actor and wait for it to finish
run = client.actor("scrapyx/trade-journals-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "fiercepharma",
    "freightwaves",
    "supplychaindive"
  ],
  "categories": []
}' |
apify call scrapyx/trade-journals-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,scrapyx/trade-journals-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/pgkrh8KtlNhfecwtU/builds/Lcu82wx8PPCDc45pT/openapi.json