# FreightWaves News Scraper (`muhammadafzal/freightwaves-news-scraper`) Actor

Scrape public FreightWaves news listings and article pages for headlines, excerpts, authors, dates, sections, tags, images, and visible article text. Use for freight market intelligence and news monitoring. Not for SONAR or subscriber-only data. $0.005/article plus $0.00005 start.

- **URL**: https://apify.com/muhammadafzal/freightwaves-news-scraper.md
- **Developed by:** [Muhammad Afzal](https://apify.com/muhammadafzal) (community)
- **Categories:** News, Automation, Other
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $5.00 / 1,000 article returneds

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## FreightWaves News Scraper

Scrape public FreightWaves logistics and transportation news pages into structured records for freight market intelligence, monitoring, research, and news-feed pipelines.

### What it extracts

| Field | Description |
| --- | --- |
| `articleUrl` | Canonical public FreightWaves article URL |
| `title` | Article headline |
| `dek` | Public excerpt or description |
| `author` | Byline when available |
| `publishedAt`, `updatedAt` | Publication and update timestamps |
| `section` | FreightWaves section or category |
| `tags` | Public topic labels |
| `imageUrl` | Open Graph or article image |
| `body` | Visible public article text, when requested and available |
| `sourceUrl` | Listing or input URL where the article was discovered |
| `warnings` | Non-fatal extraction notes |

The dataset also contains one final `recordType: "summary"` record with page counts, article counts, warnings, and the effective run configuration. The same summary is written to the `RUN_SUMMARY` key-value store key.

### When to use it

Use this actor when you need a bounded batch of public FreightWaves stories from a news, category, tag, pagination, or article URL. It is suitable for:

- Freight broker and carrier market monitoring
- 3PL and shipper competitive intelligence
- Transportation newsletter and RSS-style feeds
- Trucking, rail, maritime, air-cargo, and supply-chain topic research
- Search-indexed article archives and historical snapshots
- AI-agent workflows that need short, stable article records

Do not use it for FreightWaves SONAR data, private dashboards, subscriber-only material, account data, or actions that modify FreightWaves. The actor does not log in, bypass a paywall, or attempt to defeat a challenge page. If a public page exposes metadata but not the full body, the actor returns the metadata and adds a warning.

### Input

The default input scrapes the public FreightWaves News listing:

```json
{
  "startUrls": [
    { "url": "https://www.freightwaves.com/news/category/news" }
  ],
  "maxResults": 20,
  "maxPages": 3,
  "includeArticleBody": true,
  "requestDelayMs": 750
}
```

`startUrls` accepts up to 10 same-site FreightWaves URLs. Use a direct article URL for one known story, a category URL for a topic, a tag URL for a public tag archive, or a paginated listing URL for a known page. `maxResults` is the total article cap across all inputs and is bounded to 1–100. `maxPages` controls listing pagination and is bounded to 1–10. `includeArticleBody` defaults to `true`; set it to `false` for metadata-only output. `requestDelayMs` defaults to 750 milliseconds and is bounded to 250–5000 milliseconds for polite pacing.

### Output

An article record is flat and stable so it can be consumed by APIs, tables, or AI agents:

```json
{
  "recordType": "article",
  "articleUrl": "https://www.freightwaves.com/news/example-story",
  "canonicalUrl": "https://www.freightwaves.com/news/example-story",
  "articleId": "a4f1c3d89bb2",
  "title": "Freight market update",
  "dek": "A short public excerpt.",
  "author": "FreightWaves Staff",
  "publishedAt": "2026-08-01T12:00:00.000Z",
  "updatedAt": null,
  "section": "News",
  "imageUrl": "https://www.freightwaves.com/wp-content/uploads/example.jpg",
  "tags": ["trucking", "logistics"],
  "body": "Visible public article text...",
  "sourceUrl": "https://www.freightwaves.com/news/category/news",
  "scrapedAt": "2026-08-02T10:00:00.000Z",
  "warnings": [],
  "summary": null
}
```

When the actor reaches the article cap, the first records are kept and the run stops. The final summary record includes `articlesScraped`, `listingPagesVisited`, `articlePagesVisited`, `requestedMaxResults`, `warnings`, and `buildNumber`. If the target returns a valid empty page, the run completes with a truthful summary instead of manufacturing article records.

### Pricing

| Event | Price |
| --- | ---: |
| Actor start | $0.00005 per run |
| Article returned | $0.005 per unique article |

For example, a run capped at 20 articles has a maximum article-event cost of $0.10 plus the start event. Apify compute and proxy usage may also apply when the pay-per-usage billing option is selected. The actor prints its maximum article-event cost before the first article charge and reports the actual article charges at completion.

### Reliability and limits

The crawler uses a single concurrent request worker, a bounded request queue, session persistence, retry handling for transient upstream errors, and same-host URL validation. It connects directly to the public site by default so the actor can run with Apify Limited permissions and without requiring account-level proxy APIs. The public News listing is used as the default canary path. HTTP 403/429 responses retire the session and retry; repeated request failures are recorded in the run summary. A completely blocked run fails honestly with an actionable status message rather than returning a misleading empty dataset.

The source site can change its HTML, date labels, navigation, or article-body layout. Extraction uses metadata, JSON-LD, semantic selectors, and generic fallbacks. Optional fields are explicit `null` values when not exposed. Public-page availability is not a guarantee that every story has full text, and this actor should be operated in compliance with FreightWaves terms, applicable law, and the target site's published access preferences.

### API example

JavaScript:

```js
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const { defaultDatasetId } = await client.actor('umqHl4LhqmPHcBh0L/freightwaves-news-scraper').call({
  maxResults: 10,
  startUrls: [{ url: 'https://www.freightwaves.com/news/category/news' }]
});
const { items } = await client.dataset(defaultDatasetId).listItems();
console.log(items);
```

Python:

```python
from apify_client import ApifyClient
import os

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("umqHl4LhqmPHcBh0L/freightwaves-news-scraper").call(
    run_input={"maxResults": 10}
)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["title"])
```

### FAQ

#### Can I scrape a single article?

Yes. Pass the article URL in `startUrls` and set `maxResults` to `1`.

#### Does it scrape FreightWaves SONAR?

No. SONAR, private dashboards, and subscriber-only content are outside the actor's public-page scope.

#### Why is `body` null?

The public page may expose metadata without full text, or you may have set `includeArticleBody` to `false`. Check the record's `warnings` array.

#### Does the actor publish or modify content?

No. It is a read-only scraper for public pages.

#### How should I report a markup change?

Include the input URL, run ID, missing field, and a small example of the changed page in the Apify issue thread. Do not share account credentials or private content.

### Legal note

This actor is provided for lawful access to public web content. You are responsible for complying with FreightWaves' terms, robots/access preferences, copyright rules, privacy law, and any other applicable requirements. Do not use it to bypass authentication, paywalls, rate limits, or technical access controls.

# Actor input Schema

## `startUrls` (type: `array`):

Use this when you want articles from a FreightWaves news, category, tag, pagination, or article URL. Pass full URLs such as https://www.freightwaves.com/news/category/news. Defaults to the public News listing and does not accept SONAR or subscriber-only URLs.

## `maxResults` (type: `integer`):

Use this to cap the number of article records returned and billed. Accepts 1 to 100 articles; the default is 20. This is a total across all input URLs, not a per-page limit and not a request count.

## `maxPages` (type: `integer`):

Use this to follow pagination from a FreightWaves listing before stopping. Accepts 1 to 10 pages and defaults to 3. This controls listing pages only, not article detail pages or the maximum article count.

## `includeArticleBody` (type: `boolean`):

Use this when you need the visible article text for monitoring or downstream analysis. Defaults to true; set false for a smaller, metadata-only dataset. This does not bypass paywalls or return subscriber-only text.

## `requestDelayMs` (type: `integer`):

Use this to add polite pacing between requests to FreightWaves. Accepts 250 to 5000 milliseconds and defaults to 750. This is not a concurrency control or a way to bypass access restrictions.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://www.freightwaves.com/news/category/news"
    }
  ],
  "maxResults": 20,
  "maxPages": 3,
  "includeArticleBody": true,
  "requestDelayMs": 750
}
```

# Actor output Schema

## `articles` (type: `string`):

Structured article records and one final summary record.

## `runSummary` (type: `string`):

Latest run diagnostics stored in the default key-value store.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://www.freightwaves.com/news/category/news"
        }
    ],
    "maxResults": 20
};

// Run the Actor and wait for it to finish
const run = await client.actor("muhammadafzal/freightwaves-news-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://www.freightwaves.com/news/category/news" }],
    "maxResults": 20,
}

# Run the Actor and wait for it to finish
run = client.actor("muhammadafzal/freightwaves-news-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://www.freightwaves.com/news/category/news"
    }
  ],
  "maxResults": 20
}' |
apify call muhammadafzal/freightwaves-news-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,muhammadafzal/freightwaves-news-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/hmg4rXN6wIZdSOqSf/builds/HveuOBY4OJF2eMOv7/openapi.json