# Freight Surcharge Monitor - FreightSurcharge Watch (`leons0016/freight-surcharge-monitor`) Actor

Freight surcharge monitor: records the EIA weekly U.S. diesel price (an index, not a carrier surcharge) and watches the fuel surcharge, GRI and tariff pages and PDFs you list. Effective-dated change ledger, PDF fingerprints, weekly digest. Plain HTTP, robots.txt respected.

- **URL**: https://apify.com/leons0016/freight-surcharge-monitor.md
- **Developed by:** [Leon s](https://apify.com/leons0016) (community)
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $24.00 / 1,000 source reads

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Freight Surcharge Monitor - FreightSurcharge Watch

**Freight surcharge monitor: the EIA weekly diesel index plus the fuel surcharge, GRI and tariff pages and PDFs that you list, in one effective-dated change ledger.** FreightSurcharge Watch records the weekly U.S. on-highway diesel price from the U.S. Energy Information Administration (EIA), the index most parcel and truckload fuel surcharge tables follow, and it watches the **documents you give it** (carrier fuel surcharge, GRI and accessorial pages and PDF tariffs). Every change is written to a ledger with its effective date, PDF tariffs are fingerprinted so a file that is **silently replaced at the same address** is noticed, and a weekly digest lists what moved. Plain HTTP only, no browser, no proxy, robots.txt respected.

Actor page: `https://apify.com/leons0016/freight-surcharge-monitor`

### Read this first: narrow coverage

- **Built in: one source only.** The EIA weekly diesel price (11 U.S. areas, public domain). It is an **index, not a carrier's own surcharge percentage**.
- **Everything else is your own list.** You add the fuel surcharge, GRI or tariff pages and PDFs you are entitled to read under the site's terms (**Documents to watch**). The Actor does not ship any carrier's surcharge table.
- **Many carrier sites cannot be read with plain requests.** Large parcel and ocean carriers often answer an automated request with an access-denied page, a JavaScript check or a captcha, and some forbid automated use in their terms. The Actor never works around this: that address gets a `blocked` or `failed` row with the reason, and it is not charged. Test your own addresses with the free-plan run before you rely on them.
- **No carrier rate comparison, no quotes, no invoice audit.** Rate lines are found by rule and can be missed or misread.

Not affiliated with or endorsed by the U.S. Energy Information Administration or by any carrier. Carrier and company names that you enter or that appear on the pages you list belong to their owners. This Actor is an independent product of LeonS. It is not affiliated with, endorsed by or sponsored by Apify Technologies, the data sources named here or any third party whose name is mentioned; all product and company names are trademarks of their respective owners and are used only to identify the data source or the software the Actor works with.

### What it covers today

| Source | Status |
|---|---|
| EIA weekly diesel price (11 areas) | **Covered.** Public domain, plain HTML, robots.txt allows it. |
| Your own carrier pages and PDF tariffs | **Covered only if the site serves them to a plain request and allows it.** You list them; the Actor reports `blocked` or `failed` with the reason when a site refuses. |
| Carrier pages that answer a plain request with an access-denied page, a JavaScript check or a captcha | Not covered; each run summary repeats the kinds of pages that were not read and why. |
| Carrier pages whose terms of use restrict automated or commercial use | Not covered; whether you may list such a page is your responsibility. |

The batch of carrier pages this Actor was designed around was tested before anything was built (one plain request each, robots.txt and terms read); most of them refuse a plain request, so the product was narrowed to the EIA index plus your own documents. A carrier turns the EIA index into a percentage with its own table, so read the carrier's percentage from the carrier's page or PDF by listing it under **Documents to watch**.

### Use cases

- **Early warning for freight accruals.** The EIA price moves every Monday; the ledger shows the new week before the carrier invoice does.
- **Silent tariff replacement.** A PDF tariff replaced at the same address has a new fingerprint; if the text is identical the row says so and is free, if a rate line changed you get the old and new value.
- **Rate-change ledger for FP\&A.** `LEDGER.csv` lists every recorded change with effective date, previous and new value for the last 24 months.
- **Weekly digest** for logistics, procurement and finance, by Slack, email or webhook.

### What you get

- **An effective-dated ledger.** Each row has the effective date (from the line, or from the "effective ..." sentence of the document), the previous and new value, the change and the evidence line.
- **File and text fingerprints** (SHA-256) of every document, and PDF text extraction with pdfplumber. Scanned PDFs without a text layer are tracked by file hash only and flagged.
- **One row per source, always.** Every source gets a row with the result `ok`, `failed` or `blocked` and a sentence that says why. A run is never empty.
- **A starting point, not a bill.** The first reading of any source or area is stored without a charge.
- **Protection from noise.** Ignore patterns for time stamps and counters; a page that changes on every run stops being charged after three runs; one event for a restructured tariff instead of dozens.

### How to use

1. Open the Actor and press **Start**. The default input reads the EIA U.S. diesel price (about 10 seconds).
2. Choose more **EIA areas** (East Coast, Gulf Coast, California, ...) if your lanes need them.
3. Under **Documents to watch** add addresses of your carriers' fuel or tariff pages and PDFs, one per line, optionally `Label | address`. Only list documents you are entitled to read under the site's terms.
4. Click **Save as new task** and add a **Schedule**, for example daily. The task keeps its ledger in a named key-value store in your account (**Ledger name**).
5. Optional: add Slack, Gmail or a webhook under **Integrations**.

### Input

| Field | What it does |
|---|---|
| EIA weekly diesel price | Default on. |
| EIA areas | Default U.S. Eleven areas: U.S., PADD 1, 1A, 1B, 1C, 2, 3, 4, 5, West Coast less California, California. |
| Documents to watch | Up to 40 pages or PDFs, `Label | address` or an address. Free plan: 2. |
| Freight keywords | Words that make a line with a percentage or an amount a rate line. Defaults: surcharge, fuel, GRI, peak season, bunker, accessorial, fee, tariff, and more. |
| Ignore patterns | Regular expressions removed before comparing (a "last updated" stamp). |
| Changes per document and run | Default 25. Above it, one event for the whole document. |
| Changes per run | Default 30. Changes beyond it are found again by the next run. |
| Weekly digest | *Weekly* (default), *Off* or *Every run*. Only produced when something changed. |
| Ledger name, Webhook URL, Slack webhook | Optional. |

Sample input:

```json
{
  "eiaAreas": ["US", "PADD3", "CALIFORNIA"],
  "watchUrls": ["Example Line trade surcharges | https://example.com/tariffs/trade-surcharges.pdf"],
  "digest": "weekly"
}
```

The input is checked before anything is charged.

### Output

- **Sources** (default view): one row per source with result, reason, HTTP status, fingerprints, rate lines found and changes recorded.
- **Rate changes**: source, series, change type (`baseline`, `changed`, `revised`, `added`, `removed`, `document-changed`, `document-restructured`, `file-reissued-same-text`), effective date, previous and new value, unit, change in percent, evidence, and whether it was billed.
- **Weekly digest**: the period, counts per source and the list of changes.
- **Run summary**: one uncharged row, always the last, with counts, warnings per code, the digest result and the carrier pages that are not covered.
- Key-value store: `OUTPUT` (JSON), `REPORT` (HTML), `CHANGES.csv` (this run) and `LEDGER.csv` (all changes of the last 24 months).

### Pricing

Pay per event. You pay only for what a run delivers:

| Event | Price (USD) | When |
|---|---|---|
| `source-checked` | 0.03 | Once for each source that was read successfully, in every run. Sources that are blocked or fail are not charged. |
| `rate-change-detected` | 0.25 | Once for each effective-dated change that enters the ledger. Starting points, removed lines and identical-text PDF replacements are free. |
| `weekly-digest` | 0.50 | Once per digest, at most one every 7 days, only when it lists a change. |

**Your monthly bill is: sources x runs per month x 0.03, plus 0.25 for every change, plus 0.50 for every digest.** Measured example: the default input (EIA U.S. diesel, daily, weekly digest) is 30 x 0.03 = 0.90, plus 4 weekly price changes at 0.25 = 1.00, plus 4 digests at 0.50 = 2.00, about 3.90 a month. Eleven EIA areas plus six of your own documents, daily, comes to about 20.80 (the document changes in that figure are an assumption). The default Apify start event applies as usual; there is no pass-through of platform cost.

Set a maximum charge per run if you want a hard cap; when it is reached the run stops cleanly, every source still has a row, and unrecorded changes are found again by the next run. The free plan has no separate trial: Apify's free allowance and the low `source-checked` price let you test with the EIA page and 2 documents.

### Limits and honest notes

- **Index, not invoice.** The EIA price is not your carrier's surcharge. Documents are copies of what a page showed when it was read. Check against your contract and invoice; this is not rate advice.
- **Rule-based reading.** Rate lines are found by rule (a percentage or amount next to a freight keyword). Unusual layouts can be missed or misread. When a document changes but no rate line differs, you get a `document-changed` row with a few new lines instead of silence. If rate lines disappear the Actor warns (`RATE_LINES_GONE`) and keeps the old lines.
- **Plain HTTP only.** Pages built by JavaScript, bot checks and captchas are reported (`NEEDS_JAVASCRIPT`, `BOT_CHECK`, `ACCESS_DENIED`) and never worked around. Scanned PDFs have no OCR.
- **Navigation, footer, script and style text is ignored** when comparing HTML pages, so counters there do not cause changes.
- **robots.txt and terms.** The Actor never requests an address that robots.txt disallows. Some carriers forbid automated access in their terms; whether you may list their pages is your responsibility.
- **Weekly resolution.** EIA publishes weekly (Monday data, usually released Monday or Tuesday); the ledger shows what each run saw.

### FAQ

**Which carriers can I watch?** Those whose fuel or tariff page or PDF is served to a plain request and allowed by robots.txt and the site's terms. Many large carrier sites are not (see the table above); the run tells you per address.

**Does it use a browser or a proxy?** No. A few plain HTTPS requests per run, identified in the User-Agent.

**Where is my data?** In your own Apify account: the run's dataset and key-value store, and the named ledger store `leons-freightsurcharge-<name>-ledger`. The publisher receives nothing. See the privacy policy.

**Can I keep ledgers apart?** Yes, use a different **Ledger name** per region or client.

**Support:** atulhritik@hotmail.com

Source acknowledgment: "Source: U.S. Energy Information Administration" (U.S. government data in the public domain); keep it when you reuse EIA rows.

Third-party licences and attributions: see `docs/THIRD_PARTY_NOTICES.txt` in the Actor source.

# Changelog

This Actor's version history is a separate document: https://apify.com/leons0016/freight-surcharge-monitor/changelog.md

# Actor input Schema

## `includeEiaDiesel` (type: `boolean`):

The U.S. Energy Information Administration publishes the retail on-highway diesel price every week (public domain). Most parcel and truckload fuel surcharge tables are indexed to it, so a change here is the early signal. One source-checked event per run.

## `eiaAreas` (type: `array`):

Which EIA areas to follow. Each area whose weekly price changed is one rate-change-detected event per week. Example: US

## `watchUrls` (type: `array`):

One address per line, optionally with a label: My carrier fuel table | https://example.com/fuel.html. Pages and PDF tariffs are read with one plain request; the Actor respects robots.txt and does not try to get around a refusal (a refused address is reported as blocked). Only list documents you are entitled to monitor under the site terms. Up to 40 lines. Example: https://example.com/tariffs/fuel-surcharge.pdf

## `rateKeywords` (type: `array`):

A line with a percentage or an amount counts as a rate line when it, or one of the three lines above it for table rows, contains one of these words. Add the words your carriers use. Example: GRI

## `ignorePatterns` (type: `array`):

Text matching these regular expressions is removed before comparing, for parts of a page that change on every visit (a time stamp, a visitor counter). Example: Last updated:? \[\d/ .-]+

## `maxChangesPerDocument` (type: `integer`):

If more rate lines than this change in one document at once (a restructured tariff), the Actor records one rate-change-detected event for the whole document instead of one per line.

## `maxChangesPerRun` (type: `integer`):

The most rate-change-detected events one run records. Changes beyond it are not lost: the next run finds them again.

## `digest` (type: `string`):

A digest lists every change since the previous one and is billed as one weekly-digest event. An empty digest is never produced or charged.

## `ledgerName` (type: `string`):

Name of the ledger kept in your account. Use another name to keep a separate ledger, for example per region. Lower-case letters, digits and hyphens. Example: europe

## `webhookUrl` (type: `string`):

Optional. The run output is posted here as JSON after each run. Must be https. Example: https://example.com/hooks/freight

## `slackWebhookUrl` (type: `string`):

Optional. A short message with the new changes is posted to this Slack incoming webhook. Example: https://hooks.slack.com/services/T000/B000/XXXX

## Actor input object example

```json
{
  "includeEiaDiesel": true,
  "eiaAreas": [
    "US"
  ],
  "watchUrls": [],
  "rateKeywords": [
    "surcharge",
    "fuel",
    "GRI",
    "rate increase",
    "peak season",
    "bunker",
    "BAF",
    "accessorial",
    "fee",
    "tariff",
    "congestion",
    "emergency",
    "detention",
    "demurrage",
    "per container",
    "per TEU",
    "per package",
    "per shipment",
    "per kg"
  ],
  "ignorePatterns": [],
  "maxChangesPerDocument": 25,
  "maxChangesPerRun": 30,
  "digest": "weekly",
  "ledgerName": "default"
}
```

# Actor output Schema

## `sources` (type: `string`):

One row per source: ok, failed or blocked, with the reason.

## `changes` (type: `string`):

Effective-dated changes recorded in this run.

## `digest` (type: `string`):

The digest, when one was delivered.

## `runSummary` (type: `string`):

Counts, warnings, carrier pages not read and the stopped reason.

## `report` (type: `string`):

One-page report.

## `changesCsv` (type: `string`):

Changes recorded in this run.

## `ledgerCsv` (type: `string`):

Every change in the ledger (last 24 months), for the freight accrual forecast.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "eiaAreas": [
        "US"
    ],
    "watchUrls": [],
    "rateKeywords": [
        "surcharge",
        "fuel",
        "GRI",
        "rate increase",
        "peak season",
        "bunker",
        "BAF",
        "accessorial",
        "fee",
        "tariff",
        "congestion",
        "emergency",
        "detention",
        "demurrage",
        "per container",
        "per TEU",
        "per package",
        "per shipment",
        "per kg"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("leons0016/freight-surcharge-monitor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "eiaAreas": ["US"],
    "watchUrls": [],
    "rateKeywords": [
        "surcharge",
        "fuel",
        "GRI",
        "rate increase",
        "peak season",
        "bunker",
        "BAF",
        "accessorial",
        "fee",
        "tariff",
        "congestion",
        "emergency",
        "detention",
        "demurrage",
        "per container",
        "per TEU",
        "per package",
        "per shipment",
        "per kg",
    ],
}

# Run the Actor and wait for it to finish
run = client.actor("leons0016/freight-surcharge-monitor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "eiaAreas": [
    "US"
  ],
  "watchUrls": [],
  "rateKeywords": [
    "surcharge",
    "fuel",
    "GRI",
    "rate increase",
    "peak season",
    "bunker",
    "BAF",
    "accessorial",
    "fee",
    "tariff",
    "congestion",
    "emergency",
    "detention",
    "demurrage",
    "per container",
    "per TEU",
    "per package",
    "per shipment",
    "per kg"
  ]
}' |
apify call leons0016/freight-surcharge-monitor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,leons0016/freight-surcharge-monitor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/RahhQoW94qVq0fDG8/builds/uQZbpcI2mdaPhhtsS/openapi.json