US Historical Freight Flows — Lane Records, Per Record

Turn the U.S. Commodity Flow Survey historical file into clean, per-record lane-demand history - one row per origin x destination x commodity x mode x year (1997-2017), with value, tons and ton-miles, ready to backtest freight-lane demand over two decades.

Each source row becomes one clean, flat record with numeric fields coerced to real numbers, a stable source-native record_id, and provenance stamped on every row: source, resource id, the licence notice, the required attribution, a UTC retrieval timestamp and an interpretation caveat.

What one record represents

The source is anet-6eas — Historical CFS Data 1997-2017 on the U.S. DOT open-data portal (data.transportation.gov). Each record is one historical freight lane: origin and destination CFS areas, a commodity (SCTG), a transport mode and a survey year from 1997 to 2017, with shipped value, tons and ton-miles plus coefficients of variation.

For each record you get a composite record_id built from the source-native key, the analytic columns listed below (reproduced verbatim, numbers as numbers), and the provenance block. Columns include origin, dest, sctg/comm, mode, year, val, ton, tmile (ton-miles), avgmile and the *_cv coefficient-of-variation fields.

Coverage and volume

The live historical file holds 26,615 lane-year records spanning the 1997-2017 Commodity Flow Surveys - that is the record capacity of a full pull.

Live count: 26,615 records - matches the Wave-3 index figure exactly.

The Actor pages the source with keyless SODA $query requests ordered by the source-native key for a stable total order, and stops as soon as your Maximum records cap is met. The Actor names every wanted column explicitly in a SoQL $query so no column hidden by a default view is lost.

Licence and attribution

This is a public-domain U.S. Government work (17 U.S.C. §105) — free to use, redistribute and build on. The full notice travels on every record:

U.S. DOT / Bureau of Transportation Statistics. Public-domain U.S. Government work (17 U.S.C. 105). Reproduced verbatim; no third-party content or logos.

The required attribution — U.S. DOT / Bureau of Transportation Statistics (Commodity Flow Survey) — travels on every record.

Interpretation caveat

Each record is a CFS survey estimate for a historical lane. This is the Historical CFS file (anet-6eas, 1997-2017), distinct from the 2012-2022 hazmat, export and temperature-control files, each a separate published dataset. Coefficients of variation indicate estimate reliability.

Values are reproduced verbatim: the Actor never rescales, re-derives or editorialises a number.

Person-data policy

Origins, destinations, ports and commodities are geographies and goods classes — there are no name, email, phone, personal-address or personal-identifier fields. A per-record assertion enforces the person-field allow-list at write time.

Data quality and freshness

Numeric fields are coerced from the source's string encoding into real numbers (integers where whole, floats otherwise); genuinely missing cells are delivered as null, never as zero. Text is passed through verbatim. Every run re-reads the live source, so the data is as fresh as the portal itself, and each record's observed_at stamp records exactly when the row was retrieved. Delivery order is fixed by the source-native key, so a capped sample and a later full pull agree on their overlap and a repeated run returns rows in the same order. The RUN_RECEIPT reports source rows scanned and records delivered and charged for a per-run reconciliation.

Provenance, licensing and compliance

Every run begins with a live source-preflight: the Actor reads the exact host's robots.txt at runtime and refuses to proceed if the crawl policy disallows the data path. The gate result — URL, HTTP status, byte length and a SHA-256 of the policy — is written to the run's RUN_RECEIPT, so each run carries its own audit trail. The Actor identifies itself with a transparent, non-impersonating User-Agent and never bypasses a block, solves a challenge, or fetches through a cache or mirror. When the door is genuinely unavailable the run fails loudly and bills nothing.

Inputs

• Maximum records (maxRecords) — hard cap on records delivered and billed. Raise it to pull the full set; lower it to sample cheaply. Records arrive in a stable, source-native order.

Output

Records land in the Actor's default dataset and export as JSON, CSV, Excel or via the Apify API. A tabular overview view surfaces the most useful columns for quick inspection while the full record retains every selected field and provenance stamp.

Fields in detail

The record leads with record_id — a stable composite key drawn from the source's own grain — followed by the analytic columns described above and closed by a provenance block: source, source_dataset (the Socrata resource id), licence, attribution, caveat and observed_at. Every one of those provenance fields is present on every record, so a single row is self-describing: hand it to a colleague or a downstream system and it carries its own origin, licence and retrieval time without reference back to this page. Because delivery is ordered by the source-native key, the same record always carries the same record_id across runs, which makes the dataset safe to diff, deduplicate, or upsert into a warehouse. Nothing in the record is computed or inferred beyond the explicit count where one is stated — every other value is the source's own, reproduced byte-for-byte.

Sibling Actors

This Actor is the historical (1997-2017) cut of the Commodity Flow Survey. Its FAF-family siblings are separate CFS datasets: us-hazardous-freight-flow-records, us-export-freight-mode-records and us-cold-chain-freight-records (all 2012-2022). It also complements the owned Supply Chain MCP tool. This Actor also shares its engineering — the runtime robots gate, push-then-charge billing and verbatim-value discipline — with the fleet's other public-data records Actors.

Pricing

This Actor uses Apify's pay-per-event model: a flat $0.05 per record actually delivered to the dataset, and nothing else — no monthly rental, no per-run base fee, no compute charge. Deliver 40 records and you pay $2.00; deliver 10,000 and you pay $500.00. Billing is wired after delivery — each record is pushed first and only then does the per-record event fire — so a mid-run failure can only ever under-charge you, never over-charge. Use Maximum records to cap spend precisely.

Scaling and limits

Set Maximum records low to sample the leading slice cheaply, or high to pull the full set. The Actor paginates server-side and delivers incrementally, so memory stays flat regardless of how many records you request, and you are billed only for what is actually delivered. Because the source is a live public API, extremely deep pagination is ultimately bounded by the source's own paging behaviour; for the vast majority of uses — sampling, a full refresh, or a scheduled top-up — the default paging is more than sufficient. Schedule the Actor on Apify to keep a downstream table current: each run re-reads the live source and re-stamps observed_at, so a nightly or weekly run gives you a dated, reproducible snapshot.

Typical uses

Backtest lane demand over two decades; track how a commodity's tonnage and value moved between areas; compare modal split over time; build a long-run freight-demand baseline; or feed a supply-chain or infrastructure model with clean historical lane records.

What this Actor does not do

It does not forecast or model, does not merge multiple source tables into one record, and it contains no personal data — only geographies, commodities and freight metrics. It gives you faithful, analysis-ready records — with a provenance trail you can audit on every run.