Federal Register Data Scraper (Rules & Notices JSON)

Monitor US federal rulemaking documents from the official, keyless Federal Register API. Filter by publication date, document type, agency or keyword, and get one structured dataset item per document. No browser, no login, no API key.

What it does

The Actor calls https://www.federalregister.gov/api/v1/documents.json, the Federal Register's official developer API, and returns one dataset item per matching document with its document number, type, publication date, publishing agencies, abstract, citation, effective and comment dates, docket and regulation identifiers, and official HTML/PDF links.

• Monitor mode (default): keeps a per-filter deduplication state in the Actor's key-value store and emits only documents whose document_number was never delivered by an earlier run with the same filters. Re-running the same monitor never bills the same document twice.
• Search mode: emits every document in the selected window, deduplicated within the run. Use it for one-off research or backfills.
• Filters: publication date range (since/until, default last 7 days), document type (RULE, PRORULE, NOTICE, PRESDOCU), agency (official slug or human name, e.g. securities-and-exchange-commission or Securities and Exchange Commission) and full-text keyword.
• Schema validation: every item is validated against .actor/dataset_schema.json before it reaches the dataset. With strictSchema: true (the default) an invalid item fails the run instead of pushing malformed data.

The Federal Register states that programmatic access to its sites is limited to its developer APIs, so this Actor only uses the official API and never scrapes the HTML site. The data is public US government information. The Actor returns documents and agencies only: no personal data and no contact fields.

Example output

{
    "url": "https://www.federalregister.gov/documents/2026/10/20/C1-2026-20360/interval-fund-modernization-expansion-of-multiple-share-class-to-registered-closed-end-management",
    "title": "Interval Fund Modernization; Expansion of Multiple Share Class to Registered Closed-End Management Investment Companies and Business Development Companies",
    "scrapedAt": "2026-10-20T08:30:00.000Z",
    "document_number": "C1-2026-20360",
    "type": "Proposed Rule",
    "publication_date": "2026-10-20",
    "agencies": ["Securities and Exchange Commission"],
    "abstract": "",
    "citation": "91 FR 66293",
    "effective_on": "2026-10-05",
    "comments_close_on": "2026-12-04",
    "docket_ids": ["Release No. 33-11444", "34-106534", "IC-36351", "File No. S7-2026-34"],
    "regulation_id_numbers": ["3235-AN83"],
    "html_url": "https://www.federalregister.gov/documents/2026/10/20/C1-2026-20360/interval-fund-modernization-expansion-of-multiple-share-class-to-registered-closed-end-management",
    "pdf_url": "https://www.govinfo.gov/content/pkg/FR-2026-10-20/pdf/C1-2026-20360.pdf",
    "source_url": "https://www.federalregister.gov/api/v1/documents/C1-2026-20360.json"
}

source_url is the official per-document API endpoint for the record. Optional dates (effective_on, comments_close_on) and pdf_url are omitted when the source has no value; abstract and citation are empty strings in that case, and identifier arrays are empty.

Input

Minimal run (monitor mode, last 7 days by default):

{
    "since": "2026-10-01",
    "maxResults": 10
}

A focused monitor:

{
    "mode": "monitor",
    "since": "2026-10-01",
    "type": "PRORULE",
    "agency": "Securities and Exchange Commission",
    "keyword": "fund",
    "maxResults": 500
}

Field	Type	Description
mode	monitor or search	Default monitor: only document numbers not delivered by earlier runs with the same filters. search: every matching document in the window (deduplicated within the run).
since	string	Earliest publication date (YYYY-MM-DD, inclusive). Defaults to 7 days before the run date (UTC).
until	string	Latest publication date (YYYY-MM-DD, inclusive). Defaults to the run date (UTC).
type	RULE, PRORULE, NOTICE or PRESDOCU	Optional document type.
agency	string	Optional agency slug or name. Names are resolved against the official /agencies endpoint.
keyword	string	Optional full-text search term.
maxResults	integer	Default 500, maximum 5000. Paging stops as soon as that many new documents are collected.
strictSchema	boolean	Default true; fail the run on schema violations.

The monitor state is keyed by the filter signature (type, agency, keyword, case-insensitive); the date window and maxResults are deliberately excluded, so widening a date range does not re-deliver documents already seen by the same monitor.

Project structure

.actor/actor.json                    Actor definition (payPerEvent, dataset schema, output schema)
.actor/input_schema.json             INPUT options (mode, since, until, type, agency, keyword, maxResults)
.actor/dataset_schema.json           Item JSON Schema (fields) + table view for the Store
.actor/pay_per_event.json            PPE pricing (two synthetic platform events)
src/federal-register-client.ts       Federal Register API client: filters, pagination, retries, agency lookup
src/extract.ts                       Pure mapping and deduplication (fixture-tested, no network)
src/monitor-state.ts                 Pure monitor-state key and merge rules
src/main.ts                          Entry point: fetch, validate, push, persist monitor state
src/validate-output.ts               Ajv validator shared by the run and the tests
test/                                Offline JSON fixtures and Vitest suites

Pay-per-event pricing

The Actor uses Apify pay-per-event pricing with the two synthetic platform events:

Event	Charged when	Price
apify-actor-start	a run starts (synthetic, automatic)	0.00005 USD per run (Apify-recommended default; includes 5 seconds of free compute)
apify-default-dataset-item	a new, validated document is saved to the dataset (synthetic, automatic)	0.002 USD per document = 2.00 USD per 1,000 documents

There is no custom per-item event and no Actor.charge() call: the platform already charges apify-default-dataset-item per pushed item, so adding another event would bill users twice. Empty results, API errors, retries and documents skipped by monitor deduplication are never charged.

The price sits at the top of the direct-competitor band on the Apify Store (0.001-0.002 USD per document = 1-2 USD per 1,000) and inside Steadfetch's 1-10 USD band, justified by complete API coverage and cross-run deduplication.

Cost per 1,000 documents

A 1,000-document run needs one JSON request (per_page 1000), a few seconds of a 256 MB container and no proxy. Measured locally with apify run: 1,000 documents fetched, mapped, schema-validated and written in 7.5 seconds end to end (including Node/CLI startup), using 256 MB. At Apify compute rates that is roughly 0.0001 USD of compute plus the 0.00005 USD run-start event per 1,000 documents, well below 0.01 USD per 1,000. At the 2.00 USD price the gross margin is above 99%.

Development

npm install
npm test                 # lint + typecheck + offline Vitest suites (no credit)
npm run publish:check    # Steadfetch publish gate
npm run schemas:validate # apify-cli schema validation
npm run smoke            # local Apify run against the live API (no platform credit)

The default test suite runs against committed API fixtures and never touches the network or the Apify platform. It covers the real API response shape, pagination, agency-name resolution, monitor deduplication, malformed records and schema validation.

FAQ

Do I need a Federal Register API key? No. The documents API is keyless and public.

Does it need a browser? No. The Federal Register returns structured JSON; plain HTTP is roughly 10x cheaper and faster than a browser-based Actor.

How does monitor deduplication work? The Actor stores the document_number of every delivered document in a named key-value store that persists between runs (the default store is run-scoped in Apify), under a key derived from the type/agency/keyword filters. The next run with the same filters filters those numbers out before counting toward maxResults, so only new documents are pushed and billed. The state keeps the 20,000 most recent document numbers.

Does it return personal data? No. Only documents and agencies are returned. There are no personal names, email addresses or phone numbers in the output schema.

When am I charged? Once per run start (apify-actor-start) and once per new document saved to the dataset (apify-default-dataset-item). Failed requests, retries, empty results and documents already delivered by a previous monitor run are not charged.

Is the output schema stable? Yes. New fields are additive and the required fields in .actor/dataset_schema.json do not change.

Maintenance

This Actor is tested daily: automated runs check that the official Federal Register API is reachable and that the output still matches the published schema. Failures are reported as issues in the Steadfetch Actors repository.