ORR Rail and Road Consultation Scraper

Turn public Office of Rail and Road consultation pages into structured lifecycle records for rail operators, licensing teams and regulatory intelligence workflows. Compare exports by sourceUrl to monitor publication dates, deadlines and authoritative consultation stages without rebuilding a manual URL inventory.

What this actor covers

• The supported individual consultation links on the first ORR consultation catalog page, optionally supplemented with explicit official detail URLs.
• One row per recognized consultation, with ORR's own status, dates, summary and reference document URLs.
• Explicitly distinguishes the source stage “ORR response to consultation” as recordType: ORR response to external consultation.

This is a bounded recent inventory, not the complete archive. It does not activate Load More, submit search/filter forms, crawl the general /search/ route, or treat “Current licence casework consultations” as one consultation. On 5 October 2026 the catalog fixture held 24 rows, including 23 supported detail links and one excluded licence-casework aggregate. Catalog composition may change. It does not fetch PDF contents, respondents, survey forms, personal submissions or third-party sites.

Inputs

Example: { "discoverRecent": true, "maxItems": 10, "status": "all", "publishedSince": "" }

• consultationUrls: up to 50 plain HTTPS www.orr.gov.uk/search-consultations/slug URLs; explicit URLs are checked first. No custom host, port, query, encoded path, fragment or arbitrary start URL.
• discoverRecent (default true): discover supported detail links from the first catalog HTML page.
• maxItems (default 10, maximum 50): maximum matching output rows. Independently, at most 50 detail URLs are checked, so restrictive filters can return fewer rows.
• status: all, open, reviewing responses, conclusions published, or orr response to consultation; exact case-insensitive match to the printed source stage. Use all for unrecognized/new stages. No status is inferred from the current date.
• publishedSince: optional inclusive ISO YYYY-MM-DD. Missing or unparseable publication dates do not match this filter.

Output

title, publishedDate, closingDate, publishedDateRaw, closingDateRaw, status, recordType, summary, topics, breadcrumbs, documentUrls, responseSummaryUrls, sourceUrl, retrievedAt, warnings, attribution.

Dates are ISO calendar dates parsed from the visible ORR labels, not promises of a particular deadline time. Raw labels remain available. Missing dates become null; parse failures add warnings. Topics are read only when an explicit topic field exists; empty means not provided in the recognized detail HTML. Breadcrumbs are navigation context, not inferred topics. Summary comes from the source description metadata. Document links are deduplicated official /sites/default/files/ file references and are never downloaded. responseSummaryUrls is a conservative subset whose anchor explicitly mentions a consultation/response summary or conclusions; it may be empty even where other response material exists.

A verified detail fixture is ORR discussion document: modifying non-GBR licences in the context of rail reform: published 2026-07-27, closing 2026-09-18, source stage “Reviewing responses”, one official PDF. Keeping that source stage matters: a passed deadline is not a concluded policy outcome.

RUN_SUMMARY reports records, detail pages fetched, requests, errors and coverage. Errors preserve already emitted rows but fail the run as incomplete. A successful zero-result run means no records matched within the checked subset, not that ORR has no consultations.

Source access and limits

Read-only direct HTTPS, one request at a time, minimum two seconds between request starts (increased for robots crawl-delay/request-rate), maximum 52 total requests including robots/catalog, maximum 50 detail pages, 3 MiB per response, 25-second socket timeout, bounded response read time and a 20-minute request-start budget. Robots delay above 60 seconds fails closed. The actor reads and checks robots before pages; invalid/unavailable robots stops the run. HTTP 401/403/429 stop immediately, without retries. Redirects stop the run. The standard runtime network transport is retained, including infrastructure egress when required by the host environment. No actor-configured scraper proxies, browser fallback, challenge solving, credentials or access-control bypass.

Reuse and attribution

Source: Office of Rail and Road terms and conditions. Contains public sector information from the Office of Rail and Road licensed under the Open Government Licence v3.0. ORR logos and identified third-party material are excluded. This actor exports factual metadata and ORR-authored description text; linked documents and responses retain their own rights and are not republished. No affiliation or endorsement is implied. Retain the output attribution when redistributing records.

Why this source-specific feed

Compliance.ai's published regulatory coverage explicitly lists Office of Rail and Road, UK-ORR, Agency_ID 10000058, with Consultation coverage. This is evidence of a commercial regulatory-intelligence use case, not a claim that this actor is used by that company. Apify Store checks on 5 October 2026 found no dedicated matching actor for ORR consultations or orr.gov.uk; broader Office Rail Road results were inspected and unrelated. Store availability changes and generic page monitoring tools exist. The gap addressed here is source discovery plus typed ORR stages, deadline fields, document references and external-response classification, rather than raw text for manually supplied URLs.

Run and test

Python 3.12; install pinned requirements: pip install -r requirements.txt.

Run portable offline checks from this folder: python -m unittest discover -s tests -v.

Run with the Apify SDK: python main.py. Local SDK input can be placed in storage/key_value_stores/default/INPUT.json. On Apify, supply input in the actor input editor; dataset exports support JSON/CSV/Excel through Apify. The included Dockerfile runs the same entry point.

Fixtures are real source HTML captured on 5 October 2026, stored within tests/fixtures so the backup is independently testable. LIVE_CHECK.json documents the local live SDK check and its limits. Deployment and private cloud validation are separate from local tests; no public Store publication is implied.