Fox News Liveblogs Scraper

Collects Fox News live-news blogs and their full timelines. For each live event you get the liveblog itself and one row per update, with that update's own headline, body text, author and timestamp.

The archive holds 1,353 live events going back to June 2021, each with roughly 12 to 53 updates, which is on the order of 40,000 timeline entries.

No login, no API key, no browser. Plain HTTP, Python only.

What you get

One dataset, four kinds of row, told apart by recordType.

LIVEBLOG — one per live event: headline, description, URL, the coverage window (when Fox opened and closed the blog), publish and modified dates, the contributing authors, publisher, lead image, and how many updates the timeline holds versus how many this run returned.

LIVEBLOG_UPDATE — one per timeline entry: its own headline, body text, authors, publish and modified timestamps and image. Join it back to its parent on liveblogUrl, and order it with updateRank or updateDatePublished.

SEARCH_SUMMARY — one per run: corpus size, how many liveblogs were discovered after the date filter, how many were returned, how many pages were fetched, how many updates were truncated, and how many rows each filter dropped.

ERROR — one per URL that failed, so a failure can never look like an empty result.

Examples

The ten most recent live events with their complete timelines:

{ "maxItems": 10 }

Headlines only, no timeline rows:

{ "maxItems": 50, "includeUpdates": false }

The five latest updates from each of the twenty newest events:

{ "maxItems": 20, "maxUpdatesPerLiveblog": 5 }

Everything Fox live-blogged in a past window:

{ "maxItems": 100, "dateFrom": "2024-07-01", "dateTo": "2024-07-31" }

Live events that mention a term anywhere on their timeline:

{ "maxItems": 10, "keywords": ["hurricane"] }

Limits, stated plainly

One page fetch per liveblog, always. Unlike Fox's article feeds and video sitemap, the liveblog sitemap carries no title, no body and no dates beyond last-modified. There is no cheaper path, so maxItems is the real cost control.

The date filter is free, the keyword filter is not. Dates come from the sitemap, so a date-bounded run narrows the corpus before fetching anything and pays only for what it returns. A keyword can be buried in any update, so whether a liveblog matches is unknowable until its page is fetched. Keyword runs therefore fetch more pages than they return liveblogs, and stop at eight pages per requested item so a term that matches nothing cannot walk the whole archive.

The keyword filter runs here, not at Fox. Fox exposes no search parameter. It narrows what a run retrieved rather than widening what it can reach.

Update caps keep the newest. maxUpdatesPerLiveblog truncates from the newest end, and liveblogUpdatesAvailable still reports the true total so you always know what was left behind.

Dates are the last-modified date. That is the only date the sitemap carries, so it is what the date filter compares. Each liveblog row also carries its coverage window and publish date from the page itself.

Articles and videos are separate Actors. A liveblog is a parent plus a timeline, which is a shape neither of those has.

Blocking and proxies

No bot-mitigation gate was found on any Fox News surface during development: the sitemap and the live-news pages answered cold across four different TLS fingerprints, with no challenge, no cookie and no token. The Actor defaults to Apify's shared datacenter proxy, included in your plan at no extra cost. Residential is available in the proxy editor; Apify bills it per gigabyte.

Politeness

robots.txt at foxnews.com sets no Crawl-delay and names no AI-crawler group. It disallows the site search, the wires and the printer-friendly paths, none of which this Actor touches: it reads the liveblog sitemap and public live-news pages. Request starts are paced by minRequestInterval, which defaults to a quarter second.