Fox News Videos Scraper

Collects video metadata from foxnews.com: title, description, duration, publish date, thumbnail, the direct content URL, and optionally the topic and show tags.

It reads Fox's own video sitemaps, which carry every field of a video's record inline. That means a run of any size costs roughly one request per 4,825 videos and fetches no video pages at all, unless you ask for topic tags. The archive holds about 670,000 videos across 140 partitions.

No login, no API key, no browser. Plain HTTP, Python only.

What you get

One dataset, three kinds of row, told apart by recordType.

VIDEO — one per video: page URL, title, description, thumbnail, direct content URL, duration in both seconds and ISO 8601, publish date, and Fox's own subscription and live flags. With topic tags on, also the show and topic tags and the embed URL.

SEARCH_SUMMARY — one per run: how many videos were discovered, how many came back, how many sitemap partitions were read out of how many exist, whether the archive walk ended early, how many duplicates were removed, and how many rows each filter dropped.

ERROR — one per URL that failed, so a failure can never look like an empty result.

Examples

The fifty newest videos, two requests total:

{ "maxItems": 50 }

Full episodes only, ten minutes or longer:

{ "maxItems": 100, "minDurationSeconds": 600, "includeArchive": true, "maxArchivePages": 3 }

Short clips from a date range:

{
  "maxItems": 200,
  "maxDurationSeconds": 120,
  "dateFrom": "2026-08-01",
  "dateTo": "2026-08-31",
  "includeArchive": true,
  "maxArchivePages": 3
}

With topic and show tags, which cost one request per video:

{ "maxItems": 25, "includeTopics": true }

Limits, stated plainly

Topic tags need the video page. The sitemaps have no field for them, so includeTopics fetches each video's page. That is the only setting that turns one request per thousands of videos into one request per video. Everything else about a video is returned either way.

Expiry dates are rare. Fox supplies expiration_date on only about 1.7% of entries, measured across a full 4,825-entry partition. The field is passed through when present and left empty otherwise, never filled with a guess.

Subscription and live flags were uniform. Across a full partition, every entry reported no for both. They are passed through verbatim because they are upstream's own flags, but no filter is offered on them: a filter whose field has one observed value looks like a choice and never changes a result.

The keyword filter runs here, not at Fox. Fox exposes no video search parameter, so keywords narrow what a run already retrieved. To search wider, widen the run with includeArchive and maxArchivePages.

The duration filter is exact. Duration is an integer Fox supplies for every entry, so nothing is estimated there.

Undated videos are dropped inside a date range. If a date bound is set and a video carries no usable date, it is excluded rather than included on a guess.

Partitions are large. Each is roughly 4,825 videos in a multi-megabyte file. One is only fetched when the newest slice cannot already satisfy your request, and a partition falling entirely before your start date ends the walk.

Articles are a separate Actor. Fox's article pages carry a body, author and word count that videos have no equivalent of, so they have their own scraper rather than being folded in here with half the columns empty.

Blocking and proxies

No bot-mitigation gate was found on any Fox News surface during development: the sitemaps and the video pages answered cold across four different TLS fingerprints, with no challenge, no cookie and no token. The Actor defaults to Apify's shared datacenter proxy, included in your plan at no extra cost. Residential is available in the proxy editor if you ever need it; Apify bills it per gigabyte.

One quirk worth knowing, because it looks like a block and is not: Fox's video sitemap answers a partition that does not exist with HTTP 502 and an eight byte body, where its article sitemap would answer 404. This Actor recognises that exact response as the end of the archive and stops cleanly, instead of retrying a page that can never exist and reporting a transport failure.

Politeness

robots.txt at foxnews.com sets no Crawl-delay and names no AI-crawler group. It disallows the site search, the video search, the wires and the printer-friendly paths, none of which this Actor touches: it reads the video sitemaps that robots.txt itself advertises, and public video pages. Request starts are paced by minRequestInterval, which defaults to a quarter second.