Fox News Articles Scraper

Collects articles from foxnews.com across all 38 of its sections, with the headline, the complete body text, the author, keywords, word count, section, publish and modified dates, and the lead image.

Two public surfaces are used, and every row says which one produced it:

• Publisher feeds — 13 sections have a Fox News syndication feed that already contains the complete article text. Those sections cost one request for 25 full articles.
• Sitemaps — the other 25 sections, deep archive runs and date-filtered runs read Fox's own sitemaps and then each article's page. The archive holds roughly 1.67 million article URLs going back to November 2014.

No login, no API key, no browser. Plain HTTP, Python only.

What you get

One dataset, three kinds of row, told apart by recordType.

ARTICLE — one per article: URL, headline, description, full body text (and the body's original HTML when it came from a feed), publish and modified dates, authors, section list, keywords, word count, lead image, and whether Fox marks it free to read.

SEARCH_SUMMARY — one per requested section. It reports which surface was used, how many URLs were discovered, how many rows came back, how many article pages were fetched, how many archive partitions were read or skipped, and how many rows each filter dropped. If a number looks wrong, this row tells you why.

ERROR — one per input or URL that failed, so every request you make maps to at least one row and a failure can never look like an empty result.

Sections

latest means no section filter.

These 13 have a publisher feed, so they return 25 full articles for one request: latest, politics, world, us, opinion, entertainment, sports, science, tech, health, lifestyle, travel, media.

These 25 exist only in the sitemaps and need one page fetch per article: analysis, area-51, auto, culture, deals, faith-values, family, food-drink, forum, fox-news, games, great-outdoors, midterms-2018, official-polls, outkick, outkick-analysis, outkick-betting, outkick-culture, outkick-shows, outkick-sports, real-estate, story, transcript, true-crime, uncategorized, weather.

A section outside that list is refused before any request is sent, rather than quietly returning nothing.

Examples

Twenty-five full politics articles, one request:

{ "sections": ["politics"], "maxItemsPerSection": 25 }

Three sections at once, ten each:

{ "sections": ["politics", "world", "tech"], "maxItemsPerSection": 10 }

A section with no feed — body comes from the article page:

{ "sections": ["outkick-sports"], "maxItemsPerSection": 20 }

Everything published in July 2026 that mentions a term:

{
  "sections": ["politics"],
  "dateFrom": "2026-07-01",
  "dateTo": "2026-07-31",
  "includeArchive": true,
  "maxArchivePages": 3,
  "keywords": ["tariff"],
  "maxItemsPerSection": 100
}

Fast headline sweep, no article pages fetched at all:

{ "sections": ["latest"], "maxItemsPerSection": 150, "includeFullBody": false }

Limits, stated plainly

Feed rows carry no author. Fox's feeds leave the author element empty on every item of every feed. The feed gives you the headline, description, full body, publish date, image and topic tags. Author, keywords, word count and modified date exist only on the article page. Set enrichFeedArticles to true to fetch those pages and fill the gap — it costs one request per article, which is the whole reason it is off by default.

The keyword filter runs here, not at Fox. No public Fox surface offers a working search parameter, so keywords narrow what a run already retrieved. To search wider, widen the run with includeArchive and maxArchivePages.

A feed holds exactly 25 articles. Asking for more than 25 from a feed section without includeArchive cannot return more than 25.

Archive partitions are large. Each is 10,000 URLs in a multi-megabyte file. One is fetched only when the rolling 48-hour window cannot already satisfy your request, and partitions that fall entirely outside a date range are skipped without being downloaded. maxArchivePages is your cost control.

Undated rows are dropped inside a date range. If a date bound is set and an article carries no usable date, it is excluded rather than included on a guess.

Videos are a separate Actor. Fox's video pages carry a VideoObject, not an article body. They have their own scraper rather than being folded in here with half the columns empty.

Blocking and proxies

No bot-mitigation gate was found on any Fox News surface during development: the feed host, the sitemaps and the article pages all answered cold across four different TLS fingerprints, with no challenge, no cookie and no token. The Actor therefore defaults to Apify's shared datacenter proxy, which is included in your plan at no extra cost. Residential is available in the proxy editor if you ever need it; Apify bills it per gigabyte.

The Actor rotates TLS fingerprints and retries with exponential backoff on transport errors, and if a proxy group your account cannot use is requested, it falls back to a direct connection instead of failing the run.

Politeness

robots.txt at foxnews.com sets no Crawl-delay and names no AI-crawler group. It disallows the site search, the wires and the printer-friendly paths, none of which this Actor touches: it reads the publisher feeds, the sitemaps that robots.txt itself advertises, and public article pages. Request starts are paced by minRequestInterval, which defaults to a quarter of a second.