News Article Scraper – Headlines & Full Text

Scrape headlines and full article text from any news site, with no per-site setup and no CSS selectors. Point it at a page that lists things — a news section, a blog archive, a press-release index — and get a row per entry; point it at a story page and get that story's full text, byline and publish date.

There are no CSS selectors to write and nothing to configure per site — a language model reads the page's structure, so it works on sites it has never seen, including small local publishers whose markup no scraper targets. Built for media monitoring, news aggregation, content research and dataset building.

$1.50 per 1,000 listing pages, $0.50 per 1,000 articles. Listing pages cost more because working out which of a page's links are the list always needs the model; most articles are read straight from the publisher's own structured data and never reach it. A page that could not be read is never charged.

You need your own OpenAI API key. Paste it into the input, or set it once as an OPENAI_API_KEY environment variable on the Actor. Most article pages cost nothing in model usage at all — see What it costs to run below.

Features

• No selectors, no per-site rules — works on a site it has never seen, and does not break when that site is redesigned.
• Two modes off one input — list for index pages, article for stories.
• Every entry, not a sample — group completion takes a whole card layout rather than the subset a model names link by link.
• Links are never invented — the model sees short handles, not URLs, so every link is copied out of the page's own markup.
• Dates are never invented either — a date whose source text is nowhere on the page is dropped, so date: null means the page genuinely showed none.
• Full article text in the publisher's own words — paragraphs are joined from the page, not retold by a model.
• Wrapper links followed — Google News links, Facebook shims and t.co URLs are resolved in a browser to the real article.
• Resolved wrappers cached — Google News links are remembered permanently across runs, so polling a feed resolves each story only once.
• Browser rendering when needed — JavaScript shells and bot challenges are detected and retried in a real browser automatically.
• UTC timestamps where the page states one — datetime beside the plain date, never promoted from a bare date to a fake midnight.
• Incremental polling — onlyNewerThan takes a date or a full instant.
• Runs stay green. Every page problem, including a rejected API key, is a row.

Input

{
  "startUrls": [{ "url": "https://www.fdot.gov/info/newsroom.shtm" }],
  "type": "list",              // "list" for index pages, "article" for story pages
  "openaiApiKey": "sk-...",    // or set OPENAI_API_KEY on the Actor
  "model": "gpt-4o-mini",      // the cheapest model that reads listing pages reliably
  "maxItems": 50,              // stop after this many rows from each page
  "onlyNewerThan": "2026-08-01",  // list mode: skip entries published before this
  "cacheHours": 24,            // reuse wrapped links followed in the last N hours
  "renderJavaScript": "auto",  // "auto" | "never" | "always"
  "proxy": { "useApifyProxy": true }
}

Field	Type	Default	What it does
startUrls	array	—	Required. One or more URLs to read — listing pages, or story pages in article mode.
type	string	list	list for index pages, article for story pages.
openaiApiKey	string	—	Your key, stored encrypted. Can also be an OPENAI_API_KEY environment variable on the Actor.
model	string	gpt-4o-mini	Which model reads the page. The default is the cheapest one that reads listing pages reliably.
maxItems	integer	50	Stop after this many rows from each page.
onlyNewerThan	string	—	List mode: leave out entries published before a date (2026-08-01) or instant (2026-08-01T09:00:00Z).
cacheHours	integer	24	How long a followed wrapper link stays reusable. 0 disables it.
renderJavaScript	string	auto	auto, never or always.
proxy	object	Apify Proxy	Some publishers refuse datacenter addresses — switch to residential if a page comes back refused.

Output

Give it the URL of a page that lists things — a news section, a blog archive, a press-release index — and get back one row per entry:

field	what it is
title	the entry's headline, as the page shows it
link	the full URL of the entry, taken from the page's own markup
date	the published date as YYYY-MM-DD, or null if the page shows none
datetime	the full published instant in UTC, when the page carries one — usually null on listing pages, see below
position	the entry's place in the list, starting at 1
pageUrl	the listing page the row came from

Article mode

Set What the pages are to article and give it story URLs — the ones list mode just handed you — and each returns one row:

field	what it is
title	the headline
author	the byline, reduced to the name, or null if the page names none
date	the published date as YYYY-MM-DD, or null
datetime	the full published instant in UTC, e.g. 2026-08-19T19:14:00.000Z, or null
text	the full article text
url	where the article actually lives, after any redirect
requestedUrl	the link you sent

Links that stand in front of an article

Give it a Google News link, a Facebook link shim or a t.co URL and it follows that link to the real article and reads that. Those wrappers do not resolve over plain HTTP — a Google News URL answers with a normal page and no redirect at all, and only a browser completes the hop — so they are rendered on purpose rather than hopefully.

url is then the publisher's address and requestedUrl is the wrapper you sent, which is how you match a row back to the link it came from when you send a batch:

{
  "title": "Science Hill named Top 20 high school in Tennessee",
  "author": "From staff reports",
  "date": "2026-08-19",
  "url": "https://johnsoncitypress.com/news/387210/science-hill-named-top-20-high-school-in-tennessee/",
  "requestedUrl": "https://news.google.com/rss/articles/CBMilwFBVV95cUxPUHA4djlCZExmODJSc0M…"
}

A wrapper that never reaches an article fails that row rather than describing the wrapper page as though it were the story.

Many publishers already state the headline, date, byline and body in their page's structured data. Where they do, that is read directly and the run costs nothing in model usage; the model is only called for pages that do not.

This is also where dates come from for sites that do not print them on their listing pages — a common pattern, where the date only exists on the story itself.

Example

Input:

{
  "startUrls": [{ "url": "https://www.fdot.gov/info/newsroom.shtm" }],
  "openaiApiKey": "sk-..."
}

Output:

{
  "title": "FDOT Announces Lane Closures on I-95",
  "link": "https://www.fdot.gov/info/newsroom/2026/lane-closures-i95.shtm",
  "date": "2026-08-19",
  "position": 1,
  "pageUrl": "https://www.fdot.gov/info/newsroom.shtm"
}

Dates, and when you get a time as well

date is always the day alone, because that is what publishers reliably show and what most callers sort on. datetime carries the full instant — time and timezone, normalised to UTC — whenever the page actually states one.

Where it comes from differs by mode, and it is worth knowing which to expect:

• Article pages often state a precise timestamp in their structured data, and that is used as-is. Of a mixed sample of article pages, 40% carried a full timestamp with a timezone.
• Listing pages rarely do. Of seven news index pages checked, not one had a single machine-readable <time> element — they show "Aug 19, 2026" or "2 hours ago" and nothing more. So expect datetime to be null on most listing rows, and read the article itself when you need the exact time.

A bare date is never promoted to midnight. Inventing 00:00:00 for a story the publisher only dated would look like a time while sorting arbitrarily, so the field stays null instead.

Polling a section for what is new

Only entries newer than takes a date (2026-08-01) or a full instant (2026-08-01T09:00:00Z) and leaves out anything published before it, so a scheduled run returns only what has appeared since you last looked.

It follows from the section above that an entry with no date on the listing page is kept, not dropped. Many news platforms print no date in their index at all — the date only exists on the story — and dropping those entries would return an empty dataset on exactly those sites, which here means "this section published nothing". The run logs how many it could not judge. If you need those entries filtered too, read them in article mode, where the date almost always comes from the publisher's own structured data.

How to use

From Apify Console

1. Open the Actor and click Try for free / Start.
2. Paste your OpenAI key into OpenAI API key, or set OPENAI_API_KEY once as an environment variable on the Actor so you never paste it again.
3. Put the section or archive URLs into Listing pages and leave What the pages are on list. Send a dozen at once — batching is nearly 3x cheaper per page and costs you nothing to adopt.
4. Run it. Each row is one entry, with its title, link and date.
5. Feed those link values back in as a second run with What the pages are set to article to get the full text of the ones you want.
6. Export the Dataset as JSON, CSV or Excel.

Check the rows for an error field — the run stays green even when every page in it failed.

From the API

curl -s "https://api.apify.com/v2/acts/simple.actors~universal-article-extractor/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{
    "startUrls": [{"url": "https://www.fdot.gov/info/newsroom.shtm"}],
    "type": "list",
    "openaiApiKey": "sk-..."
  }'

The two-step pattern — list a section, then read the stories — with the JavaScript client:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const actor = client.actor('simple.actors/universal-article-extractor');

// 1. What has this section published since yesterday?
const listed = await actor.call({
    startUrls: [{ url: 'https://www.fdot.gov/info/newsroom.shtm' }],
    type: 'list',
    onlyNewerThan: '2026-08-28',
});
const { items: entries } = await client.dataset(listed.defaultDatasetId).listItems();

// 2. Read them all in one batch — batching is ~3x cheaper per page than one per run.
const read = await actor.call({
    startUrls: entries.filter((e) => !e.error).map((e) => ({ url: e.link })),
    type: 'article',
});
const { items: articles } = await client.dataset(read.defaultDatasetId).listItems();
for (const a of articles) {
    if (a.error) continue;
    console.log(a.title, a.author, a.datetime ?? a.date, a.text.length);
}

Use cases

• Media monitoring and PR clip tracking — poll a set of publisher sections with onlyNewerThan and pull the full text of anything new.
• News aggregation — build a feed from local publishers whose markup no off-the-shelf scraper targets, without writing a selector for each.
• Press-release and regulator monitoring — government, agency and corporate newsrooms are listing pages like any other; watch them for new notices.
• Content and competitor research — headlines, bylines and publish dates across a set of competing publications.
• Training and NLP dataset building — full article text in the publisher's own words, with the byline and timestamp attached.
• Resolving wrapped links at scale — turn a pile of Google News, Facebook or t.co links into publisher URLs and article text, with each row matched back by requestedUrl.

How it works

A normal scraper needs a CSS selector per site, and that selector breaks the next time the site is redesigned. This one has no per-site rules at all. It reduces the page to its links and the text around each of them, then asks a language model which of those links are the list — the same judgment a person makes at a glance, and the part that plain selectors cannot generalise.

That means it works on a site it has never seen, including small local publishers whose markup no scraper targets.

Every entry, not a sample of them

Deciding link by link, a model tends to stop early: on one news index it named 26 of the 59 stories in the same repeated card layout and left the rest, and it left a different subset each run. So once it has clearly chosen a group of cards — several of them, and a good share of the group — the rest of that group is taken with it. On the same page that moved the result from 32-39 entries varying run to run to a steady 61.

Two limits are worth knowing. Repeated listings of one recurring event — the same production once per showtime — are folded down to a single entry. And an entry completed this way has no date unless the page states one in machine-readable form, because the date is never inferred from surrounding text without the model having read it. Read the entry in article mode when you need its date.

The link is never guessed

The model is shown each link by a short handle rather than by its address, and returns handles. The URL you get is copied out of the page's markup — a model cannot invent a link it was never able to type. Dates get the same treatment: the model has to report the exact text it read a date from, and a date whose source text is nowhere on the page is dropped rather than published. A row with date: null means the page genuinely showed no date.

Article mode works the same way: the model says which blocks are the story and the text is joined from the page's own paragraphs, so what you get is the publisher's wording rather than a model's retelling of it. An author or date the page never showed is dropped rather than invented.

Pages that need a browser

Most sites send their listing in the HTML, which is read without a browser — faster, cheaper, and what happens by default. Some smaller publishers send an empty shell and load the listing afterwards with JavaScript, and some sites answer with a bot challenge. Both are detected and retried in a real browser automatically, so you do not have to know in advance which kind of site you have. Set Run the page's JavaScript to never or always if you do want to pin it.

Usage notes

Both modes work through their input several at a time, and a run has a fixed start-up cost that one link has to carry alone. Measured on the same twelve listing pages: $0.00042 a page sent together against $0.00120 sent one per run — 2.9x cheaper, for the same pages and the same result. You are billed per page either way, so batching costs you nothing to adopt. Beyond a dozen or so the gain flattens out, so there is no need for enormous batches.

Wrapped links you have already followed are reused. In article mode, a Google News link, Facebook shim or t.co address followed within the last 24 hours returns its article from store instead of being fetched again — no request to the publisher, no model call. Those are the links worth remembering, because the address is not the article's identity and you cannot tell a repeat from the link alone.

An ordinary article URL is always read fresh, on the basis that your own code already knows which articles it has. Change Reuse wrapped links read in the last (hours) to widen or narrow the window, or set it to 0 to disable. Every row carries retrievedAt, the moment the article was actually read from the web, so you can always tell.

Listing pages are never reused. A listing page is a question about what is on it now, and answering that from store would report yesterday's news as today's.

Google News links are remembered once resolved, permanently and across runs. A link always points at the same article, and Google keeps handing out the same link for the same story — 99 of 100 unchanged across a poll twenty seconds apart, and 23 of 24 still matching a cache written four hours earlier. So polling a feed repeatedly pays to resolve each story only once, and a repeat read of the same set costs about half.

What it costs to run

The Actor charges $1.50 per 1,000 listing pages and $0.50 per 1,000 articles. Your OpenAI usage is separate and billed by OpenAI, not here — and it is smaller than you might expect, because most article pages are read from the publisher's own structured data with no model call at all. Measured over 100 articles from a mix of news sites, 85 needed no model call and the whole batch cost about a tenth of a cent.

Listing pages always use the model, since deciding which links are the list is the judgment you are paying for. Measured over 49 real listing pages that is about 6,700 tokens in and 1,000 out — roughly $0.0017 of OpenAI per page on the default model, against about $0.0001 for an average article.

Empty output means something

An empty dataset means the pages were read successfully and listed nothing. A page that could not be read — refused, challenged, or returning links that could not be matched — comes back as its own row carrying error and errorDescription, so one blocked page never costs you the rest of the batch and an empty result is never a disguised error.

The rows are where failures are reported, not the run status. The run succeeds even when every page in it failed, and even when it could not start at all — a missing or rejected OpenAI key is itself a row saying so, and its first sentence is the run's status message. So do not read a green run as "every page worked": check the rows for an error field. Only a bug in the Actor fails a run.

Limits

• It reads the page you give it. It does not follow "next page" links, so maxItems can only ever trim what that one page lists.
• List mode extracts the list, not the stories. Feed those links back in as article to get the text.
• Sites behind an unsolvable bot challenge come back as an error row rather than as text. Residential proxy clears most of them.

FAQ

Is scraping news sites legal? This Actor reads publicly visible pages and returns the publisher's own wording rather than a rewrite. Copyright in the article text belongs to the publisher, so check the platform's ToS and each site's own terms before republishing what you collect — reading for monitoring or analysis is a different question from redistributing full text.

Do I need an OpenAI API key? Yes, your own. Paste it into the input or set OPENAI_API_KEY once as an environment variable on the Actor. Your OpenAI usage is billed by OpenAI separately, and it is small: of 100 articles measured, 85 needed no model call at all because the publisher's structured data already had what was needed.

Do I need to write CSS selectors for each site? No — that is the point. There are no per-site rules at all, so it works on a site it has never seen and does not break when that site is redesigned.

Does it support pagination or "next page" links? No. It reads the page you give it and does not follow pagination, so maxItems can only trim what that one page lists. Give it each page URL you want read.

Why is date null on my listing rows? Because the page showed no date. Dates are never inferred — the model must report the exact text it read one from, and a date whose source text is not on the page is dropped. Of seven news index pages checked, not one carried a machine-readable <time> element. Read the entry in article mode when you need its date.

Why is datetime null when date has a value? A bare date is never promoted to midnight, because inventing 00:00:00 would look like a real time while sorting arbitrarily. datetime appears only when the page actually states an instant — about 40% of article pages do, and almost no listing pages.

Can it follow Google News or t.co links? Yes, in article mode. Those wrappers do not resolve over plain HTTP, so they are rendered in a browser on purpose. url is then the publisher's address and requestedUrl the wrapper you sent, so you can match rows back to your input. Resolved Google News links are remembered permanently across runs.

How do I make a large job cheaper? Send links in batches rather than one per run — measured 2.9x cheaper per page for the same pages and the same result. You are billed per page either way.

Why did my run succeed when nothing came back? Check the rows for an error field. Every page problem is a row, including a missing or rejected OpenAI key, and the run stays green even when every page failed. Only a bug in the Actor fails a run.