Any News Site Article Extractor
Pricing
from $0.49 / 1,000 results
Any News Site Article Extractor
Point this actor at a news website's homepage and get back structured articles: title, authors, publish date, full body text, top image, keywords and a short summary, with keyword and minimum-length filtering.
Pricing
from $0.49 / 1,000 results
Rating
0.0
(0)
Developer
vexaloom-labs
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
a day ago
Last modified
Categories
Features
Share
Turn news pages into structured article rows without writing a scraper per publication. Paste article links you already have, or give the actor a news site and let it find the stories itself.
Two ways to run it
Known links. Put one or more article URLs in articleUrls. Each page is fetched and parsed directly — useful when you have a list of links from a feed, a newsletter, or a search result.
Site discovery. Put a homepage, section page, or newsroom URL in websiteUrl. The actor reads the page's current story links first, then walks the publication's sitemaps (including robots.txt declarations, sitemap indexes, and gzipped sitemap files) to fill the rest of your quota. A section URL such as https://example.com/technology keeps discovery focused on that section.
Both paths can be combined in one run. Known links are processed first, then discovery continues until maxArticles is reached.
What each article contains
- The canonical article URL and the URL that was requested
- Headline, author names, publish and last-modified timestamps
- Full body text with navigation, related-story rails, newsletter prompts, and ad labels removed
- Word count, detected language, section, and meta description
- Top image plus every other image inside the article body (related-story and author-card images are excluded)
- Optional top-10 keyword list and a three-sentence summary
- Source domain and extraction timestamp
Fields a page genuinely does not publish are left out. The actor never writes null, empty strings, or empty arrays into a dataset row.
How the extraction cascade works
Each page is parsed with three fallbacks, in order:
- JSON-LD —
NewsArticle,Article,BlogPosting, and related schema.org types, including@graphandmainEntitywrappers. - Open Graph / article meta —
og:*,article:published_time,article:author,article:section,og:locale, and canonical links. - The rendered HTML — the most article-specific body container (
.entry-content,.article-body,[itemprop=articleBody], and similar), author bylines,<time datetime>, and in-article images.
Structured data wins over meta tags, which win over DOM heuristics. This keeps the output consistent across publishers while still returning clean text on older sites that never adopted JSON-LD.
Inputs
| Field | Type | Default | What it does |
|---|---|---|---|
articleUrls | string[] | – | Article links to extract directly |
websiteUrl | string | – | Homepage, section, or newsroom URL to discover articles from |
maxArticles | integer | 20 | Total article cap for the run (1–500) |
keywordFilter | string | – | Boolean expression over title + body: AND, OR, NOT, parentheses |
minWordCount | integer | 100 | Drop articles whose body is shorter than this |
concurrency | integer | 5 | Parallel article fetches (1–20) |
extractKeywords | boolean | true | Add a top-10 keyword list to each row |
extractSummary | boolean | true | Add a three-sentence summary to each row |
language | string | auto | Language hint / filter — auto or one of 40 ISO codes |
autoProxyFallback | boolean | true | Retry a blocked direct fetch through Apify residential proxy |
At least one of articleUrls or websiteUrl is required.
Output fields
| Field | Description |
|---|---|
articleUrl | URL that was fetched |
articleCanonicalUrl | Publisher's canonical URL, when declared |
articleTitle | Headline |
articleAuthors[] | Author names, cleaned and deduplicated |
articlePublishDate | ISO 8601 publish timestamp |
articleModifiedDate | ISO 8601 last-modified timestamp |
articleSection | Section or desk, when declared |
articleDescription | Meta / JSON-LD description |
articleText | Full body text |
articleWordCount | Words in articleText |
articleTopImage | Hero / lead image URL |
articleImages[] | All article image URLs, hero first |
articleLanguage | Detected or requested language code |
articleSiteName | Publication name, when declared |
articleKeywords[] | Top content words by frequency (optional) |
articleSummary | Three-sentence summary (optional) |
sourceDomain | Hostname of the article |
scrapedAt | ISO 8601 extraction timestamp |
If a run finds no usable articles it writes one diagnostic row with type: "news_crawler_error" and a reason, so a scheduled job can tell "empty result" apart from "no output".
Examples
Extract three links directly:
{"articleUrls": ["https://www.bbc.com/news/articles/example-one","https://apnews.com/article/example-two"],"maxArticles": 3}
Discover the latest stories from a technology publication and keep only AI coverage:
{"websiteUrl": "https://techcrunch.com","maxArticles": 25,"keywordFilter": "(AI OR \"machine learning\") AND NOT crypto","minWordCount": 200}
Scan a French news site in French:
{"websiteUrl": "https://www.franceinfo.fr","maxArticles": 20,"language": "fr"}
Common uses
- Media monitoring — watch a publication or section and route only matching stories to alerts.
- Research corpora — collect article text and metadata from many outlets into one dataset.
- Brand and executive tracking — filter coverage by names, products, or competitors.
- Digest building — take a list of links from a feed and turn it into clean rows for a newsletter or briefing.
- RAG pipelines — use the body text and canonical URLs as document chunks with reliable provenance.
FAQ
Does it work on every news site? Most publications that expose article metadata or a sitemap work without special handling. Pages that are fully client-side rendered, paywalled behind a login, or served as a bot challenge may return thin or no body text.
What happens on sites that block datacenter IPs?
The direct fetch is tried first. If it is blocked and autoProxyFallback is on, the same requests are retried through Apify residential proxy. Sites that work directly never use proxy bandwidth.
Are duplicate links returned twice? No. URLs are deduplicated before fetching, so the same article appears once even if it is both pasted and discovered.
Why did I get fewer rows than maxArticles?
The cap is a ceiling, not a promise. A small site, a shallow sitemap, the keyword filter, and the minimum word count can all reduce the number of qualifying articles.
Can I use the keyword filter with phrases?
Yes. Quote multi-word phrases: "artificial intelligence" AND NOT "machine learning". Bare terms match on word boundaries, so AI will not match email.
Is this affiliated with any publisher? No. It is an independent tool that reads publicly accessible pages from any news site. It is not endorsed by or connected to any publication.