# Any News Site Article Extractor (`vexaloom-labs/any-news-site-article-extractor`) Actor

Point this actor at a news website's homepage and get back structured articles: title, authors, publish date, full body text, top image, keywords and a short summary, with keyword and minimum-length filtering.

- **URL**: https://apify.com/vexaloom-labs/any-news-site-article-extractor.md
- **Developed by:** [vexaloom-labs](https://apify.com/vexaloom-labs) (community)
- **Categories:** News, Automation, Other
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.49 / 1,000 results

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Any News Site Article Extractor

Turn news pages into structured article rows without writing a scraper per publication. Paste article links you already have, or give the actor a news site and let it find the stories itself.

### Two ways to run it

**Known links.** Put one or more article URLs in `articleUrls`. Each page is fetched and parsed directly — useful when you have a list of links from a feed, a newsletter, or a search result.

**Site discovery.** Put a homepage, section page, or newsroom URL in `websiteUrl`. The actor reads the page's current story links first, then walks the publication's sitemaps (including `robots.txt` declarations, sitemap indexes, and gzipped sitemap files) to fill the rest of your quota. A section URL such as `https://example.com/technology` keeps discovery focused on that section.

Both paths can be combined in one run. Known links are processed first, then discovery continues until `maxArticles` is reached.

### What each article contains

- The canonical article URL and the URL that was requested
- Headline, author names, publish and last-modified timestamps
- Full body text with navigation, related-story rails, newsletter prompts, and ad labels removed
- Word count, detected language, section, and meta description
- Top image plus every other image inside the article body (related-story and author-card images are excluded)
- Optional top-10 keyword list and a three-sentence summary
- Source domain and extraction timestamp

Fields a page genuinely does not publish are left out. The actor never writes `null`, empty strings, or empty arrays into a dataset row.

### How the extraction cascade works

Each page is parsed with three fallbacks, in order:

1. **JSON-LD** — `NewsArticle`, `Article`, `BlogPosting`, and related schema.org types, including `@graph` and `mainEntity` wrappers.
2. **Open Graph / article meta** — `og:*`, `article:published_time`, `article:author`, `article:section`, `og:locale`, and canonical links.
3. **The rendered HTML** — the most article-specific body container (`.entry-content`, `.article-body`, `[itemprop=articleBody]`, and similar), author bylines, `<time datetime>`, and in-article images.

Structured data wins over meta tags, which win over DOM heuristics. This keeps the output consistent across publishers while still returning clean text on older sites that never adopted JSON-LD.

### Inputs

| Field | Type | Default | What it does |
|---|---|---|---|
| `articleUrls` | string\[] | – | Article links to extract directly |
| `websiteUrl` | string | – | Homepage, section, or newsroom URL to discover articles from |
| `maxArticles` | integer | `20` | Total article cap for the run (1–500) |
| `keywordFilter` | string | – | Boolean expression over title + body: `AND`, `OR`, `NOT`, parentheses |
| `minWordCount` | integer | `100` | Drop articles whose body is shorter than this |
| `concurrency` | integer | `5` | Parallel article fetches (1–20) |
| `extractKeywords` | boolean | `true` | Add a top-10 keyword list to each row |
| `extractSummary` | boolean | `true` | Add a three-sentence summary to each row |
| `language` | string | `auto` | Language hint / filter — `auto` or one of 40 ISO codes |
| `autoProxyFallback` | boolean | `true` | Retry a blocked direct fetch through Apify residential proxy |

At least one of `articleUrls` or `websiteUrl` is required.

### Output fields

| Field | Description |
|---|---|
| `articleUrl` | URL that was fetched |
| `articleCanonicalUrl` | Publisher's canonical URL, when declared |
| `articleTitle` | Headline |
| `articleAuthors[]` | Author names, cleaned and deduplicated |
| `articlePublishDate` | ISO 8601 publish timestamp |
| `articleModifiedDate` | ISO 8601 last-modified timestamp |
| `articleSection` | Section or desk, when declared |
| `articleDescription` | Meta / JSON-LD description |
| `articleText` | Full body text |
| `articleWordCount` | Words in `articleText` |
| `articleTopImage` | Hero / lead image URL |
| `articleImages[]` | All article image URLs, hero first |
| `articleLanguage` | Detected or requested language code |
| `articleSiteName` | Publication name, when declared |
| `articleKeywords[]` | Top content words by frequency (optional) |
| `articleSummary` | Three-sentence summary (optional) |
| `sourceDomain` | Hostname of the article |
| `scrapedAt` | ISO 8601 extraction timestamp |

If a run finds no usable articles it writes one diagnostic row with `type: "news_crawler_error"` and a `reason`, so a scheduled job can tell "empty result" apart from "no output".

### Examples

Extract three links directly:

```json
{
  "articleUrls": [
    "https://www.bbc.com/news/articles/example-one",
    "https://apnews.com/article/example-two"
  ],
  "maxArticles": 3
}
```

Discover the latest stories from a technology publication and keep only AI coverage:

```json
{
  "websiteUrl": "https://techcrunch.com",
  "maxArticles": 25,
  "keywordFilter": "(AI OR \"machine learning\") AND NOT crypto",
  "minWordCount": 200
}
```

Scan a French news site in French:

```json
{
  "websiteUrl": "https://www.franceinfo.fr",
  "maxArticles": 20,
  "language": "fr"
}
```

### Common uses

- **Media monitoring** — watch a publication or section and route only matching stories to alerts.
- **Research corpora** — collect article text and metadata from many outlets into one dataset.
- **Brand and executive tracking** — filter coverage by names, products, or competitors.
- **Digest building** — take a list of links from a feed and turn it into clean rows for a newsletter or briefing.
- **RAG pipelines** — use the body text and canonical URLs as document chunks with reliable provenance.

### FAQ

**Does it work on every news site?**
Most publications that expose article metadata or a sitemap work without special handling. Pages that are fully client-side rendered, paywalled behind a login, or served as a bot challenge may return thin or no body text.

**What happens on sites that block datacenter IPs?**
The direct fetch is tried first. If it is blocked and `autoProxyFallback` is on, the same requests are retried through Apify residential proxy. Sites that work directly never use proxy bandwidth.

**Are duplicate links returned twice?**
No. URLs are deduplicated before fetching, so the same article appears once even if it is both pasted and discovered.

**Why did I get fewer rows than `maxArticles`?**
The cap is a ceiling, not a promise. A small site, a shallow sitemap, the keyword filter, and the minimum word count can all reduce the number of qualifying articles.

**Can I use the keyword filter with phrases?**
Yes. Quote multi-word phrases: `"artificial intelligence" AND NOT "machine learning"`. Bare terms match on word boundaries, so `AI` will not match `email`.

**Is this affiliated with any publisher?**
No. It is an independent tool that reads publicly accessible pages from any news site. It is not endorsed by or connected to any publication.

# Actor input Schema

## `articleUrls` (type: `array`):

Paste one or more article URLs to extract directly. Any news site works, and no discovery is needed. When both this and News website URL are set, these URLs are processed first.

## `websiteUrl` (type: `string`):

Homepage, section page, or newsroom URL to discover articles from. The actor reads the site's sitemaps first (including robots.txt entries and gzipped sitemaps) and falls back to article-shaped links on the page itself. Leave blank when using Article URLs.

## `maxArticles` (type: `integer`):

Hard cap on the number of articles returned across all input URLs.

## `keywordFilter` (type: `string`):

Boolean expression applied to each article's title and body. Supports AND, OR, NOT and parentheses, case-insensitive. Examples: `AI AND NOT crypto`, `(startup OR funding) AND SaaS`. Leave blank to keep every article.

## `minWordCount` (type: `integer`):

Drop articles whose extracted body text is shorter than this.

## `concurrency` (type: `integer`):

Number of articles fetched in parallel. Higher is faster but more likely to trigger rate limits on small sites.

## `extractKeywords` (type: `boolean`):

Compute the top 10 most frequent content words per article (stopwords removed).

## `extractSummary` (type: `boolean`):

Emit the first three sentences of each article as a summary, with a meta-description fallback.

## `language` (type: `string`):

Language hint for the articles. Use `auto` to detect from site metadata. An explicit language also filters out articles that declare a different one.

## `autoProxyFallback` (type: `boolean`):

When a direct datacenter-IP fetch is blocked (0 candidates or 0 extracted articles), retry the same requests once through Apify residential proxy. Sites that work directly never trigger it. Turn off to guarantee zero proxy spend.

## Actor input object example

```json
{
  "articleUrls": [
    "https://www.bbc.com/news/articles/example-article"
  ],
  "websiteUrl": "https://techcrunch.com",
  "maxArticles": 1,
  "keywordFilter": "AI AND startup",
  "minWordCount": 100,
  "concurrency": 5,
  "extractKeywords": true,
  "extractSummary": true,
  "language": "auto",
  "autoProxyFallback": true
}
```

# Actor output Schema

## `articles` (type: `string`):

Dataset containing all scraped news articles.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "websiteUrl": "https://techcrunch.com",
    "maxArticles": 1,
    "extractKeywords": true,
    "extractSummary": true,
    "autoProxyFallback": true
};

// Run the Actor and wait for it to finish
const run = await client.actor("vexaloom-labs/any-news-site-article-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "websiteUrl": "https://techcrunch.com",
    "maxArticles": 1,
    "extractKeywords": True,
    "extractSummary": True,
    "autoProxyFallback": True,
}

# Run the Actor and wait for it to finish
run = client.actor("vexaloom-labs/any-news-site-article-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "websiteUrl": "https://techcrunch.com",
  "maxArticles": 1,
  "extractKeywords": true,
  "extractSummary": true,
  "autoProxyFallback": true
}' |
apify call vexaloom-labs/any-news-site-article-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,vexaloom-labs/any-news-site-article-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/DBmfP5hmm0BrHG5lx/builds/RT9eB8Rtre1mEYvQS/openapi.json
