# article-scrapper (`credible_sandal/article-scrapper`) Actor

A flexible and powerful Apify Actor for scraping articles from tech news websites. This scraper can work with any tech news site - either from predefined presets or custom URLs

- **URL**: https://apify.com/credible_sandal/article-scrapper.md
- **Developed by:** [RK K](https://apify.com/credible_sandal) (community)
- **Categories:** News
- **Stats:** 30 total users, 0 monthly users, 100.0% runs succeeded, 1 bookmarks
- **User rating**: No ratings yet

## Pricing

$3.00 / 1,000 result_sets

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Tech News Article Scraper

A powerful Apify Actor for scraping articles from 14 popular tech news sources or any custom URL. Extracts full article content, metadata, sentiment, word count, and reading time. Supports RSS feeds for fast and reliable scraping.

### Features

- **14 Preset Sources** — The Verge, TechCrunch, Wired, Hacker News, BBC Tech, MIT Technology Review, VentureBeat, Dev.to, Product Hunt, and more
- **RSS Feed Support** — Faster and more reliable than HTML scraping; automatically used for all preset sources
- **Custom URLs** — Scrape any news site by providing your own URLs
- **Smart Extraction** — Automatically detects titles, authors, dates, content, images, and tags
- **Sentiment Analysis** — Labels each article as positive, neutral, or negative with a confidence score
- **Date Range Filtering** — Filter by date range or relative window (e.g. last 7 days)
- **Flexible Filters** — Filter by keywords, title text, or description with AND/OR logic
- **Word Count & Reading Time** — Automatically calculated for every article
- **Robust Error Handling** — Retry logic, graceful fallbacks, and detailed logging

***

### Preset Sources

| Key | Source | Focus |
|-----|--------|-------|
| `verge` | The Verge | Tech news and media |
| `techcrunch` | TechCrunch | Startups and technology |
| `wired` | Wired | Technology and culture |
| `cnet` | CNET | Product reviews and news |
| `arstechnica` | Ars Technica | In-depth tech analysis |
| `engadget` | Engadget | Consumer electronics |
| `theguardian-tech` | The Guardian Tech | Tech news from The Guardian |
| `thenextweb` | The Next Web | International tech news |
| `hackernews` | Hacker News | Developer and startup community |
| `bbc-tech` | BBC Technology | Global tech news |
| `mit-tech-review` | MIT Technology Review | Deep tech and AI research |
| `venturebeat` | VentureBeat | AI, startups, and enterprise tech |
| `devto` | Dev.to | Developer articles and tutorials |
| `producthunt` | Product Hunt | Daily startup and product launches |

***

### Input Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `usePresets` | boolean | `true` | Use preset sources or custom URLs |
| `presetSources` | array | `["verge", "techcrunch", "wired"]` | List of preset source keys |
| `customUrls` | array | `[]` | Custom URLs to scrape (when `usePresets` is false) |
| `maxArticlesPerSource` | integer | `10` | Max articles per source (1–100) |
| `maxPages` | integer | `1` | Max listing pages to check per source (1–10) |
| `includeContent` | boolean | `true` | Extract full article text |
| `useRssFeeds` | boolean | `true` | Use RSS feeds for preset sources (faster, more reliable) |
| `includeSentiment` | boolean | `true` | Add sentiment analysis to each article |
| `sentimentFilter` | string | `"all"` | Only return articles with this sentiment: `all`, `positive`, `neutral`, `negative` |
| `filterMode` | string | `"any"` | `any` = article passes if it matches at least one filter (OR). `all` = article must pass every filter (AND) |
| `searchKeywords` | array | `[]` | Filter articles containing any of these keywords in title or summary |
| `titleContains` | string | `""` | Only include articles with this text in the title |
| `descriptionContains` | string | `""` | Only include articles with this text in the summary or content |
| `dateFrom` | string | `""` | Only include articles published on or after this date (YYYY-MM-DD) |
| `dateTo` | string | `""` | Only include articles published on or before this date (YYYY-MM-DD) |
| `publishedWithin` | string | `""` | Only include articles published within this window. Examples: `24h`, `7d`, `30d`, `2w` |

***

### Output Format

Each article is saved with the following fields:

```json
{
  "title": "OpenAI Releases GPT-5",
  "url": "https://techcrunch.com/2025/01/15/openai-gpt5",
  "author": "Jane Doe",
  "published_date": "2025-01-15T10:30:00+00:00",
  "content": "Full article text...",
  "summary": "OpenAI today announced GPT-5, its most capable model yet...",
  "source": "TechCrunch",
  "tags": ["AI", "OpenAI", "GPT"],
  "image_url": "https://techcrunch.com/images/gpt5.jpg",
  "word_count": 847,
  "reading_time_minutes": 4,
  "sentiment": {
    "label": "positive",
    "score": 0.6249
  },
  "scraped_at": "2025-01-15T11:00:00+00:00"
}
```

> `sentiment` is only present when `includeSentiment` is `true`.

***

### Usage Examples

#### 1. Morning AI News Briefing

Get the latest AI articles from the last 24 hours across major sources:

```json
{
  "usePresets": true,
  "presetSources": ["techcrunch", "verge", "wired", "mit-tech-review"],
  "maxArticlesPerSource": 20,
  "publishedWithin": "24h",
  "searchKeywords": ["AI", "artificial intelligence", "ChatGPT", "LLM"],
  "filterMode": "any",
  "useRssFeeds": true,
  "includeSentiment": true
}
```

#### 2. Brand Monitoring — Negative Press Only

Track negative coverage about a topic:

```json
{
  "usePresets": true,
  "presetSources": ["techcrunch", "verge", "arstechnica", "bbc-tech"],
  "maxArticlesPerSource": 30,
  "searchKeywords": ["Apple", "iPhone"],
  "filterMode": "any",
  "includeSentiment": true,
  "sentimentFilter": "negative",
  "publishedWithin": "7d"
}
```

#### 3. Startup Launch Tracker

Track new product launches and funding news:

```json
{
  "usePresets": true,
  "presetSources": ["producthunt", "techcrunch", "venturebeat"],
  "maxArticlesPerSource": 20,
  "searchKeywords": ["launch", "funding", "raises", "Series A"],
  "filterMode": "any",
  "dateFrom": "2025-01-01"
}
```

#### 4. Developer Community Digest

Pull community articles from Dev.to and Hacker News:

```json
{
  "usePresets": true,
  "presetSources": ["devto", "hackernews"],
  "maxArticlesPerSource": 50,
  "includeContent": true,
  "useRssFeeds": true
}
```

#### 5. Custom Site Scraping

Scrape any site not in the preset list:

```json
{
  "usePresets": false,
  "customUrls": [
    "https://9to5mac.com",
    "https://9to5google.com"
  ],
  "maxArticlesPerSource": 15,
  "maxPages": 2,
  "includeContent": true
}
```

#### 6. Long-form Articles Only

Use word count to filter out short posts and stubs:

```json
{
  "usePresets": true,
  "presetSources": ["wired", "arstechnica", "mit-tech-review"],
  "maxArticlesPerSource": 20,
  "includeContent": true
}
```

> Tip: After export, filter `word_count > 800` in your spreadsheet to get long-form articles only.

***

### Filtering Guide

Three filter types are available — `filterMode` controls how they combine:

| filterMode | Behaviour |
|------------|-----------|
| `any` (recommended) | Article passes if it matches **at least one** active filter |
| `all` | Article must pass **every** active filter simultaneously |

**Example with `filterMode: "any"`:**

- `searchKeywords: ["AI"]` matches → article included, regardless of other filters

**Example with `filterMode: "all"`:**

- Article must match `searchKeywords` AND `titleContains` AND `descriptionContains` — very strict

***

### Viewing Results

1. **Dataset Tab** — View all articles in a table after the run completes
2. **Export** — Download as JSON, CSV, XML, or Excel
3. **API** — Access results programmatically via the Apify API
4. **Schedule** — Set up automated daily or weekly runs via Apify Scheduler

***

### Troubleshooting

**Getting 0 articles with multiple filters set**

- Switch `filterMode` to `"any"` — AND logic (`"all"`) is very strict
- Increase `maxArticlesPerSource` so more articles are available to filter
- Check the logs — each rejected article shows which filter removed it

**Missing content or short word counts**

- RSS feeds provide summaries, not full articles — set `useRssFeeds: false` for full HTML scraping
- Some sites have unique page structures the generic scraper may miss

**Connection timeouts or blocked requests**

- Reduce `maxArticlesPerSource` to avoid rate limiting
- Some sites block automated requests — try a different source

**JavaScript-heavy sites not working**

- This scraper uses static HTML parsing — JS-rendered pages are not supported
- Use RSS mode (`useRssFeeds: true`) for preset sources to avoid this entirely

***

### FAQ

**Can I scrape paywalled content?**
No — only publicly accessible content is scraped.

**How fast is it?**
RSS mode: ~10 articles per second. HTML mode: ~1–2 articles per second due to polite rate limiting.

**Can I add my own sources permanently?**
Use `customUrls` for one-off scraping, or open a GitHub issue to request a new preset source.

**Is this legal?**
Web scraping legality depends on the site's terms of service and your jurisdiction. Always check `robots.txt`, avoid overloading servers, and use data responsibly. This tool is for legitimate use cases only.

***

### License

Provided as-is for educational and legitimate scraping purposes. Always respect website terms of service and robots.txt files.

# Actor input Schema

## `usePresets` (type: `boolean`):

Use predefined news sites or provide custom URLs

## `presetSources` (type: `array`):

Select from predefined tech news sources (only used if 'Use Preset Sites' is checked)

## `customUrls` (type: `array`):

Enter custom URLs to scrape (one per line). Can be homepages or article listing pages. Used when 'Use Preset Sites' is unchecked.

## `maxArticlesPerSource` (type: `integer`):

Maximum number of articles to scrape from each source

## `maxPages` (type: `integer`):

Maximum number of listing pages to check per source

## `includeContent` (type: `boolean`):

Extract full article text (may increase scraping time)

## `searchKeywords` (type: `array`):

Filter articles by keywords. Only articles containing ANY of these keywords in title or description will be included. Leave empty to scrape all articles.

## `filterMode` (type: `string`):

'any' — article passes if it matches at least one filter (OR logic, recommended). 'all' — article must pass every filter simultaneously (AND logic, very strict).

## `titleContains` (type: `string`):

Only scrape articles where the title contains this text (case-insensitive). Leave empty to scrape all articles.

## `descriptionContains` (type: `string`):

Only scrape articles where the summary/description contains this text (case-insensitive). Leave empty to scrape all articles.

## `dateFrom` (type: `string`):

Only include articles published on or after this date. Format: YYYY-MM-DD or ISO datetime (e.g. 2025-01-15). Leave empty for no restriction.

## `dateTo` (type: `string`):

Only include articles published on or before this date. Format: YYYY-MM-DD or ISO datetime (e.g. 2025-03-31). Leave empty for no restriction.

## `publishedWithin` (type: `string`):

Only include articles published within this time window. Examples: '24h', '7d', '30d', '2w'. Takes precedence over 'Published After' when both are set.

## `useRssFeeds` (type: `boolean`):

For preset sources, use their RSS/Atom feed instead of HTML scraping. Feeds are faster, more reliable, and return cleaner data.

## `includeSentiment` (type: `boolean`):

Analyze each article's sentiment (positive / neutral / negative) based on its title and summary. Adds a 'sentiment' field to the output.

## `sentimentFilter` (type: `string`):

Only return articles matching this sentiment. Requires 'Include Sentiment Analysis' to be enabled. Use 'all' to return everything.

## Actor input object example

```json
{
  "usePresets": true,
  "presetSources": [
    "verge",
    "techcrunch",
    "wired"
  ],
  "maxArticlesPerSource": 10,
  "maxPages": 1,
  "includeContent": true,
  "searchKeywords": [
    "AI",
    "ChatGPT",
    "iPhone"
  ],
  "filterMode": "any",
  "titleContains": "AI",
  "descriptionContains": "artificial intelligence",
  "useRssFeeds": true,
  "includeSentiment": true,
  "sentimentFilter": "all"
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "searchKeywords": [
        "AI",
        "ChatGPT",
        "iPhone"
    ],
    "titleContains": "AI",
    "descriptionContains": "artificial intelligence"
};

// Run the Actor and wait for it to finish
const run = await client.actor("credible_sandal/article-scrapper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "searchKeywords": [
        "AI",
        "ChatGPT",
        "iPhone",
    ],
    "titleContains": "AI",
    "descriptionContains": "artificial intelligence",
}

# Run the Actor and wait for it to finish
run = client.actor("credible_sandal/article-scrapper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "searchKeywords": [
    "AI",
    "ChatGPT",
    "iPhone"
  ],
  "titleContains": "AI",
  "descriptionContains": "artificial intelligence"
}' |
apify call credible_sandal/article-scrapper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,credible_sandal/article-scrapper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/gewZImvXGOpOOac1q/builds/1qEqCybmJwdLJ045S/openapi.json