Article Extractor & News Scraper

Extract articles, news content, and blog posts from any website. Get clean, structured data with title, full text, authors, publication date, images, keywords, and metadata—powered by 7 specialized extraction engines.

🔗 Run this Actor on Apify | 📖 API Documentation

---

Table of Contents

• Features
• Use Cases
• How It Works
• Extraction Engines Comparison
• Quick Start
• Input Configuration
• Output Format
• Example Outputs
• Anti-Blocking Features
• Performance Tips
• Integrations
• Troubleshooting
• FAQ
• Changelog

---

✨ Features

Core Capabilities

• 🔍 7 Specialized Extraction Engines — Choose from Newspaper4k, Trafilatura, Boilerpy3, News-Please, Goose3, Article Parser, and JusText
• 🌐 Universal Website Compatibility — Works with news sites, blogs, magazines, and any article-based content
• 📊 Complete Content Extraction — Captures title, description, full text, authors, publication date, images, keywords, and metadata
• 🔄 Smart Fallback System — Automatically tries alternative extractors if the primary one fails

Anti-Blocking Technology

• 🎭 Browser Impersonation — Uses curl_cffi and impit to send requests with a real browser's TLS fingerprint and headers
• 🔀 Proxy Rotation — Automatic proxy rotation with support for residential proxies
• ⏱️ Intelligent Rate Limiting — Domain-specific delays and concurrency control

Output Options

• 📝 Plain Text — Clean, extracted article text
• 🔖 Article HTML — Preserved formatting with links and media
• 📄 Full Page HTML — Complete webpage source for custom processing
• 📋 Structured JSON — All metadata in a standardized format

---

🎯 Use Cases

Industry	Application
Media Monitoring	Track news coverage, brand mentions, and competitor activity
Research & Academia	Collect data for NLP, sentiment analysis, and content studies
Content Aggregation	Build news feeds, curated content platforms, and newsletters
SEO Analysis	Analyze competitor content, keywords, and publishing patterns
Market Intelligence	Monitor industry news, trends, and market developments
Web Archiving	Preserve article content with full metadata
AI/ML Training	Generate training datasets for language models

---

⚙️ How It Works

graph LR
    A[Input URLs] --> B[Fetch Pages]
    B --> C{Anti-Bot Check}
    C -->|Blocked| D[Rotate Proxy/Headers]
    D --> B
    C -->|Success| E[Extract Content]
    E --> F{Extraction OK?}
    F -->|No| G[Try Fallback Engine]
    G --> E
    F -->|Yes| H[Output JSON]

1. Input Processing — Accepts a list of article URLs
2. Smart Fetching — Impersonates a real browser and rotates proxies
3. Anti-Bot Evasion — Detects block pages and retries with a fresh proxy
4. Content Extraction — Applies the selected extraction engine
5. Fallback Logic — Automatically tries alternative engines if extraction fails
6. Output Generation — Returns structured JSON with all extracted data

---

📊 Extraction Engines Comparison

Engine	Best For	Speed	Metadata	NLP Features	Special Capabilities
Newspaper4k	General news	⚡⚡⚡	✅ Full	✅ Yes	Summary, keywords, NER
Trafilatura	News & blogs	⚡⚡⚡⚡	✅ Full	❌ No	Language detection, categories
Boilerpy3	Simple articles	⚡⚡⚡⚡⚡	⚠️ Basic	❌ No	Text density metrics
News-Please	Rich metadata	⚡⚡	✅ Full	❌ No	Multiple fallback methods
Goose3	Image extraction	⚡⚡⚡	✅ Full	❌ No	Top image detection
Article Parser	HTML/Markdown	⚡⚡⚡	⚠️ Basic	❌ No	Multiple output formats
JusText	Boilerplate removal	⚡⚡⚡⚡	⚠️ Basic	❌ No	Language-aware filtering

Recommended Engines by Content Type

• 📰 News Sites → Newspaper4k or Trafilatura
• 📝 Blog Posts → Trafilatura or Goose3
• 📚 Long-form Articles → Newspaper4k (with NLP for summarization)
• 🖼️ Image-heavy Content → Goose3
• ⚡ High-volume Scraping → Boilerpy3 or Trafilatura
• 🔤 Non-English Content → JusText (40+ languages supported)

---

🚀 Quick Start

Run on Apify Platform

{
  "startUrls": [
    "https://www.nytimes.com/2024/01/15/technology/ai-developments.html",
    "https://www.theguardian.com/world/2024/jan/15/breaking-news"
  ],
  "extractorEngine": "newspaper4k"
}

Run Locally with Apify CLI

# Install Apify CLI
npm install -g apify-cli

# Clone and run
apify pull article-extractor-news-scraper
cd article-extractor-news-scraper
apify run --input='{"startUrls": ["https://example.com/article"]}'

Call via API

curl -X POST "https://api.apify.com/v2/acts/YOUR_USERNAME~article-extractor-news-scraper/runs" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": ["https://www.bbc.com/news/world-12345"],
    "extractorEngine": "newspaper4k"
  }'

---

📝 Input Configuration

Parameter	Type	Default	Description
startUrls	array	required	List of article URLs to extract
extractorEngine	string	newspaper4k	Extraction engine to use
useFallbackExtractors	boolean	true	Try alternative engines if primary fails
saveHtml	boolean	false	Include full page HTML in output
saveArticleHtml	boolean	false	Include cleaned article HTML
maxRetries	integer	15	Retry attempts for failed requests
useHeaderGenerator	boolean	true	Impersonate the browsers set in headerGeneratorOptions (otherwise alternate Chrome and Firefox)
headerGeneratorOptions	object	{}	browsers (chrome, firefox, safari, edge) and devices (desktop, mobile) to impersonate
customHeaders	object	{}	Additional HTTP headers
proxyConfiguration	object	residential	Proxy settings

Full Input Example

{
  "startUrls": [
    "https://www.nytimes.com/2024/01/15/world/article.html",
    "https://www.theguardian.com/world/2024/jan/15/story",
    "https://www.bbc.com/news/world-12345678"
  ],
  "extractorEngine": "newspaper4k",
  "useFallbackExtractors": true,
  "saveHtml": false,
  "saveArticleHtml": true,
  "maxRetries": 15,
  "useHeaderGenerator": true,
  "headerGeneratorOptions": {
    "browsers": ["chrome", "firefox", "safari", "edge"],
    "devices": ["desktop"]
  },
  "customHeaders": {},
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  }
}

---

📤 Output Format

Each extracted article produces a JSON object with the following fields:

Common Fields (All Engines)

Field	Type	Description
url	string	Original article URL
title	string	Article headline
text	string	Full article text (cleaned)
sourceDomain	string	Website domain
extractorEngine	string	Engine used for extraction
extractedAt	string	ISO 8601 timestamp

Extended Fields (Engine-Dependent)

Field	Type	Available In
description	string	newspaper4k, goose3, news-please
author	array	newspaper4k, news-please
publishedDate	string	newspaper4k, trafilatura, news-please
image	string	newspaper4k, goose3, news-please
keywords	array	newspaper4k, goose3
summary	string	newspaper4k
language	string	newspaper4k, trafilatura, justext
categories	array	trafilatura
tags	array	trafilatura
allImages	array	newspaper4k
metaData	object	newspaper4k
siteName	string	newspaper4k
favicon	string	newspaper4k

Metadata Fields

Field	Type	Description
fallbackUsed	boolean	Whether a fallback engine was used
originalExtractor	string	Originally requested engine (if fallback used)

---

📋 Example Outputs

---

🛡️ Anti-Blocking Features

This Actor includes advanced anti-blocking technology to maximize success rates:

Browser Impersonation

Uses curl_cffi and impit to make each request look like it comes from a real browser:

• Chrome, Firefox, Safari, and Edge (desktop or mobile)
• The browser's real TLS fingerprint, so it matches the headers
• The browser's own headers, including sec-ch-ua client hints
• By default, retries alternate between Chrome and Firefox, since sites block them differently

Proxy Rotation

• Automatic proxy rotation on 403/429 errors
• Support for residential, datacenter, and custom proxies
• Domain-specific proxy strategies

Intelligent Rate Limiting

• Per-domain concurrency control
• Adaptive delays based on site response
• Strict mode for heavily protected sites

---

⚡ Performance Tips

For Maximum Speed

{
  "extractorEngine": "boilerpy3",
  "maxRetries": 5,
  "useFallbackExtractors": false
}

For Maximum Success Rate

{
  "extractorEngine": "newspaper4k",
  "maxRetries": 15,
  "useFallbackExtractors": true,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  }
}

For Rich Metadata

{
  "extractorEngine": "newspaper4k",
  "saveArticleHtml": true,
  "useFallbackExtractors": true
}

---

🔌 Integrations

Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")

run_input = {
    "startUrls": ["https://www.example.com/article"],
    "extractorEngine": "newspaper4k"
}

run = client.actor("YOUR_USERNAME/article-extractor-news-scraper").call(run_input=run_input)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"Title: {item['title']}")
    print(f"Text: {item['text'][:200]}...")

JavaScript/Node.js

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });

const run = await client.actor('YOUR_USERNAME/article-extractor-news-scraper').call({
    startUrls: ['https://www.example.com/article'],
    extractorEngine: 'newspaper4k'
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach(item => {
    console.log(`Title: ${item.title}`);
    console.log(`Text: ${item.text.substring(0, 200)}...`);
});

Webhooks

Configure webhooks to receive results automatically:

{
  "webhooks": [
    {
      "eventTypes": ["ACTOR.RUN.SUCCEEDED"],
      "requestUrl": "https://your-server.com/webhook"
    }
  ]
}

Zapier / Make (Integromat)

Use the Apify integration in Zapier or Make to connect extracted articles to:

• Google Sheets
• Notion databases
• Slack notifications
• Email newsletters
• CRM systems

---

🔧 Troubleshooting

Common Issues

Issue	Cause	Solution
Empty text output	Anti-bot blocking	Enable residential proxies, reduce concurrency
403/429 errors	Rate limiting	Increase maxRetries
Timeout errors	Slow server response	Increase maxRetries, use residential proxies
Missing metadata	Engine limitation	Switch to a different extraction engine
Garbled text	Encoding issues	Try trafilatura or newspaper4k

Reporting Issues

If you encounter persistent issues:

1. Check if the URL works in a regular browser
2. Try different extraction engines
3. Open an issue with:
   • The problematic URL
   • Your input configuration

---

❓ Frequently Asked Questions

---

📝 Changelog

v1.0.0 (December 2025)

• ✨ Initial public release
• 🔍 7 extraction engines: Newspaper4k, Trafilatura, Boilerpy3, News-Please, Goose3, Article Parser, JusText
• 🛡️ Advanced anti-blocking with browser impersonation
• 🔄 Automatic fallback extraction
• 📊 Multiple dataset views (Overview, Content, Metadata)
• ⚙️ Configurable concurrency and retry settings

---

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

---

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

---

Built with ❤️ for the data extraction community

Keywords: article extractor, news scraper, web scraping, content extraction, newspaper4k, trafilatura, apify actor, python scraper, text extraction, metadata extraction, NLP, news monitoring, content aggregation