Article Content Extractor

📰 Article Content Extractor – Scrape Article Text, Metadata & Author Data from Any URL

The Article Content Extractor is a lightweight article scraper that turns any list of web page URLs into clean, structured article data — title, description, full body content, author, published date, source domain, lead image, outbound links and tags. Point it at news articles, blog posts, press releases, documentation pages or editorial listings, and it returns one tidy JSON record per URL, ready for a content database, an NLP pipeline or a retrieval-augmented generation index.

Unlike a generic HTML dump, this article content extractor focuses on the fields that actually matter for content analysis. It parses the page's metadata layer and the main body region, normalises them into a stable schema, and stamps every record with the time it was collected. Because it runs on plain HTTP requests and an HTML parser rather than a headless browser, runs are fast and cheap on static and server-rendered pages.

---

📊 What Data Can You Extract with This Article Scraper?

The article scraper returns a flat, predictable record for every URL you submit. Every field below is part of the real output schema — nothing more, nothing less.

Category	Fields	What it gives you
🔗 Identity & provenance	url, source	The canonical URL of the scraped page and the publishing domain it came from, so you always know where a record originated.
📝 Headline & summary	title, description	The article headline and the long-form description or summary text used for previews, meta descriptions and social cards.
📄 Body content	content	The extracted textual content of the article — the main reading material, ready for text analysis, summarisation or embedding.
✍️ Attribution & timing	author, publishedDate	The byline and the publication date, the two fields most content teams need for credibility scoring and freshness filters.
🖼️ Media	image	The lead image or social preview image URL associated with the article.
🧭 Navigation & topics	links, tags	Outbound and internal links collected from the page, plus any topic tags or keywords attached to the article.
⏱️ Collection metadata	scrapedAt	An ISO 8601 timestamp recording exactly when the record was captured.

One field that quietly does a lot of work is scrapedAt. When you re-run the same URL list on a schedule, the timestamp lets you diff versions of an article over time — catching silent edits to headlines, corrections to body copy or changes in the byline that publishers rarely announce.

---

🌟 Key Features of the Article Content Extractor

Feature	Description
🎯 URL-list driven	Feed a simple array of article URLs. No selectors to configure, no site-specific templates to maintain.
⚡ HTTP-first architecture	Built on httpx and BeautifulSoup rather than a headless browser, so static and server-rendered pages are parsed quickly and with a small compute footprint.
🧱 Stable flat schema	Every record has the same eleven keys in the same shape, so downstream loaders, CSV exports and database inserts never break on a missing column.
📝 Full body extraction	The content field captures the article's textual content rather than just a truncated meta description.
🏷️ Metadata harvesting	author, publishedDate, image, tags and description are pulled from the page's metadata layer where publishers expose them.
🔗 Link collection	The links array captures URLs discovered on the page, useful for citation mapping and seed generation for deeper crawls.
🕒 Timestamped records	Each item carries a scrapedAt timestamp, making change detection and incremental loads straightforward.
🧯 Null-safe output	Every field is typed as string-or-null (or array-or-null), so partial pages produce usable records instead of failed runs.
📦 Dataset-native output	Results land in an Apify dataset and export to JSON, CSV, Excel, XML or HTML, or stream straight out through the API.

---

🚀 Why Choose This Article Content Extractor?

No per-site configuration. Most article scraping projects die under the weight of maintaining CSS selectors for dozens of publishers. This article content extractor works from a single input — a list of URLs — and applies a generic extraction strategy to each one. Adding a new publisher to your pipeline costs nothing more than appending a URL.

Built for text pipelines, not screenshots. The content and description fields are plain text you can pass directly into a tokeniser, an embedding model, a summariser or a keyword extractor. There is no post-processing step to strip navigation chrome out of a full-page HTML blob before your analysis can begin.

Fast and lightweight by design. The actor uses direct HTTP requests with an HTML parser instead of launching a browser per page. For the static and server-rendered pages that make up the bulk of news and blog content, that means shorter runs and lower compute consumption.

Predictable, machine-readable records. The output schema is fixed and every field is null-safe. That reliability matters when the article scraper is one stage in a longer chain — a scheduled run feeding a warehouse table, a webhook triggering an enrichment step, or an integration pushing rows into a spreadsheet.

---

📥 Input

The Article Content Extractor takes a single required input: the list of article URLs you want to extract.

{
  "urls": [
    "https://www.fancode.com/pickleball/schedule"
  ]
}

🔧 Article Content Extractor Input Fields

Field	Type	Required	Default	Description
urls	array of strings	✅ Yes	["https://www.fancode.com/pickleball/schedule"]	List of article URLs to extract content from. Each entry is fetched and parsed into one output record.

💡 Input Examples

Single article

{
  "urls": ["https://example.com/news/2026/market-report"]
}

Batch of articles from several publishers

{
  "urls": [
    "https://example.com/blog/quarterly-outlook",
    "https://another-site.org/press/product-launch",
    "https://third-site.net/opinion/editorial-column"
  ]
}

A monitoring list you re-run on a schedule

{
  "urls": [
    "https://example.com/company/newsroom",
    "https://example.com/company/press-releases",
    "https://example.com/investors/announcements"
  ]
}

---

📤 Output

Each URL produces one JSON object in the dataset. Below is a real record from an actual run, lightly trimmed for readability.

{
  "url": "https://www.fancode.com/pickleball/schedule",
  "title": "Cricket Schedule - Check International and Domestic matches on FanCode",
  "description": "Afghanistan tour of Ireland, 2026 Asian Legends League, 2026 Caribbean Premier League, 2026 Delhi Prem…",
  "content": "<div><div><p><label>Afghanistan tour of Ireland, 2026</label></p></div><div><p><label>Asian Legends…",
  "author": "",
  "publishedDate": "",
  "source": "fancode.com",
  "image": "https://www.fancode.com/skillup-uploads/fc-web/fc_logo_og_image.png",
  "links": [
    "https://www.fancode.com/pickleball/schedule"
  ],
  "tags": [],
  "scrapedAt": "2026-08-08T14:41:38.460712Z"
}

🧾 Article Content Extractor Output Fields

Field	Type	Description
url	string | null	Canonical URL of the scraped item.
title	string | null	Title of the item — usually the article headline.
description	string | null	Long-form description text, typically the summary or meta description.
content	string | null	Extracted textual content of the article body.
author	string | null	Author of the item. Empty when the page exposes no byline.
publishedDate	string | null	Publication date of the item where the page declares one.
source	string | null	Source the item came from, normally the publishing domain.
image	string | null	Image URL — the lead or social preview image.
links	array | null	Links collected for the item.
tags	array | null	Tags attached to the item.
scrapedAt	string | null	ISO 8601 timestamp recording when the record was scraped.

Note that author, publishedDate and tags reflect what the page actually publishes. Listing pages, schedules and index pages often carry no byline or date, so those fields come back empty — that is faithful extraction, not a failure.

---

💻 How to Use the Article Content Extractor (Step by Step)

Step 1: Assemble Your Article URL List

Start by collecting the URLs you want to extract. These can come from an RSS feed, a sitemap, a search result export, a competitor watchlist or a manual shortlist of publications you track. The article scraper accepts a plain array of strings, so almost any source of links can be reshaped into valid input with a spreadsheet formula or a two-line script. Keep the list focused: a tight list of genuinely relevant articles produces a cleaner dataset than an indiscriminate dump of every link on a domain.

Step 2: Open the Actor and Paste Your URLs

In the Apify Console, open the Article Content Extractor and go to the Input tab. The urls field is rendered as a string list editor, so you can add entries one per line or paste a block of URLs at once. The field is pre-filled with a sample URL that you can replace. If you prefer working in raw JSON, switch to the JSON editor and supply the object shown in the Input section above.

Step 3: Run the Article Scraper

Click Start and watch the log. The actor works through the URL list, fetching each page and parsing it into the standard record shape. Because the extraction is HTTP-based rather than browser-based, throughput is high and the run log stays readable — you will see progress per URL rather than long stretches of browser startup noise.

Step 4: Review the Dataset Preview

When the run finishes, open the Storage → Dataset tab. The preview table shows one row per URL with all eleven fields as columns. Scan the title and source columns first: if those look right, the extraction found the correct page. Then spot-check content on two or three rows to confirm the body text you expected is present and reasonably clean.

Step 5: Export the Article Data

Use the export button to download the dataset as JSON, CSV, Excel, XML or HTML. JSON preserves the links and tags arrays exactly as scraped, which matters if you plan to process them programmatically. CSV and Excel flatten those arrays and are the better choice when the data is going to an analyst, a content editor or a stakeholder report.

Step 6: Automate with Schedules and Webhooks

For ongoing monitoring, attach a schedule to the actor so the same URL list is re-scraped daily, hourly or weekly. Combine that with a webhook that fires on run success to push new records into your own system the moment they land. The scrapedAt field makes it trivial to work out which rows are new since your last import.

Step 7: Connect the Output to Your Pipeline

Finally, wire the dataset into whatever consumes it — a vector store for RAG, a content warehouse, an internal search index, a Slack digest or a Google Sheet. The API section below shows the two most common patterns: a synchronous cURL call that returns items directly, and a Python client call for scripted workflows.

---

🔌 API Access & Integrations

Run the article content extractor directly from the Apify API and get the dataset items back in a single synchronous call.

curl -X POST "https://api.apify.com/v2/acts/scrapers-hub~article-content-extractor/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": [
      "https://example.com/news/2026/market-report",
      "https://example.com/blog/quarterly-outlook"
    ]
  }'

The same run from Python using the official client:

from apify_client import ApifyClient

client = ApifyClient("YOUR_TOKEN")

run_input = {
    "urls": [
        "https://example.com/news/2026/market-report",
        "https://example.com/blog/quarterly-outlook",
    ]
}

run = client.actor("scrapers-hub/article-content-extractor").call(run_input=run_input)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["source"], "—", item["title"])
    print(item["content"][:300] if item.get("content") else "(no content)")

Beyond the API, the actor plugs into Zapier, Make, Google Sheets, Slack and generic HTTP webhooks, so extracted article data can flow into your CRM, content calendar, data warehouse or notification channels without custom glue code.

---

💡 Best Use Cases for Article Content Data

📚 Building a Content Corpus for AI and RAG

The content, title and description fields give you clean, plain-text documents to chunk and embed for a retrieval-augmented generation index or a fine-tuning dataset. Pair each chunk with url and source so your model's answers can cite the original publisher. The publishedDate field lets you weight recent material more heavily when relevance depends on freshness.

📈 Competitive Content and SEO Analysis

Extract every article a competitor publishes and analyse title patterns, tags taxonomy and content length to reverse-engineer their editorial strategy. Because source normalises the publishing domain, you can group thousands of records by publisher and compare topic coverage across an entire competitive set in one pivot table.

📰 Media Monitoring and Brand Mentions

Feed the article scraper a rotating list of news URLs and search results, then scan content and title for brand names, product names and executive names. Combining publishedDate with scrapedAt tells you both when a story ran and when you caught it — the gap between the two is a direct measure of how fast your monitoring loop reacts.

🧠 Sentiment and Topic Modelling Pipelines

Text analytics needs body copy, not snippets. The content field supplies enough text for meaningful sentiment scoring, named entity recognition and topic modelling. Group results by author to see how individual journalists frame a subject, or by source to compare editorial tone across outlets.

🔗 Citation Mapping and Link Graph Research

The links array captures the URLs a page points to. Collect articles across a domain or a topic and you can build a link graph showing which sources cite which, where authority concentrates, and which outbound destinations a publisher favours. That graph is a strong seed list for the next round of article scraping.

🗂️ Archiving and Change Detection

Publishers edit articles quietly. Re-running the same urls list on a schedule and comparing title, content and author between snapshots surfaces stealth edits, retitled headlines and removed bylines. The scrapedAt timestamp anchors each snapshot to a point in time, which is exactly what you need for a defensible archive.

🎯 Newsletter and Content Curation Workflows

Curators can pull title, description, image and url for a shortlist of articles and drop them straight into a newsletter template. The image field supplies the thumbnail, description supplies the blurb, and tags help you sort candidate stories into the right newsletter section before a human ever reads them.

---

⚙️ Tips for Better Article Scraping Results

• Point at article pages, not category hubs. Index pages, schedules and tag archives are valid inputs, but they rarely carry an author or publishedDate. If your analysis depends on those fields, submit the individual article permalinks instead.
• Batch sensibly. Sending a few hundred URLs per run keeps logs readable and makes it easy to identify which batch produced an anomaly. Very large lists are better split across several scheduled runs.
• Deduplicate before you submit. The same article often appears under tracking parameters, AMP variants and syndication mirrors. Stripping query strings and normalising URLs before submission avoids paying to scrape the same story three times.
• Expect empty strings on some fields. author, publishedDate and tags come back empty when the page does not publish them. Treat empty as "not declared by the publisher" rather than as an error, and build your downstream filters accordingly.
• Check content on a small sample first. Run five representative URLs from a new publisher before committing a large batch. That tells you immediately whether the site's markup yields the body text you need.
• Use scrapedAt for incremental loads. Rather than re-importing whole datasets, filter on scrapedAt to pull only records captured since your last sync.

---

🛠️ Troubleshooting

Why is the content field empty or very short for some URLs? Some pages render their body text with client-side JavaScript after the initial HTML loads. Because this article content extractor fetches the served HTML directly rather than executing a browser, content injected purely at runtime will not appear. Sites that server-render or ship their text in the initial HTML — the majority of news and blog platforms — extract normally.

Why are author and publishedDate blank? These fields are only populated when the page actually declares them. Listing pages, schedules, product pages and many corporate sites publish no byline or date at all. Verify by viewing the page source: if there is no author or date in the markup, there is nothing for the scraper to extract.

A URL returned nothing at all — what happened? Check that the URL is publicly reachable without a login, a cookie wall or a regional redirect. Paywalled and members-only articles typically serve a stub page to anonymous requests, so the extractor sees only the teaser that the publisher chose to expose.

Why does content contain HTML tags? The content field carries the extracted textual content as found in the page's main region, which on some sites includes inline markup. If you need strictly plain text, strip tags in your downstream step — a single regex or an HTML-to-text helper handles it, and keeping the raw form gives you the option of preserving structure when you need it.

The run finished but produced fewer records than URLs submitted. Duplicate URLs in the input list and URLs that fail to resolve are the usual causes. Compare the url values in the dataset against your input list to identify which entries dropped out, then re-check those specific links in a browser.

---

❓ Frequently Asked Questions About Article Scraping

What does the Article Content Extractor actually do? It takes a list of article URLs, fetches each page, and returns structured records containing the title, description, body content, author, published date, source domain, lead image, links, tags and a scrape timestamp.

Do I need to write CSS selectors or XPath for each website? No. The article scraper applies a generic extraction approach to every URL you submit, so there is no per-site configuration and nothing to maintain when a publisher redesigns their template.

What input does the article content extractor require? Exactly one field: urls, an array of article URLs. It is the only input property and it is required.

Can I scrape multiple websites in a single run? Yes. The urls array can mix domains freely — the extractor handles each URL independently and stamps every record with its source domain so you can group results by publisher afterwards.

Does this article scraper use a headless browser? No. It uses direct HTTP requests with an HTML parser, which makes runs faster and lighter on static and server-rendered pages. Pages that build their entire body in the browser after load are the trade-off.

Can it extract full article text, or just summaries? The content field carries the extracted textual content of the page body, not just a meta description. The separate description field holds the summary text where the publisher provides one.

How do I get the data out of Apify? Export the dataset as JSON, CSV, Excel, XML or HTML from the Console, or pull items programmatically through the API and the apify_client library. Both approaches are shown in the API Access section above.

Can I schedule the article content extractor to run automatically? Yes. Attach an Apify schedule to run the same URL list on any cadence you like, and add a webhook so downstream systems are notified the moment fresh records land.

Is there a limit on how many URLs I can submit? The input schema does not impose a fixed cap. Practical limits come from run duration and memory, so very large lists are best split into several runs.

Why do some records have empty tags arrays? Because the page publishes no tags. The field is preserved as an empty array rather than dropped so that your output schema stays consistent across every record.

Can I use this article scraper for paywalled content? Only what a publisher serves to anonymous visitors is retrievable. If an article is behind a hard paywall, the extractor sees the same teaser a logged-out reader sees.

What is the links field useful for? It contains URLs found on the scraped page. Use it to map citations between publishers, to discover related articles worth scraping next, or to audit a site's outbound linking behaviour.

Does the actor handle redirects? Standard HTTP redirects are followed as part of the fetch, and the url field reflects the canonical URL of the scraped item so you can tell where a request ultimately landed.

How do I detect when an article has been edited? Re-run the same URL on a schedule and compare title, content and author between snapshots, using scrapedAt to order them. Any difference between two consecutive captures indicates an edit.

What formats can I export article data in? Apify datasets export to JSON, JSONL, CSV, Excel, XML, HTML and RSS. JSON is the best choice when you want the links and tags arrays preserved intact.

---

🆘 Support & Feedback

Found a bug, hit an edge case, or seen a page that extracts poorly? Open a ticket on the actor's Issues tab with the exact URL and a short description of what you expected — reproducible reports are fixed fastest.

Need a customised version — extra fields, a different extraction strategy, deeper crawling, or a private actor built around your specific publisher set? Email scraperhubapi@gmail.com and describe your use case.

If the Article Content Extractor saves you time, please leave a review on the Apify Store. Ratings and written feedback genuinely shape which improvements get prioritised next.

---

⚖️ Disclaimer

The Article Content Extractor is designed to collect publicly available data — pages that any visitor can reach without logging in, bypassing a paywall or circumventing an access control. It does not attempt to defeat authentication or subscription barriers.

You are responsible for how you use the extracted article data. Before running the article scraper at scale, review the target website's Terms of Service and robots.txt, and respect any restrictions they set out. Reasonable request volumes and sensible scheduling are part of scraping responsibly.

Article pages frequently contain personal data, most obviously author names in the author field. If you process such data for people in the EU, UK or other regulated jurisdictions, you must comply with the GDPR and equivalent privacy laws — establish a lawful basis, honour data subject rights, minimise what you retain, and delete what you no longer need. Copyright also applies: article text remains the property of its publisher, and extraction does not transfer any licence to republish it.

If you believe data collected by this actor relates to you and you would like it removed, contact scraperhubapi@gmail.com with the relevant details and we will action the request.