Article Extractor for LLM / RAG

Turn a list of article URLs into clean, LLM-ready JSON. No browser, no boilerplate -- just the content.

What it does

Give it a list of article, blog, or news URLs. For each one it fetches the page and extracts:

• title -- article headline
• author -- byline, when present
• publishedDate -- publish date, when present
• siteName -- publisher/site name
• language -- detected language
• text -- full body text, with navigation, ads, comments, and other boilerplate stripped out
• wordCount -- word count of the extracted text
• excerpt -- first ~400 characters
• contentHash -- short hash of the text, useful for dedup/change detection in a RAG pipeline

Built on trafilatura, a widely used content-extraction library, run over a plain HTTP fetch. No headless browser and no proxy needed for the large majority of article/blog/news pages.

Input

{
  "urls": [
    "https://example.com/some-article",
    "https://example.com/another-article"
  ],
  "maxConcurrency": 5
}

• urls (required) -- list of article URLs to process.
• maxConcurrency (optional, default 5) -- how many URLs to fetch in parallel.

Output

One dataset item per URL:

{
  "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "status": "success",
  "title": "Retrieval-augmented generation - Wikipedia",
  "author": null,
  "publishedDate": "2023-11-05",
  "siteName": "Wikimedia Foundation, Inc.",
  "text": "Retrieval-augmented generation\nRetrieval-augmented generation (RAG) is a technique...",
  "wordCount": 2070,
  "excerpt": "Retrieval-augmented generation\nRetrieval-augmented generation (RAG) is a technique...",
  "contentHash": "a1b2c3d4e5f6a7b8",
  "httpStatus": 200,
  "extractedAt": "2026-08-21T04:00:00.000000"
}

If a URL fails or has no extractable article content, status will be failed or no_content_extracted and error will explain why -- you'll always get one dataset item per input URL, so batch runs are easy to reconcile.

Notes

• This actor does not render JavaScript. Sites that require JS to inject their article body (rare for news/blog content) will come back as no_content_extracted.
• Every input URL produces exactly one dataset item, including failures, so success/failure counts always add up to your input count.