AI Article Extractor

Turn public web pages into clean article text you can read, save, or feed into an AI tool. Give it article links, a sitemap, or an RSS/Atom feed, and it returns one tidy row per article with the title, author, date, language, word count, and the full article body. It is for anyone who wants the useful part of a page without menus, ads, cookie banners, or other clutter.

What you get

One result row per article, with these fields:

• Link — the address you asked for, plus the page's own preferred address for the same article when it provides one.
• Title — the article headline.
• Author — the byline, or several names separated by commas.
• Published date — in a standard date format when the page gives a readable date, otherwise the date as shown on the page.
• Site name — the publication or website name.
• Language — a short language code such as en or en-US.
• Word count — counted by spaces in the cleaned article text.
• Article body — the main text only. You choose Markdown (the default, with working links, headings, lists, tables, and code blocks), plain text, or cleaned HTML.
• Chunks — optional smaller pieces of the article, each with a piece number, the text, an approximate token count (a token is a small unit of text used by AI tools), and a character count.
• Status — ok means the main article was found; fallback means a simpler reading method found it; failed means no usable article text was found, but the row is still kept so you can see what was missed.
• Reason — a short explanation when a page failed or only partly worked.
• Fetched at — when the row was created.

How to use

1. Open the actor in Apify Console.
2. Fill in the input form. You need at least one source: article links, a sitemap, or a feed.
   • Article URLs — paste full web addresses, one per line. You can combine these with a sitemap or feed. If this is empty, the other sources are used; if all three source fields are empty, the run stops and tells you to add a source.
   • Sitemap URL — paste the address of an XML sitemap. It can be a sitemap that points to other sitemaps; up to 25 sitemap files are followed, three levels deep. Leave it empty to skip sitemaps. Links that look like files, such as images, PDFs, or ZIP archives, are ignored.
   • RSS/Atom feed URL — paste the address of an RSS or Atom feed to use its entries. Leave it empty to skip feeds.
   • Max articles — the total number of articles to process, from 1 to 1000. The default is 10. Empty means 10. Sources are read in this order: article links, then sitemap, then feed. Once the limit is reached, later sources are not read.
   • Output format — choose Markdown, plain text, or HTML. Empty means Markdown.
   • Chunk size — optional. Set a number from 32 to 8000 to split each article into smaller pieces of roughly that many tokens (small units of text used by AI tools). Empty means no chunks.
   • Request timeout — how long to wait for one page, from 1 to 300 seconds. Empty means 30 seconds.
   • Max requests per minute — a politeness limit from 1 to 600 requests. Empty means 60.
   • Respect robots.txt — on by default. When on, pages the site's robots.txt rules disallow are skipped and do not appear in the results. Leave it on unless you own the site.
3. Press Start.
4. Wait for the run to finish. The run shows how many articles it has processed. Large runs take longer because requests are spread out.
5. Open the Output or Dataset tab. Each article is one row. Download the results as JSON, CSV, or Excel.
6. You can also fetch results through the Apify API or connect them to other tools, without writing code.

Example input

{
    "urls": ["https://blog.apify.com/what-is-web-scraping/"],
    "maxArticles": 1,
    "chunkSize": 200
}

Example output

The example below is shortened: the article text and the chunk texts are cut, and the author's name is replaced with a fake name.

{
    "url": "https://blog.apify.com/what-is-web-scraping/",
    "canonicalUrl": "https://blog.apify.com/what-is-web-scraping/",
    "title": "What is web scraping?",
    "author": "Jane Doe",
    "publishedAt": "2024-10-15T14:32:00.000Z",
    "siteName": "Apify Blog",
    "language": "en",
    "wordCount": 1427,
    "outputFormat": "markdown",
    "extractionStatus": "ok",
    "extractionError": null,
    "fetchedAt": "2026-10-27T06:00:13.627Z",
    "markdown": "Web scraping is the process of automatically extracting data from a website. You use a program called a web scraper to access a web page, interpret the data, and extract what you need. The data is…",
    "chunks": [
        {
            "index": 0,
            "text": "Web scraping is the process of automatically extracting data from a website. You use a program called a web scraper to access a web page, interpret the data, and extract what you need. The data is…",
            "tokenCount": 165,
            "charCount": 659
        },
        {
            "index": 1,
            "text": "Web scraping is the **automated process of extracting data from a website.** Web scraping is also known as **web harvesting**, **data extraction**, **web data extraction, data scraping,** and **data…",
            "tokenCount": 94,
            "charCount": 376
        }
    ]
}

This output comes from a real run of the actor on that link.

Tips and limits

• Public pages only. Login-only, private, and paywalled content cannot be read.
• Some pages need extra steps to show their text. The actor reads the page as it is first delivered, so those pages may come back empty. It never invents article content.
• Some sites block automated visits or show different content to them. A blocked request is retried twice, then its row is marked failed with the reason.
• Missing details stay empty. Some pages do not state an author, date, or language. A date may be the page's last-updated date rather than the original publish date.
• Images, videos, and formulas are left out of Markdown and plain text. HTML output keeps images. Links, tables, lists, and code blocks are kept.
• Duplicate links are compared after cleanup. A trailing #section and common tracking parameters are removed first. The same article under two different addresses still produces two rows.
• Pages disallowed by robots.txt are skipped without a row. This is why the number of rows can be lower than the number of links you supplied.
• Sitemaps are taken in their listed order, not newest first. Up to 25 sitemap files are read, three levels deep.
• Word and token counts are estimates. The word count uses spaces, so languages that do not separate words with spaces are undercounted. Chunk token counts are approximate and do not match any one AI model exactly.
• A large run can be slow. The default limit is 60 requests per minute, so 1,000 articles can take a while. Start with a small number to check the output.
• The actor does not summarize, translate, or rewrite articles. It collects and cleans what the page already contains. It does not follow links inside an article or crawl beyond the sources you provide.
• Use content responsibly. Respect each site's terms, robots.txt rules, and copyright. Extracted pages may contain personal data, so collect only what you need and follow the privacy rules that apply to you.

FAQ

Do I need to know how to code? No. Fill in the form, press Start, and download the results from the Output or Dataset tab.

Why did an article fail? The page may be blocked, paywalled, empty, or only show its text after extra page steps. Open the row's reason field to see what happened. If a page was disallowed by robots.txt, it is skipped and has no row at all.

Can I process a whole blog? Yes, use its sitemap. The actor follows up to 25 sitemap files, three levels deep, and the max articles setting caps the total. For a few pages, pasting article links is simpler.

Can I get images in the output? Not in Markdown or plain text. Choose HTML if you need the cleaned article with its images.

Are the chunks ready to use in my AI tool? They are plain pieces of the article, split near paragraphs or sentences and never overlapping. The token counts are estimates, so check them against your tool's limits before uploading very large sets.