Article Extractor — News & Blog Text, Date and Markdown by URL
Pricing
from $0.80 / 1,000 article extracteds
Article Extractor — News & Blog Text, Date and Markdown by URL
Article extractor: Main text, dates and Markdown from news and blog URLs. $0.001 per article extracted; start extra; usage included; failed results uncharged.
Pricing
from $0.80 / 1,000 article extracteds
Rating
0.0
(0)
Developer
Logan Parker
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
3 days ago
Last modified
Features
Share
Article extractor for news, blogs and documentation: return the main text, title, date and lead image as text and optional Markdown, with menus and sidebars removed. AI agents can call and pay through Apify MCP. List event rates are $0.001 per article extracted; start is extra, usage is included and failed results are uncharged.
Example result
Example record for the minimal input under How to use it, as this Actor returned it on 2026-10-27 in a run of its production image; the record is kept unchanged with the release.
{"url": "https://docs.apify.com/academy/scraping-basics-javascript","ok": true,"finalUrl": "https://docs.apify.com/academy/scraping-basics-javascript","title": "Web scraping basics for JavaScript devs | Academy | Apify Documentation","publishedAt": null,"language": "en","siteName": "docs.apify.com","leadImage": "https://apify.com/og-image/docs-article?title=Web+scraping+basics+for+JavaScript+devs","text": "Web scraping basics for JavaScript devs\nLearn how to use JavaScript to extract information from websites in this practical course, starting from the absolute basics.\nIn this course we'll use JavaScript to create an application for watching prices. It'll be able to scrape all product pages of an e-commerce website and record prices. Data from several runs of such program would be useful for seeing trends in price changes, detecting discounts, etc.\n- Inspect pages using browser DevTools.\n- Download web pages using the Fetch API.\n- Extract data from web pages using the Cheerio library.\n- Save extracted data in various formats (e.g. CSV which MS Excel or Google Sheets can open) using the json2csv library.\n- Follow links programmatically (crawling).\n- Save time and effort with frameworks, such as Crawlee, and scraping platforms, such as Apify.\nAnyone with basic knowledge of developing programs in JavaScript who wants to start with web scraping can take this course. The course does not expect you to have any prior knowledge of other web technologies or scraping.\n- A macOS, Linux, or Windows machine with a web browser and Node.js installed.\n- Familiarity with JavaScript basics: variables, conditions, loops, functions, strings, arrays, objects, files, classes, promises, imports, and exceptions.\n- Comfort with building a Node.js package and installing dependencies with npm.\n- Familiarity with running commands in Terminal (macOS/Linux) or Command Prompt (Windows).\nLet's explore the key reasons to take this course. What is web scraping good for, and what career opportunities does it enable for you?\nThe internet is full of useful data, but most of it isn't offered in a structured way that's easy to process programmatically. That's why you need scraping, a set of approaches to download websites and extract data from them.\nScraper development is also a fun and challenging way to learn web development, web technologies, and understand the internet. You'll reverse-engineer websites, understand how they work internally, discover what technologies they use, and learn how they communicate with servers. You'll also master your chosen programming language and core programming concepts. Understanding web scraping gives you a head start in learning web technologies such as HTML, CSS, JavaScript, frontend frameworks (like React or Next.js), HTTP, REST APIs, GraphQL APIs, and more.\nScrapers are programs specifically designed to mine data from the internet. Point-and-click or no-code scraping solutions do exist, but they only take you so far. While simple to use, they lack the flexibility and optimization needed to handle advanced cases. Only custom-built scrapers can tackle more difficult challenges. And unlike ready-made solutions, they can be fine-tuned to perform tasks more efficiently, at a lower cost, or with greater precision.\nAs a scraper developer, you are not limited by whether certain data is available programmatically through an official API - the entire web becomes your API! Here are some things you can do if you understand scraping:\n- Improve your productivity by building personal tools, such as your own real estate or rare sneakers watchdog.\n- Companies can hire you to build custom scrapers mining data important for their business.\n- Become an invaluable asset to data journalism, data science, or nonprofit teams working to make the world a better place.\n- You can publish your scrapers on platforms like the Apify Store and earn money by letting others use them.\nWe are Apify, a web scraping and automation platform. We do our best to build this course on top of open source technologies. That means what you learn applies to any scraping project, and you'll be able to run your scrapers on any computer. We will show you how a scraping platform can simplify your life, but that lesson is optional and designed to fit within our free tier.\n📄️ DevTools: Inspecting\nLesson about using the browser tools for developers to inspect and manipulate the structure of a website.\n📄️ DevTools: Locating HTML elements\nLesson about using the browser tools for developers to manually find products on an e-commerce website.\n📄️ DevTools: Extracting data\nLesson about using the browser tools for developers to manually extract product data from an e-commerce website.\n📄️ Downloading HTML\nLesson about building a Node.js application for watching prices. Using the Fetch API to download HTML code of a product listing page.\n📄️ Parsing HTML\nLesson about building a Node.js application for watching prices. Using the Cheerio library to parse HTML code of a product listing page.\n📄️ Locating HTML elements\nLesson about building a Node.js application for watching prices. Using the Cheerio library to locate products on the product listing page.\n📄️ Extracting data from HTML\nLesson about building a Node.js application for watching prices. Using string manipulation to extract and clean data scraped from the product listing page.\n📄️ Saving data\nLesson about building a Node.js application for watching prices. Using the json2csv library to save data scraped from product listing pages in both JSON and CSV.\n📄️ Getting links from HTML\nLesson about building a Node.js application for watching prices. Using the Cheerio library to locate links to individual product pages.\n📄️ Crawling websites\nLesson about building a Node.js application for watching prices. Using the Fetch API to follow links to individual product pages.\n📄️ Scraping product variants\nLesson about building a Node.js application for watching prices. Using browser DevTools to figure out how to extract product variants and exporting them as separate items.\n📄️ Using a framework\nLesson about building a Node.js application for watching prices. Using the Crawlee framework to simplify creating a scraper.\n📄️ Using a platform\nLesson about building a Node.js application for watching prices. Using the Apify platform to deploy a scraper.","markdown": "# Web scraping basics for JavaScript devs\n\n**Learn how to use JavaScript to extract information from websites in this practical course, starting from the absolute basics.**\n\nIn this course we'll use JavaScript to create an application for watching prices. It'll be able to scrape all product pages of an e-commerce website and record prices. Data from several runs of such program would be useful for seeing trends in price changes, detecting discounts, etc.\n\n- Inspect pages using browser DevTools.\n- Download web pages using the Fetch API.\n- Extract data from web pages using the Cheerio library.\n- Save extracted data in various formats (e.g. CSV which MS Excel or Google Sheets can open) using the json2csv library.\n- Follow links programmatically (crawling).\n- Save time and effort with frameworks, such as Crawlee, and scraping platforms, such as Apify.\n\nAnyone with basic knowledge of developing programs in JavaScript who wants to start with web scraping can take this course. The course does not expect you to have any prior knowledge of other web technologies or scraping.\n\n- A macOS, Linux, or Windows machine with a web browser and Node.js installed.\n- Familiarity with JavaScript basics: variables, conditions, loops, functions, strings, arrays, objects, files, classes, promises, imports, and exceptions.\n- Comfort with building a Node.js package and installing dependencies with `npm`.\n- Familiarity with running commands in Terminal (macOS/Linux) or Command Prompt (Windows).\n\nLet's explore the key reasons to take this course. What is web scraping good for, and what career opportunities does it enable for you?\n\nThe internet is full of useful data, but most of it isn't offered in a structured way that's easy to process programmatically. That's why you need scraping, a set of approaches to download websites and extract data from them.\n\nScraper development is also a fun and challenging way to learn web development, web technologies, and understand the internet. You'll reverse-engineer websites, understand how they work internally, discover what technologies they use, and learn how they communicate with servers. You'll also master your chosen programming language and core programming concepts. Understanding web scraping gives you a head start in learning web technologies such as HTML, CSS, JavaScript, frontend frameworks (like React or Next.js), HTTP, REST APIs, GraphQL APIs, and more.\n\nScrapers are programs specifically designed to mine data from the internet. Point-and-click or no-code scraping solutions do exist, but they only take you so far. While simple to use, they lack the flexibility and optimization needed to handle advanced cases. Only custom-built scrapers can tackle more difficult challenges. And unlike ready-made solutions, they can be fine-tuned to perform tasks more efficiently, at a lower cost, or with greater precision.\n\nAs a scraper developer, you are not limited by whether certain data is available programmatically through an official API - the entire web becomes your API! Here are some things you can do if you understand scraping:\n\n- Improve your productivity by building personal tools, such as your own real estate or rare sneakers watchdog.\n- Companies can hire you to build custom scrapers mining data important for their business.\n- Become an invaluable asset to data journalism, data science, or nonprofit teams working to make the world a better place.\n- You can publish your scrapers on platforms like the [Apify Store](https://apify.com/store) and earn money by letting others use them.\n\nWe are [Apify](https://apify.com), a web scraping and automation platform. We do our best to build this course on top of open source technologies. That means what you learn applies to any scraping project, and you'll be able to run your scrapers on any computer. We will show you how a scraping platform can simplify your life, but that lesson is optional and designed to fit within our [free tier](https://apify.com/pricing).\n\n[📄️ DevTools: InspectingLesson about using the browser tools for developers to inspect and manipulate the structure of a website.](https://docs.apify.com/academy/scraping-basics-javascript/devtools-inspecting)\n\n[📄️ DevTools: Locating HTML elementsLesson about using the browser tools for developers to manually find products on an e-commerce website.](https://docs.apify.com/academy/scraping-basics-javascript/devtools-locating-elements)\n\n[📄️ DevTools: Extracting dataLesson about using the browser tools for developers to manually extract product data from an e-commerce website.](https://docs.apify.com/academy/scraping-basics-javascript/devtools-extracting-data)\n\n[📄️ Downloading HTMLLesson about building a Node.js application for watching prices. Using the Fetch API to download HTML code of a product listing page.](https://docs.apify.com/academy/scraping-basics-javascript/downloading-html)\n\n[📄️ Parsing HTMLLesson about building a Node.js application for watching prices. Using the Cheerio library to parse HTML code of a product listing page.](https://docs.apify.com/academy/scraping-basics-javascript/parsing-html)\n\n[📄️ Locating HTML elementsLesson about building a Node.js application for watching prices. Using the Cheerio library to locate products on the product listing page.](https://docs.apify.com/academy/scraping-basics-javascript/locating-elements)\n\n[📄️ Extracting data from HTMLLesson about building a Node.js application for watching prices. Using string manipulation to extract and clean data scraped from the product listing page.](https://docs.apify.com/academy/scraping-basics-javascript/extracting-data)\n\n[📄️ Saving dataLesson about building a Node.js application for watching prices. Using the json2csv library to save data scraped from product listing pages in both JSON and CSV.](https://docs.apify.com/academy/scraping-basics-javascript/saving-data)\n\n[📄️ Getting links from HTMLLesson about building a Node.js application for watching prices. Using the Cheerio library to locate links to individual product pages.](https://docs.apify.com/academy/scraping-basics-javascript/getting-links)\n\n[📄️ Crawling websitesLesson about building a Node.js application for watching prices. Using the Fetch API to follow links to individual product pages.](https://docs.apify.com/academy/scraping-basics-javascript/crawling)\n\n[📄️ Scraping product variantsLesson about building a Node.js application for watching prices. Using browser DevTools to figure out how to extract product variants and exporting them as separate items.](https://docs.apify.com/academy/scraping-basics-javascript/scraping-variants)\n\n[📄️ Using a frameworkLesson about building a Node.js application for watching prices. Using the Crawlee framework to simplify creating a scraper.](https://docs.apify.com/academy/scraping-basics-javascript/framework)\n\n[📄️ Using a platformLesson about building a Node.js application for watching prices. Using the Apify platform to deploy a scraper.](https://docs.apify.com/academy/scraping-basics-javascript/platform)","wordCount": 944}
How to use it
Start a run in Apify Console, by API or from an agent with an input like this one — it is the input Apify runs every day to test this Actor:
{"urls": ["https://docs.apify.com/academy/scraping-basics-javascript"]}
Start with the minimal JSON example above, then adjust the fields in the table below. Read the Example result and Output fields sections for the returned record, Errors for failure codes, and Events and prices for result events and the separate platform start.
| Field | Type | Meaning |
|---|---|---|
urls | array | Public article pages to read; one record each. |
includeMarkdown | boolean | Also return the article as Markdown with headings, lists and links. |
Events and prices
| Event | Title | Charged when | FREE / BRONZE | SILVER | GOLD / PLATINUM / DIAMOND |
|---|---|---|---|---|---|
article-extracted | Article extracted | One page fetched and its main text returned. | $0.001 | $0.0009 | $0.0008 |
Platform usage is not passed through. Apify's automatic actor-start event remains $0.00005 per started GB (one GB or less counts once). The synthetic dataset-item event is absent. Charge limits stop further paying work; unprocessed inputs receive uncharged errors. Completed readable pending events are settled on abort.
Worked cost example: ten successful article extracted events cost $0.01 at list, plus actor start. Failures are not included in that count.
Call it from an agent
Through Apify's MCP server (https://mcp.apify.com) use call-actor:
{"actor": "hermeskd/article-extractor","input": {"urls": ["https://docs.apify.com/academy/scraping-basics-javascript"]},"callOptions": {"maxTotalChargeUsd": 0.01,"timeout": 600},"waitSecs": 30}
Then call get-dataset-items with the returned defaultDatasetId. Dedicated tool URL: https://mcp.apify.com?tools=hermeskd/article-extractor
Output fields
| Field | Meaning |
|---|---|
url | The input URL. |
ok | True for a returned result; a failure is never charged. |
finalUrl | The URL after redirects. |
title | The article's headline. |
publishedAt | The publish date the page declares, ISO 8601. |
language | The page's declared or detected language. |
siteName | The publishing site's name. |
leadImage | The article's lead image URL. |
text | The main text, boilerplate removed. |
markdown | The main text as Markdown, when includeMarkdown is on. |
wordCount | Words in the main text. |
error | Failure, always uncharged. |
Limits and run time
1–100 URLs a run. Each page is fetched once, up to 5 MB, under the 30-second per-request deadline; only HTML pages are read, and a page with under 20 words of main text is no_article_text. Bylines and author names are not returned. How it differs from URL to Markdown: that Actor converts the whole page, sections, tables and navigation text included; this one keeps only the article and adds its publish date, site name and lead image — use URL to Markdown for a full page, this for news, blog and documentation articles.
The default run timeout is 600 seconds. Typical run time: not yet measured; written from the first deploy.
Errors
| Code | Meaning |
|---|---|
http_error | The source answered an unsuccessful HTTP status. |
download_failed | The request or connection failed within its bounded deadline. |
unsupported_content_type | The URL did not return an HTML page. |
no_article_text | No main text was found on the page. |
robots_disallowed | robots.txt forbids fetching the URL; no request was made to it. |
blocked_host | The host is on the deny-list or resolves to a private or local address; no request was made. |
too_large | The response exceeded the size cap. |
timeout | The input did not finish within the run's time budget. |
invalid_input | The run input passed the input schema but cannot be used; the message names the field. |
internal_error | Unexpected failure; the message carries the error class. |
charge_limit_reached | The run reached its maximum total charge; this input was not processed. |
No failure record is charged. A run fails only when all input outcomes are Actor-side failures; caller errors and a mixture of caller/Actor failures do not fail the run.
Sources and terms
Only process content you have the right to process. URL fetches obey HermesBot robots-group selection, the shared deny-list, public-address checks on connections and redirects, honest identity, deadlines and rate limits. No caller credentials, cookies, CAPTCHA solving, residential proxies or evasion. Public pages the caller names are fetched under robots.txt; text extraction by trafilatura (Apache-2.0). Process only content you have the right to process.
Not affiliated
This Actor is not affiliated with Apify or the owners of the pages it reads. Third-party names describe compatibility and source material.
Related Hermes Actors
- AI Crawler Checker: Which AI Bots robots.txt Allows per URL — check declared robots.txt permissions for six AI crawlers at URL paths.
- ATS Job Search — Jobs Confirmed Open on Company Career Boards — search open jobs in the Hermes directory of Greenhouse and Lever boards, each confirmed live.
- Speech to Text — Transcribe Audio & Video URLs, Speaker Labels — speech to text with SRT and VTT subtitles.
- Video Converter — MP4 to MP3, Trim, GIF and Audio Extraction — audio and video conversion: MP4 to MP3, trim, GIF.
- Broken Links Checker — Links, Assets, Redirects and Their Pages — broken links, 404s and redirects on a site.
- ATS Detector — Company Hiring System & Job Board Lookup — which hiring system and job board a company uses.
- SEC EDGAR Filings — Recent Company Forms, Dates and XBRL Flags — Read recent company financial filing metadata from the official SEC JSON API
- DOI Lookup — Crossref Titles, Venues, Dates and Citation Counts — crossref doi metadata.
- CVE Vulnerability Lookup — NVD CVSS Scores and Exploited Flags — look up NVD CVE dates, status, numeric CVSS scores and severity.
- Merge JSON Datasets — Deduplicate Rows, Export JSON and CSV — json dataset cleaner.
- DNS Lookup — Records, SPF Lookup Counts, DKIM & DMARC by Domain — DNS records and SPF, DKIM and DMARC checks.
- DOCX to Markdown — Word, Excel, PowerPoint and EPUB Converter — Word, Excel, PowerPoint and other office files to Markdown.
- Domain Availability Checker — Bulk RDAP Status and Registrar — whether domain names are available, from registry RDAP.
- USGS Earthquake Lookup — Event Magnitude, Depth, Coordinates — Read numeric USGS earthquake metadata for known event identifiers.
- Currency Exchange Rates — ECB Daily Euro Rates and Cross Pairs — Read latest ECB daily currency reference rates and calculated cross pairs.
- Bulk File Downloader — Save URL Lists to Storage With SHA-256 — files and images from URLs, saved with a public link each.
- GitHub Repo Stats — Stars, Forks, Subscribers and Open Issues — Read numeric metrics for known public GitHub repositories.
- Greenhouse Jobs — Live Openings, Offices and Apply Links — every open job on a company's Greenhouse board.
- Hacker News Story Metrics — Scores, Comment Counts, Timestamps — read numeric Hacker News story scores, timestamps and comment counts by ID.
- HTTP GET Request — Bulk Response Bodies, Status and SHA-256 — public http get.
- Security Headers Checker — HSTS, CSP, Framing, Nosniff by URL — Check seven declared HTTP security header families without returning raw values.
- Image Background Remover — U2NetP Cutouts for JPEG, PNG, WebP — Remove image backgrounds with the small U2NetP model and return transparent PNG pixels as base64
- Image Converter — JPEG, PNG and WebP With Resize and Quality — image format converter.
- Image Metadata Remover — EXIF and GPS Scrub, Pixels Verified — GPS location, camera serial numbers and other private metadata removed from photos, losslessly.
- Image to Text OCR — Text With Word Positions and Confidence — OCR: the text in images.
- Image Upscaler — Real-ESRGAN 2x/4x for Photos and Graphics — image upscaling 2x or 4x, and photo restoration.
- Lever Jobs Scraper — Job Board Postings With Salary, US & EU — every published job on a company's Lever site, global or EU.
- Lighthouse Audit — Lab Web Vitals and Scores, Mobile & Desktop — Lighthouse scores and Core Web Vitals.
- npm Package Lookup — Latest Version, License, Dependency Count — read latest npm package versions, license tokens and dependency counts.
- Website Change Monitor — Page Text Diffs Against Your Snapshots — what changed on a page since the last check.
- PDF to Markdown — PDF Parser & Text Extractor With OCR — PDF to Markdown and text, with OCR for scanned pages.
- Public Holidays Calendar — Holiday Dates for Five Countries — build offline public-holiday calendars for five countries and bounded years.
- Remote Jobs Search — Remote OK Jobs by Keyword, Tag and Salary — current remote jobs from Remote OK, with salary and listing links.
- Research Paper Search — OpenAlex Papers, DOIs, Citation Counts — Search OpenAlex for work-level paper metadata without authors, affiliations, abstracts or full text
- RSS Feed Reader — Atom and JSON Feed Parser With Enclosures — every item of an RSS, Atom or JSON feed.
- SEO Audit — On-Page Score, Meta Tags, Schema & Fix Guidance — an on-page SEO audit: meta tags, schema, headings and links.
- Sitemap Extractor — Unique URLs From XML, Gzip & Index Sitemaps — the URLs in a site's sitemaps.
- Text to Speech — Offline English, German, Spanish WAV Voices — Convert short nonpersonal texts to offline formant WAV speech for agents
- Time Zone Converter API — Convert one offset-qualified instant to named IANA time zones using pinned data.
- URL to Markdown — Clean Text for LLM & RAG, With Selectors — a public job page as clean Markdown where the site allows it (deny-listed job sites are refused).
- Extract Video Frames — Timed Stills by Interval or Scene Cut — frames, thumbnails and scenes from a video.
- Website Color Palette — CSS Colors With Alpha and Usage Counts — Extract declared website CSS color literals and counts from public HTML and bounded linked stylesheets
- Favicon Extractor — Icons, Touch Icons, Manifests, Theme Color — declared website icons, theme colors and manifest links.
- Website Screenshot — Full Pages, Elements, Dark Mode & PDF — screenshots and PDF prints of web pages.
- Tech Stack Detector — Website Technologies With Match Evidence — the technologies a website runs, detected live.
- Website Crawler to Markdown — Same-Site Pages for LLM and RAG — bounded whole-site Markdown with robots and crawl-delay enforced
Changelog
- 1.0 (2026-10-27) — first release.