Webpage Text Extractor

Extract clean text content from web pages. Strips HTML and returns structured text with headings, links, metadata, and word count.

What does Webpage Text Extractor do?

This actor fetches web pages and extracts their clean text content by stripping all HTML tags, scripts, and styles. It identifies the main content area (article, main, etc.), extracts headings structure, page links, and metadata like author, publish date, and language. Use it for LLM input preparation, content analysis, text mining, or feeding clean text into downstream data pipelines.

Who is it for?

• AI/LLM engineers -- convert web pages to clean text for RAG pipelines, fine-tuning datasets, or prompt context
• Content analysts -- extract text for sentiment analysis, topic modeling, keyword extraction, or NLP processing
• Data journalists -- collect article text from multiple news sources for comparison and analysis
• Accessibility auditors -- extract text structure and heading hierarchy to verify correct semantic markup
• Data pipeline builders -- feed clean, structured text into downstream processing tools and databases

Why use Webpage Text Extractor?

• AI-ready clean text and optional Markdown -- keeps the existing mainText output and can add Markdown that preserves headings, paragraphs, lists, links, and blockquotes for LLM, RAG, and publishing workflows
• Rich metadata -- extracts title, meta description, author, publish date, language, and Open Graph tags
• Heading structure -- returns all headings with their level (H1-H6) for document outline analysis
• Export-ready links -- resolves HTTP(S) links against the final page URL and identifies external hostnames
• Response provenance -- reports the submitted URL, final URL after redirects, HTTP status, and content type
• Honest batch errors -- returns structured, uncharged error rows for malformed URLs and failed HTTP pages without stopping valid URLs
• Configurable metadata -- toggle metadata inclusion with the includeMetadata option to control output size
• Pay-per-event pricing -- uses plan-tier pricing per successfully extracted URL with no monthly subscription

Input parameters

Parameter	Type	Required	Default	Description
urls	string[]	Yes	--	List of web page URLs to extract text from
includeMetadata	boolean	No	true	Include links and extra metadata in the output
includeMarkdown	boolean	No	false	Add a markdown string while retaining the existing mainText field

Example input

{
    "urls": [
        "https://en.wikipedia.org/wiki/Web_scraping",
        "https://blog.apify.com"
    ],
    "includeMetadata": true,
    "includeMarkdown": true
}

Output example

{
    "url": "https://en.wikipedia.org/wiki/Web_scraping",
    "finalUrl": "https://en.wikipedia.org/wiki/Web_scraping",
    "statusCode": 200,
    "contentType": "text/html; charset=UTF-8",
    "title": "Web scraping - Wikipedia",
    "metaDescription": "...",
    "author": null,
    "publishedDate": null,
    "language": "en",
    "mainText": "Web scraping is the process of...",
    "markdown": "# Web scraping\n\nWeb scraping is the process of...",
    "headings": [
        { "level": 1, "text": "Web scraping" },
        { "level": 2, "text": "Techniques" }
    ],
    "links": [
        { "text": "data extraction", "href": "https://en.wikipedia.org/wiki/Data_extraction", "isExternal": false }
    ],
    "wordCount": 3450,
    "charCount": 21000,
    "error": null,
    "extractedAt": "2026-03-01T12:00:00.000Z"
}

Output fields

Field	Type	Description
url	string	The submitted URL (retained even after redirects)
finalUrl	string | null	Final response URL after redirects, when known
statusCode	number | null	HTTP response status, when a response was received
contentType	string | null	Response Content-Type header, when available
title	string	The page title
metaDescription	string	The meta description tag content
author	string	Author name if detected from meta tags
publishedDate	string	Publish date if detected from meta tags
language	string	Page language from the lang attribute
mainText	string	Clean text content with HTML stripped; always retained regardless of Markdown settings
markdown	string	Optional Markdown preserving headings, paragraphs, lists, links, and blockquotes when includeMarkdown is true
headings	array	List of headings with level (1-6) and text
links	array	Absolute HTTP(S) links with text and an isExternal hostname flag; unsupported schemes are skipped
wordCount	number	Total words in the extracted text
charCount	number	Total characters in the extracted text
error	string	Error message if extraction failed, null otherwise
extractedAt	string	ISO timestamp of the extraction

How to extract text from web pages

1. Open Webpage Text Extractor on Apify.
2. Enter one or more web page URLs in the urls field.
3. Choose whether to include metadata with includeMetadata and whether to add Markdown with includeMarkdown.
4. Click Start and wait for the run to finish.
5. Download the extracted text as JSON, CSV, or Excel from the Dataset tab.

How much does it cost to extract text from web pages?

Webpage Text Extractor uses Apify's pay-per-event pricing model. You only pay for what you use.

Event	Price	Description
Start	$0.035	One-time per run
URL extracted	Plan-tier price	Charged only after a page is successfully extracted

The per-URL price is $0.00115 on Free, $0.001 on Bronze, and decreases on higher plans (to $0.00028 on Diamond). Failed HTTP pages and invalid URL rows do not incur the url-extracted event. The $0.035 start charge is unchanged.

Example Bronze costs:

• 10 successful pages: $0.035 + 10 x $0.001 = $0.045
• 100 successful pages: $0.035 + 100 x $0.001 = $0.135
• 1,000 successful pages: $0.035 + 1,000 x $0.001 = $1.035

API usage

You can start Webpage Text Extractor programmatically from your own applications using the Apify API. The following examples show how to run the actor and retrieve results in both Node.js and Python.

Node.js

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_TOKEN' });
const run = await client.actor('automation-lab/webpage-text-extractor').call({
    urls: ['https://en.wikipedia.org/wiki/Web_scraping'],
    includeMetadata: true,
    includeMarkdown: true,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python

from apify_client import ApifyClient

client = ApifyClient('YOUR_TOKEN')
run = client.actor('automation-lab/webpage-text-extractor').call(run_input={
    'urls': ['https://en.wikipedia.org/wiki/Web_scraping'],
    'includeMetadata': True,
    'includeMarkdown': True,
})
items = client.dataset(run['defaultDatasetId']).list_items().items
print(items)

cURL

curl -X POST "https://api.apify.com/v2/acts/automation-lab~webpage-text-extractor/runs?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://en.wikipedia.org/wiki/Web_scraping"],
    "includeMetadata": true,
    "includeMarkdown": true
  }'

Use with Claude AI (MCP)

This actor is available as a tool in Claude AI through the Model Context Protocol (MCP). Add it to Claude Desktop, Cursor, Windsurf, or any MCP-compatible client.

Setup for Claude Code

$claude mcp add --transport http apify "https://mcp.apify.com?tools=automation-lab/webpage-text-extractor"

Setup for Claude Desktop, Cursor, or VS Code

Add this to your MCP config file:

{
    "mcpServers": {
        "apify": {
            "url": "https://mcp.apify.com?tools=automation-lab/webpage-text-extractor"
        }
    }
}

Example prompts

• "Extract the main text content from this article: https://example.com/blog/post"
• "Get clean text from these web pages and summarize them"
• "How many words are on this page and what is the heading structure?"

Learn more in the Apify MCP documentation.

Integrations

Webpage Text Extractor works with all major automation platforms available on Apify. Export results to Google Sheets to build a text content database for analysis. Use Zapier or Make to trigger text extraction whenever new URLs are added to a watchlist. Send extracted text to Slack channels for quick review. Pipe results into n8n workflows to feed clean text into LLM APIs, vector databases, or NLP pipelines. Set up webhooks to get notified when extraction finishes and automatically pass text to downstream processing.

Tips and best practices

• Set includeMetadata to false if you only need the main text -- this reduces output size significantly, especially for pages with hundreds of links
• Enable includeMarkdown for LLM, RAG, or publishing workflows when heading, list, link, and blockquote structure is useful; it adds markdown without replacing mainText
• Use finalUrl, statusCode, and contentType for provenance checks before sending extracted content downstream; failed HTTP pages are returned as empty, structured error rows
• Use the headings array to understand document structure before feeding text into LLMs -- heading hierarchy provides valuable context for summarization and Q&A
• Filter by language when processing multilingual sites to route text to the correct NLP model or translation pipeline
• Combine with Content Readability Checker to get both the raw text and readability scores for each page
• Chain with Sitemap URL Extractor to first get all URLs from a sitemap, then extract clean text from every page for a complete content export

Data handling, retention, and AI

The Actor requests only the public URLs you provide and writes extracted public page content and response provenance to your Apify dataset. It does not store response cookies or credentials, create an external cache, or send content to an AI model. Runs, datasets, key-value-store records, and logs follow your Apify account retention settings and can be deleted with Apify storage controls.

Legality

This tool analyzes publicly accessible web content. Automated analysis of public web resources is standard practice in SEO and web development. Always respect robots.txt directives and rate limits when analyzing third-party websites. For personal data processing, ensure compliance with applicable privacy regulations.

Support

For product questions or reproducible extraction problems, use the Issues tab on the Actor's Apify Store page and include a sanitized URL, input options, and run ID. Do not include credentials or private page content.

FAQ

Does the actor render JavaScript? No. The actor uses plain HTTP requests and extracts text from the initial HTML response. Pages that load content dynamically via JavaScript after page load may return incomplete text. Enabling includeMarkdown does not render JavaScript, crawl linked pages, or create RAG chunks.

Does Markdown replace mainText? No. includeMarkdown is opt-in and defaults to false. When enabled, it adds a markdown string to successful results while preserving the existing mainText field and all other output fields.

What is the mainText field? It contains the clean text content extracted from the page's main content area, with all HTML tags, scripts, styles, and navigation elements stripped out. This is the primary output field for most use cases.

What happens when a URL is invalid or returns an HTTP error? The actor keeps processing the rest of the batch and returns one structured error row with empty text and counts. When a server responded, finalUrl, statusCode, and contentType remain available. These rows are not charged as successful URL extractions.

Can I extract text from PDF or Word documents? No. The actor is designed for HTML web pages. contentType exposes mislabeled or non-HTML responses so downstream workflows can detect them; use a dedicated file-processing tool for documents.

The extracted text includes navigation menu and footer text. How do I get only the article content? The actor tries to detect the main content area using semantic HTML elements (<article>, <main>). If the website does not use these elements, it falls back to cleaned <body> text after removing common non-content elements. In that fallback, mainText can include navigation or footer text when the page lacks semantic content markup.

The actor returns very little or no text for a page that has content. Why? The page likely loads its content via client-side JavaScript (React, Angular, Vue, etc.). The actor uses plain HTTP requests and parses the initial HTML response without executing JavaScript. For JavaScript-heavy sites, you may need a browser-based scraping solution.

Related tools

• Word Counter -- count words, sentences, and paragraphs on any web page
• Website Language Detector -- detect the language of web pages from HTML attributes
• Website Performance Checker -- measure TTFB, page size, and compression
• Website Carbon Calculator -- estimate the carbon footprint of any web page
• Website Health Report -- comprehensive website health audit with scoring