Web Scraper for LLMs: Clean Markdown and HTML From Any URL

Web Scraper for LLMs turns any web page into clean markdown or HTML that is ready for a language model, a RAG pipeline, or a search index. Give it a list of URLs to scrape them directly, or turn on crawl mode and give it a starting page to follow its links and pull in everything underneath it. Every page comes back as one dataset record with its content, title and basic page details (language, favicon and canonical URL), ready to download as JSON, CSV or Excel, or to read straight through the API.

Why This Scraper?

• Two ways to collect pages. Scrape a fixed list of URLs, or turn on crawl mode and let the actor follow links from a seed page to discover and read the rest of a site.
• Markdown and HTML output. Choose one format or both, per run, so the content slots straight into a prompt, a vector store, or a content pipeline.
• Automatic content cleaning. Navigation, headers, footers, sidebars, ads and tracking elements are stripped by default, so what you get is the page's actual content, not the surrounding layout.
• Handles slow and script heavy pages. A page is read with a quick request first, and the actor automatically falls back to full page rendering when a site needs JavaScript to show its content or blocks a simple request.
• Page details with every record. Title, language, favicon and canonical URL come back alongside the content, no extra parsing required.
• Fine grained control when you need it. CSS selectors to keep or drop specific elements, regex patterns to include or exclude URLs during a crawl, and a selector to wait for before reading a page.
• Respects robots.txt automatically. Disallowed URLs are skipped or refused rather than fetched, and a site's own crawl delay is honored during a crawl.

Use Cases

• RAG pipelines and AI knowledge bases: turn a list of pages or a whole site into clean markdown chunks ready for embedding.
• Documentation and content migration: crawl a docs site or blog and export every page as markdown in one run.
• Research and monitoring: pull articles, product pages or listings on a recurring schedule and compare the results over time.
• AI agent tooling: give an agent a single action that reads any URL and returns clean text instead of raw HTML.

Data You Get

Sample shape: values are illustrative placeholders, not from a live record.

Field	Example
url	"https://example.com/blog/how-rag-pipelines-work"
title	"How RAG Pipelines Work"
description	null (not currently returned)
markdown	the page content as clean markdown (when markdown is a requested format)
html	the page content as cleaned HTML (when html is a requested format)
success	true
error	null (set to a message when the page could not be read)
duration	4210 (milliseconds spent reading this page)
scrapedAt	"2026-02-24T03:36:28.990Z"
metadata.title	"How RAG Pipelines Work"
metadata.language	"en"
metadata.favicon	"https://example.com/favicon.ico"
metadata.canonical	"https://example.com/blog/how-rag-pipelines-work"
metadata.description	null (not currently returned)
metadata.openGraph	null (not currently returned)
metadata.twitter	null (not currently returned)

How to Use

1. Add one or more URLs. Use a single seed URL if you plan to crawl from it.
2. Turn on Crawl Mode to follow links from each URL and scrape the pages it discovers, and set Crawl Depth and Max Pages to Discover to control how far it goes.
3. Choose output formats and content cleaning options, then adjust Concurrency, Max Retries and Timeout to fit the run.
4. Start the run, then download the dataset as JSON, CSV or Excel, or read it through the API.

Scrape a short list of URLs:

{
  "urls": ["https://example.com", "https://example.com/about"]
}

Scrape one page with both formats and custom content cleaning:

{
  "urls": ["https://example.com/blog/post-1"],
  "formats": ["markdown", "html"],
  "onlyMainContent": true,
  "removeAds": true,
  "excludeTags": [".newsletter-signup", ".comments"]
}

Crawl a documentation site:

{
  "urls": ["https://docs.example.com"],
  "crawl": true,
  "crawlDepth": 2,
  "crawlMaxPages": 50,
  "timeoutMs": 120000,
  "includePatterns": ["/docs/"]
}

Scrape several pages with a proxy and tuned concurrency:

{
  "urls": ["https://example.com/product/1", "https://example.com/product/2"],
  "concurrency": 5,
  "timeoutMs": 45000,
  "maxRetries": 3,
  "proxyConfiguration": { "useApifyProxy": true }
}

Run it from your code

Python:

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("abotapi/web-scraper-for-llms").call(run_input={"urls": ["https://example.com"]})
for page in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(page["url"], page["success"])
    print(page["markdown"])

JavaScript:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_APIFY_TOKEN>' });
const run = await client.actor('abotapi/web-scraper-for-llms').call({ urls: ['https://example.com'] });
const { items } = await client.dataset(run.defaultDatasetId).listItems();

Or connect it to Make, Zapier, n8n, Google Sheets or webhooks from the Integrations tab.

Input Parameters

Parameter	Type	Default	Description
urls	array	sample	URLs to scrape. When Crawl Mode is on, each URL is used as a seed to discover linked pages. Required.
crawl	boolean	false	Follow links from each URL and scrape the pages it discovers, instead of only the URLs you listed.
crawlDepth	integer	1	How many link hops from the seed URL to follow. Ignored when Crawl Mode is off.
crawlMaxPages	integer	20	Maximum pages to discover per seed URL. Applies separately to each URL in the list. Ignored when Crawl Mode is off.
formats	array	["markdown"]	Content formats to return: markdown, html, or both.
concurrency	integer	3	Number of URLs processed in parallel.
maxRetries	integer	2	Retry attempts for a URL that fails to load. Ignored when Crawl Mode is on.
timeoutMs	integer	30000	Timeout per URL, in milliseconds. In Crawl Mode it instead caps the total link discovery time for each seed URL, so raise it for larger crawls.
onlyMainContent	boolean	true	Keep only the main content and strip navigation, headers, footers and sidebars. Ignored when Crawl Mode is on.
removeAds	boolean	true	Remove ad and tracking elements from the page.
removeBase64Images	boolean	true	Strip base64 encoded images to keep output size down.
includeTags	array	(none)	CSS selectors for elements to keep, everything else is dropped. Ignored when Crawl Mode is on.
excludeTags	array	(none)	CSS selectors for elements to remove from the output. Ignored when Crawl Mode is on.
includePatterns	array	(none)	Regex patterns for URL filtering. Meaningful only in Crawl Mode, where it controls which discovered links are followed. Has no effect on URLs you list directly.
excludePatterns	array	(none)	Regex patterns for URL filtering. Meaningful only in Crawl Mode, where it controls which discovered links are skipped. Has no effect on URLs you list directly.
waitForSelector	string	(none)	CSS selector to wait for before reading a page, useful for content that loads after the initial page render. Applies when the page is fully rendered. Ignored when Crawl Mode is on.
proxyConfiguration	object	(none)	Apify proxy settings. Pages are fetched with a direct connection unless this is turned on.

Output Example

Sample shape: values are illustrative placeholders, not from a live record.

{
  "url": "https://example.com/blog/how-rag-pipelines-work",
  "title": "How RAG Pipelines Work",
  "description": null,
  "markdown": "# How RAG Pipelines Work\n\nRetrieval augmented generation combines a search step with a language model, so answers can draw on content the model was never trained on.",
  "html": "<article><h1>How RAG Pipelines Work</h1><p>Retrieval augmented generation combines a search step with a language model.</p></article>",
  "metadata": {
    "title": "How RAG Pipelines Work",
    "language": "en",
    "favicon": "https://example.com/favicon.ico",
    "canonical": "https://example.com/blog/how-rag-pipelines-work",
    "description": null,
    "openGraph": null,
    "twitter": null
  },
  "duration": 4210,
  "scrapedAt": "2026-02-24T03:36:28.990Z",
  "success": true,
  "error": null
}

Plan Requirement

This actor runs on standard Apify compute, no special plan tier is required. Pages are fetched with a direct connection unless you turn on Proxy Configuration. For sites that block direct requests or lean heavily on JavaScript, Apify Proxy with the residential group gives the most reliable results at scale. Memory scales automatically with your URL count, crawl settings and concurrency.

FAQ

How much does it cost?

You pay for the compute the run consumes: URLs scraped or pages crawled, full page rendering when a page needs it, and proxy bandwidth if you turn a proxy on. The Pricing tab on the actor page shows the current rate card. Lower Concurrency, Max Pages to Discover, or Crawl Depth to cap the cost of any run.

Is it legal to scrape any URL with this?

This actor is a general purpose reader, it does not choose or endorse the pages you point it at. It respects robots.txt rules and skips or refuses URLs a site has blocked. You are responsible for the URLs you provide: follow each site's terms of use and the laws that apply to you, and get legal advice before using scraped content commercially or on pages that contain personal data.

Can I schedule it to run automatically, or only fetch new content?

You can schedule this actor from the Schedules tab like any other Apify actor. It does not track what changed between runs on its own, every run scrapes the URLs you give it from scratch. If you only want new or changed content, keep the same seed URLs between runs and compare the resulting datasets, or crawl to a fixed depth each time and diff the pages yourself.

Why did a crawl return far more pages than Max Pages to Discover?

Max Pages to Discover is a limit per seed URL, not a total for the run. If you list five seed URLs with Crawl Mode on, the actor can return up to five times that number, one batch of discovered pages per seed. Reduce the number of seed URLs, or lower Max Pages to Discover, to control the total.

Why did my run fail instead of returning results?

With Crawl Mode off, the run fails outright only when the input has no URLs. If a specific page cannot be read, blocked, timed out, taken down, or refused by robots.txt, its row comes back with success set to false and an error message, and the run keeps going. In Crawl Mode, pages that cannot be opened or are disallowed by robots.txt during link discovery are skipped without a row. A crawl whose pages take more than about five minutes to read can stop the run, but rows already written are kept. Lower Max Pages to Discover or split large crawls into several runs.

Can I use this with AI agents or MCP?

Yes. It is built to hand language models and RAG pipelines clean markdown instead of raw HTML, and like any Apify actor it can be called from an AI agent or an MCP client through Apify's integrations.

🔗 Want more AI data?

Pair this actor with these related scrapers from the same team:

🤖 AI Agent Web Fetcher An advanced web fetcher that can fetch almost all websites and convert them to...	🧩 Doc To Markdown MCP Server An MCP server that converts documents to clean Markdown. Convert PDFs, Word docs, Excel...
🤖 Doc To Markdown Convert documents (PDF, Word, PowerPoint, Excel, HTML, images) to clean Markdown...	🤖 AI Search Tool Give your AI agents real-world knowledge. This Actor provides high-quality web, news...
🧩 Yandex SERP Scraper Scrape Yandex web, image and video results with URLs, snippets and organic rankings for...	📱 Clutch Scraper Scrape Clutch company directories and profiles. Get names, ratings, review counts, hourly...

👉 Browse all abotapi scrapers

💬 Support & custom scrapers

• 🐞 Found a bug or a missing field? Open a ticket on the Issues tab. We usually reply within hours.
• 🛠️ Need another site, extra fields or a private build? Email contact@abotapi.com or message Telegram @abotapi.
• ⭐ Enjoying it? A quick review on the actor page helps other users find it.