Web Scraper for LLMs: Clean Markdown and HTML From Any URL

Web Scraper for LLMs turns any web page into clean markdown or HTML that is ready for a language model, a RAG pipeline, or a search index. Give it a list of URLs to scrape them directly, or turn on crawl mode and give it a starting page to follow its links and pull in everything underneath it. Every page comes back as one dataset record with its content, title and basic page details (language, favicon and canonical URL), ready to download as JSON, CSV or Excel, or to read straight through the API.

Why This Scraper?

• Two ways to collect pages. Scrape a fixed list of URLs, or turn on crawl mode and let the actor follow links from a seed page to discover and read the rest of a site.
• Markdown and HTML output. Choose one format or both, per run, so the content slots straight into a prompt, a vector store, or a content pipeline.
• Automatic content cleaning. Navigation, headers, footers, sidebars, ads and tracking elements are stripped by default, so what you get is the page's actual content, not the surrounding layout.
• Handles slow and script heavy pages. A page is read with a quick request first, and the actor automatically falls back to full page rendering when a site needs JavaScript to show its content or blocks a simple request.
• Page details with every record. Title, language, favicon and canonical URL come back alongside the content, no extra parsing required.
• Fine grained control when you need it. CSS selectors to keep or drop specific elements, regex patterns to include or exclude URLs during a crawl, and a selector to wait for before reading a page.
• Respects robots.txt automatically. Disallowed URLs are skipped or refused rather than fetched, and a site's own crawl delay is honored during a crawl.

Use Cases

• RAG pipelines and AI knowledge bases: turn a list of pages or a whole site into clean markdown chunks ready for embedding.
• Documentation and content migration: crawl a docs site or blog and export every page as markdown in one run.
• Research and monitoring: pull articles, product pages or listings on a recurring schedule and compare the results over time.
• AI agent tooling: give an agent a single action that reads any URL and returns clean text instead of raw HTML.

Data You Get

Sample shape: values are illustrative placeholders, not from a live record.

Field	Example
url	"https://example.com/blog/how-rag-pipelines-work"
title	"How RAG Pipelines Work"
description	"A short guide to building RAG pipelines." (the page meta description, null when the page has none)
text	the plain readable text of the page, without markdown syntax
markdown	the page content as clean markdown (when markdown is a requested format)
html	the page content as cleaned HTML (when html is a requested format)
success	true
error	null (pages that could not be read are not written here; see the run summary)
duration	4210 (milliseconds spent reading this page)
scrapedAt	"2026-02-24T03:36:28.990Z"
metadata.title	"How RAG Pipelines Work"
metadata.language	"en"
metadata.favicon	"https://example.com/favicon.ico"
metadata.canonical	"https://example.com/blog/how-rag-pipelines-work"
metadata.description	the meta description, null when the page has none
metadata.author, metadata.keywords	the author and keywords meta tags, null when absent
metadata.languageCode	"en"
metadata.canonicalUrl	"https://example.com/blog/how-rag-pipelines-work"
metadata.openGraph, metadata.twitter	the Open Graph and Twitter card tags, null when the page has none
metadata.jsonLd	the structured data (JSON-LD) blocks found on the page, [] when none
metadata.headers	the HTTP response headers of the page
crawl.loadedUrl, crawl.loadedTime	the final URL after redirects and when it was loaded
crawl.referrerUrl, crawl.depth	the page that linked here and the link hops from the seed (0 for listed URLs)
crawl.httpStatusCode, crawl.contentType, crawl.contentLengthBytes	status code, content type and size of the response

How to Use

1. Add one or more URLs. Use a single seed URL if you plan to crawl from it. If you leave the URLs field out entirely, the actor scrapes four example pages, one of each shape: a bare homepage, a homepage with a query string, a static documentation page and a large wiki article. Each page is a normal result row.
2. Turn on Crawl Mode to follow links from each URL and scrape the pages it discovers, and set Crawl Depth and Max Pages to Discover to control how far it goes.
3. Choose output formats and content cleaning options, then adjust Concurrency, Max Retries and Timeout to fit the run.
4. Start the run, then download the dataset as JSON, CSV or Excel, or read it through the API.

Scrape a short list of URLs:

{
  "urls": ["https://example.com", "https://example.com/about"]
}

Scrape one page with both formats and custom content cleaning:

{
  "urls": ["https://example.com/blog/post-1"],
  "formats": ["markdown", "html"],
  "onlyMainContent": true,
  "removeAds": true,
  "excludeTags": [".newsletter-signup", ".comments"]
}

Crawl a documentation site:

{
  "urls": ["https://docs.example.com"],
  "crawl": true,
  "crawlDepth": 2,
  "crawlMaxPages": 50,
  "timeoutMs": 120000,
  "includePatterns": ["/docs/"]
}

Scrape several pages with a proxy and tuned concurrency:

{
  "urls": ["https://example.com/product/1", "https://example.com/product/2"],
  "concurrency": 5,
  "timeoutMs": 45000,
  "maxRetries": 3,
  "proxyConfiguration": { "useApifyProxy": true }
}

Run it from your code

Python:

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("abotapi/web-scraper-for-llms").call(run_input={"urls": ["https://example.com"]})
for page in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(page["url"], page["success"])
    print(page["markdown"])

JavaScript:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_APIFY_TOKEN>' });
const run = await client.actor('abotapi/web-scraper-for-llms').call({ urls: ['https://example.com'] });
const { items } = await client.dataset(run.defaultDatasetId).listItems();

Or connect it to Make, Zapier, n8n, Google Sheets or webhooks from the Integrations tab.

URL Filters and Discovery

All settings below are optional and off or neutral by default, so existing inputs behave as before.

• Include/Exclude URL globs: glob versions of the regex patterns, for example https://example.com/blog/**. They apply to crawl-mode links and to sitemap or llms.txt URLs.
• Use sitemaps / Use llms.txt: also scrape the pages listed in the site's sitemap.xml or llms.txt. Those two files are fetched directly rather than through the proxy.
• Respect robots.txt: on by default; turn it off only when you are allowed to.
• Max results: stop after a total number of saved pages. Failed URLs do not count toward it. In Crawl Mode each seed may fetch up to Max Pages to Discover pages and stops saving at Max Results, so the cap counts saved rows only.
• Only pages modified since: skip pages whose Last-Modified header or sitemap date is older than the given date. Pages without that information are kept. In Crawl Mode, pages skipped by this filter or by incremental mode still count toward the per-seed Max Pages to Discover limit, but not toward Max Results.
• Fetch engine: auto (HTTP first, browser fallback), HTTP only, or browser only. Crawl-mode link discovery always uses the browser.
• Custom HTTP headers / Initial cookies: sent with each page request; they are not applied during crawl-mode link discovery.
• Keep URL fragments: keep #fragment on discovered links.

Failed URLs are not written to the dataset and are not billed. They are listed with their error message under failedUrls in the run summary (the OUTPUT record of the key-value store), together with per-URL success and failure counts.

Send results into your apps (MCP connectors)

Optionally pipe a condensed summary of each scraped page into the apps you already use, via Model Context Protocol (MCP) connectors. This is an extra delivery step after the scrape: the Apify dataset is never changed, and failed pages are not exported.

1. Authorize a connector once under Apify > Settings > Integrations (Notion, Linear, Airtable, or Apify).
2. Select it in the Pipe results into your apps field (mcpConnectors). If the picker is empty, you have not authorized a connector yet.
3. For Notion, also set notionParentPageUrl to the page where items should be created.
4. maxNotifyListings caps how many pages are written to each connector per run (default 50). It does not affect the dataset.

The connection is mediated by Apify's MCP proxy, so this actor never sees your third-party credentials. Leave the field empty to skip.

Input Parameters

Parameter	Type	Default	Description
urls	array	sample	URLs to scrape. When Crawl Mode is on, each URL is used as a seed to discover linked pages. Required.
crawl	boolean	false	Follow links from each URL and scrape the pages it discovers, instead of only the URLs you listed.
crawlDepth	integer	1	How many link hops from the seed URL to follow. Ignored when Crawl Mode is off.
crawlMaxPages	integer	20	Maximum pages to discover per seed URL. Applies separately to each URL in the list. Ignored when Crawl Mode is off.
formats	array	["markdown"]	Content formats to return: markdown, html, or both.
concurrency	integer	3	Number of URLs processed in parallel.
maxRetries	integer	2	Retry attempts for a URL that fails to load. Ignored when Crawl Mode is on.
timeoutMs	integer	30000	Timeout per URL, in milliseconds. In Crawl Mode it instead caps the total link discovery time for each seed URL, so raise it for larger crawls.
onlyMainContent	boolean	true	Keep only the main content and strip navigation, headers, footers and sidebars. Ignored when Crawl Mode is on.
removeAds	boolean	true	Remove ad and tracking elements from the page.
removeBase64Images	boolean	true	Strip base64 encoded images to keep output size down.
includeTags	array	(none)	CSS selectors for elements to keep, everything else is dropped. Ignored when Crawl Mode is on.
excludeTags	array	(none)	CSS selectors for elements to remove from the output. Ignored when Crawl Mode is on.
includePatterns	array	(none)	Regex patterns for URL filtering. Meaningful only in Crawl Mode, where it controls which discovered links are followed. Has no effect on URLs you list directly.
excludePatterns	array	(none)	Regex patterns for URL filtering. Meaningful only in Crawl Mode, where it controls which discovered links are skipped. Has no effect on URLs you list directly.
waitForSelector	string	(none)	CSS selector to wait for before reading a page, useful for content that loads after the initial page render. Applies when the page is fully rendered. Ignored when Crawl Mode is on.
incrementalMode	boolean	false	Only write new or changed pages. Each page's content fingerprint is compared with the previous run; unchanged pages are skipped, not written to the dataset and not billed as results. Failed pages are always listed in the run summary.
ignoreFieldsForChanges	array	["duration","scrapedAt","crawl.loadedTime","metadata.headers"]	Incremental mode only. Output fields left out of the change fingerprint (they stay in the output). Dot paths reach nested keys.
mcpConnectors	array	empty	MCP connectors that receive a condensed summary of each page.
notionParentPageUrl	string	empty	Notion parent page (Notion connector only).
maxNotifyListings	integer	50	Max pages exported to each connector per run.
crawlerType	string	auto	Fetch engine: auto (HTTP first, browser fallback), http or browser.
maxResults	integer	1000	Stop after this many saved pages (0 means 1,000,000, the maximum). Failed URLs do not count.
customHttpHeaders	object	(none)	Extra HTTP headers sent with each page request.
initialCookies	array	(none)	Cookies sent with each page request.
includeUrlGlobs	array	(none)	Glob patterns a URL must match (crawl links, sitemap and llms.txt URLs).
excludeUrlGlobs	array	(none)	Glob patterns that exclude a URL.
useSitemaps	boolean	false	Also scrape pages listed in the site's sitemap.xml.
useLlmsTxt	boolean	false	Also scrape pages listed in the site's llms.txt.
respectRobotsTxtFile	boolean	true	Skip URLs disallowed by robots.txt.
keepUrlFragments	boolean	false	Keep #fragment on discovered links.
onlyCrawlPagesModifiedSince	string	(none)	Skip pages older than this date; pages without date information are kept.
proxyConfiguration	object	(none)	Apify proxy settings. Pages are fetched with a direct connection unless this is turned on.

Output Example

Sample shape: values are illustrative placeholders, not from a live record.

{
  "url": "https://example.com/blog/how-rag-pipelines-work",
  "title": "How RAG Pipelines Work",
  "description": null,
  "markdown": "# How RAG Pipelines Work\n\nRetrieval augmented generation combines a search step with a language model, so answers can draw on content the model was never trained on.",
  "html": "<article><h1>How RAG Pipelines Work</h1><p>Retrieval augmented generation combines a search step with a language model.</p></article>",
  "metadata": {
    "title": "How RAG Pipelines Work",
    "language": "en",
    "favicon": "https://example.com/favicon.ico",
    "canonical": "https://example.com/blog/how-rag-pipelines-work",
    "description": null,
    "openGraph": null,
    "twitter": null
  },
  "duration": 4210,
  "scrapedAt": "2026-02-24T03:36:28.990Z",
  "success": true,
  "error": null
}

Plan Requirement

This actor runs on standard Apify compute. Pages are fetched with a direct connection unless you turn on Proxy Configuration, which uses Apify Proxy and so needs a plan that includes it. This is a general purpose reader with no single target site, and the test pages (simple public sites) loaded without a proxy. Datacenter and residential proxy lanes were not measured, so the proxy group to use is your choice: if a site blocks direct requests or leans heavily on JavaScript, try a proxy and compare results. Memory scales automatically with your URL count, crawl settings and concurrency.

FAQ

How much does it cost?

Billing is pay per event: one actor start event per run, plus one dataset item event for every row written to the default dataset. The Pricing tab on the actor page shows the current rate for each event. URLs that could not be read, in either mode, are not written to the dataset and are not billed; they are listed under failedUrls in the run summary. Pages skipped as unchanged in incremental mode and pages older than the modified-since date also write no row and are not billed. Platform usage such as proxy traffic is not part of these events. Lower Max Results, Max Pages to Discover, or Crawl Depth to cap the number of rows in any run.

Is it legal to scrape any URL with this?

This actor is a general purpose reader, it does not choose or endorse the pages you point it at. It respects robots.txt rules and skips or refuses URLs a site has blocked. You are responsible for the URLs you provide: follow each site's terms of use and the laws that apply to you, and get legal advice before using scraped content commercially or on pages that contain personal data.

Can I schedule it to run automatically, or only fetch new content?

You can schedule this actor from the Schedules tab like any other Apify actor. By default every run scrapes the URLs you give it from scratch. Turn on Only emit new or changed pages (incrementalMode) for scheduled runs: the actor keeps a content fingerprint per page in a named key-value store and skips pages whose content is unchanged, so they are neither written to the dataset nor billed as results. The fields in ignoreFieldsForChanges (default duration, scrapedAt, crawl.loadedTime and metadata.headers) are not compared. The fingerprint is kept per output format, so changing formats re-emits pages once. Crawl Mode follows links breadth first, so there is no page number to resume from: each run walks the seeds again, and incremental mode only filters what is written.

Why did a crawl return far more pages than Max Pages to Discover?

Max Pages to Discover is a limit per seed URL, not a total for the run. If you list five seed URLs with Crawl Mode on, the actor can return up to five times that number, one batch of discovered pages per seed. Reduce the number of seed URLs, or lower Max Pages to Discover, to control the total.

Why did my run fail instead of returning results?

With Crawl Mode off, the run ends as failed when the input has no URLs or when no URL at all could be loaded (every URL is listed under failedUrls in the run summary). If a specific page cannot be read, blocked, timed out, taken down, or refused by robots.txt, it is listed under failedUrls with an error message, no dataset row is written for it, and the run keeps going. In Crawl Mode, pages that cannot be opened or are disallowed by robots.txt during link discovery are skipped and not listed. A crawl whose pages take more than about five minutes to read can stop the run, but rows already written are kept. Lower Max Pages to Discover or split large crawls into several runs.

Can I use this with AI agents or MCP?

Yes. It is built to hand language models and RAG pipelines clean markdown instead of raw HTML, and like any Apify actor it can be called from an AI agent or an MCP client through Apify's integrations.

🔗 Want more AI data?

Pair this actor with these related scrapers from the same team:

🤖 AI Agent Web Fetcher An advanced web fetcher that can fetch almost all websites and convert them to...	🏨 Doc To Markdown MCP Server An MCP server that converts documents to clean Markdown. Convert PDFs, Word docs, Excel...
🤖 AI Search Tool Give your AI agents real-world knowledge. This Actor provides high-quality web, news...	🤖 Doc To Markdown Convert documents (PDF, Word, PowerPoint, Excel, HTML, images) to clean Markdown...
🧩 Yandex Search Scraper Scrape Yandex web, image and video search results with URLs, snippets and organic...	⚽ JioHotstar Scraper Scrape the JioHotstar catalog by content type or URL. Extract shows, movies, episodes...

👉 Browse all abotapi scrapers

💬 Support

• 🐞 Found a bug or a missing field? Open a ticket on the Issues tab. We usually reply within hours.
• 🛠️ Need another site, extra fields or a private build? Email contact@abotapi.com.
• ⭐ Enjoying it? A quick review on the actor page helps other users find it.