🕸️ RAG Web Crawler: Markdown & Text Chunks

Turn public web pages into clean Markdown, plain text, and ready-to-embed text chunks for your retrieval-augmented generation (RAG) pipeline. Give it a list of URLs or a search query, optionally let it follow links within each site, and get one Dataset item per page. Every chunk keeps its source URL, page title, and exact character offsets, so your application can cite the page an answer came from.

The crawler respects robots.txt and stops at the page, request, and output limits you set, so you know the most a run can do before it starts.

✨ Main features

• 📝 Clean Markdown and plain text with headings, lists, links, code blocks, and basic tables.
• ✂️ Ready-made RAG chunks that overlap and end at paragraph or sentence breaks where possible. Each chunk has an id that stays the same across runs while the page and your chunk settings don't change, plus its index, startChar/endChar offsets, sourceUrl, and title.
• 🧹 Main content first: menus, headers, footers, sidebars, and similar navigation are left out, and you can remove more with CSS selectors.
• 🧭 Link following within a site up to the depth you choose. Duplicate links are fetched once: tracking parameters and in-page anchors are ignored, and redirects to an already saved page are skipped.
• 🔎 Search to find pages: give a query instead of URLs, and each result page is crawled like a start URL. Needs a paid Apify plan.
• 🤖 robots.txt respected by default, with polite pacing between requests to the same site.
• 🌍 English, Chinese, and other languages, kept in their original language without translation.
• 📄 HTML, Markdown, and plain-text pages, including Markdown documents with front-matter titles and descriptions.
• 📊 Transparent results: every page gets an outcome with a status and an error code, and every run gets a summary with counts and a stop reason.
• 🛡️ Bounded runs with limits on pages, network requests, response size, text length, and total output.

🚀 Quick start

1. In the Input tab, replace the example URLs in startUrls with the public pages you want to crawl, or add a search query to find pages first.
2. Keep maxDepth at 0 to fetch only those pages, or raise it to follow links within each site.
3. Start the run. 1 GB of memory and the default concurrency of 3 work well.
4. Open the Output tab to browse pages and chunks. Export the Dataset as JSON to keep the nested chunks.

Example input

{
  "startUrls": [
    { "url": "https://www.w3.org/WAI/tutorials/tables/one-header/" },
    { "url": "https://www.mozilla.org/zh-CN/about/manifesto/" }
  ],
  "maxPages": 2,
  "maxDepth": 0,
  "maxSearchRequests": 0
}

Example output

Each saved page is one Dataset item. This one comes from a real run, with the text shortened and one of its five chunks shown:

{
  "schemaVersion": "1.0",
  "url": "https://www.w3.org/WAI/tutorials/tables/one-header/",
  "title": "Tables with One Header",
  "description": "Accessibility resources free online from the international standards organization: W3C Web Accessibility Initiative (WAI).",
  "languageCode": "en",
  "canonicalUrl": "https://www.w3.org/WAI/tutorials/tables/one-header/",
  "markdown": "# Tables with One Header\n\nin [Tables Tutorial](https://www.w3.org/WAI/tutorials/tables/)\n\n## Overview\n\n…",
  "text": "Tables with One Header\n\nin Tables Tutorial\n\nOverview\n\nThis page covers tables that have one simple header for rows or one simple header for columns. …",
  "chunks": [
    {
      "id": "495cb4e58e13bbf9b5cdbe922f7ea652973ca4dfda61b952aaa021db045a450c",
      "index": 0,
      "text": "Tables with One Header\n\nin Tables Tutorial\n\nOverview\n\nThis page covers tables that have one simple header for rows or one simple header for columns. …",
      "startChar": 0,
      "endChar": 988,
      "sourceUrl": "https://www.w3.org/WAI/tutorials/tables/one-header/",
      "title": "Tables with One Header"
    }
  ],
  "contentTruncated": false,
  "sources": [],
  "crawl": {
    "depth": 0,
    "httpStatusCode": 200,
    "loadedAt": "2026-10-19T07:03:53.643Z",
    "contentType": "text/html",
    "renderingMode": "http",
    "extractionMethod": "main"
  }
}

🎛️ Input

Give startUrls, a search query or queries, or both. Everything else has a default.

What to crawl

Input	Default	Description
startUrls	None	Public page URLs, as { "url": "…" } objects or plain strings. Up to 1,000.
maxDepth	0	How many links deep to follow within each start URL's site, meaning the same scheme, host, and port. 0 fetches only the start URLs. Range: 0–5.
maxPages	20	Maximum distinct pages scheduled, including pages that fail or that robots.txt excludes. Range: 1–1,000.
respectRobotsTxt	true	Check each site's robots.txt before fetching its pages.

Content and chunking

Input	Default	Description
onlyMainContent	true	Extract the page's main content. When false, the whole page body is used, still without scripts and navigation.
removeSelectors	[]	Up to 30 CSS selectors, each at most 300 characters, for elements to remove before extraction.
chunkSize	1000	Chunk length in Unicode code points, not tokens. Range: 200–10,000.
chunkOverlap	150	Code points shared by consecutive chunks. Must be smaller than chunkSize.
maxContentChars	50000	Maximum text and Markdown length per page in code points. Truncation sets contentTruncated. Range: 1,000–100,000.
renderingMode	"http"	Keep http. JavaScript rendering is not available in the hosted release.

Workload limits

Input	Default	Description
maxConcurrency	3	Pages processed at the same time. Range: 1–10.
maxFetchRequests	100	All network attempts in the run, including robots.txt, redirects, and retries. Range: 0–10,000.
requestTimeoutSecs	30	Timeout for each request, not for the whole run. Range: 5–120 seconds.
maxResponseBytes	2097152	Largest page response accepted: 2 MiB by default. Range: 1 KiB–10 MiB.
maxOutputBytes	20971520	Total page-record and search-result JSON per run: 20 MiB by default. Each page record is also limited to 1 MiB; a larger page is not saved, and the run continues. Range: 1–100 MiB.

These settings limit the work a run does. They are not a dollar budget: to cap charges, set a maximum cost per run in the run options.

Search

Search finds pages for you: each result is crawled like a start URL, and its query and rank are kept in the page's sources. Search needs a paid Apify plan; on the free plan, use startUrls.

Input	Default	Description
query	None	One search query.
queries	None	More search queries, merged with query. Duplicate queries are searched once.
maxResultsPerQuery	5	Search results to crawl for each query. Fewer may come back. Range: 1–50.
maxSearchRequests	10	Search requests per run, including retries. 0 turns search off. Range: 0–1,000.

Queries support these operators:

Find or filter	Query example
Search a specific site	asyncio site:docs.python.org
Exclude a site	RAG -site:reddit.com
Match an exact phrase	"retrieval augmented generation"
Exclude a keyword	RAG tutorial -advertisement
Exclude an exact phrase	RAG -"release notes"
Search multiple sites	RAG site:github.com OR site:huggingface.co
Restrict result languages	RAG lang:zh,en
Find pages published after a date	RAG after:2026-01-01
Find pages published before a date	RAG before:2026-10-01

Each query must be non-blank and at most 128 Unicode code points, including operators and spaces. site: narrows search results; it does not limit which links the crawler follows. Some result pages need JavaScript to show their content and fail with NO_CONTENT; the search is still charged when it returned results.

📦 Output

Dataset: one item per page

Field	Description
url	The page's final public URL, after redirects.
title, description, languageCode, canonicalUrl	Page metadata. A missing title is an empty string; the other three are null when missing. canonicalUrl is the canonical link the page declares on its own site.
markdown	The extracted content as Markdown.
text	The plain-text version. Chunks are cut from this field.
chunks[]	id, index, text, startChar (inclusive), endChar (exclusive), sourceUrl, and title. Offsets count Unicode code points in text.
contentTruncated	true when the text or Markdown reached maxContentChars.
sources[]	Search discovery details. Empty when no search led to the page.
crawl	depth, httpStatusCode, loadedAt (fetch time, not publication time), contentType, renderingMode, and extractionMethod.

Run records

The Output tab also links to three key-value store records:

• REQUESTS: one entry per page and search, with status (ok, empty, error, or skipped), code, message, depth, and attempts.
• OUTPUT: the run summary, with status (complete, partial, failed, or stopped), stopReason, counts of pages, chunks, requests, errors, and skips, and the events charged (chargedEvents, chargedUsd).
• SEARCH_RESULTS: candidate URLs for each search query. Empty for URL-only runs.

This is the OUTPUT of a real 10-page run, made before event pricing was enabled, so nothing was charged:

{
  "schemaVersion": "1.0",
  "status": "complete",
  "stopReason": "completed",
  "pagesSaved": 10,
  "chunksSaved": 108,
  "pageRequests": 18,
  "searchRequests": 0,
  "outputBytes": 397937,
  "errorCount": 0,
  "skippedCount": 0,
  "chargedEvents": { "apify-actor-start": 0, "search-success": 0, "fetch-success": 0 },
  "chargedUsd": 0,
  "billingUnknown": false,
  "startedAt": "2026-10-19T07:03:51.258Z",
  "finishedAt": "2026-10-19T07:03:55.634Z"
}

A run can succeed while some pages were skipped or failed, so review REQUESTS and OUTPUT alongside the Dataset.

🔌 Use the chunks in your RAG pipeline

The chunks are ready to embed, with no further splitting needed. Use id as the record ID in your vector store and keep sourceUrl, title, startChar, and endChar as metadata for citations.

Run the Actor and read the chunks with the Apify Python client:

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("autofacts/rag-web-crawler").call(run_input={
    "startUrls": [{"url": "https://www.w3.org/WAI/tutorials/tables/"}],
    "maxDepth": 1,
    "maxPages": 20,
})

for page in client.dataset(run["defaultDatasetId"]).iterate_items():
    for chunk in page["chunks"]:
        # Embed chunk["text"] and store it with its citation metadata.
        print(chunk["id"], chunk["sourceUrl"], chunk["startChar"], chunk["endChar"])

To load them as LangChain documents:

from langchain_core.documents import Document

documents = [
    Document(
        page_content=chunk["text"],
        metadata={"source": chunk["sourceUrl"], "title": chunk["title"], "chunk_id": chunk["id"]},
    )
    for page in client.dataset(run["defaultDatasetId"]).iterate_items()
    for chunk in page["chunks"]
]

To export the results instead, download the Dataset as JSON to keep each page's nested chunks. JSON Lines also gives one page per line, not one chunk per line.

The Actor prepares source content. Embeddings, vector storage, retrieval, and answer generation happen in your own workflow.

💰 Pricing

This Actor uses pay-per-event pricing. You pay for these events:

Event	Price	When it is charged
Crawled page	$0.0015 per page ($1.50 per 1,000 pages)	Once for each page saved to the Dataset. Failed, skipped, and duplicate pages are free.
Actor start	$0.00005	When a run starts, once per GB of memory and at least once.
Search query	$0.003 per query	Once for each search query that returns results. Needs a paid Apify plan.

For example, a run with 1 GB of memory that saves 100 pages costs $0.15005.

• Set a maximum cost per run in the run options (maxTotalChargeUsd in the API) to cap what a run can charge. The start charge counts toward it, and the crawler stops before the next page would go over it.
• Start with a few pages and check the results before raising maxPages or maxDepth.
• OUTPUT.chargedEvents and OUTPUT.chargedUsd show what the run charged, including the start event.

🚧 Limitations

• Search needs a paid Apify plan. On the free plan, crawl start URLs instead.
• No JavaScript rendering. Pages that need JavaScript to show their content can fail with NO_CONTENT or come back incomplete.
• Public pages only. Logins, cookies, paywalls, CAPTCHA solving, and proxies are not supported. Private-network addresses and URLs that carry credentials or signatures are rejected.
• HTML, Markdown, and plain text only. PDF extraction and optical character recognition (OCR) are not included.
• Extraction isn't perfect. Related links or editorial notes can remain, useful content can be dropped, and complex tables can lose structure. Test a few of your target pages first.
• Same-site links only. Links are followed only within each start URL's site. Pages are deduplicated within a run, but a new run fetches them again.
• Secrets are removed. API keys and URLs that carry credentials are replaced with [REDACTED] or [REDACTED URL] in page text.

🛠️ Troubleshooting

What you see	What to check
NO_CONTENT	No readable body text was found. Check that the URL is a public content page whose text is available without JavaScript.
HTTP_403 or HTTP_429	The site denied or rate-limited the request. Lower maxConcurrency and try a small batch later; some sites stay inaccessible.
ROBOTS_DISALLOWED	The site's robots.txt excludes the page.
ROBOTS_UNAVAILABLE	The site's robots.txt returned an error, so its pages were not fetched. Try again later.
UNSAFE_URL	A start URL points to a private address, includes a username or password, or has a sensitive parameter such as token or key. The run stops before crawling; use the public page URL.
UNSUPPORTED_CONTENT	The URL returned something other than HTML, Markdown, or plain text, such as a PDF or an image.
RESPONSE_TOO_LARGE	The page response exceeded maxResponseBytes.
RECORD_TOO_LARGE	One page's record (text, Markdown, and chunks together) exceeded 1 MiB, so that page was not saved; the run continued. Lower maxContentChars or chunkOverlap.
OUTPUT_LIMIT	The run's combined output reached maxOutputBytes, so the remaining pages were skipped. Try fewer pages or a smaller maxContentChars.
FETCH_LIMIT or PAGE_LIMIT	A workload limit was reached. Review your limits and the pages already delivered.
DUPLICATE_PAGE	Nothing to fix: the URL led to a page this run had already delivered, usually through a redirect. Its content is in the Dataset under the final URL.
Fewer pages than expected	Check your input for duplicate URLs, then review REQUESTS and OUTPUT for exclusions, errors, and stop reasons.
Too much unrelated text	Keep onlyMainContent: true, and add CSS selectors for the unwanted elements to removeSelectors.

❓ FAQ

Do I need an API key?

No external API key is needed, for crawling or for search. You need an Apify account, and API access uses your Apify token.

Can I retry only the failed pages?

Yes. Take the failed URLs from REQUESTS and start a new run with them. Deduplication applies within a run, so a new run can fetch the same pages again and use more resources.

Can I automate runs?

Yes. Use the API tab to copy a request with your input, schedule the Actor, or call it from an integration.

Is it legal to crawl websites?

Crawling publicly available, non-personal data is generally legal, but website content is often protected by copyright, and some sites restrict crawling in their terms. The crawler follows robots.txt by default. You are responsible for how you use the content; if you are unsure, seek legal advice.

🔗 Related Actors

More Actors from Autofacts that turn web content into text and data for LLMs:

• YouTube Subtitle & Transcript Scraper: subtitles and transcripts from videos, Shorts, playlists, and channels as JSON, SRT, VTT, or LLM-ready text.
• TikTok Transcript & Subtitle Scraper: TikTok captions as timestamped JSON, SRT, VTT, plain text, or LLM-ready text.
• SEC EDGAR Scraper: filing metadata, section-parsed 10-K and 10-Q text for RAG, and XBRL financial facts.
• CourtListener Scraper: US court opinions with full text, dockets, and RAG-ready chunks.
• Schema Markup Scraper & SEO Auditor: JSON-LD, Microdata, RDFa, Open Graph, and Twitter Card data with a 0–100 SEO audit.
• Universal Web Printer: web pages and HTML rendered to PDF, PNG, JPEG, or WebP.

💬 Feedback

Found a problem or need a feature? Open an issue in the Actor's Issues tab.