Website Content Crawler

Website content crawler for RAG and LLM apps. Give it a website, a section of one or a list of pages, and it saves each page's own text as clean Markdown: menus, footers and cookie banners that repeat on most pages are taken out and kept once, and a page that has the same text as another address is saved once. Turn on chunks and every page comes split into pieces that know the headings above them, ready for embeddings and a vector database.

Found a bug, need a field, or want a new Actor? Open an issue on the Issues tab. If it works well for you, a short review helps others find it.

• Price: $3.00 per 1,000 saved pages on the Free plan, $2.10 on Business, platform usage included. You pay only for pages saved with their text: pages that refuse the crawler, fail, copy another page or have no text are free, and so are chunks and notices.
• HTML only: no JavaScript rendering, proxy or PDF. A website that needs them gets a free notice row.
• Max pages per website: up to 5,000 in one run. It also caps what one website can cost.
• Input: a website (example.com), a section (example.com/docs/) or single pages. An input written for apify/website-content-crawler works as it is.
• Output: one row per page, or one row per chunk, with text in Markdown, title, description, language and the headings above each chunk.
• Measured: 197 pages from 4 websites in 51 seconds on Apify. Of 100 websites in two tests, 82 gave their text, 16 refused the crawler and 2 need JavaScript.

How to use it

1. Put the address in Websites to crawl: example.com for a whole site, example.com/docs/ for one section, or several pages, one per line.
2. Choose Crawl scope: pages under the start address, the whole website, or only the addresses you gave.
3. Set Max pages per website. It also caps what one website can cost.
4. For a vector database, turn on Split text into chunks and set Chunk size.
5. Click Start, then open Pages or Chunks in the Output tab.

{
    "startUrls": ["https://docs.python.org/3/library/"],
    "maxPages": 60,
    "splitIntoChunks": true,
    "chunkSize": 1000,
    "chunkOverlap": 100
}

The crawl follows links nearest first, then reads the pages the website's sitemap lists that no link led to. It keeps to the scope you chose, so example.com/docs/ never leaves /docs/, and example.com/en/5.2 stays in that one version of the docs.

What it takes out and what it keeps

Part of the page	What happens
Menus, footers, cookie banners	A text block found on at least half of the pages of a website, and on at least 3, is taken out of every page and saved once, in a row of type shared
Bars under the page title	"Edit this page" or breadcrumbs that most pages have right under their title go to the shared row too
Several languages	Each language is counted on its own and gets its own shared row
Navigation elements	<nav> is left out by default. Remove HTML elements takes any CSS selector
The same page at two addresses	Saved once, under the shorter address. Copies are listed in the run summary
Tables	Markdown tables. A price list without a header row gets an empty one
Code	Fenced code blocks with every line and space as the browser shows them, and the language when the page names it. Line numbers, copy buttons and prompts you cannot select are left out
FAQ accordions	A question on a button becomes a heading over its answer, so a chunk keeps the two together
Contact links	A "Call us" link gets its number in brackets, a "Write to us" link its address
Images	Their alt text stays, file names do not
Embedded maps and booking forms	A line such as Map: https://... or Booking form: https://...
Scripts, styles, forms, other buttons	Left out

What data you get

Field	What it holds
type	page, chunk, shared (the text repeated across the website) or notice (why something could not be read)
url, title, description, language	The address after redirects, the page title, meta description and the lang of the page
text, chars	The page's own text as Markdown, and its length in characters
chunkIndex, chunkCount, headings	For chunks: the place of the chunk in its page, how many the page has, and the headings above it
depth, foundBy	Links from a start address, and whether the page came from a start address, a link or the sitemap
site, crawledAt, error	The website, the time, and on a notice row what happened

A real chunk row, the first of 22 from one Python documentation page:

{
    "type": "chunk",
    "url": "https://docs.python.org/3/library/",
    "title": "The Python standard library — Python 3.15.2 documentation",
    "text": "# The Python standard library\n\nThis library reference manual describes the standard library distributed with Python. It also describes some of the optional components that are commonly included in Python distributions.\n\nElsewhere, The Python Language Reference describes the exact syntax and semantics of the Python language, and Python built-ins reference describes the built-in functions.",
    "chars": 390,
    "chunkIndex": 0,
    "chunkCount": 22,
    "headings": ["The Python standard library"],
    "language": "en",
    "depth": 0,
    "foundBy": "start",
    "site": "https://docs.python.org"
}

A long table is cut between rows with its header repeated, a long code block between lines with its fences repeated. With Chunk overlap, a chunk starts with the last words, rows or code lines of the one before it.

A real page row with chunks off:

{
    "type": "page",
    "url": "https://www.usa.gov/phone",
    "title": "Call USAGov | USAGov",
    "text": "# Call USAGov\n\nTalk with a USAGov agent for information on federal benefits, services, programs, and more. They will answer your question or direct you to an agency that can. Or call to report an issue with or content update for the USA.gov website.\n\nFor your protection, do not share personal information like telephone or Social Security numbers.\n\nCalls may be monitored for quality assurance.\n\n## Hours of operation\n\nAgents are available Monday - Friday, 8:00 AM - 8:00 PM Eastern Time (except federal holidays).\n\n## Call USAGov\n\nTo speak to a live agent, call toll-free from the U.S. or Canada 1-844-USA-GOV1 (1-844-872-4681).\n\nLAST UPDATED: May 22, 2025",
    "chars": 658,
    "chunkIndex": null,
    "headings": null,
    "language": "en",
    "depth": 0,
    "foundBy": "start",
    "site": "https://www.usa.gov"
}

The SUMMARY record in the key-value store lists, for each website, the pages saved, the duplicates skipped, the pages that failed and why, and how many found pages the scope, the filters, robots.txt or the page limit left out.

How much does it cost?

You pay for each page saved with its text. Chunks, shared rows and notices cost nothing, and neither do pages that fail, copies of another page or pages with no text of their own.

Plan	Per 1,000 pages
Free	$3.00
Starter	$2.70
Scale	$2.40
Business	$2.10

A run also pays $0.00005 when it starts. Example: a website of 500 pages costs $1.50 on the Free plan, the same with chunks on or off. Max pages per website and the run's maximum cost both stop the crawl before it goes over.

How to build a chatbot knowledge base from a website

1. Start at the home page with Crawl scope set to the whole website, and turn on chunks.
2. Keep Remove menus, footers and banners on. The shared row then holds the footer once: contacts, addresses and opening hours stay findable without being repeated in every chunk.
3. Embed text. Keep url, title and headings as metadata, so an answer can name the page and the section it came from.

A website in two languages gives two shared rows. Filter rows by language to build one assistant per language.

How to crawl only the docs

Start at the docs section, for example https://kubernetes.io/docs/concepts/, and leave Crawl scope at pages under the start address. Use Skip pages whose address contains for parts you do not want, such as /blog/ or ?page=. To read only the main text of each page, put its element in Keep only HTML elements, for example main or article.

An input written for apify/website-content-crawler

Paste it as it is. startUrls with { "url": ... }, includeUrlGlobs (they decide the scope instead of the start path), excludeUrlGlobs, maxCrawlDepth, keepElementsCssSelector, removeElementsCssSelector, useSitemaps, respectRobotsTxtFile, customHttpHeaders and initialCookies work the same way; maxCrawlPages caps each website and maxResults the whole run. Headers go only to the hosts of the start addresses and their subdomains, cookies only where their domain matches. Settings for a proxy, a browser or files get a free notice row, and the rest of the input is used.

Limits

• It reads HTML and runs no JavaScript. A website that builds its text in the browser gets a notice, and its empty pages are not saved or charged. A part that a script fills in later, such as the ticket prices on one museum page, is missing without a notice.
• It uses no proxy. Websites with bot protection refuse it: 16 of the 100 websites in our tests did, 9 of them online shops. You get a notice row that names the protection when it can tell. A website that lets the first page through and then refuses 20 pages in a row is left, and what was read is saved.
• PDF, Word and other files are skipped. Pages behind a login are read only with the cookies or headers you give through the API.
• One website is read one request at a time, at least 0.25 seconds apart, slower when the site is slow or asks for a Crawl-delay. 60 pages of one website took between 20 and 49 seconds in our runs, 400 pages 7 to 8 minutes, and 5,000 pages of one documentation website 54 minutes. Several websites are read at the same time.
• The rows of a website are saved when its crawl ends, because what repeats across its pages is known only then. Until then the status line counts the pages read.
• Up to 5,000 pages per website in one run. Their text stays in memory until the crawl of the website ends: 5,000 documentation pages with chunks took 557 MB of the default 1 GB, and a website of very long pages needs a run with more memory.
• A run stops reading before its timeout, 12 minutes before the default 4 hours, and saves what it has. robots.txt is followed by default.
• When up to 3 pages of a website are read, or up to 3 in one language, they keep their menus and footers. A repeated line right under a section heading stays in its page.

API and integrations

Run it from the Apify API, the Apify clients for JavaScript and Python, schedules, webhooks, Make, Zapier or any MCP client. The dataset exports to JSON, CSV and Excel, and the chunks view gives just the fields an embedding step needs.

FAQ

Do I pay per chunk? No. You pay per page. A page split into 30 chunks costs the same as one row.

Do code blocks and FAQ accordions come through? Yes. On the 20 pages of the example input, 178 of 178 code blocks matched what a browser shows, line by line, without line numbers. On 5 pricing pages, 77 of 77 FAQ questions stayed with their answers.

A website gave 0 pages. Why? Look at its notice row: it says whether the site refused the crawler, robots.txt does not allow it, the address did not open, or the text needs JavaScript. You are not charged for those pages.

The dataset is empty while the run is going. Is it stuck? No. Each website's rows arrive together when its crawl ends. The status line counts the pages read so far.

Can I keep menus and footers in every page? Yes. Turn off Remove menus, footers and banners and clear Remove HTML elements.

Does it obey robots.txt? Yes, its rules and its Crawl-delay up to 10 seconds. Turn Respect robots.txt off only for a website you own or may crawl.

What chunk size should I use? The default is 1,500 characters, about 375 tokens of English text. Sizes from 400 to 20,000 characters are allowed.

This Actor is an independent tool. It has no connection to the websites it reads, and their content belongs to their owners.