🌐 Website Content Crawler

Website Content Crawler is an Apify Actor that crawls a website from your start URLs and saves every page as clean text, Markdown and metadata, ready for AI, search and content projects. Set a depth and a page limit, then export one row per page as JSON, CSV or Excel. It costs $0.005 per saved page on the Free plan and $0.0035 on the Gold, Platinum and Diamond plans. 🚀 Try it free

What can Website Content Crawler do?

• 📝 Clean text and Markdown for every page. Use it as a website to markdown converter and a website text extractor in one run. Markdown and plain text sit side by side in each row.
• 🧹 Cleaner reading content. Navigation, footers and other page noise are removed before extraction. Keep only an article body, or remove extra elements, with CSS selectors.
• 🧭 Control where the crawl goes. Set the crawl depth, a page limit and a result limit. Use include and exclude URL patterns to crawl a documentation site without the rest of the domain.
• 📍 More ways to find pages. Seed the crawl from a site's sitemap or its llms.txt file, and choose whether to respect robots.txt.
• 💻 JavaScript websites. In Auto rendering mode, pages that need rendering are loaded in a browser. You can also render every page or none.
• 💾 Rows appear while the run is going. Each page is written to the dataset as soon as it is ready, so you can watch results build up.
• 📸 Snapshots and screenshots. Save HTML snapshots of each page and, if you turn it on, a PNG screenshot.
• 🔌 Apify platform tools. Export, schedule runs, call the Actor through the API and trigger webhooks.

What data can I extract from a website?

Each saved page becomes one dataset row.

Field	Description
🔗 url, loadedUrl	The page address (the canonical address unless you ignore it) and the address that was actually loaded.
🔢 orderNo	Position of the page in the order rows were saved.
🔖 title, description, canonicalUrl	Page title, meta description and canonical link.
👤 author, keywords, languageCode	Author, keywords and language when the page declares them.
📅 datePublished, dateModified	Publish and update dates when the page declares them.
📝 text, markdown, html	Clean text, a Markdown version and the cleaned readable HTML.
🧭 depth, referrerUrl, httpStatusCode, loadedTime	How far from the start URL the page is, which page linked to it, the response status and the load time.
📂 crawl	A grouped copy of loadedUrl, loadedTime, referrerUrl, depth and httpStatusCode.
🧾 metadata	Page metadata with Open Graph tags, structured data and response headers.
📄 htmlUrl, screenshotUrl	Links to the saved HTML snapshot and the screenshot, when saved.
📎 files	Links to documents found on the page, such as PDFs.
🔧 debug, debugRequestHandlerMode, debugTotalLinks, debugStatusCode	Crawl diagnostics for the page.

Fields the page does not provide are empty. The Actor reads pages as a visitor sees them and does not log in for you.

What is Website Content Crawler used for?

Use	How Website Content Crawler helps
🤖 RAG	Use it as a website crawler for RAG. Crawl a website for RAG and feed website content to a vector database.
🧠 LLM	A website crawler for LLM projects that collects website data for an LLM knowledge base.
🎓 AI training data	A crawler for AI training data that returns clean text from web pages, ready for vector database ingestion.
📝 Markdown	A website to markdown converter and web page to markdown tool.
🗂 Site export	A site to markdown crawler for whole documentation sites.
🔍 SEO	Crawl a website for SEO audit work. Extract page metadata and Open Graph tags.
🏷 Metadata	A website metadata extractor for titles, descriptions and canonical links.
📍 Sitemaps	A sitemap crawler: crawl a site using its sitemap.
🧾 llms.txt	An llms.txt crawler option to seed the crawl from the llms.txt file.
📄 Documents	Extract text from PDF links and save HTML snapshot of web pages.
💬 Chatbots	Build a chatbot from website content, or crawl a documentation site for a help assistant.
⚡ Fast	A fast website crawler when rendering is off.
🌳 Deep	A deep website crawler when you raise the depth.
🧰 General	A web crawler and web content crawler for pages, with no code needed.
🕸 Scraping	A web scraping crawler and website scraper in one tool.
📤 Extraction	A website content extractor for website content extraction at scale.
📊 Data	A website data extractor and page content extractor.
🔤 Text	A website text scraper and web page text extractor for plain text.
🧱 HTML	An HTML to text crawler that also keeps the cleaned HTML.
🛠 Tooling	A website crawling tool and website content scraper for teams.
📈 Export	Extract text from website pages to CSV, or scrape website content to Excel.
💻 Dynamic sites	Crawl JavaScript websites and use it as a website screenshot crawler.

What can you do with the data from a website?

🤖 Build a RAG or LLM knowledge base

1. Add your documentation address to startUrls and restrict it with includeUrlGlobs, for example /docs/**.
2. Keep saveMarkdown on and turn saveHtml off for smaller rows.
3. Export the Dataset as JSON and load it into your vector database as the first step of a RAG pipeline.

🔍 Run an SEO content audit

1. Switch on useSitemaps so pages that are not linked from the start page are found.
2. Open the Metadata view for titles, descriptions, canonical URLs and language codes.
3. Export as CSV or Excel and review gaps in a spreadsheet.

📍 Find all URLs on a website

1. Raise maxCrawlDepth and maxResults. This is how to scrape all pages of a website.
2. Open the Crawl Details view: it lists each page, its referrer, depth and HTTP status.
3. Turn on storeSkippedUrls to also get a list of discovered links that were not crawled, with a reason for each.

📰 Extract clean article text

1. Set keepElementsCssSelector to article, main.
2. Turn on redactPii if emails and phone numbers should be masked.
3. Schedule the run and compare the saved text between runs.

How do I use Website Content Crawler?

1. Open Website Content Crawler on Apify and click Try for free.
2. Add one or more addresses to Start URLs.
3. Set Max crawl depth and Max rows in your dataset. The default is 10 rows.
4. Add include or exclude URL patterns if you only need part of the site.
5. Choose a Rendering mode. Auto is recommended.
6. Click Start and follow the status message while pages are saved.
7. Open the Dataset tab, pick a view and export as JSON, CSV or Excel.

How much does it cost to scrape a website?

Website Content Crawler uses pay-per-event pricing: you pay for each saved page, and Apify platform usage is billed in addition. Failed pages, skipped links and files in the key-value store are not charged as pages.

Plan	Price per saved page	Per 1,000 pages
🆓 Free	$0.005	$5.00
🥉 Bronze	$0.0045	$4.50
🥈 Silver	$0.004	$4.00
🥇 Gold, Platinum and Diamond	$0.0035	$3.50

A start event of $0.00005 per GB of memory is charged when a run begins, with a minimum of one. The default memory is 4,096 MB (4 GB), so each run adds 4 start events, or $0.0002.

Example: 500 saved pages on the Silver plan cost 500 x $0.004 = $2.00, plus the $0.0002 start events and platform usage.

Is the website content crawler free?

Running it is not free of charge: every saved page is billed at your plan price, starting at $0.005 on the Free plan, plus the start events and platform usage. To keep a first test small, leave Max rows in your dataset at its default of 10 and raise it once the output looks right.

How much does Website Content Crawler cost per 1,000 results?

One thousand saved pages cost $5.00 on the Free plan, $4.50 on Bronze, $4.00 on Silver and $3.50 on Gold, Platinum and Diamond, plus the $0.0002 start events per run and platform usage. Set a maximum cost per run in Apify to cap spending.

Input

Only Start URLs is required. Everything else has a default, and the input form groups fields into sections. Patterns use glob syntax: * stops at /, ** crosses folders, and a pattern without http:// or https:// is matched against the path.

Start and scope

Field	Type	Required	Description	Default
🌱 startUrls	array of strings	Yes	Pages where the crawl begins. Start URLs are always crawled; filters apply to discovered links.	none
🌱 referrerUrl	string	No	Value saved as referrerUrl on rows from start URLs.	empty
🌱 maxCrawlDepth	integer	No	0 = start URLs only, 1 = their direct links, and so on (0 to 50).	2
🌱 maxCrawlPages	integer	No	Cap on unique pages opened, including empty or failed ones. 0 means no extra cap (up to 1,000,000).	0
🌱 maxResults	integer	No	Maximum rows saved (1 to 100,000).	10
🌱 includeUrlGlobs	array	No	Only discovered URLs matching a pattern are queued.	empty
🌱 excludeUrlGlobs	array	No	Discovered URLs matching a pattern are skipped.	empty
🌱 keepUrlFragments	boolean	No	Treat #anchors as separate pages.	false
🌱 ignoreCanonicalUrl	boolean	No	Use the loaded URL instead of the canonical URL as the row url.	false

Discovery sources

Field	Type	Required	Description	Default
📍 useSitemaps	boolean	No	Also queue pages listed in the site's sitemaps.	false
📍 useLlmsTxt	boolean	No	Also queue URLs listed in the site's llms.txt file.	false
📍 respectRobotsTxtFile	boolean	No	Skip discovered URLs that robots.txt disallows.	false
📍 storeSkippedUrls	boolean	No	Save discovered but not crawled URLs, with reasons, to skipped-urls.json.	false
📍 extractPdfText	boolean	No	Open linked PDFs and put their text in text. They count toward your limits.	false

Rendering and page loading

Field	Type	Required	Description	Default
💻 renderingMode	string	No	auto, always or never.	auto
💻 retryFailedWithRendering	boolean	No	Retry a page that failed to load, using rendering. Ignored when the mode is never.	true
💻 dynamicContentWaitSecs	integer	No	Extra wait on rendered pages (0 to 120 seconds).	3
💻 waitForSelector	string	No	CSS selector to wait for before extracting.	empty
💻 maxScrollHeightPixels	integer	No	Scroll depth on rendered pages for lazy-loaded content. 0 turns scrolling off.	0
💻 clickElementsCssSelector	string	No	Elements to click before extracting, such as expanders.	empty
💻 blockMedia, blockAds	boolean	No	Skip images, fonts and media, or known ad and tracker hosts, on rendered pages.	false
💻 mobileView	boolean	No	Render with a mobile viewport.	false
💻 maxConcurrency	integer	No	Pages loaded at the same time (1 to 50).	10
💻 rateLimitDelaySecs	string	No	Pause before each request, for example 0.1.	0.1
💻 maxRequestRetries	integer	No	Attempts per page (1 to 20).	3
💻 requestTimeoutSecs	integer	No	Page load timeout (5 to 300 seconds).	60
💻 ignoreHttpsErrors	boolean	No	Accept invalid certificates. Use only for development sites.	false

Text extraction and output

Field	Type	Required	Description	Default
🧼 keepElementsCssSelector	string	No	Keep only matching elements before extraction.	empty
🧼 removeElementsCssSelector	string	No	Comma-separated selectors to remove.	navigation, footer and similar elements
🧼 readableTextCharThreshold	integer	No	Pages with less cleaned text are treated as too thin and, in Auto mode, collected again more thoroughly.	100
🧼 saveMarkdown	boolean	No	Include Markdown in each row.	true
🧼 saveHtml	boolean	No	Include cleaned HTML in the html field.	true
🧼 redactPii	boolean	No	Mask emails, phone numbers, card-like numbers and SSNs in text and markdown. Best effort, not a compliance guarantee.	false
🧼 kvDir	string	No	Prefix for saved snapshot files. Empty skips writing files.	kv
🧼 saveScreenshots	boolean	No	Save a PNG for every row. Needs a prefix and a rendering mode other than never.	false
🧼 screenshotFullPage	boolean	No	Capture the whole page instead of the visible area.	true

Network, headers and diagnostics

Field	Type	Required	Description	Default
🌍 proxyConfiguration	object	No	Optional network settings for the crawl.	Set in the input form
🌍 headerAcceptLanguage, headerDnt, headerUpgradeInsecureRequests	string	No	Optional request headers.	en-US,en;q=0.9, 1, 1
🌍 headerUserAgent, headerSecChUa, headerSecChUaMobile, headerSecChUaPlatform	string	No	Optional identity headers. Keep the defaults unless a site needs a specific value.	desktop browser values
🌍 customHttpHeaders	object	No	Extra headers, such as cookies. They override the headers above.	empty
🌍 initialCookies	array	No	Cookies sent with requests and loaded into rendered pages.	empty
🌍 debugLog	boolean	No	Print extra detail in the run log.	true

Example input

{
  "startUrls": ["https://example.com/docs"],
  "maxCrawlDepth": 2,
  "maxResults": 50,
  "includeUrlGlobs": ["/docs/**"],
  "excludeUrlGlobs": ["**/*.pdf"],
  "useSitemaps": true,
  "renderingMode": "auto",
  "saveMarkdown": true,
  "saveHtml": false
}

Output

Results go to the default Apify Dataset, one row per saved page. Download them as JSON, CSV, Excel, XML or HTML from the Dataset tab or through the API. The dataset has five views: Quick View, Crawl Details, Metadata, Content and Storage and Debug.

{
  "orderNo": 1,
  "url": "https://example.com/docs/getting-started",
  "loadedUrl": "https://example.com/docs/getting-started",
  "loadedTime": "2026-10-17T10:15:30.123Z",
  "referrerUrl": "",
  "depth": 0,
  "httpStatusCode": 200,
  "title": "Getting started | Example Docs",
  "description": "Learn how to set up Example in five minutes.",
  "canonicalUrl": "https://example.com/docs/getting-started",
  "author": "Example Team",
  "keywords": "example, docs, setup",
  "datePublished": "2026-01-15T09:00:00Z",
  "dateModified": "2026-09-30T12:30:00Z",
  "languageCode": "en",
  "crawl": {
    "loadedUrl": "https://example.com/docs/getting-started",
    "loadedTime": "2026-10-17T10:15:30.123Z",
    "referrerUrl": "",
    "depth": 0,
    "httpStatusCode": 200
  },
  "metadata": {
    "canonicalUrl": "https://example.com/docs/getting-started",
    "title": "Getting started | Example Docs",
    "description": "Learn how to set up Example in five minutes.",
    "author": "Example Team",
    "keywords": "example, docs, setup",
    "datePublished": "2026-01-15T09:00:00Z",
    "dateModified": "2026-09-30T12:30:00Z",
    "languageCode": "en",
    "openGraph": [
      { "property": "og:title", "content": "Getting started" }
    ],
    "jsonLd": null,
    "headers": { "content-type": "text/html; charset=utf-8" }
  },
  "screenshotUrl": null,
  "text": "Getting started\n\nInstall Example and create your first project.",
  "html": "",
  "htmlUrl": "https://example.com/files/kv__abc123_readable.html",
  "markdown": "# Getting started\n\nInstall Example and create your first project.",
  "files": ["https://example.com/files/guide.pdf"],
  "debug": {
    "requestHandlerMode": "fast",
    "pageType": "Article",
    "pageTypeReason": "MeetsReadableTextThreshold",
    "totalLinks": 24,
    "pageLinks": 23,
    "fileLinks": 1,
    "response": { "statusCode": 200, "contentType": "text/html; charset=utf-8" },
    "snapshots": {
      "originalHtmlUrl": "https://example.com/files/kv__abc123_original.html",
      "removedElementsHtmlUrl": "https://example.com/files/kv__abc123_removed.html",
      "readableTextHtmlUrl": "https://example.com/files/kv__abc123_readable.html"
    },
    "lengths": { "originalHtml": 48210, "readableHtml": 5120, "readableText": 1840, "markdown": 1902 },
    "readableTextCharThreshold": 100
  },
  "debugRequestHandlerMode": "fast",
  "debugTotalLinks": 24,
  "debugStatusCode": 200
}

Snapshot files are saved to the run's key-value store when kvDir is set, and the htmlUrl field links to the readable snapshot. If storeSkippedUrls is on and links were skipped, skipped-urls.json lists each URL with a reason such as off-domain, not-included, excluded or robots-disallow.

Notes and limitations

• The default of 10 rows is an input default, not a hard limit. Raise maxResults up to 100,000.
• The crawl stays on the domains of your start URLs. A leading www. is ignored, so add a start URL for each subdomain you need.
• robots.txt is not applied unless you turn on respectRobotsTxtFile.
• Screenshots need rendering, so they are unavailable when renderingMode is never, and they make runs slower.
• PDFs larger than 20 MB are skipped, and only the first 300 pages of a PDF are read.
• A row larger than about 8 MB is trimmed, and debug.truncated is set to true.
• Pages that return not found, are not HTML or cannot be loaded are skipped and not saved.

FAQ

Is it legal to crawl a website?

Crawling public pages is generally allowed, but the rules depend on the site, the content and your country. Check the site's terms, respect copyright and personal data rules such as GDPR and CCPA, and read Apify's guide to web scraping legality. This is not legal advice.

Do I need an account, cookies or an API key?

You need an Apify account to run Website Content Crawler, but no account or API key for the sites you crawl, because it reads public pages. If a page needs a session, you can pass your own cookies in initialCookies or headers in customHttpHeaders. Login is not performed for you.

How many pages can I get per run?

Set maxResults to any value from 1 to 100,000; the default is 10. Use maxCrawlPages to cap how many unique pages are opened, and maxCrawlDepth to limit how far the crawl follows links. Speed depends on the site and on maxConcurrency, which accepts 1 to 50.

Can I export the crawled pages to CSV, Excel or JSON?

Yes. Open the Dataset tab, choose a view such as Metadata or Content, and export JSON, CSV, Excel, XML or HTML. You can also fetch the same data through the API. Turn off saveHtml or saveMarkdown if you want smaller files and only need plain text.

Does the crawler respect robots.txt?

Only if you ask it to. Turn on respectRobotsTxtFile and discovered URLs that the site's robots.txt disallows are skipped, while your own start URLs are still crawled. The option is off by default. You can also lower maxConcurrency and raise rateLimitDelaySecs to be gentler on small sites.

Can I schedule runs, use webhooks or integrations?

Yes. Apify lets you schedule Website Content Crawler runs, trigger webhooks when a run finishes and connect the dataset to tools such as Make, Zapier or n8n. A scheduled crawl is a simple way to track changes to documentation or product pages over time.

Can I use Website Content Crawler with the Apify API or MCP?

Yes. You can start runs, read datasets and manage schedules through the Apify API, and call the Actor from an MCP client by connecting Apify's MCP server. The API tab of this Actor shows ready-made request examples; the full reference is at docs.apify.com/api/v2.

Why am I getting fewer pages than expected?

The default is 10 rows, so raise maxResults first. Then check maxCrawlDepth, the include and exclude patterns, and whether links point to another domain. Enable useSitemaps for unlinked pages. For script-built sites set renderingMode to always, and review storeSkippedUrls to see what was skipped.

Website Content Crawler not working?

First check the run log and the status message, then confirm the start URLs open in a browser and your include and exclude patterns are not too strict. If a page is still missing, open the Issues tab of this Actor with your input and the affected URL, and we will take a look.

Related Actors

Actor	Use it to
🔗 Website Lead Intelligence	Read the public contact, team and about pages of company websites and get scored leads with emails, decision-makers, phones and social links.
🔗 YouTube Transcript Scraper	Collect the transcripts, subtitles and captions of YouTube videos in bulk for research or AI projects.

Support and disclaimer

Found a bug or need a feature? Open an issue in the Issues tab. For custom solutions, integrations or questions, contact hello.dataminds@gmail.com. Website Content Crawler is an independent tool and is not affiliated with any website it crawls; trademarks belong to their owners. It collects publicly available data only, and you are responsible for complying with applicable laws and each site's terms.

Last updated: 2026-10-17