🌐 Website Content Crawler

What is Website Content Crawler?

Website Content Crawler turns any website into clean text, Markdown and page metadata, ready for RAG pipelines, LLM knowledge bases and SEO content audits. Enter one or more start URLs, set a page limit, and get one dataset row per page with the title, text, Markdown, metadata and crawl details. Run it on the Free plan at $0.005 per page, or from $0.0035 per page on the Gold plan.

Just add a start URL and click Start. No coding needed. The default input crawls up to 10 pages of a public documentation site, so you can see real output on a first test run. Platform usage is billed in addition to the page price.

• Feed RAG and LLM pipelines: collect website content as Markdown for a vector database, a chatbot or an LLM knowledge base.
• Export a documentation site: crawl a documentation site and keep headings, lists, tables and code blocks in Markdown.
• Run an SEO content audit: compare titles, descriptions, canonical URLs, languages and publish dates across many pages.
• Find all URLs on a website: follow links, sitemaps and llms.txt entries, and save skipped URLs with a reason.
• Monitor competitor content: schedule repeat crawls and compare page text and metadata between runs.
• Extract text from PDF links: turn linked PDF files into searchable text, counted as pages like any other page.

Teams use it as a site crawler and website content extractor to pull page text out of large sites. As a documentation crawler it exports help centers and developer docs. As a sitemap crawler it reaches pages that links alone miss. It also works as a website to markdown converter and an HTML to text crawler for clean copy.

You can scrape a website without code and learn how to scrape all pages of a website in a single run. It is a simple website crawling tool for page content. It also serves as a website data extractor. When you only need plain text, it acts as a page content extractor. Use the output as a website content scraper feed for search, analytics or AI. Prepare website data for an LLM knowledge base. Load it for vector database ingestion. Build a chatbot from website content. Export website content to JSON, CSV or Excel.

Because it runs on Apify, you also get API access, scheduling, webhooks, integrations and JSON, CSV, Excel, XML and HTML exports.

What can Website Content Crawler do?

• 🌐 Crawl a website from your start URLs, staying on the same site, up to the depth and page limits you choose.
• 📝 Extract clean text and Markdown from web pages, with navigation, footers and other noise removed by your own selector lists.
• 🏷 Extract page metadata such as description, canonical URL, author, keywords, language, Open Graph tags and structured data.
• 🗺 Crawl a site using its sitemap or act as an llms.txt crawler to reach pages that are not linked from your start page, with include and exclude URL patterns.
• 🖥 Crawl JavaScript websites with the rendering mode set to auto, always or never, plus scrolling and click options for lazy content.
• 📸 Save HTML snapshots and screenshots to the key-value store as a website screenshot crawler, with a clickable link on every row.
• 🔒 Mask emails, phone numbers and card-like numbers in text and Markdown with the best-effort redaction option.

---

What data can Website Content Crawler extract?

Each crawled page becomes one dataset row. The default Quick View table shows the first seven fields; the other views show crawl details, metadata, content and storage links.

Field	Description
🔢 orderNo	Position of the page in the saved results.
🔗 url	Page URL. The canonical URL is used when the page has one and you did not turn on ignoreCanonicalUrl.
📥 loadedUrl	The URL that was actually loaded.
🕒 loadedTime	UTC time when the page was loaded.
🔙 referrerUrl	The page that linked to this one, or your referrerUrl input for start URLs.
📂 depth	Number of link hops from a start URL.
✅ httpStatusCode	HTTP status of the page.
📰 title	Page title.
📝 description	Page description.
🧭 canonicalUrl	Canonical URL declared by the page.
👤 author	Author, when the page names one.
🔖 keywords	Page keywords.
📅 datePublished	Publish date, when available.
📅 dateModified	Last modified date, when available.
🌐 languageCode	Page language code.
📃 text	Readable page text.
📄 markdown	Markdown version of the page (empty when saveMarkdown is off).
🧱 html	Cleaned readable HTML (empty when saveHtml is off).
📥 htmlUrl	Link to the saved readable HTML snapshot.
📸 screenshotUrl	Link to the PNG screenshot, when screenshots are on.
📎 files	Links to files such as documents and archives found on the page.
🧾 crawl	Object with loadedUrl, loadedTime, referrerUrl, depth and httpStatusCode.
🏷 metadata	Object with canonicalUrl, title, description, author, keywords, datePublished, dateModified, languageCode, openGraph, jsonLd and headers.
🧰 debug	Object with page type, link counts, response status, snapshot links and text lengths.
🧰 debugRequestHandlerMode	Whether the page was loaded in fast or rendered mode (pdf for PDF files).
🔗 debugTotalLinks	Total number of links found on the page.
✅ debugStatusCode	HTTP status copied from the debug object.

Note: description, author, keywords, datePublished, dateModified, languageCode, canonicalUrl, screenshotUrl and files are empty (null) when the page does not provide them. PDF rows carry the PDF text and title only, so most metadata fields are empty for them.

What data can't Website Content Crawler extract?

• Content behind a login, unless you supply your own cookies or headers in the input.
• Images, video and audio content. Image links appear in Markdown, but the files are not downloaded.
• Pages outside the start URLs' site, because the crawl stays on the same domain.

For company contact details from websites, try Website Lead Intelligence.

---

How to scrape website content

Website Content Crawler is designed for an easy start, even if you have never extracted data from the web before.

1. Create a free Apify account using your email.
2. Open Website Content Crawler.
3. Add one or more pages to startUrls.
4. Set maxResults (rows to save) and maxCrawlDepth (link hops from your start URLs).
5. Click Start and wait for the run to finish.
6. Download your data in JSON, CSV, Excel, XML or HTML.

---

How much does it cost to scrape website content?

Website Content Crawler uses a pay-per-event pricing model: you pay for every page saved to the dataset, and the price drops with your Apify plan. Apify platform usage is billed in addition to the page price.

Plan	Price per page	Price per 1,000 pages
Free	$0.005	$5
Bronze	$0.0045	$4.50
Silver	$0.004	$4
Gold	$0.0035	$3.50

Each run also pays a small start event of $0.00005 per GB of memory, at least one event. The default memory is 4096 MB (4 GB), so a run starts with 4 events, which is $0.0002.

Worked example: crawling 2,000 pages on the Silver plan costs 2,000 x $0.004 = $8 in page charges, plus the start event and platform usage.

See the pricing tab for all events.

Is scraping website content free?

Yes, you can start for free. The Apify Free plan includes $5 in monthly credit, which covers about 1,000 pages at the Free price of $0.005 per page, before platform usage. Use maxResults to keep a first test small and cheap.

---

Input

The only required input is startUrls, the page or pages where the crawl begins. Set the input in the Actor interface or send it as JSON through the API. Everything else is optional and has a default.

Here is the default setup, shortened to the fields most people change. All other fields keep their defaults.

{
  "startUrls": ["https://docs.apify.com/academy/scraping-basics-javascript"],
  "referrerUrl": "",
  "maxCrawlDepth": 2,
  "maxCrawlPages": 0,
  "maxResults": 10,
  "maxConcurrency": 10,
  "rateLimitDelaySecs": "0.1",
  "renderingMode": "auto",
  "saveMarkdown": true,
  "saveHtml": true,
  "kvDir": "kv",
  "saveScreenshots": false,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}

See the input tab for the full description of every field.

Field	Type	Required	Default	Description
startUrls	array	Yes	prefilled docs page	Pages where the crawl starts. Start URLs are always crawled.
referrerUrl	string	No	empty	Value saved as referrerUrl on rows from start URLs.
blockAds	boolean	No	false	Skip known ad and tracker hosts on rendered pages.
mobileView	boolean	No	false	Render pages with a mobile viewport.
redactPii	boolean	No	false	Mask emails, phone numbers, card-like numbers and SSNs in text and markdown (best effort).
screenshotFullPage	boolean	No	true	Full-page or viewport-only screenshots.
extractPdfText	boolean	No	false	Extract text from linked PDF files into text.
includeUrlGlobs	array	No	empty	Only discovered URLs matching a pattern are queued.
excludeUrlGlobs	array	No	empty	Discovered URLs matching a pattern are skipped.
useSitemaps	boolean	No	false	Also queue pages listed in the site's sitemaps.
useLlmsTxt	boolean	No	false	Also queue URLs listed in /llms.txt.
respectRobotsTxtFile	boolean	No	false	Skip discovered URLs disallowed by robots.txt.
keepUrlFragments	boolean	No	false	Treat #anchors as separate pages.
ignoreCanonicalUrl	boolean	No	false	Use the loaded URL instead of the canonical URL in url.
maxCrawlDepth	integer	No	2	Link hops from the start URLs (0 to 50).
maxCrawlPages	integer	No	0	Cap on unique pages opened. 0 means no extra cap.
maxResults	integer	No	10	Maximum rows saved (1 to 100,000).
maxConcurrency	integer	No	10	Pages loaded at the same time (1 to 50).
rateLimitDelaySecs	string	No	0.1	Pause before each fetch, in seconds.
maxRequestRetries	integer	No	3	Total attempts per URL.
requestTimeoutSecs	integer	No	60	Page load timeout (5 to 300).
dynamicContentWaitSecs	integer	No	3	Extra wait on rendered pages (0 to 120).
waitForSelector	string	No	empty	CSS selector to wait for before extracting.
renderingMode	string	No	auto	auto, always or never.
retryFailedWithRendering	boolean	No	true	Try a failed page once more with rendering.
blockMedia	boolean	No	false	Skip images, fonts and video on rendered pages.
maxScrollHeightPixels	integer	No	0	Auto-scroll height on rendered pages. 0 turns it off.
clickElementsCssSelector	string	No	empty	Click matching elements (for example accordions) before extracting.
ignoreHttpsErrors	boolean	No	false	Allow pages with certificate problems.
keepElementsCssSelector	string	No	empty	Keep only matching elements.
removeElementsCssSelector	string	No	navigation, footer and similar elements	Comma-separated selectors to remove.
readableTextCharThreshold	integer	No	100	Pages with less cleaned text count as thin.
kvDir	string	No	kv	Prefix for saved snapshots. Empty skips writing files.
saveScreenshots	boolean	No	false	Save a full-page PNG for every saved row.
saveMarkdown	boolean	No	true	Include Markdown in each row.
saveHtml	boolean	No	true	Include cleaned HTML in each row.
proxyConfiguration	object	No	prefilled with Apify Proxy	Proxy configuration. Leave the default if unsure.
headerAcceptLanguage	string	No	en-US,en;q=0.9	Preferred content language.
headerDnt	string	No	1	Do Not Track header. Empty omits it.
headerUpgradeInsecureRequests	string	No	1	Upgrade-insecure-requests header. Empty omits it.
headerUserAgent	string	No	desktop browser string	User-Agent header.
headerSecChUa	string	No	desktop browser string	Client hint header.
headerSecChUaMobile	string	No	?0	Client hint header.
headerSecChUaPlatform	string	No	"Windows"	Client hint header.
customHttpHeaders	object	No	{}	Extra HTTP headers that override the fields above.
initialCookies	array	No	empty	Cookies sent with every request.
debugLog	boolean	No	true	Print extra notes in the run log.
storeSkippedUrls	boolean	No	false	Save skipped URLs with reasons to skipped-urls.json.

Start URLs

Paste full page addresses. A start URL without https:// is given https:// automatically. The crawl stays on the same site as your start URLs.

• ✅ Good: https://docs.example.com/guide
• ❌ Bad: docs.example.com/guide? with stray spaces or text around the address

Warning: URL filters apply only to links the crawler discovers. Your start URLs are always crawled.

URL filters

Use includeUrlGlobs and excludeUrlGlobs to keep a crawl on topic. * stops at /, and ** crosses folders. A pattern without https:// is matched against the path.

• ✅ Good: /blog/*, /docs/**, **/*.pdf
• ❌ Bad: blog (too vague to match a path)

Rendering mode

auto is recommended: most pages are collected quickly and only pages that need it are rendered. always is the most reliable for JavaScript-heavy sites but the slowest. never is the cheapest and fastest, but it cannot take screenshots or capture content built by JavaScript.

Warning: saveScreenshots needs rendering. With renderingMode set to never, screenshots are switched off and a warning appears in the log.

Keep and remove selectors

keepElementsCssSelector keeps only the matching parts of a page, for example article, main. removeElementsCssSelector then removes noise from what is left. An invalid selector is skipped with a warning instead of failing the page.

Initial cookies

Pass a JSON array such as [{"name": "sessionid", "value": "abc123"}]. Each item needs a name; value, domain and path are optional.

Warning: Only use cookies for accounts and content you are allowed to access.

---

Output

Results are stored in the run's dataset, one row per page. Open the Output or Storage tab to view them as a table or JSON, choose the fields you need, and download them in JSON, CSV, Excel, XML or HTML. Rows appear as soon as each page is ready, so you do not have to wait for the run to finish.

JSON example

Results for the input example above (values shortened for readability):

{
  "orderNo": 1,
  "url": "https://example.com/guide/getting-started",
  "loadedUrl": "https://example.com/guide/getting-started",
  "loadedTime": "2026-01-15T10:30:00.000Z",
  "referrerUrl": "",
  "depth": 0,
  "httpStatusCode": 200,
  "title": "Getting started",
  "description": "A short guide for new users.",
  "canonicalUrl": "https://example.com/guide/getting-started",
  "author": null,
  "keywords": null,
  "datePublished": null,
  "dateModified": null,
  "languageCode": "en",
  "crawl": {
    "loadedUrl": "https://example.com/guide/getting-started",
    "loadedTime": "2026-01-15T10:30:00.000Z",
    "referrerUrl": "",
    "depth": 0,
    "httpStatusCode": 200
  },
  "metadata": {
    "canonicalUrl": "https://example.com/guide/getting-started",
    "title": "Getting started",
    "description": "A short guide for new users.",
    "author": null,
    "keywords": null,
    "datePublished": null,
    "dateModified": null,
    "languageCode": "en",
    "openGraph": null,
    "jsonLd": null,
    "headers": {"content-type": "text/html; charset=utf-8"}
  },
  "screenshotUrl": null,
  "text": "Getting started\n\nWelcome to the guide.",
  "html": "<div id=\"readability-content\">...</div>",
  "htmlUrl": "https://example.com/kv__0123456789abcdef0123_readable.html",
  "markdown": "# Getting started\n\nWelcome to the guide.",
  "files": null,
  "debug": {
    "requestHandlerMode": "fast",
    "pageType": "Article",
    "pageTypeReason": "MeetsReadableTextThreshold",
    "totalLinks": 12,
    "pageLinks": 12,
    "fileLinks": 0,
    "response": {"statusCode": 200, "contentType": "text/html; charset=utf-8"},
    "snapshots": {
      "originalHtmlUrl": "https://example.com/kv__0123456789abcdef0123_original.html",
      "removedElementsHtmlUrl": "https://example.com/kv__0123456789abcdef0123_removed.html",
      "readableTextHtmlUrl": "https://example.com/kv__0123456789abcdef0123_readable.html"
    },
    "lengths": {"originalHtml": 20480, "readableHtml": 3120, "readableText": 1450, "markdown": 1500},
    "readableTextCharThreshold": 100
  },
  "debugRequestHandlerMode": "fast",
  "debugTotalLinks": 12,
  "debugStatusCode": 200
}

The key-value store also holds HTML snapshots and screenshots (keys start with your kvDir prefix) and, when storeSkippedUrls is on, a skipped-urls.json list. Each entry has a url and a reason: non-http, off-domain, not-included, excluded or robots-disallow.

Notes and limitations

• A run stops when it reaches maxResults, the maxCrawlPages cap, or the spend limit you set for the run. Rows saved before that point are kept.
• Very large rows are shortened to stay under the dataset item size limit, and the debug object then contains truncated: true.
• PDF text extraction reads up to 300 pages of PDFs up to 20 MB each. Larger PDFs are skipped.
• respectRobotsTxtFile is off by default. Turn it on if you want disallowed discovered URLs skipped.
• Redaction is best effort and is not a compliance guarantee.
• Pages that cannot be loaded after the allowed attempts are skipped and not charged, because only saved rows are charged.

---

Tips and advanced usage

How many pages can you crawl with Website Content Crawler?

You can save up to 100,000 pages per run through maxResults. The real number depends on the site size, your depth limit, your URL filters and how many pages need rendering. The best way to know is a quick test run with a small maxResults.

How to get faster or cheaper results

• Set renderingMode to never for plain content sites.
• Turn on blockMedia and blockAds for rendered crawls.
• Turn off saveHtml for smaller rows, and saveMarkdown if you only need text.
• Use includeUrlGlobs and a low maxCrawlDepth so the crawl does not wander.
• Raise maxConcurrency carefully on small sites.

Known issues

• Some sites load content only after interaction. Try maxScrollHeightPixels, clickElementsCssSelector or waitForSelector.
• Screenshots slow the run because every page is rendered.

---

Integrations and API

Can I use Website Content Crawler as an API?

Yes. The Apify API lets you start runs, schedule them and fetch results. Use the apify-client package for Node.js or Python, and see the API tab for ready-to-use code.

How to scrape website content with Python

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("data_minds/website-content-crawler").call(
    run_input={
        "startUrls": ["https://docs.apify.com/academy/scraping-basics-javascript"],
        "maxCrawlDepth": 2,
        "maxResults": 10,
        "renderingMode": "auto",
        "saveMarkdown": True,
    }
)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["url"], item["title"])

Can I integrate Website Content Crawler with other apps?

Yes. Connect it with Make, Zapier, Google Sheets, Slack and other tools through Apify integrations. Use webhooks to trigger an action when a run finishes, and schedule runs to keep your website content fresh.

Can I use Website Content Crawler with AI agents (MCP)?

Yes. Connect it to Claude and other MCP clients through the Apify MCP server. Open the MCP tab for setup steps.

---

Other website scrapers

Scraper	What it does
🔗 Google Maps Email And Phone Number Scraper	Find business emails and phone numbers behind Google Maps places: scrape listings and get emails, phones, social profiles and contact details for...
🔗 Reddit Post & Subreddit Extractor	Scrape Reddit posts, comments, communities and user profiles by keyword, URL or full subreddit — no API key, no login.
🔗 Google Search Results Scraper	Scrape Google Search: organic results, AI Overview, People Also Ask, related searches, ads, videos and knowledge panel.
🔗 YouTube Transcript & Subtitles Scraper API	Get full YouTube transcripts with timestamps, chapters and SRT/VTT/TXT subtitle files for videos, Shorts, playlists, channels and searches.

---

FAQ

Is it legal to crawl a website?

Crawling publicly available pages is generally legal, but you must respect personal data rules such as GDPR, copyright and the website's terms. Only collect personal data for a legitimate reason. This is not legal advice. Read Is web scraping legal? and ask a lawyer if unsure.

How does Website Content Crawler work?

You give it start URLs and limits. It opens those pages, follows links on the same site up to your depth, cleans each page and saves one row with text, Markdown, metadata and crawl details. You can add filters, sitemaps, rendering options and snapshots to match your use case.

Why crawl website content for RAG and LLM knowledge bases?

Language models answer better when they read your own current content. Crawling a site gives you clean Markdown that you can split, embed and load into a vector database or chatbot. Scheduled runs then refresh the knowledge base whenever the site changes, without copying pages by hand.

Why am I getting fewer pages than expected?

First check that maxResults, maxCrawlPages and maxCrawlDepth are high enough. Then check your include and exclude patterns, because the crawl stays on the start site. Pages that fail to load are skipped. Turn on storeSkippedUrls to see which URLs were left out and why.

Does Website Content Crawler respect robots.txt?

It can, but the option is off by default. Turn on respectRobotsTxtFile and discovered URLs that the site's robots.txt disallows are skipped. Your explicit start URLs are always crawled, so choose them carefully. You can also use useSitemaps to queue pages the site lists itself.

Can Website Content Crawler crawl JavaScript websites?

Yes. With renderingMode on auto, pages with little text are loaded again with rendering to capture content built by JavaScript. Set it to always for the hardest sites. You can also wait for a selector, scroll the page or click expanders. Rendering costs more time and compute.

Can Website Content Crawler extract text from PDF links?

Yes. Turn on extractPdfText and discovered PDF links are opened and their text is saved in the text field of a row. Each PDF counts as a page toward your limits and your bill. PDFs over 20 MB are skipped, and only the first 300 pages are read.

Website Content Crawler not working?

Open the run log first, then check your start URLs and filters. If the problem remains, report it on the Issues tab with your run link and input, and we will look into it. Try a smaller test run with fewer pages to narrow down the cause.

---

Support and feedback

Found a bug or have a feature request? Please create an issue on the Issues tab.

Need a custom solution or more data? Contact us at hello.dataminds@gmail.com.

Built and maintained by the data_minds team on Apify.

This Actor is independent and is not affiliated with or endorsed by any website you crawl. You are responsible for using it in line with applicable laws and the terms of the sites you crawl.