🕸️ Website Content Crawler

🚀 Crawl an entire website and export clean Markdown in seconds. Seed from sitemaps, respect robots.txt, and fall back to a real browser for JavaScript-heavy pages. No API key, no registration, no manual pipeline code.

🕒 Last updated: 2026-04-21 · 📊 18 fields per page · 🗺️ Sitemap auto-seed · 🤖 Robots-aware · 🌐 HTTP + browser fallback

The Website Content Crawler walks any website from a starting URL, following internal links up to a configurable depth. It parses sitemap.xml and sitemap_index.xml to discover thousands of URLs instantly, respects robots.txt, and can switch to a headless browser when HTTP-only fetching returns thin content. Every crawled page comes back as clean Markdown plus 17 metadata fields, ready for RAG pipelines, knowledge bases, and content audits.

Built-in include and exclude regex filters let you narrow the crawl to /docs/, skip /auth/, or ignore query-heavy URLs. Concurrency defaults to 10 parallel fetches, so a 100-page crawl typically finishes in about a minute. The output uses a consistent schema across HTTP and browser modes, so downstream consumers never have to know which fetch strategy was used.

🎯 Target Audience	💡 Primary Use Cases
AI app teams, knowledge engineers, SEO specialists, documentation writers, research scientists, content archivists	RAG knowledge bases, docs mirroring, SEO audits, competitor content analysis, research corpus assembly

---

📋 What the Website Content Crawler does

Six crawl workflows in a single run:

• 🗺️ Sitemap auto-seed. Parses sitemap.xml and index files to discover every public URL in seconds.
• 🤖 Robots.txt aware. Respects disallow rules for the * and apify user-agents.
• 🌐 Browser fallback. Uses Playwright when a page returns thin content, handling JavaScript-heavy sites automatically.
• 📝 Markdown extraction. Clean headings, paragraphs, lists, blockquotes, and code blocks. Navigation and footers stripped.
• 🔗 Link analytics. Counts internal and outbound links per page for site-structure analysis.
• 🚦 Include/exclude patterns. Regex filters to control which URLs enter the queue.

Every page ships with title, description, language, author, publishedTime, siteName, og:image, link counts, HTTP status, response time, depth, parent URL, and a timestamp.

💡 Why it matters: RAG pipelines, SEO audits, and knowledge bases all start with a clean crawl. Doing it yourself means writing link discovery, sitemap parsers, robots.txt logic, and a Markdown cleaner. This Actor ships all of that pre-packaged.

---

🎬 Full Demo

🚧 Coming soon: a 3-minute walkthrough showing sitemap seeding and browser fallback in action.

---

⚙️ Input

Input	Type	Default	Behavior
startUrls	array of URLs	required	One or more starting URLs for the crawl.
maxDepth	integer	2	Link hops from the start URLs (0 = start URLs only).
maxItems	integer	10	Pages returned. Free plan caps at 10, paid plan at 1,000,000.
sameDomain	boolean	true	Stay within the starting domain.
includeSubdomains	boolean	true	Follow subdomains of the root host.
renderingType	string	"http"	http, browser, or auto (browser fallback when HTTP content is thin).
useSitemap	boolean	true	Seed queue from sitemap.xml.
respectRobotsTxt	boolean	true	Skip URLs disallowed by robots.txt.
includeUrlPatterns	array of regex	[]	Only URLs matching any pattern are crawled.
excludeUrlPatterns	array of regex	[]	URLs matching any pattern are skipped.

Example: crawl documentation with sitemap seeding.

{
    "startUrls": [{ "url": "https://docs.apify.com" }],
    "maxDepth": 3,
    "maxItems": 500,
    "useSitemap": true,
    "respectRobotsTxt": true,
    "renderingType": "auto"
}

Example: blog crawl with URL filters.

{
    "startUrls": [{ "url": "https://example.com" }],
    "maxDepth": 5,
    "maxItems": 200,
    "includeUrlPatterns": ["/blog/"],
    "excludeUrlPatterns": ["/tag/", "/page/"]
}

⚠️ Good to Know: concurrency is capped at 10 parallel fetches to stay polite. Use browser mode only when HTTP-only returns thin content, because browser rendering is about 3x slower per page.

---

📊 Output

Each record contains 18 fields. Download the dataset as CSV, Excel, JSON, or XML.

🧾 Schema

Field	Type	Example
🔗 url	string	"https://docs.apify.com/platform/actors"
🪜 depth	number	1
🏠 parentUrl	string | null	"https://docs.apify.com"
🏷️ title	string | null	`"Actors
📝 description	string | null	"Learn how Apify Actors package scrapers."
📃 markdown	string	"# Actors\n\nAn Actor is..."
💬 text	string	"Actors An Actor is..."
🔢 wordCount	number	860
🌍 language	string | null	"en"
🧑 author	string | null	"Apify"
📅 publishedTime	ISO 8601 | null	"2024-08-15T00:00:00Z"
🏢 siteName	string | null	"Apify Documentation"
🖼️ imageUrl	string | null	"https://.../og.png"
↗️ outboundLinks	number	14
↘️ internalLinks	number	42
🟢 httpStatus	number	200
⏱️ responseTimeMs	number	210
🕒 crawledAt	ISO 8601	"2026-04-21T12:00:00.000Z"
❗ error	string | null	"Timeout" on failure

📦 Sample records

---

✨ Why choose this Actor

	Capability
🗺️	Sitemap auto-seeding. Discovers thousands of URLs from sitemap.xml instantly.
🤖	Robots-aware. Respects disallow rules out of the box.
🌐	HTTP plus browser. Auto fallback to Playwright when JavaScript matters.
📝	Clean Markdown. Strips nav, footer, aside, and scripts. Preserves content structure.
🔗	Link graph. Counts internal and outbound links per page for site analysis.
⚡	Fast. 100 pages in under a minute with HTTP concurrency of 10.
🚫	No credentials. Runs on any publicly accessible site.

📊 Clean crawling is the difference between a RAG pipeline that answers correctly and one that returns garbled navigation text. This Actor does the cleaning for you.

---

📈 How it compares to alternatives

Approach	Cost	Coverage	Refresh	Filters	Setup
⭐ Website Content Crawler (this Actor)	$5 free credit, then pay-per-use	Any public site	Live per run	depth, patterns, sitemap, robots	⚡ 2 min
Generic open-source spiders	Free	Raw HTML	Your schedule	Manual coding	🐢 Days
Cloud crawler platforms	$$$+/month	Full enterprise	Managed	Visual rules	🕒 Hours
DIY Playwright scripts	Free	Your code	Your maintenance	Whatever you build	🐢 Days

Pick this Actor when you want a clean, RAG-ready crawl with sitemap discovery and zero infrastructure.

---

🚀 How to use

1. 📝 Sign up. Create a free account with $5 credit (takes 2 minutes).
2. 🌐 Open the Actor. Go to the Website Content Crawler page on the Apify Store.
3. 🎯 Set input. Pick one or more start URLs, a depth limit, and maxItems.
4. 🚀 Run it. Click Start and let the Actor walk the site.
5. 📥 Download. Grab your results in the Dataset tab as CSV, Excel, JSON, or XML.

⏱️ Total time from signup to downloaded dataset: 3-5 minutes. No coding required.

---

💼 Business use cases

🧠 AI Knowledge Bases Feed product docs into a vector database Sync internal wikis into a RAG index Refresh chatbot context on a schedule Build training corpora from public sites	📈 SEO & Content Audits Inventory every public page on a site Map internal and outbound link structure Detect orphan and 404 pages Compare competitor content footprints
📚 Documentation Mirroring Archive documentation for offline use Snapshot support portals for compliance Monitor API reference changes over time Build plain-Markdown docs archives	🧑‍🔬 Research Corpora Extract text datasets from academic sites Gather news archives by domain Build language modeling corpora Snapshot regulatory content for analysis

---

🔌 Automating Website Content Crawler

Control the scraper programmatically for scheduled runs and pipeline integrations:

• 🟢 Node.js. Install the apify-client NPM package.
• 🐍 Python. Use the apify-client PyPI package.
• 📚 See the Apify API documentation for full details.

The Apify Schedules feature lets you trigger this Actor on any cron interval. Daily or weekly refreshes keep downstream databases aligned with the source site.

---

❓ Frequently Asked Questions

---

🔌 Integrate with any app

Website Content Crawler connects to any cloud service via Apify integrations:

• Make - Automate multi-step workflows
• Zapier - Connect with 5,000+ apps
• Slack - Get run notifications
• Airbyte - Pipe content into your warehouse
• GitHub - Trigger runs from commits
• Google Drive - Export Markdown to Docs

You can also use webhooks to push freshly crawled content into vector databases and RAG pipelines.

---

🔗 Recommended Actors

• 🤖 RAG Web Browser - Search or fetch URLs with LLM-ready output
• 📰 Smart Article Extractor - Extract clean article text from news sites
• 🔍 Google Search Scraper - SERP results with rank and description
• 📧 Contact Info Scraper - Emails, phones, and socials from URLs
• 📸 URL Screenshot Tool - Full-page screenshots as PNG, JPEG, or PDF

💡 Pro Tip: browse the complete ParseForge collection for more AI-ready web tools.

---

🆘 Need Help? Open our contact form to request a new scraper, propose a custom data project, or report an issue.

---

⚠️ Disclaimer: this Actor is an independent tool and is not affiliated with any website or crawler framework. Only publicly accessible pages are crawled. Robots.txt rules are respected by default. Always honor the terms of service of the sites you crawl.