# Website Content Crawler (Advanced Feature) (`scrapevanta/website-content-crawler`) Actor
Website Content Crawler extracts clean text, Markdown, HTML, metadata, and links from websites. It checks broken links, redirects, and external URLs while supporting SEO audits, content analysis, web crawling, LLM/RAG pipelines, and search indexing.
- **URL**: https://apify.com/scrapevanta/website-content-crawler.md
- **Developed by:** [ScrapeVanta](https://apify.com/scrapevanta) (community)
- **Categories:**
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet
## Pricing
from $3.99 / 1,000 results
This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.
Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event
## What's an Apify Actor?
Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".
## How to integrate an Actor?
If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.
- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).
For usage examples, see the [API](#api) section below.
For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).
# README
A website content crawler helps crawl one or more websites, discover pages, extract readable content, and collect structured page data at scale. It can capture website text, Markdown, HTML, metadata, URLs, headings, and links for SEO research, web audits, content analysis, AI workflows, and data collection.
The crawler also checks discovered links and reports broken links, HTTP status codes, redirects, and unreachable URLs. With configurable crawl depth, URL filters, concurrency, browser fallback, proxy settings, and extraction controls, you can build repeatable website crawling workflows without manually checking every page.
### 🆘 Support & Feedback
- 🐞 Bug reports: Open a ticket in the repository's Issues section.
- ✨ Custom solutions or feature requests: Contact our team 📧 Email: ScrapeCraze@outlook.com
### 🌟 Key Features Website Content Crawler
#### 🔎 Website Crawling
Crawl one or more websites from your Start URLs and discover accessible pages across the selected crawl scope. The crawler follows internal links based on your configured crawling depth and result limits.
#### 📝 Website Content Extraction
Extract readable website content, webpage text, headings, links, and page information from crawled pages. This makes the Actor useful for website content scraping, research, indexing, and data collection.
#### 🧹 Clean Text Extraction
Remove unwanted page elements such as navigation, scripts, styles, footers, dialogs, and other interface content before extracting the main text. This produces cleaner data for website text extraction and content analysis.
#### 📄 Markdown Extraction
Convert crawled webpage content into Markdown and return it with each result. Markdown output works well for documentation, content processing, AI applications, knowledge bases, and RAG pipelines.
#### 🌐 URL Discovery
Discover internal URLs and connected pages while crawling a website. Use URL patterns, crawling depth, and page limits to control which pages the website crawler visits.
#### 🔗 Broken Link Checking
Live-check links found on crawled pages and identify broken links, redirected URLs, server errors, and unreachable domains. This turns the crawler into a website link checker for site audits.
#### 🚦 HTTP Status Monitoring
Record the HTTP status of discovered links, including successful responses, errors, and redirects. Redirect information also shows the destination URL when a link points somewhere else.
#### 🏷️ Metadata Extraction
Extract available webpage metadata such as page titles, descriptions, canonical URLs, language information, Open Graph data, robots directives, and response headers.
#### 🕸️ URL Filtering
Use include and exclude URL patterns to control the crawl scope. This helps focus the website content crawler on specific directories, content types, blog sections, product pages, or other URL groups.
#### 🤖 Browser Fallback
Use a headless browser when a standard HTTP request cannot access or properly render a page. This helps process JavaScript-heavy websites and pages that require browser rendering.
#### 🛡️ Anti-Bot Support
The crawler supports browser humanization, challenge waiting, proxy configuration, and browser fingerprint settings. These options help handle websites that apply different levels of automated-request protection.
#### 📊 Structured Output
Return website data in structured records containing the crawled URL, page type, crawl information, extracted text, HTML, Markdown, metadata, links, HTTP statuses, redirect details, and debugging information.
#### ⚙️ Automated Website Audits
Combine website crawling, content extraction, URL discovery, and link checking in one automated workflow. Use the collected data for SEO audits, technical website checks, content research, and web operations.
#### 📦 Bulk Website Processing
Add multiple Start URLs and process them within the same Actor run. This supports bulk website crawling, website content collection, and large-scale webpage extraction without manually processing each URL.
#### 🤖 AI, LLM & RAG Workflows
Use the extracted website text, Markdown, HTML, metadata, and links as input for AI applications. The output can support LLM datasets, RAG systems, vector databases, semantic search, knowledge bases, and AI-powered website analysis.
#### 🔍 SEO & Website Analysis
Use extracted page content and link-status information for SEO website crawling, technical SEO audits, broken-link analysis, content analysis, metadata checks, and internal linking research.
#### 🗃️ Website Content Data Collection
Collect structured webpage data for website content datasets, website research, content migration, search indexing, web archiving, and custom data-processing workflows.
#### 🌐 Multiple Extraction Formats
Use the Actor as a website scraper tool, website content extraction tool, website text scraper, website text extractor, website HTML scraper, website HTML extractor, or webpage content extractor, depending on the required output format.
#### 🔗 API & Data Pipeline Integration
Connect extracted website data with APIs, databases, search indexes, AI systems, analytics workflows, or custom applications. This makes the crawler suitable for repeatable web data extraction and website content processing workflows.
The Website Content Crawler combines website content scraping, website content extraction, URL discovery, and link checking in one workflow. It can work as a website crawler tool, website scraper tool, website content extraction tool, or website content scraping tool, depending on your project.
For text-focused projects, it works as a website text scraper, website text extractor, or website text crawler. For structural extraction, it can process pages as a website HTML scraper, website HTML extractor, or webpage content extractor.
The crawler also supports website content data collection, website content dataset creation, website content analysis, website content research, and automated web data workflows. Developers can connect the results with APIs, databases, search systems, AI applications, or custom processing pipelines.
### How does it work?
Website Content Crawler works in three main stages:
1. Crawling – Starts with the provided Start URLs and discovers accessible pages based on the configured crawl scope.
2. Content processing – Loads each page, removes unwanted elements, extracts readable content, and collects metadata and links.
3. Link checking and output – Checks discovered links and returns page content, metadata, crawl details, and link status information.
#### 🕸️ Crawling
The crawler starts with one or more Start URLs and processes pages according to the configured crawl depth and result limits.
You can control which pages are included with include URL globs and excluded with exclude URL globs. This helps focus a website URL crawler, website link crawler, or website content discovery tool on specific sections.
The crawler can process multiple starting points, making it useful for bulk website content crawling, multiple website crawling, batch website crawling, and large-scale website content data collection.
#### 🧹 Content processing
After loading a page, the crawler processes its HTML to identify useful readable content.
It can remove unwanted elements such as navigation, footer content, scripts, styles, SVG elements, images, alerts, banners, and modal interface elements. This produces cleaner content for a website content extractor, website page content extractor, or webpage text extractor workflow.
The crawler can save extracted content as plain text and Markdown. HTML saving can also be configured when the original page structure is important.
#### 🔗 Link checking
The crawler can check links discovered on each page, including external links when enabled.
Each checked link can include its URL, anchor text, internal or external classification, HTTP status code, broken status, redirect status, redirected destination, and redirect chain.
This makes the Actor useful as a broken link checker, website link checker, broken link crawler, 404 link checker, and redirect checker.
For example, the output can identify a 404 page as broken, detect a 500 server response, or identify a link that redirects to another URL.
#### 🤖 Browser fallback
The crawler can begin with HTTP-based requests and use a headless browser fallback when browser rendering is required.
This approach helps process JavaScript-heavy pages where important content does not appear in the initial HTML response. Browser settings can also control humanization, challenge waiting, fingerprint profiles, and related crawling behavior.
This supports JavaScript website crawling, dynamic website crawling, headless browser crawling, and automated web content extraction.
### 💻 How to Use
#### 1. Sign in or create an account
First, log in to your Apify account or sign up for a new account. Open the Website Content Crawler Actor.
#### 2. Add Start URLs
Enter one or more website URLs in Start URLs. These URLs define where the website content crawler begins processing.
#### 3. Set the crawl scope
Configure the maximum results, crawl depth, URL inclusion rules, and exclusion rules according to your project.
#### 4. Configure content extraction
Choose the content and text extraction settings you want. You can save Markdown and control which page elements are removed during processing.
#### 5. Configure link checking
Enable link status checking when you want to identify broken links, redirects, HTTP errors, and unreachable URLs.
You can also enable external link checking and configure the maximum links checked per page, timeout, and link-check concurrency.
#### 6. Configure browser and proxy settings
For dynamic or protected websites, configure browser fallback, headless mode, humanization, challenge waiting, CAPTCHA solving, proxy settings, and browser fingerprint options where required.
#### 7. Start the Actor
Click Start and let the Actor crawl the configured website pages, extract content, and check discovered links.
#### 8. Review the results
Open the dataset after the run completes. Review the extracted page content, metadata, URLs, crawl information, links, status codes, redirects, and diagnostic information.
#### 9. Export the data
Export the results for SEO audits, content research, website analysis, AI applications, RAG pipelines, databases, or other downstream workflows. 📥
### 📥 Input
The primary input is one or more Start URLs. These URLs tell the website page crawler where to begin crawling and discovering content.
The Website Content Crawler accepts one or more starting URLs and lets you control crawling, content extraction, browser fallback, proxy behavior, concurrency, and link auditing.
#### Example input
```json
{
"startUrls": [
"https://paste.rs/aOPsH"
],
"maxResults": 1,
"maxCrawlDepth": 0,
"readableTextCharThreshold": 50,
"checkLinkStatus": true,
"checkExternalLinks": true,
"maxLinksPerPage": 50,
"linkCheckTimeoutSecs": 12,
"linkCheckConcurrency": 6,
"includeUrlGlobs": [],
"excludeUrlGlobs": [],
"keepUrlFragments": false,
"ignoreCanonicalUrl": false,
"removeElementsCssSelector": "nav, footer, script, style, noscript, svg, img[src^=\"data:\"], [role=\"alert\"], [role=\"banner\"], [role=\"dialog\"], [role=\"alertdialog\"], [role=\"region\"][aria-label*=\"skip\" i], [aria-modal=\"true\"]",
"saveMarkdown": true,
"saveHtmlAsFile": false,
"saveScreenshots": false,
"debugLog": false,
"browserFallback": true,
"browserHeadless": true,
"browserHumanize": true,
"challengeWaitSecs": 30,
"waitForSelector": "",
"capsolverEnabled": false,
"capsolverMaxSolves": 50,
"maxConcurrency": 10,
"perDomainConcurrency": 4,
"perDomainDelaySecs": 1,
"maxRequestRetries": 3,
"requestTimeoutSecs": 60,
"proxyConfiguration": {
"useApifyProxy": false
},
"fingerprintProfile": "chrome_windows",
"customHttpHeaders": {}
}
```
#### Main input options
- Start URLs — URLs where the crawl begins.
- Max results — Maximum number of successfully collected pages.
- Max crawl depth — Number of internal link levels the crawler follows.
- Include URL patterns — Restricts crawling to matching URL globs.
- Exclude URL patterns — Prevents specific URLs from being crawled.
- Minimum readable text length — Filters out pages with very little extracted text.
- Save Markdown — Adds cleaned Markdown to every page result.
- Save HTML snapshots — Stores HTML versions for debugging and analysis.
- Save screenshots — Captures full-page screenshots when browser rendering is used.
- Browser fallback — Automatically switches to a browser when HTTP extraction fails or JavaScript is required.
- Captcha solving — Optionally enables CapSolver for supported challenges.
- Max concurrency — Controls the number of pages processed simultaneously.
- Proxy configuration — Controls proxy usage and automatic proxy escalation.
- Link status checking — Checks discovered links for broken responses and redirects.
- External link checking — Includes links pointing outside the crawled domain.
### Output
The crawler returns one structured dataset item for each successfully collected page. Each result can include the page URL, crawl information, metadata, extracted text, HTML, Markdown, debugging information, individual link status results, and a link-check summary.
#### Example output
```json
{
"url": "https://paste.rs/aOPsH",
"type": "page",
"isChild": false,
"crawl": {
"loadedUrl": "https://paste.rs/aOPsH",
"loadedTime": "2026-09-04T13:09:13.466Z",
"referrerUrl": "https://apify.com",
"httpStatusCode": 200,
"depth": 0,
"contentLengthBytes": 913,
"contentType": "text/plain; charset=utf-8"
},
"metadata": {
"title": "Link Check Fixture",
"description": null,
"author": null,
"keywords": null,
"languageCode": "",
"jsonLd": null,
"canonical": null,
"robotsDirectives": [],
"htmlLang": null,
"viewport": null,
"openGraph": {},
"mixedContentRefs": [],
"headers": {
"server": "nginx",
"date": "Fri, 04 Sep 2026 13:09:13 GMT",
"content-type": "text/plain; charset=utf-8",
"content-length": "913"
}
},
"screenshotUrl": "",
"text": "Link Check Fixture Link Check Fixture Static test page for verifying the broken-link and redirect checker.",
"html": "
...
",
"htmlUrl": "",
"markdown": "# Link Check Fixture\n\nStatic test page for verifying the broken-link and redirect checker.",
"debug": {
"requestHandlerMode": "http",
"pageType": "Article",
"pageTypeReason": "HasMinWordsCrawleeHtmlToText",
"totalLinks": 6,
"response": {
"statusCode": 200
},
"snapshots": {},
"antibot": {
"tier": "http",
"vendor": "",
"proxyTier": "none",
"solvedWith": [],
"attempts": 1
},
"lengths": {
"extractus": 0,
"readableText": 416,
"none": 891,
"defuddle": 0,
"readableTextIfPossible": 416,
"result": 416,
"markdown": 675
}
},
"links": [
{
"url": "https://httpbin.org/status/200",
"anchorText": "Healthy link 200",
"isInternal": false,
"statusCode": 200,
"isBroken": false,
"isRedirect": false,
"redirectedTo": null,
"redirectChain": [],
"checkedAt": "2026-09-04T13:09:13.544Z"
},
{
"url": "https://httpbin.org/status/404",
"anchorText": "Broken link 404",
"isInternal": false,
"statusCode": 404,
"isBroken": true,
"isRedirect": false,
"redirectedTo": null,
"redirectChain": [],
"checkedAt": "2026-09-04T13:09:13.544Z"
},
{
"url": "https://httpbin.org/status/500",
"anchorText": "Broken link 500",
"isInternal": false,
"statusCode": 500,
"isBroken": true,
"isRedirect": false,
"redirectedTo": null,
"redirectChain": [],
"checkedAt": "2026-09-04T13:09:13.544Z"
},
{
"url": "https://httpbin.org/redirect-to?url=https://example.com&status_code=301",
"anchorText": "Redirecting link 301",
"isInternal": false,
"statusCode": 200,
"isBroken": false,
"isRedirect": true,
"redirectedTo": "https://example.com",
"redirectChain": [
"https://httpbin.org/redirect-to?url=https://example.com&status_code=301"
],
"checkedAt": "2026-09-04T13:09:13.544Z"
},
{
"url": "https://this-domain-does-not-exist-abc123xyz.invalid/",
"anchorText": "Unreachable domain",
"isInternal": false,
"statusCode": null,
"isBroken": true,
"isRedirect": false,
"redirectedTo": null,
"redirectChain": [],
"checkedAt": "2026-09-04T13:09:13.544Z"
},
{
"url": "https://example.com",
"anchorText": "Healthy external link",
"isInternal": false,
"statusCode": 200,
"isBroken": false,
"isRedirect": false,
"redirectedTo": null,
"redirectChain": [],
"checkedAt": "2026-09-04T13:09:13.544Z"
}
],
"linkCheckSummary": {
"checked": 6,
"broken": 3,
"redirected": 1
}
}
```
#### What the output contains
| Output field | Description |
| --- | --- |
| url | Original URL processed by the crawler |
| crawl | Load URL, status code, crawl depth, content type, and timing |
| metadata | Title, description, canonical URL, robots directives, headers, and other page metadata |
| text | Clean readable text extracted from the page |
| html | Processed HTML content |
| markdown | Markdown version of the extracted page content |
| links | Links discovered on the page with their audit results |
| debug | Request, extraction, browser, anti-bot, and content-length details |
| screenshotUrl | Screenshot location when screenshots are enabled |
| htmlUrl | Stored HTML snapshot location when HTML snapshots are enabled |
| linkCheckSummary | Total checked links, broken links, and redirects |
The links array is particularly useful for broken link checking, 404 detection, redirect auditing, external link analysis, and SEO site audits. Each link includes its HTTP status, broken status, redirect status, destination, and redirect chain.
### 🎯 Best Use Cases
#### 🔎 SEO Website Audits
Use the crawler to collect page titles, descriptions, headings, URLs, readable text, canonical information, and links for SEO website content analysis.
It can also identify broken URLs and redirects for technical SEO reviews.
#### 🔗 Broken Link Audits
Use the broken link checker to inspect internal and external links across crawled pages. Status codes help identify 404, 500, unreachable, and redirected URLs.
#### 🤖 AI and RAG Data Preparation
A website content crawler for AI can collect readable text and Markdown for LLM applications, RAG systems, knowledge bases, vector databases, and AI search workflows.
#### 🏆 Competitor Research
Use a competitor website content crawler to collect publicly accessible competitor pages, metadata, URLs, headings, and content for structured research.
#### 📝 Content Audits
Create a website content inventory containing URLs, page titles, descriptions, text, metadata, and links. This supports website content audit, content analysis, and content planning.
#### 🔄 Website Migration
Use the crawler before a website migration to create a record of existing URLs and content. The results can support URL mapping, content inventories, and redirect planning.
#### 📚 Content Research
Use a website content research tool to collect web pages for market research, competitor research, topic research, and structured information discovery.
#### 👨💻 Developer Workflows
Developers can use the website content crawler API, website content extraction API, or automated crawling workflows to send collected data into applications, databases, search systems, and processing pipelines.
#### ⚙️ Automated Website Crawling
Automate repeated website content extraction, URL discovery, link checking, and content collection without manually opening each page.
#### 📦 Bulk Website Data Collection
Process multiple Start URLs and collect website content in bulk. This supports bulk website content crawler, batch website content scraper, and multiple website content crawler workflows.
#### 📊 Website Data Analysis
Export structured website data for analytics, content comparison, technical audits, SEO research, and website performance investigations.
### 💰 How much does Website Content Crawler cost?
Website Content Crawler costs $3.99 per 1,000 results. Your total cost depends on the number of pages successfully collected during the Actor run.
#### Pay only for the results you collect
For example, collecting 1,000 results costs $3.99, while collecting 500 results costs approximately $2.00 at the same rate. Link checking can generate additional requests during a crawl, but the result-based pricing is based on collected results.
### ⚖️ Is it legal to scrape websites?
Website crawling is not automatically legal or illegal. The legality depends on the website, the data collected, your purpose, applicable laws, and the site's terms or access restrictions.
Use this website content crawler for legitimate research, auditing, analysis, and automation. Respect copyright, privacy requirements, robots directives, access restrictions, and website terms where applicable.
Do not collect confidential, restricted, or personal information without appropriate authorization. Before starting a large website content scraping project, verify that your planned collection and intended use are permitted.
### ⚙️ Website Content Crawler Input Parameters
#### 🔗 Start URLs
startUrls contains the website URLs where the crawler begins.
Example:
```json
{
"startUrls": \[
"https://example.com/"
\]
}
```
#### 🎯 Max Results
maxResults limits the number of page results returned by the crawl.
The supplied configuration uses:
"maxResults": 1
#### 🕸️ Crawl Scope & URL Filters
maxCrawlDepth, includeUrlGlobs, excludeUrlGlobs, keepUrlFragments, and canonical URL settings control which pages the crawler processes.
These options help create focused website crawling, website URL discovery, and website content extraction workflows.
#### 🧹 Content & Text Extraction
The content extraction settings control how readable page content is processed.
The configured CSS selector removes interface elements such as navigation, footer, scripts, styles, SVG elements, images, alerts, banners, dialogs, and modal content.
This produces cleaner output for website text extraction, web content extraction, and AI data preparation.
#### 🗃️ Snapshots, Screenshots & Debug Logs
Screenshots, HTML files, and debug logs can be enabled when additional inspection data is useful.
These options help investigate extraction behavior, page classification, crawling problems, and browser fallback behavior.
#### 🛡️ Anti-Bot Bypass & Headless Browser
Browser fallback can process pages that require JavaScript rendering.
The configuration supports headless browser execution, browser humanization, challenge waiting, and browser fingerprint selection.
#### 🔓 CAPTCHA Solver
capsolverEnabled controls optional CAPTCHA-solving functionality.
You can also configure the maximum number of CAPTCHA solving attempts for a run.
#### 🚀 Speed, Concurrency & Rate Limits
maxConcurrency controls overall parallel processing.
perDomainConcurrency limits simultaneous requests for one domain, while perDomainDelaySecs controls the delay between requests to the same domain.
maxRequestRetries and requestTimeoutSecs control retry and timeout behavior.
#### 🕵️ Proxy Rotation & Browser Fingerprint
Proxy configuration controls how requests are routed. The crawler also supports a fingerprint profile for browser-based requests.
These settings can help with websites that apply different behavior based on request origin or browser characteristics.
#### 🚦 Broken Link & 404 Redirect Audit
Enable checkLinkStatus to check links discovered on each page.
With checkExternalLinks enabled, external links are also checked. The crawler can report:
- HTTP status codes
- Broken links
- Redirects
- Redirect destinations
- Redirect chains
- Unreachable domains
- Anchor text
- Internal or external classification
- Link-check timestamps
This makes the Actor useful for website broken link audits, 404 detection, redirect auditing, and technical SEO workflows.
### ⏳ During the Actor run
#### Start with HTTP requests
The crawler first attempts to retrieve pages using fast HTTP requests. This approach reduces browser overhead and works well for static HTML pages.
#### Use browser fallback when required
If a page requires JavaScript rendering or blocks the HTTP request, the crawler can switch to a headless browser. Browser humanization and fingerprint settings help handle sites that require browser-like requests.
#### Extract and audit each page
After loading a page, the Actor extracts readable text, Markdown, HTML, and metadata. It also discovers links and can check their live HTTP status, redirects, and unreachable destinations.
#### Continue crawling within your limits
The crawler follows eligible internal links according to your crawl depth, URL filters, and maximum result settings. Concurrency, retries, request delays, and timeouts control how quickly the run processes pages.
### 📝 Final Note
Website Content Crawler combines website crawling, content extraction, metadata collection, URL discovery, and link checking in one workflow.
Use it to build website content datasets, perform SEO audits, identify broken links, prepare content for AI and RAG systems, research competitor websites, support migrations, or automate recurring website data collection.
Configure the crawl depth, result limits, URL filters, extraction settings, browser options, concurrency, and link-check settings based on your project scope.
### ❓ FAQ
#### What is a Website Content Crawler?
Website Content Crawler is a web crawling tool that discovers website pages and extracts readable text, Markdown, HTML, metadata, URLs, and links.
#### What data can Website Content Crawler extract?
It can extract page URLs, loaded URLs, titles, descriptions, readable text, Markdown, HTML, headings, metadata, canonical information, language data, links, HTTP status codes, redirects, and crawl information when available.
#### Can it crawl multiple websites?
Yes. You can provide multiple Start URLs and process them within the configured crawl limits and URL scope.
#### Can it check broken links?
Yes. Enable link status checking to inspect links discovered on crawled pages. External links can also be checked when checkExternalLinks is enabled.
#### Can it detect 404 errors?
Yes. The link checker can identify links returning HTTP 404 responses and mark them as broken.
#### Can it detect redirects?
Yes. The output can identify redirected links and provide the destination URL and redirect chain when available.
#### Can it crawl JavaScript websites?
Yes. Browser fallback can process pages that require JavaScript rendering. Headless browser settings control this behavior.
#### Can I extract Markdown from websites?
Yes. Enable saveMarkdown to save processed website content in Markdown format.
#### Can I extract HTML?
The crawler processes HTML internally and can optionally save HTML files when the corresponding setting is enabled.
#### Can it extract website metadata?
Yes. Metadata can include page titles, descriptions, author information, language, canonical URLs, robots directives, Open Graph data, and response headers when available.
#### Can I use it for SEO audits?
Yes. It can collect URLs, page content, metadata, links, status codes, and redirects for SEO website audits, content inventories, and technical analysis.
#### Can it prepare content for AI or RAG?
Yes. Readable text and Markdown output can be used as source material for LLM applications, RAG pipelines, vector databases, knowledge bases, and AI search systems.
#### Can developers automate website crawling?
Yes. The Actor can support automated website crawling and API-based workflows for developers who want to integrate extracted website data into other systems.
# Actor input Schema
## `startUrls` (type: `array`):
One or more URLs to start crawling from. Add as many rows as you like — bulk input is fully supported. By default the crawler also follows internal links found on these pages (see **Max crawling depth** below).
## `maxResults` (type: `integer`):
Stop once this many pages have been successfully collected. This is your main "how much do I get" dial — the crawl stops the moment it's reached, even if more pages are still queued.
## `includeUrlGlobs` (type: `array`):
Only crawl links matching one of these glob patterns, e.g. `https://example.com/blog/**`. Leave empty to follow all same-domain sub-pages of your Start URLs.
## `excludeUrlGlobs` (type: `array`):
Skip any discovered link matching one of these glob patterns, e.g. `https://example.com/login/**`.
## `maxCrawlDepth` (type: `integer`):
How many link-hops away from your Start URLs to follow. `0` = only the Start URLs themselves; `1` = also pages directly linked from them; and so on.
## `maxCrawlPages` (type: `integer`):
A hard ceiling on how many pages the crawler is allowed to visit in total, including ones that turn out empty or duplicate. Leave empty to let **Max results** alone decide when to stop.
## `keepUrlFragments` (type: `boolean`):
If the target site uses `#fragment` URLs to represent different pages (common on some single-page apps), enable this so `page#a` and `page#b` are crawled as separate pages instead of being deduplicated together.
## `ignoreCanonicalUrl` (type: `boolean`):
By default, a page's own `` tag decides its final URL and dedup identity. Enable this to always use the actually-visited URL instead — useful for sites that report incorrect canonical URLs.
## `removeElementsCssSelector` (type: `string`):
Comma-separated CSS selectors removed from a page before text/HTML extraction. The default strips navigation, footers, scripts, and common cookie/alert dialogs.
## `readableTextCharThreshold` (type: `integer`):
A page whose extracted text falls below this many characters is treated as empty/thin and dropped instead of being saved as a result.
## `saveMarkdown` (type: `boolean`):
Convert each page's HTML to Markdown and include it in the `markdown` output field — handy for feeding LLMs directly.
## `saveHtmlAsFile` (type: `boolean`):
Save the original, cleaned, and readable HTML for each page to the key-value store, linked from the `htmlUrl` / `debug.snapshots` output fields. Off by default to keep storage usage down.
## `saveScreenshots` (type: `boolean`):
Save a full-page PNG screenshot for every page rendered by the browser tier, linked from the `screenshotUrl` output field. Useful for debugging; adds storage cost.
## `debugLog` (type: `boolean`):
Log low-level crawl internals (tier escalations, vendor detection, session bookkeeping) at DEBUG level. Off by default — normal runs only show clean, high-level progress in the log.
## `browserFallback` (type: `boolean`):
When a fast HTTP request is blocked or a page needs JavaScript to render, escalate to a real, fingerprint-resistant headless browser. Turning this off makes the crawler HTTP-only — faster, but it will fail on JS-heavy or heavily-protected sites.
## `browserHeadless` (type: `boolean`):
Keep this on for normal runs. Disabling only makes sense when debugging locally with a visible browser window, and has no effect on the Apify platform.
## `browserHumanize` (type: `boolean`):
Move the mouse along realistic paths instead of teleporting the cursor. Adds roughly 1-2 seconds per browser page but meaningfully helps against behavior-based anti-bot systems.
## `challengeWaitSecs` (type: `integer`):
How many seconds to let the browser wait for a bot-detection interstitial to clear on its own before giving up on that page.
## `waitForSelector` (type: `string`):
When using the browser tier, wait for this CSS selector to appear before extracting content — useful for pages whose content loads asynchronously.
## `capsolverEnabled` (type: `boolean`):
If enabled and an API key is supplied below, the crawler pays a solver service (CapSolver.com) to clear interactive captchas it can't get past on its own.
## `capsolverApiKey` (type: `string`):
Your CapSolver.com API key. Required only when **Enable captcha solving** is on. Stored encrypted and never shown in logs.
## `capsolverMaxSolves` (type: `integer`):
Hard ceiling on how many paid captcha solves this run is allowed to spend, regardless of how many pages remain.
## `maxConcurrency` (type: `integer`):
The overall number of pages the crawler works on in parallel, across all domains combined.
## `perDomainConcurrency` (type: `integer`):
How many requests to the *same* domain run in parallel. This automatically halves after a block and slowly ramps back up after a run of clean responses.
## `perDomainDelaySecs` (type: `number`):
Starting spacing (seconds, with jitter) between requests to the same domain. Doubles automatically after a block, eases back down after a run of clean responses.
## `maxRequestRetries` (type: `integer`):
How many times to retry a single page over the fast HTTP tier before escalating it to the browser tier. Also acts as the floor for the proxy ladder's own no-proxy → datacenter → residential retry sequence.
## `requestTimeoutSecs` (type: `integer`):
Timeout in seconds for a single HTTP request or browser page navigation.
## `proxyConfiguration` (type: `object`):
**Recommended: leave this on "No proxy".** Every request starts as a direct, no-proxy connection and *automatically* escalates to Apify's Datacenter proxy, then Residential proxy, only when a target site actually blocks it — each escalation is logged clearly so you can see exactly when and why it happened, and once Residential is reached it's used for the rest of the run. Pick a specific proxy group here only if you want to force one from the very first request.
## `fingerprintProfile` (type: `string`):
The TLS/HTTP2 fingerprint and User-Agent identity used for fast HTTP requests (tier 0 of the anti-bot ladder). All three are kept internally self-consistent — mixing them is a bigger giveaway than sending no headers at all.
## `customHttpHeaders` (type: `object`):
Extra HTTP headers merged into every request (applied last, so they override the fingerprint profile's own values). Useful for authentication headers required by the target site. Example: `{"Authorization": "Bearer ..."}`.
## `checkLinkStatus` (type: `boolean`):
For every link found on a crawled page, send a live request to it and record its HTTP status, whether it's broken (4xx/5xx or unreachable), and whether it redirects. This is what turns the crawl into a broken-link/redirect audit — turn it off to crawl exactly like the base content crawler.
## `checkExternalLinks` (type: `boolean`):
Also live-check links that point to other domains, not just links that stay on the crawled site. Turn off to only audit the site's own internal links (faster, no requests leave the domain being crawled).
## `maxLinksPerPage` (type: `integer`):
Cap on how many of a page's links get a live status check. Pages with more links than this only have their first N checked, in the order found on the page. Keeps a link-heavy page from ballooning the run's request count.
## `linkCheckTimeoutSecs` (type: `integer`):
Timeout in seconds for a single link's status check request, independent of the page-load timeout above.
## `linkCheckConcurrency` (type: `integer`):
How many link status checks run in parallel, across all pages combined.
## Actor input object example
```json
{
"startUrls": [
"https://apify.com"
],
"maxResults": 20,
"includeUrlGlobs": [],
"excludeUrlGlobs": [],
"maxCrawlDepth": 3,
"keepUrlFragments": false,
"ignoreCanonicalUrl": false,
"removeElementsCssSelector": "nav, footer, script, style, noscript, svg, img[src^=\"data:\"], [role=\"alert\"], [role=\"banner\"], [role=\"dialog\"], [role=\"alertdialog\"], [role=\"region\"][aria-label*=\"skip\" i], [aria-modal=\"true\"]",
"readableTextCharThreshold": 100,
"saveMarkdown": true,
"saveHtmlAsFile": false,
"saveScreenshots": false,
"debugLog": false,
"browserFallback": true,
"browserHeadless": true,
"browserHumanize": true,
"challengeWaitSecs": 30,
"waitForSelector": "",
"capsolverEnabled": false,
"capsolverMaxSolves": 50,
"maxConcurrency": 10,
"perDomainConcurrency": 4,
"perDomainDelaySecs": 1,
"maxRequestRetries": 3,
"requestTimeoutSecs": 60,
"proxyConfiguration": {
"useApifyProxy": false
},
"fingerprintProfile": "chrome_windows",
"customHttpHeaders": {},
"checkLinkStatus": true,
"checkExternalLinks": true,
"maxLinksPerPage": 200,
"linkCheckTimeoutSecs": 10,
"linkCheckConcurrency": 10
}
```
# Actor output Schema
## `results` (type: `string`):
Every crawled page in a compact overview: URL, title, status code, depth, and a text preview.
## `content` (type: `string`):
Full extracted plain text, Markdown, and HTML for each page.
## `seo` (type: `string`):
Title, description, author, keywords, language, canonical URL and Open Graph tags for each page.
## `antibot` (type: `string`):
Which fetch tier and proxy tier each page went through, link counts, and captcha-solver activity.
## `linkAudit` (type: `string`):
One row per broken or redirecting link found anywhere in the crawl — the flat broken-link/redirect audit.
## `allItemsJson` (type: `string`):
The complete, unfiltered dataset — every field on every record.
## `csv` (type: `string`):
All results exported as CSV.
# API
You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.
## JavaScript example
```javascript
import { ApifyClient } from 'apify-client';
// Initialize the ApifyClient with your Apify API token
// Replace the '' with your token
const client = new ApifyClient({
token: '',
});
// Prepare Actor input
const input = {
"startUrls": [
"https://apify.com"
],
"includeUrlGlobs": [],
"excludeUrlGlobs": [],
"proxyConfiguration": {
"useApifyProxy": false
},
"customHttpHeaders": {}
};
// Run the Actor and wait for it to finish
const run = await client.actor("scrapevanta/website-content-crawler").call(input);
// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
console.dir(item);
});
// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs
```
## Python example
```python
from apify_client import ApifyClient
# Initialize the ApifyClient with your Apify API token
# Replace '' with your token.
client = ApifyClient("")
# Prepare the Actor input
run_input = {
"startUrls": ["https://apify.com"],
"includeUrlGlobs": [],
"excludeUrlGlobs": [],
"proxyConfiguration": { "useApifyProxy": False },
"customHttpHeaders": {},
}
# Run the Actor and wait for it to finish
run = client.actor("scrapevanta/website-content-crawler").call(run_input=run_input)
# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
print(item)
# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start
```
## CLI example
```bash
echo '{
"startUrls": [
"https://apify.com"
],
"includeUrlGlobs": [],
"excludeUrlGlobs": [],
"proxyConfiguration": {
"useApifyProxy": false
},
"customHttpHeaders": {}
}' |
apify call scrapevanta/website-content-crawler --silent --output-dataset
```
## MCP server setup
```json
{
"mcpServers": {
"apify": {
"type": "http",
"url": "https://mcp.apify.com/?tools=fetch-actor-details,scrapevanta/website-content-crawler"
}
}
}
```
The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).
## OpenAPI specification
Download the OpenAPI definition: https://api.apify.com/v2/actors/Ohn8Lwars9IPAKcX4/builds/Qa5NECJFicquZ4uDL/openapi.json