Website Content Crawler (Advanced Feature)

A website content crawler helps crawl one or more websites, discover pages, extract readable content, and collect structured page data at scale. It can capture website text, Markdown, HTML, metadata, URLs, headings, and links for SEO research, web audits, content analysis, AI workflows, and data collection. The crawler also checks discovered links and reports broken links, HTTP status codes, redirects, and unreachable URLs. With configurable crawl depth, URL filters, concurrency, browser fallback, proxy settings, and extraction controls, you can build repeatable website crawling workflows without manually checking every page.

🆘 Support & Feedback

• 🐞 Bug reports: Open a ticket in the repository's Issues section.

• ✨ Custom solutions or feature requests: Contact our team 📧 Email: ScrapeCraze@outlook.com

🌟 Key Features Website Content Crawler

🔎 Website Crawling

Crawl one or more websites from your Start URLs and discover accessible pages across the selected crawl scope. The crawler follows internal links based on your configured crawling depth and result limits.

📝 Website Content Extraction

Extract readable website content, webpage text, headings, links, and page information from crawled pages. This makes the Actor useful for website content scraping, research, indexing, and data collection.

🧹 Clean Text Extraction

Remove unwanted page elements such as navigation, scripts, styles, footers, dialogs, and other interface content before extracting the main text. This produces cleaner data for website text extraction and content analysis.

📄 Markdown Extraction

Convert crawled webpage content into Markdown and return it with each result. Markdown output works well for documentation, content processing, AI applications, knowledge bases, and RAG pipelines.

🌐 URL Discovery

Discover internal URLs and connected pages while crawling a website. Use URL patterns, crawling depth, and page limits to control which pages the website crawler visits.

🔗 Broken Link Checking

Live-check links found on crawled pages and identify broken links, redirected URLs, server errors, and unreachable domains. This turns the crawler into a website link checker for site audits.

🚦 HTTP Status Monitoring

Record the HTTP status of discovered links, including successful responses, errors, and redirects. Redirect information also shows the destination URL when a link points somewhere else.

🏷️ Metadata Extraction

Extract available webpage metadata such as page titles, descriptions, canonical URLs, language information, Open Graph data, robots directives, and response headers.

🕸️ URL Filtering

Use include and exclude URL patterns to control the crawl scope. This helps focus the website content crawler on specific directories, content types, blog sections, product pages, or other URL groups.

🤖 Browser Fallback

Use a headless browser when a standard HTTP request cannot access or properly render a page. This helps process JavaScript-heavy websites and pages that require browser rendering.

🛡️ Anti-Bot Support

The crawler supports browser humanization, challenge waiting, proxy configuration, and browser fingerprint settings. These options help handle websites that apply different levels of automated-request protection.

📊 Structured Output

Return website data in structured records containing the crawled URL, page type, crawl information, extracted text, HTML, Markdown, metadata, links, HTTP statuses, redirect details, and debugging information.

⚙️ Automated Website Audits

Combine website crawling, content extraction, URL discovery, and link checking in one automated workflow. Use the collected data for SEO audits, technical website checks, content research, and web operations.

📦 Bulk Website Processing

Add multiple Start URLs and process them within the same Actor run. This supports bulk website crawling, website content collection, and large-scale webpage extraction without manually processing each URL.

🤖 AI, LLM & RAG Workflows

Use the extracted website text, Markdown, HTML, metadata, and links as input for AI applications. The output can support LLM datasets, RAG systems, vector databases, semantic search, knowledge bases, and AI-powered website analysis.

🔍 SEO & Website Analysis

Use extracted page content and link-status information for SEO website crawling, technical SEO audits, broken-link analysis, content analysis, metadata checks, and internal linking research.

🗃️ Website Content Data Collection

Collect structured webpage data for website content datasets, website research, content migration, search indexing, web archiving, and custom data-processing workflows.

🌐 Multiple Extraction Formats

Use the Actor as a website scraper tool, website content extraction tool, website text scraper, website text extractor, website HTML scraper, website HTML extractor, or webpage content extractor, depending on the required output format.

🔗 API & Data Pipeline Integration

Connect extracted website data with APIs, databases, search indexes, AI systems, analytics workflows, or custom applications. This makes the crawler suitable for repeatable web data extraction and website content processing workflows.

The Website Content Crawler combines website content scraping, website content extraction, URL discovery, and link checking in one workflow. It can work as a website crawler tool, website scraper tool, website content extraction tool, or website content scraping tool, depending on your project.

For text-focused projects, it works as a website text scraper, website text extractor, or website text crawler. For structural extraction, it can process pages as a website HTML scraper, website HTML extractor, or webpage content extractor.

The crawler also supports website content data collection, website content dataset creation, website content analysis, website content research, and automated web data workflows. Developers can connect the results with APIs, databases, search systems, AI applications, or custom processing pipelines.

How does it work?

Website Content Crawler works in three main stages:

1. Crawling – Starts with the provided Start URLs and discovers accessible pages based on the configured crawl scope.

2. Content processing – Loads each page, removes unwanted elements, extracts readable content, and collects metadata and links.

3. Link checking and output – Checks discovered links and returns page content, metadata, crawl details, and link status information.

🕸️ Crawling

The crawler starts with one or more Start URLs and processes pages according to the configured crawl depth and result limits.

You can control which pages are included with include URL globs and excluded with exclude URL globs. This helps focus a website URL crawler, website link crawler, or website content discovery tool on specific sections.

The crawler can process multiple starting points, making it useful for bulk website content crawling, multiple website crawling, batch website crawling, and large-scale website content data collection.

🧹 Content processing

After loading a page, the crawler processes its HTML to identify useful readable content.

It can remove unwanted elements such as navigation, footer content, scripts, styles, SVG elements, images, alerts, banners, and modal interface elements. This produces cleaner content for a website content extractor, website page content extractor, or webpage text extractor workflow.

The crawler can save extracted content as plain text and Markdown. HTML saving can also be configured when the original page structure is important.

🔗 Link checking

The crawler can check links discovered on each page, including external links when enabled.

Each checked link can include its URL, anchor text, internal or external classification, HTTP status code, broken status, redirect status, redirected destination, and redirect chain.

This makes the Actor useful as a broken link checker, website link checker, broken link crawler, 404 link checker, and redirect checker.

For example, the output can identify a 404 page as broken, detect a 500 server response, or identify a link that redirects to another URL.

🤖 Browser fallback

The crawler can begin with HTTP-based requests and use a headless browser fallback when browser rendering is required.

This approach helps process JavaScript-heavy pages where important content does not appear in the initial HTML response. Browser settings can also control humanization, challenge waiting, fingerprint profiles, and related crawling behavior.

This supports JavaScript website crawling, dynamic website crawling, headless browser crawling, and automated web content extraction.

💻 How to Use

1. Sign in or create an account

First, log in to your Apify account or sign up for a new account. Open the Website Content Crawler Actor.

2. Add Start URLs

Enter one or more website URLs in Start URLs. These URLs define where the website content crawler begins processing.

3. Set the crawl scope

Configure the maximum results, crawl depth, URL inclusion rules, and exclusion rules according to your project.

4. Configure content extraction

Choose the content and text extraction settings you want. You can save Markdown and control which page elements are removed during processing.

5. Configure link checking

Enable link status checking when you want to identify broken links, redirects, HTTP errors, and unreachable URLs.

You can also enable external link checking and configure the maximum links checked per page, timeout, and link-check concurrency.

6. Configure browser and proxy settings

For dynamic or protected websites, configure browser fallback, headless mode, humanization, challenge waiting, CAPTCHA solving, proxy settings, and browser fingerprint options where required.

7. Start the Actor

Click Start and let the Actor crawl the configured website pages, extract content, and check discovered links.

8. Review the results

Open the dataset after the run completes. Review the extracted page content, metadata, URLs, crawl information, links, status codes, redirects, and diagnostic information.

9. Export the data

Export the results for SEO audits, content research, website analysis, AI applications, RAG pipelines, databases, or other downstream workflows. 📥

📥 Input

The primary input is one or more Start URLs. These URLs tell the website page crawler where to begin crawling and discovering content.

The Website Content Crawler accepts one or more starting URLs and lets you control crawling, content extraction, browser fallback, proxy behavior, concurrency, and link auditing.

Example input

{
  "startUrls": [
    "https://paste.rs/aOPsH"
  ],
  "maxResults": 1,
  "maxCrawlDepth": 0,
  "readableTextCharThreshold": 50,
  "checkLinkStatus": true,
  "checkExternalLinks": true,
  "maxLinksPerPage": 50,
  "linkCheckTimeoutSecs": 12,
  "linkCheckConcurrency": 6,
  "includeUrlGlobs": [],
  "excludeUrlGlobs": [],
  "keepUrlFragments": false,
  "ignoreCanonicalUrl": false,
  "removeElementsCssSelector": "nav, footer, script, style, noscript, svg, img[src^=\"data:\"], [role=\"alert\"], [role=\"banner\"], [role=\"dialog\"], [role=\"alertdialog\"], [role=\"region\"][aria-label*=\"skip\" i], [aria-modal=\"true\"]",
  "saveMarkdown": true,
  "saveHtmlAsFile": false,
  "saveScreenshots": false,
  "debugLog": false,
  "browserFallback": true,
  "browserHeadless": true,
  "browserHumanize": true,
  "challengeWaitSecs": 30,
  "waitForSelector": "",
  "capsolverEnabled": false,
  "capsolverMaxSolves": 50,
  "maxConcurrency": 10,
  "perDomainConcurrency": 4,
  "perDomainDelaySecs": 1,
  "maxRequestRetries": 3,
  "requestTimeoutSecs": 60,
  "proxyConfiguration": {
    "useApifyProxy": false
  },
  "fingerprintProfile": "chrome_windows",
  "customHttpHeaders": {}
}

Main input options

• Start URLs — URLs where the crawl begins.

• Max results — Maximum number of successfully collected pages.

• Max crawl depth — Number of internal link levels the crawler follows.

• Include URL patterns — Restricts crawling to matching URL globs.

• Exclude URL patterns — Prevents specific URLs from being crawled.

• Minimum readable text length — Filters out pages with very little extracted text.

• Save Markdown — Adds cleaned Markdown to every page result.

• Save HTML snapshots — Stores HTML versions for debugging and analysis.

• Save screenshots — Captures full-page screenshots when browser rendering is used.

• Browser fallback — Automatically switches to a browser when HTTP extraction fails or JavaScript is required.

• Captcha solving — Optionally enables CapSolver for supported challenges.

• Max concurrency — Controls the number of pages processed simultaneously.

• Proxy configuration — Controls proxy usage and automatic proxy escalation.

• Link status checking — Checks discovered links for broken responses and redirects.

• External link checking — Includes links pointing outside the crawled domain.

Output

The crawler returns one structured dataset item for each successfully collected page. Each result can include the page URL, crawl information, metadata, extracted text, HTML, Markdown, debugging information, individual link status results, and a link-check summary.

Example output

{
  "url": "https://paste.rs/aOPsH",
  "type": "page",
  "isChild": false,
  "crawl": {
    "loadedUrl": "https://paste.rs/aOPsH",
    "loadedTime": "2026-09-04T13:09:13.466Z",
    "referrerUrl": "https://apify.com",
    "httpStatusCode": 200,
    "depth": 0,
    "contentLengthBytes": 913,
    "contentType": "text/plain; charset=utf-8"
  },
  "metadata": {
    "title": "Link Check Fixture",
    "description": null,
    "author": null,
    "keywords": null,
    "languageCode": "",
    "jsonLd": null,
    "canonical": null,
    "robotsDirectives": [],
    "htmlLang": null,
    "viewport": null,
    "openGraph": {},
    "mixedContentRefs": [],
    "headers": {
      "server": "nginx",
      "date": "Fri, 04 Sep 2026 13:09:13 GMT",
      "content-type": "text/plain; charset=utf-8",
      "content-length": "913"
    }
  },
  "screenshotUrl": "",
  "text": "Link Check Fixture Link Check Fixture Static test page for verifying the broken-link and redirect checker.",
  "html": "<div id=\"readability-content\"><div id=\"readability-page-1\" class=\"page\">...</div></div>",
  "htmlUrl": "",
  "markdown": "# Link Check Fixture\n\nStatic test page for verifying the broken-link and redirect checker.",
  "debug": {
    "requestHandlerMode": "http",
    "pageType": "Article",
    "pageTypeReason": "HasMinWordsCrawleeHtmlToText",
    "totalLinks": 6,
    "response": {
      "statusCode": 200
    },
    "snapshots": {},
    "antibot": {
      "tier": "http",
      "vendor": "",
      "proxyTier": "none",
      "solvedWith": [],
      "attempts": 1
    },
    "lengths": {
      "extractus": 0,
      "readableText": 416,
      "none": 891,
      "defuddle": 0,
      "readableTextIfPossible": 416,
      "result": 416,
      "markdown": 675
    }
  },
  "links": [
    {
      "url": "https://httpbin.org/status/200",
      "anchorText": "Healthy link 200",
      "isInternal": false,
      "statusCode": 200,
      "isBroken": false,
      "isRedirect": false,
      "redirectedTo": null,
      "redirectChain": [],
      "checkedAt": "2026-09-04T13:09:13.544Z"
    },
    {
      "url": "https://httpbin.org/status/404",
      "anchorText": "Broken link 404",
      "isInternal": false,
      "statusCode": 404,
      "isBroken": true,
      "isRedirect": false,
      "redirectedTo": null,
      "redirectChain": [],
      "checkedAt": "2026-09-04T13:09:13.544Z"
    },
    {
      "url": "https://httpbin.org/status/500",
      "anchorText": "Broken link 500",
      "isInternal": false,
      "statusCode": 500,
      "isBroken": true,
      "isRedirect": false,
      "redirectedTo": null,
      "redirectChain": [],
      "checkedAt": "2026-09-04T13:09:13.544Z"
    },
    {
      "url": "https://httpbin.org/redirect-to?url=https://example.com&status_code=301",
      "anchorText": "Redirecting link 301",
      "isInternal": false,
      "statusCode": 200,
      "isBroken": false,
      "isRedirect": true,
      "redirectedTo": "https://example.com",
      "redirectChain": [
        "https://httpbin.org/redirect-to?url=https://example.com&status_code=301"
      ],
      "checkedAt": "2026-09-04T13:09:13.544Z"
    },
    {
      "url": "https://this-domain-does-not-exist-abc123xyz.invalid/",
      "anchorText": "Unreachable domain",
      "isInternal": false,
      "statusCode": null,
      "isBroken": true,
      "isRedirect": false,
      "redirectedTo": null,
      "redirectChain": [],
      "checkedAt": "2026-09-04T13:09:13.544Z"
    },
    {
      "url": "https://example.com",
      "anchorText": "Healthy external link",
      "isInternal": false,
      "statusCode": 200,
      "isBroken": false,
      "isRedirect": false,
      "redirectedTo": null,
      "redirectChain": [],
      "checkedAt": "2026-09-04T13:09:13.544Z"
    }
  ],
  "linkCheckSummary": {
    "checked": 6,
    "broken": 3,
    "redirected": 1
  }
}

What the output contains

Output field	Description
url	Original URL processed by the crawler
crawl	Load URL, status code, crawl depth, content type, and timing
metadata	Title, description, canonical URL, robots directives, headers, and other page metadata
text	Clean readable text extracted from the page
html	Processed HTML content
markdown	Markdown version of the extracted page content
links	Links discovered on the page with their audit results
debug	Request, extraction, browser, anti-bot, and content-length details
screenshotUrl	Screenshot location when screenshots are enabled
htmlUrl	Stored HTML snapshot location when HTML snapshots are enabled
linkCheckSummary	Total checked links, broken links, and redirects

The links array is particularly useful for broken link checking, 404 detection, redirect auditing, external link analysis, and SEO site audits. Each link includes its HTTP status, broken status, redirect status, destination, and redirect chain.

🎯 Best Use Cases

🔎 SEO Website Audits

Use the crawler to collect page titles, descriptions, headings, URLs, readable text, canonical information, and links for SEO website content analysis.

It can also identify broken URLs and redirects for technical SEO reviews.

🔗 Broken Link Audits

Use the broken link checker to inspect internal and external links across crawled pages. Status codes help identify 404, 500, unreachable, and redirected URLs.

🤖 AI and RAG Data Preparation

A website content crawler for AI can collect readable text and Markdown for LLM applications, RAG systems, knowledge bases, vector databases, and AI search workflows.

🏆 Competitor Research

Use a competitor website content crawler to collect publicly accessible competitor pages, metadata, URLs, headings, and content for structured research.

📝 Content Audits

Create a website content inventory containing URLs, page titles, descriptions, text, metadata, and links. This supports website content audit, content analysis, and content planning.

🔄 Website Migration

Use the crawler before a website migration to create a record of existing URLs and content. The results can support URL mapping, content inventories, and redirect planning.

📚 Content Research

Use a website content research tool to collect web pages for market research, competitor research, topic research, and structured information discovery.

👨‍💻 Developer Workflows

Developers can use the website content crawler API, website content extraction API, or automated crawling workflows to send collected data into applications, databases, search systems, and processing pipelines.

⚙️ Automated Website Crawling

Automate repeated website content extraction, URL discovery, link checking, and content collection without manually opening each page.

📦 Bulk Website Data Collection

Process multiple Start URLs and collect website content in bulk. This supports bulk website content crawler, batch website content scraper, and multiple website content crawler workflows.

📊 Website Data Analysis

Export structured website data for analytics, content comparison, technical audits, SEO research, and website performance investigations.

💰 How much does Website Content Crawler cost?

Website Content Crawler costs $3.99 per 1,000 results. Your total cost depends on the number of pages successfully collected during the Actor run.

Pay only for the results you collect

For example, collecting 1,000 results costs $3.99, while collecting 500 results costs approximately $2.00 at the same rate. Link checking can generate additional requests during a crawl, but the result-based pricing is based on collected results.

⚖️ Is it legal to scrape websites?

Website crawling is not automatically legal or illegal. The legality depends on the website, the data collected, your purpose, applicable laws, and the site's terms or access restrictions.

Use this website content crawler for legitimate research, auditing, analysis, and automation. Respect copyright, privacy requirements, robots directives, access restrictions, and website terms where applicable.

Do not collect confidential, restricted, or personal information without appropriate authorization. Before starting a large website content scraping project, verify that your planned collection and intended use are permitted.

⚙️ Website Content Crawler Input Parameters

🔗 Start URLs

startUrls contains the website URLs where the crawler begins.

Example:

{

  "startUrls": \[

    "https://example.com/"

  \]

}

🎯 Max Results

maxResults limits the number of page results returned by the crawl.

The supplied configuration uses:

"maxResults": 1

🕸️ Crawl Scope & URL Filters

maxCrawlDepth, includeUrlGlobs, excludeUrlGlobs, keepUrlFragments, and canonical URL settings control which pages the crawler processes.

These options help create focused website crawling, website URL discovery, and website content extraction workflows.

🧹 Content & Text Extraction

The content extraction settings control how readable page content is processed.

The configured CSS selector removes interface elements such as navigation, footer, scripts, styles, SVG elements, images, alerts, banners, dialogs, and modal content.

This produces cleaner output for website text extraction, web content extraction, and AI data preparation.

🗃️ Snapshots, Screenshots & Debug Logs

Screenshots, HTML files, and debug logs can be enabled when additional inspection data is useful.

These options help investigate extraction behavior, page classification, crawling problems, and browser fallback behavior.

🛡️ Anti-Bot Bypass & Headless Browser

Browser fallback can process pages that require JavaScript rendering.

The configuration supports headless browser execution, browser humanization, challenge waiting, and browser fingerprint selection.

🔓 CAPTCHA Solver

capsolverEnabled controls optional CAPTCHA-solving functionality.

You can also configure the maximum number of CAPTCHA solving attempts for a run.

🚀 Speed, Concurrency & Rate Limits

maxConcurrency controls overall parallel processing.

perDomainConcurrency limits simultaneous requests for one domain, while perDomainDelaySecs controls the delay between requests to the same domain.

maxRequestRetries and requestTimeoutSecs control retry and timeout behavior.

🕵️ Proxy Rotation & Browser Fingerprint

Proxy configuration controls how requests are routed. The crawler also supports a fingerprint profile for browser-based requests.

These settings can help with websites that apply different behavior based on request origin or browser characteristics.

🚦 Broken Link & 404 Redirect Audit

Enable checkLinkStatus to check links discovered on each page.

With checkExternalLinks enabled, external links are also checked. The crawler can report:

• HTTP status codes

• Broken links

• Redirects

• Redirect destinations

• Redirect chains

• Unreachable domains

• Anchor text

• Internal or external classification

• Link-check timestamps

This makes the Actor useful for website broken link audits, 404 detection, redirect auditing, and technical SEO workflows.

⏳ During the Actor run

Start with HTTP requests

The crawler first attempts to retrieve pages using fast HTTP requests. This approach reduces browser overhead and works well for static HTML pages.

Use browser fallback when required

If a page requires JavaScript rendering or blocks the HTTP request, the crawler can switch to a headless browser. Browser humanization and fingerprint settings help handle sites that require browser-like requests.

Extract and audit each page

After loading a page, the Actor extracts readable text, Markdown, HTML, and metadata. It also discovers links and can check their live HTTP status, redirects, and unreachable destinations.

Continue crawling within your limits

The crawler follows eligible internal links according to your crawl depth, URL filters, and maximum result settings. Concurrency, retries, request delays, and timeouts control how quickly the run processes pages.

📝 Final Note

Website Content Crawler combines website crawling, content extraction, metadata collection, URL discovery, and link checking in one workflow.

Use it to build website content datasets, perform SEO audits, identify broken links, prepare content for AI and RAG systems, research competitor websites, support migrations, or automate recurring website data collection.

Configure the crawl depth, result limits, URL filters, extraction settings, browser options, concurrency, and link-check settings based on your project scope.

❓ FAQ

What is a Website Content Crawler?

Website Content Crawler is a web crawling tool that discovers website pages and extracts readable text, Markdown, HTML, metadata, URLs, and links.

What data can Website Content Crawler extract?

It can extract page URLs, loaded URLs, titles, descriptions, readable text, Markdown, HTML, headings, metadata, canonical information, language data, links, HTTP status codes, redirects, and crawl information when available.

Can it crawl multiple websites?

Yes. You can provide multiple Start URLs and process them within the configured crawl limits and URL scope.

Can it check broken links?

Yes. Enable link status checking to inspect links discovered on crawled pages. External links can also be checked when checkExternalLinks is enabled.

Can it detect 404 errors?

Yes. The link checker can identify links returning HTTP 404 responses and mark them as broken.

Can it detect redirects?

Yes. The output can identify redirected links and provide the destination URL and redirect chain when available.

Can it crawl JavaScript websites?

Yes. Browser fallback can process pages that require JavaScript rendering. Headless browser settings control this behavior.

Can I extract Markdown from websites?

Yes. Enable saveMarkdown to save processed website content in Markdown format.

Can I extract HTML?

The crawler processes HTML internally and can optionally save HTML files when the corresponding setting is enabled.

Can it extract website metadata?

Yes. Metadata can include page titles, descriptions, author information, language, canonical URLs, robots directives, Open Graph data, and response headers when available.

Can I use it for SEO audits?

Yes. It can collect URLs, page content, metadata, links, status codes, and redirects for SEO website audits, content inventories, and technical analysis.

Can it prepare content for AI or RAG?

Yes. Readable text and Markdown output can be used as source material for LLM applications, RAG pipelines, vector databases, knowledge bases, and AI search systems.

Can developers automate website crawling?

Yes. The Actor can support automated website crawling and API-based workflows for developers who want to integrate extracted website data into other systems.