Under maintenance

Pricing

Pay per usage

Try for free

Go to Apify Store

Website Content Crawler for LLM's

Under maintenance

Try for free

Extract contact information + turn any website into clean, structured content ready for LLM's (e.g. AI lead magnets, RAG pipelines, and outbound personalization). Most web scrapers dump raw HTML or unstructured text. This crawler is purpose-built for LLM's, and optimized for lead generation.

Pricing

Pay per usage

Rating

0.0

(0)

Developer

SalesBlaster AI

Actor stats

Bookmarked

Total users

Monthly active users

3 days ago

Last modified

[2.0.0] - 2026-01-09

Breaking Changes

Input Schema

Changed: startUrls (array) replaced with single startUrl (string)
- Old: { "startUrls": [{ "url": "https://example.com" }] }
- New: { "startUrl": "https://example.com" }
Renamed: maxRequestsPerCrawl → maxPages (clearer naming)
Removed: maxRequestsPerUrl (no longer applicable for single-site crawling)
Added: New configuration options:
- sitemapStrategy: Control sitemap discovery (AUTO, SITEMAP_FIRST, CRAWL_LINKS)
- includePaths: Whitelist URL path prefixes
- excludePaths: Blacklist URL path prefixes (default excludes admin/login/cart)
- excludeUrlRegex: Regex pattern to exclude URLs (default excludes media/binary files)
- chunkingOptions: Configure chunk size and overlap
- extractContacts: Enable/disable contact extraction

Output Format

Changed from full-page records to semantic chunks:

Old output: One record per page with full markdownContent field

New output: Multiple records per page (chunks) with the following schema:

{
  "site": "example.com",
  "url": "https://example.com/about",
  "title": "About Us",
  "chunkIndex": 0,
  "chunkCount": 3,
  "headingPath": "About > Team > Leadership",
  "markdown": "# Team\n\n...",
  "contentType": "marketing",
  "quality": { "score": 85, "textLength": 1500, "linkDensity": 0.03 },
  "crawledAt": "2026-01-09T12:00:00Z",
  "datasetName": "default"
}

Removed fields: metaDescription, linkCount
Added fields: chunkIndex, chunkCount, headingPath, contentType, quality, site

Contact Data Storage

Contacts now stored in KeyValue store under OUTPUT key (not in dataset)

Format:

{
  "summary": {
    "totalEmails": 5,
    "totalPhones": 3,
    "totalSocialLinks": 8,
    "socialBreakdown": { "linkedin": 3, "twitter": 2, "facebook": 3 }
  },
  "contacts": {
    "emails": ["contact@example.com"],
    "phones": ["+14155552671"],
    "social": [{ "platform": "linkedin", "url": "..." }]
  },
  "crawlStats": {
    "pagesVisited": 20,
    "pagesSkipped": 0,
    "errors": 0
  },
  "generatedAt": "2026-01-09T12:00:00Z"
}

Added

LLM-Optimized Content Extraction

Mozilla Readability integration: Extracts main content, removing nav/footer/cookie banners
Fallback selectors: Uses heuristics (main, article, [role="main"]) if Readability fails
Markdown post-processing: Removes boilerplate patterns (cookie policy, copyright, social share buttons)
Quality scoring: Assigns quality score (0-100) based on text length, link density, and structure
Content type classification: Categorizes pages as blog, docs, legal, product, marketing, or other

Semantic Chunking

Heading-based chunking: Splits content by heading hierarchy (h1-h6)
Configurable chunk size: Default 2000 chars with 200 char overlap
Heading path tracking: Maintains breadcrumb trail (e.g., "Home > About > Team")
Smart overlap: Tries to break at sentence/paragraph boundaries

Contact Extraction

Email extraction: Regex-based with false positive filtering (excludes example.com, image extensions)
Phone number extraction: From visible text and tel: links
Phone normalization: E.164 format using libphonenumber-js
Social link extraction: Detects and categorizes LinkedIn, Instagram, Facebook, X/Twitter, TikTok, YouTube, GitHub
Deduplication: Automatic deduplication across all crawled pages

Sitemap Support

Three strategies:
- AUTO: Try sitemap first, fall back to link crawling
- SITEMAP_FIRST: Fail if no sitemap found (strict mode)
- CRAWL_LINKS: Ignore sitemaps, only follow links
Sitemap index support: Recursively parses sitemap indexes
Path filtering: Applies includePaths/excludePaths to sitemap URLs

URL Filtering

Path-based filtering: Include/exclude by URL path prefix
Regex filtering: Exclude URLs matching regex pattern
Default exclusions: Automatically skips common non-content paths (admin, login, wp-admin, cart, checkout)
Default file exclusions: Skips binary/media files (PDF, images, videos, CSS, JS)

Changed

Crawl Behavior

From multi-domain to single-site: Focus on one domain per crawl
Conservative concurrency: Default 5 concurrent requests (was 1000)
Simple page budget: Uses maxPages limit instead of per-domain counters
Same-site enforcement: Uses eTLD+1 (registrable domain) comparison via tldts
Removed KV store domain counters: No longer needed for single-site crawling

Architecture

New module structure: Organized into lib/ with url, sitemap, extraction, chunking, summary modules
Explicit dependencies: All dependencies now listed in package.json (apify, crawlee, playwright, @mozilla/readability, jsdom, etc.)
Type-safe input: Zod schema validation with helpful error messages
CJS build: Kept CommonJS for Apify compatibility

Docker

Multi-stage build: Builder stage for TypeScript compilation, slim production image
Removed root Dockerfile: Only .actor/Dockerfile is used (Apify convention)
Explicit dependency installation: No reliance on base image for dependencies

Removed

Legacy INPUT_SCHEMA.json: Deleted to avoid confusion (use .actor/input_schema.json)
Per-domain KV counters: No longer needed for single-site crawling
Multi-domain support: Simplified to single-site per run
Full body HTML extraction: Now extracts only main content

Migration Guide

If you were using v1.x:

Update input format:

// OLD (v1.x)
{
  "startUrls": [{ "url": "https://example.com" }],
  "maxRequestsPerCrawl": 50000,
  "maxRequestsPerUrl": 5,
  "datasetName": "my-dataset"
}

// NEW (v2.0)
{
  "startUrl": "https://example.com",
  "maxPages": 20,
  "datasetName": "my-dataset"
}

Update downstream consumers:
- Change from processing one record per page to multiple chunks per page
- Access contact data from KV store OUTPUT key (not dataset)
- Adapt to new fields: chunkIndex, chunkCount, headingPath, quality, contentType
Review filtering options:
- Set includePaths if you only want specific sections (e.g., /blog, /docs)
- Adjust excludePaths if defaults don't match your needs
- Use excludeUrlRegex for advanced filtering
Configure chunking (optional):
- Adjust chunkingOptions.maxChars for your LLM's context window
- Increase chunkingOptions.overlapChars for better context preservation
Disable contacts extraction (optional):
- Set extractContacts: false if not needed

Dependencies

New dependencies added:

@mozilla/readability: ^0.5.0 (content extraction)
jsdom: ^24.0.0 (DOM parsing for Readability)
libphonenumber-js: ^1.11.0 (phone normalization)
tldts: ^6.1.0 (domain extraction)
xml2js: ^0.6.2 (sitemap parsing)
zod: ^3.24.0 (input validation)

Existing dependencies (now explicit):

apify: ^3.2.0
crawlee: ^3.11.0
playwright: ^1.40.0
node-html-markdown: ^1.3.0

Technical Details

Module Structure

src/
├── main.ts                    # Actor entry point
├── routes.ts                  # Playwright router
├── types.ts                   # Shared types
└── lib/
    ├── url/                   # URL utilities (normalize, filter, domain)
    ├── sitemap/               # Sitemap discovery and parsing
    ├── extraction/            # Content + contact extraction, quality scoring
    ├── chunking/              # Markdown chunking by headings
    └── summary/               # Contact aggregation

Performance

Default crawl settings: 5 concurrency, 20 pages max
Memory efficient: Chunks are streamed to dataset, not held in memory
Conservative session pool: maxPoolSize = concurrency * 2

Quality Assurance

Input validation with Zod (catches errors early)
Error handling: Graceful degradation, saves error records to dataset
Logging: Structured logs with context for debugging

[1.0.0] - 2025-01-01

Initial multi-domain crawler implementation.

Multi-site crawling with per-domain limits
Full body HTML → Markdown conversion
Dataset output (one record per page)
Aggressive concurrency (1000 workers)

Website Content Crawler for AI and RAG

tropical_quince/website-content-crawler-rag

Donny Nguyen

AI Content Crawler

kai-agent/ai-content-crawler

Crawl any website and get clean, AI-ready content in markdown format. Perfect for RAG pipelines, LLM training data, and vector database ingestion. Features smart chunking, metadata extraction, and multiple output formats.

Kai Agent

Website Content to Markdown for LLM Training

easyapi/website-content-to-markdown-for-llm-training

🚀 Transform web content into clean, LLM-ready Markdown! 📘 Scrape multiple pages, extract main content, and convert to Markdown format. Perfect for AI researchers, data scientists, and LLM developers. Fast, efficient, and customizable. Supercharge your AI training data today! 🌐📝🧠

EasyApi

244

5.0

Web-to-Markdown Generator for AI & RAG Pipelines

profitstack/web-to-markdown-generator-for-ai-rag-pipelines

Convert any website into clean, heading-based chunking, LLM-ready Markdown for RAG and AI agents.

Manas Mantri

Website To Markdown

smart_api/website-to-markdown

Convert any webpage into clean, LLM-ready Markdown in seconds — perfect for AI training data, RAG pipelines, and content archiving.

SmartApi

5.0

Website Content Crawler

nexgendata/website-content-crawler

Crawl any website and extract clean text content, headings, links, and metadata. Configurable depth, domain restriction, and output formats. Ideal for AI/LLM training data preparation and content analysis.

Stephan Corbeil

Website Content Crawler

apify/website-content-crawler

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.

Apify

103K

4.6

🔥 AI HTML to JSON Extractor (Fast, Free LLM for Data)

autoscaler/ai-html-to-json-extractor

Eliminate messy HTML cleanup and high LLM costs. This Actor uses a high-speed, zero-cost large language model to turn unstructured content (HTML, text, reviews, blog posts) into valid, structured JSON.

Mooo

LLM-Ready Web Scraper

devoted_helix/llm-web-scraper

Convert web pages to clean, LLM-friendly text. Perfect for RAG pipelines, AI chatbot training, and fine-tuning datasets. Removes ads,menus, and clutter automatically.

batuhan senavci

Website Content Crawler

jasondev/website-content-crawler

A powerful web crawler that extracts text content from websites, optimized for AI models, Large Language Models (LLMs), vector databases, and Retrieval-Augmented Generation (RAG) pipelines.

Jason Giang

Website Content Crawler for LLM's

Changelog

[2.0.0] - 2026-01-09

Breaking Changes

Input Schema

Output Format

Contact Data Storage

Added

LLM-Optimized Content Extraction

Semantic Chunking

Contact Extraction

Sitemap Support

URL Filtering

Changed

Crawl Behavior

Architecture

Docker

Removed

Migration Guide

Dependencies

Technical Details

Module Structure

Performance

Quality Assurance

[1.0.0] - 2025-01-01

You might also like

Website Content Crawler for AI and RAG

AI Content Crawler

Website Content to Markdown for LLM Training

Web-to-Markdown Generator for AI & RAG Pipelines

Website To Markdown

Website Content Crawler

Website Content Crawler

🔥 AI HTML to JSON Extractor (Fast, Free LLM for Data)

LLM-Ready Web Scraper

Website Content Crawler

Changelog

[2.0.0] - 2026-01-09

Breaking Changes

Input Schema

Output Format

Contact Data Storage

Added

LLM-Optimized Content Extraction

Semantic Chunking

Contact Extraction

Sitemap Support

URL Filtering

Changed

Crawl Behavior

Architecture

Docker

Removed

Migration Guide

Dependencies

Technical Details

Module Structure

Performance

Quality Assurance

[1.0.0] - 2025-01-01