Pricing

from $1.20 / 1,000 results

Try for free

Go to Apify Store

Website Content Crawler – Markdown for AI, Emails & Contacts

Try for free

Website content crawler that turns a whole site into clean Markdown for LLM and RAG pipelines, with the role emails, social profiles and WhatsApp numbers found on each page. Seeded from the sitemap, honours robots.txt, plain HTTP with no browser.

Pricing

from $1.20 / 1,000 results

Rating

0.0

(0)

Developer

Locomint

Actor stats

Bookmarked

Total users

Monthly active users

5 days ago

Last modified

Categories

Developer tools

SEO tools

Websites to crawl

startUrls

Required

One or more site homepages (full URLs with https://). Each site is crawled separately, up to 50 sites per run.

Type:array

Max pages per site

maxPagesPerSite

Optional

Stop after this many pages on a site. You pay per page delivered.

Type:integer

Minimum:1

Maximum:2000

Default:50

Max link depth

maxDepth

Optional

How far to follow links: 0 = follow none, 1 = pages linked from the start page, and so on. Sitemap pages are crawled whatever this is set to; turn off 'Use the sitemap' for a start-page-only run.

Type:integer

Minimum:0

Maximum:10

Default:3

Include subdomains

includeSubdomains

Optional

Also follow links to blog.example.com, shop.example.com, etc.

Type:boolean

Default:false

Only crawl URLs containing

includeUrlPatterns

Optional

Optional. Keep only URLs containing one of these texts, e.g. ["/docs/"]. The start page is always included.

Type:array

Skip URLs containing

excludeUrlPatterns

Optional

Optional. Skip URLs containing any of these texts, e.g. ["/tag/", "?page="].

Type:array

Use the sitemap

useSitemap

Optional

Seed the crawl from sitemap.xml when the site has one - finds pages that are not linked from the homepage.

Type:boolean

Default:true

Respect robots.txt

respectRobots

Optional

Filter pages the site's robots.txt disallows out of the queue before they are fetched. A disallowed page is never fetched or charged either way: every page is checked against robots.txt right before its fetch.

Type:boolean

Default:true

Content format

outputFormat

Optional

Markdown keeps headings, lists, links and code; plain text is just the words.

Type:string

Default:markdown

Options:

markdowntext

Max characters per page

maxChars

Optional

Longer content is cut here and marked truncated.

Type:integer

Minimum:500

Maximum:500000

Default:50000

Extract business contacts

includeContacts

Optional

Email addresses (company mailboxes first; named staff addresses are also listed in named_emails), social profiles, staff profile links (named_profiles), WhatsApp, contact page URL, tech stack - from every page, no extra cost.

Type:boolean

Default:true

Include outgoing links

includeLinks

Optional

Add the list of links found on each page (up to 500).

Type:boolean

Default:false

Parallel fetches

concurrency

Optional

Pages fetched at the same time per site.

Type:integer

Minimum:1

Maximum:10

Default:5

Website Email & Social Media Scraper

saregaa/website-contacts-scraper

Extract emails and social media links (Facebook, Instagram, LinkedIn, TikTok, YouTube and more) from any list of websites. Perfect for enriching Google Maps leads, CRM exports, or business directories. Pay only for results actually found - no subscription, no wasted spend.

Saregaa

⚡ Web to LLM Clean Markdown & RAG Crawler

unrivaled_fortress/llm-markdown-clean-web-crawler

Convert any webpage into clean, LLM-ready markdown. Strips ads, navbars, and cookie banners to optimize AI context tokens for ChatGPT, Claude, and RAG.

David A

Website Content Extractor - Markdown for LLM, RAG, Vector DB

snow_leo_data/website-to-markdown

Readable text from 73.1% of 52 live pages over plain HTTP, including 10 of 10 documentation sites. Website to LLM text and a website to markdown converter in one run: clean Markdown, chunks with overlap for a vector database, 46 flat fields, and what changed since the last run.

Snow Leo Data

Public Business Contact Extractor

mottier-automation/public-business-contact-extractor

Extract public business emails, phone numbers, WhatsApp links, social profiles, and contact pages from websites supplied by the user.

lautaro mottier

X (Twitter) Followers Contact Extractor - Emails & Websites

seemuapps/x-followers-contact-extractor

Extract emails, websites, and social links from the bios of any X (Twitter) account's followers or following. Filter by keyword, location, and follower count.

Seemu Scraping

Website Email Scraper – Contact Details, Socials & WhatsApp

locomint/website-content-contact-extractor

Contact details scraper for any list of website URLs: role emails, social profiles, WhatsApp click-to-chat numbers, the contact page and tech stack, plus the page as clean Markdown or text from the same fetch. No browser, robots.txt honoured.

Locomint

Website Email, Phone & Social Finder

scrapesignal_labs/website-contact-social-finder

Find public business emails, phone numbers, contact forms, addresses, and LinkedIn, Instagram, Facebook, X, YouTube, TikTok, and GitHub profiles from website URLs.

ScrapeSignal Labs

Website Crawler - Site Pages to Markdown and Text

leadproof/website-crawler

Crawl one public website within page, depth and time limits: clean text, Markdown, links and metadata per page, plus a coverage report. Private build under development.

Lead Proof

Website Content to Markdown & llms.txt Crawler - AI & RAG Ready

abi4/website-content-llmstxt-crawler

High-performance, lightweight asynchronous crawler converting web pages into clean Markdown, computing token estimates, and generating standard /llms.txt manifests for AI and RAG ingestion.