Pricing

from $1.00 / 1,000 page crawleds

Try for free

Go to Apify Store

Website to Markdown Crawler for LLM & RAG (llms.txt)

Try for free

Crawl any website and get every page as clean Markdown for LLMs, RAG and AI agents. Main-content extraction, sitemap discovery, RAG chunks, llms.txt and llms-full.txt. Fast HTTP crawler with a flat price per page: no browser, no compute surprises.

Pricing

from $1.00 / 1,000 page crawleds

Rating

0.0

(0)

Developer

Catalin Friptu

Actor stats

Bookmarked

Total users

Monthly active users

a day ago

Last modified

Categories

Developer tools

SEO tools

Start URLs

startUrls

Optional

Where the crawl starts. Only links under each start URL's path are followed: https://example.com/docs/ crawls the docs section, https://example.com/ crawls the whole site.

Type:array

Max pages

maxPages

Optional

Stop after saving this many pages. You pay only for saved pages, so this also caps the cost of a run.

Type:integer

Minimum:1

Maximum:1000000

Default:100

Max crawl depth

maxCrawlDepth

Optional

How many links away from a start URL the crawler may go. 0 = only the start URLs themselves. Pages found in sitemaps count as depth 1.

Type:integer

Minimum:0

Maximum:1000

Default:20

Use sitemaps

useSitemaps

Optional

Also queue every in-scope URL listed in the site's sitemap.xml (found through robots.txt). Finds pages that no menu links to.

Type:boolean

Default:false

Include URLs (globs)

includeUrlGlobs

Optional

Crawl only URLs that match one of these patterns, instead of the default 'under the start URL' rule. Example: https://example.com/{blog,docs}/** (** matches anything, * matches anything except a slash).

Type:array

Exclude URLs (globs)

excludeUrlGlobs

Optional

Never crawl URLs that match one of these patterns. Example: /tag/ or https://example.com/blog/page/*

Type:array

Respect robots.txt

respectRobotsTxt

Optional

Skip pages the site's robots.txt disallows. Turn off only for your own site.

Type:boolean

Default:true

Max concurrency

maxConcurrency

Optional

Pages fetched at the same time. Lower it for small or slow sites.

Type:integer

Minimum:1

Maximum:50

Default:10

Content

contentMode

Optional

Main content: the article or documentation text only, without menus, cookie banners, sidebars and footers (best for RAG). Full page: everything in the page body.

Type:string

Default:main

Options:

mainfull

Keep links in Markdown

includeLinks

Optional

Keep hyperlinks as text. Off gives cleaner text for embeddings.

Type:boolean

Default:false

Keep images in Markdown

includeImages

Optional

Keep images as alt .

Type:boolean

Default:false

Also save plain text

saveText

Optional

Add a 'text' field with the content as plain text (no Markdown syntax).

Type:boolean

Default:false

Also save raw HTML

saveHtml

Optional

Add an 'html' field with the page's original HTML. Makes the dataset much bigger.

Type:boolean

Default:false

Generate llms.txt

generateLlmsTxt

Optional

Write llms.txt (an index of the crawled pages in the llmstxt.org format) and llms-full.txt (every page's Markdown in one file) to the run's key-value store. Free.

Type:boolean

Default:true

Detect language

detectLanguage

Optional

Fill 'language' from the text when the page doesn't declare one in its html lang attribute.

Type:boolean

Default:true

Chunk size (characters)

chunkSize

Optional

Also split each page's Markdown into chunks of about this many characters, ready for embeddings (a 'chunks' field). 0 turns chunking off. 1000-2000 suits most embedding models.

Type:integer

Minimum:0

Default:0

Chunk overlap (characters)

chunkOverlap

Optional

How many characters neighbouring chunks share, so a sentence cut at a boundary still appears whole in one chunk.

Type:integer

Minimum:0

Default:200

Tweet Scraper V2 - X Scraper / Twitter Scraper / X.com Scraper

apidojo/tweet-scraper

Lightning-fast search, URL, list, and profile scraping, with customizable filters. At $0.40 per 1000 tweets, and 30-80 tweets per second, it is ideal for researchers, entrepreneurs, and businesses! Get comprehensive insights from Twitter (X) now!

API Dojo

113K

3.9

(196)

Facebook Search Scraper — People, Pages, Posts & More

memo23/facebook-search-scraper

The most complete Facebook search: people, pages, places, events, posts, videos and top — in one actor, no login and no cookies. Get names, URLs, profile pictures, verified badges, event dates, and post text with authors. Auto-paginates across results. Export to JSON, CSV or Excel.

Muhamed Didovic

5.6K

5.0

(2)

Facebook Search Posts

scrapeforge/facebook-search-posts

Search Facebook posts, pages, groups, people, videos and events by keyword. Returns detailed data including reactions breakdown, media URLs, and engagement metrics.

ScrapeForge

68K

3.6

(10)

TikTok Scraper

clockworks/tiktok-scraper

Extract data from TikTok videos, hashtags, and users. Use URLs or search queries to scrape TikTok profiles, hashtags, posts, URLs, shares, followers, hearts, names, video, and music-related data. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Clockworks

354K

4.7

(370)

Youtube Scraper (Pay Per Result)

apidojo/youtube-scraper

Experience unparalleled efficiency and depth with its ultra-fast capabilities in searching, URL, playlist, channel, and profile scraping. It comes equipped with a variety of filters to customize your data collection precisely. Plus, it's incredibly affordable at only $0.50 per 1000 videos!

API Dojo

35K

3.3

(21)

TikTok Search Scraper: Keywords, Hashtags, Creators

xmolodtsov/tiktok-search-scraper

Scrape TikTok search results by keyword or hashtag and discover creators. Pay only per result. Views, likes, comments, shares, bookmarks, channel stats, music, subtitles and direct video URLs as JSON. Output compatible with popular TikTok scrapers. No proxies or login needed. Works via API and MCP.

GoFetch

66K

5.0

(1)

Search Threads By Keywords

watcher.data/search-threads-by-keywords

Fast and reliable scraper for searching posts on Threads by Meta. Perfect for social media monitoring, research, and business intelligence.

Giichi Arai

8.4K

3.4

(10)

Twitter (X.com) Scraper Unlimited: No Limits

apidojo/twitter-scraper-lite

Introducing Twitter Scraper Unlimited, the most comprehensive Twitter data extraction solution available. Our enterprise-grade scraper offers unmatched capabilities with a transparent event-based pricing model, making it perfect for both small-scale and large-scale data extraction needs.

API Dojo

40K

4.8

(100)

Threads Search Scraper

igview-owner/threads-search-scraper

Powerful Meta Threads search scraper for posts and users. Filter by keywords or usernames, export structured data for analytics, research, monitoring and reporting.

Sachin Kumar Yadav

12K

3.5

(6)

Facebook Search Scraper

igview-owner/facebook-old-posts-search

Search Facebook posts in seconds. Find viral posts, photos and videos by keyword, date range and engagement – perfect for research, content ideas, brand monitoring and sentiment analysis.

Sachin Kumar Yadav

3.2K

4.8

(3)