Pricing

from $1.20 / 1,000 page conversions

Try for free

Go to Apify Store

Website Content to Markdown Crawler for LLMs

Try for free

Website content crawler: crawl a site and convert pages to clean Markdown for LLMs, RAG and AI training. HTTP only, no browser. Sitemap-first discovery, main content only, tables and code kept. Adds title, language, word count, token estimate, RAG chunks and llms.txt files.

Pricing

from $1.20 / 1,000 page conversions

Rating

0.0

(0)

Developer

Malik Said

Actor stats

Bookmarked

Total users

Monthly active users

17 hours ago

Last modified

Categories

Developer tools

Start URLs

startUrls

Optional

Pages where the crawl starts. By default the crawl stays inside the folder of each start URL. For example, https://example.com/docs/intro crawls pages under https://example.com/docs/.

Type:array

Default:

[
  {
    "url": "https://crawlee.dev/js/docs/quick-start"
  }
]

Maximum pages

maxPages

Optional

Stop after converting this many pages. The default keeps test runs short and cheap. Set 0 for no limit.

Type:integer

Minimum:0

Default:30

Crawl scope

scope

Optional

Which links the crawl follows. 'Same folder' stays under the start URL's folder. 'Same domain' covers the whole host (www. is ignored). 'Domain and subdomains' also covers blog.example.com and similar.

Type:string

Default:path

Options:

pathdomainsubdomains

Include URL patterns

includeGlobs

Optional

Optional. Crawl only URLs that match one of these patterns. They replace the crawl scope above. * matches any text except /, ** matches any text. Example: https://example.com/docs/**

Type:array

Default:

[]

Exclude URL patterns

excludeGlobs

Optional

Optional. Skip URLs that match one of these patterns. Example: https://example.com/**/tag/**

Type:array

Default:

[]

Read sitemaps first

useSitemaps

Optional

Read the site's XML sitemaps (from robots.txt, or /sitemap.xml) and queue their in-scope pages before following links. This finds pages faster and finds pages that no link points to.

Type:boolean

Default:true

Use the site's llms.txt

useLlmsTxt

Optional

Also read the site's own /llms.txt and crawl the in-scope pages it links to. robots.txt still applies.

Type:boolean

Default:false

Follow links

followLinks

Optional

Follow in-scope links found on each page. Turn off to convert only the start URLs and sitemap pages.

Type:boolean

Default:true

Maximum link depth

maxCrawlDepth

Optional

How many links away from a start URL the crawl may go. Start URLs are depth 0. Sitemap pages are depth 1.

Type:integer

Minimum:0

Default:20

Main content only

mainContentOnly

Optional

Keep only the main article text (Mozilla Readability). Menus, headers, footers and sidebars are removed. Turn off to convert the whole page body without header, footer and navigation.

Type:boolean

Default:true

Content selector

contentSelector

Optional

Optional. Convert only the elements that match this CSS selector, for example article.post or #content. Pages where it matches nothing use the normal extraction. Leave empty to let the crawler find the main content.

Type:string

Default:

Remove elements

removeSelectors

Optional

Optional. Drop elements that match these CSS selectors before conversion, for example .ads, #comments, .related-posts. Separate selectors with commas.

Type:string

Default:

Keep links

includeLinks

Optional

Keep links as Markdown links with absolute URLs. Turn off to keep only the link text.

Type:boolean

Default:true

Keep images

includeImages

Optional

Keep images as Markdown image links. Off by default, because image links rarely help an LLM.

Type:boolean

Default:false

Include linked PDF files

includePdfs

Optional

Also convert in-scope PDF files linked from crawled pages to Markdown (text layer only, no OCR). Each PDF counts as one page. Files over 20 MB, encrypted files and files robots.txt disallows are skipped free. Up to 300 pages of each PDF are read.

Type:boolean

Default:false

Split into chunks for RAG

chunking

Optional

Add a list of chunks to each page, ready for a vector database. Each chunk has its heading path and a token estimate. A new chunk starts at each H1 to H3 heading and when the size limit is reached.

Type:boolean

Default:true

Chunk size (tokens)

chunkSize

Optional

Largest chunk, in estimated tokens. The estimate is about 4 characters per token for English.

Type:integer

Minimum:50

Maximum:8000

Default:500

Chunk overlap (tokens)

chunkOverlap

Optional

Estimated tokens repeated from the end of the previous chunk. At most half the chunk size.

Type:integer

Minimum:0

Maximum:2000

Default:0

Save llms.txt and llms-full.txt

saveLlmsTxt

Optional

Save an llms.txt index of all converted pages and an llms-full.txt file with all their Markdown to the run's key-value store. llms-full.txt stops at 20 MB.

Type:boolean

Default:true

Change-only mode

changeOnly

Optional

Save only pages that are new or whose Markdown changed since the last change-only run with the same state key. Unchanged pages are charged at a lower price. Schedule the Actor with this on to keep a knowledge base fresh.

Type:boolean

Default:false

Modified since

modifiedSince

Optional

Only load pages changed on or after this date, for example 2026-10-01 or 30 days. Uses the sitemap's last-modified date and the If-Modified-Since header. Start URLs and pages with no date are always loaded. Skipped pages cost nothing.

Type:string

State key

stateKey

Optional

Name of the memory that change-only mode uses. Use a different key for each site you watch.

Type:string

Default:default

Parallel requests

maxConcurrency

Optional

Most pages loaded at the same time. A robots.txt Crawl-delay lowers this to 1.

Type:integer

Minimum:1

Maximum:20

Default:3

Requests per minute

maxRequestsPerMinute

Optional

Most page requests per minute across the run. A robots.txt Crawl-delay lowers this. Keep it polite on sites you do not own.

Type:integer

Minimum:1

Maximum:600

Default:120

Proxy

proxyConfiguration

Optional

Optional. Most sites need no proxy. A proxy does not get past bot checks: pages behind a challenge are skipped.

Type:object

Default:

{
  "useApifyProxy": false
}

Instagram Scraper

apify/instagram-scraper

Extract Instagram posts, reels, profiles, places, hashtags, carousels, and comments. Get data from Instagram using one or more Instagram URLs or search queries: content, context, metrics, metadata. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Apify

446K

4.7

(603)

Instagram Profile Scraper

apify/instagram-profile-scraper

Scrape all Instagram profile info. Just add Instagram usernames, IDs or URLs and extract name, join date, number of followers, location, bio, website, related profiles, video&post count, latest posts. Export scraped data, schedule scraper via API, and integrate with other tools or AI workflows.

Apify

261K

4.7

(175)

TikTok Scraper

clockworks/tiktok-scraper

Extract data from TikTok videos, hashtags, and users. Use URLs or search queries to scrape TikTok profiles, hashtags, posts, URLs, shares, followers, hearts, names, video, and music-related data. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Clockworks

354K

4.7

(370)

Google Maps Scraper

compass/crawler-google-places

Extract data from thousands of Google Maps locations and businesses, including reviews, reviewer details, images, contact info, including full name, email, and job title, opening hours, prices & more. Export data, run via API, schedule and monitor runs, or integrate with other tools.

Compass

660K

4.7

(1,818)

LinkedIn Profile Scraper + Email (No Cookies)

harvestapi/linkedin-profile-scraper

Extract detailed information from LinkedIn profiles in bulk, including complete work experience, education history, skills and more. No cookies or account required.

HarvestAPI

86K

4.6

(93)

Google Search Results Scraper

apify/google-search-scraper

Scrape Google Search Engine Results Pages (SERPs). Select the country or language and extract organic and paid results, AI Mode, AI overviews, ads, queries, People Also Ask, prices, reviews, like a Google SERP API. Export data, run the scraper via API, schedule runs, or integrate with other tools.

Apify

213K

4.6

(220)

Instagram Reel Scraper

apify/instagram-reel-scraper

Scrape or download Instagram reels. Just add Instagram profile or reel URL, and get caption, timestamp, transcript, hashtags, mentions, tagged users, comments, likes, shares, views, duration, and downloaded video. Export scraped data, schedule via API, and integrate with other tools or AI workflows.

Apify

173K

4.4

(103)

Facebook Posts Scraper

apify/facebook-posts-scraper

Extract posts, videos, and engagement metrics from Facebook pages. Get text captions, reactions, video transcripts, images, external links, collaborators, and more from Facebook pages and profiles. Export ad data, schedule runs via API, and integrate with other tools or AI workflows.

Apify

130K

4.7

(236)

Instagram Post Scraper

apify/instagram-post-scraper

Scrape Instagram posts. Just add one or more Instagram usernames and get your data in seconds including caption, metrics, images, mentions, coauthors, recent comments, sponsored status, video duration, views. Export scraped data, schedule scraper via API, integrate with other tools or AI workflows.

Apify

147K

4.3

(138)

TikTok Profile Scraper

clockworks/tiktok-profile-scraper

Extract data from TikTok profiles and videos they post. Get TikTok profile data, URLs, numbers of shares, followers, comments, hearts, video and music metadata. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Clockworks

58K

4.7

(73)