Pricing

from $2.00 / 1,000 extracted rag pages

Try for free

Go to Apify Store

Website to Markdown & RAG Crawler: Clean Text for LLM

Try for free

Convert entire websites into clean Markdown and semantic token chunks for LLMs, RAG vector pipelines, Claude, and OpenAI embeddings.

Pricing

from $2.00 / 1,000 extracted rag pages

Rating

0.0

(0)

Developer

Lakshay Papneja

Actor stats

Bookmarked

Total users

Monthly active users

19 hours ago

Last modified

Categories

Developer tools

Automation

Start URLs

startUrls

Required

List of URLs to extract clean Markdown and RAG chunks from.

Type:array

Default:

[
  {
    "url": "https://docs.apify.com/platform/actors"
  }
]

Max Pages to Crawl

maxPages

Optional

Maximum number of pages to process.

Type:integer

Minimum:1

Maximum:1000

Default:10

Follow Internal Links

followInternalLinks

Optional

Whether to automatically discover and crawl internal documentation/article links on the same domain.

Type:boolean

Default:false

Target RAG Chunk Size (Tokens)

chunkSizeTokens

Optional

Target token count per semantic chunk for vector database ingestion (approx. 4 chars = 1 token).

Type:integer

Minimum:100

Maximum:4000

Default:500

Chunk Overlap (Tokens)

chunkOverlapTokens

Optional

Token overlap between consecutive chunks to preserve context.

Type:integer

Minimum:0

Maximum:500

Default:50

CSS Selectors to Strip

removeSelectors

Optional

Comma-separated tags/selectors to remove before converting to Markdown.

Type:string

Default:nav, footer, header, script, style, noscript, iframe, .cookie-banner, .ads, #sidebar

Instagram Scraper

apify/instagram-scraper

Extract Instagram posts, reels, profiles, places, hashtags, carousels, and comments. Get data from Instagram using one or more Instagram URLs or search queries: content, context, metrics, metadata. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Apify

457K

4.7

(603)

TikTok Scraper

clockworks/tiktok-scraper

Extract data from TikTok videos, hashtags, and users. Use URLs or search queries to scrape TikTok profiles, hashtags, posts, URLs, shares, followers, hearts, names, video, and music-related data. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Clockworks

369K

4.7

(370)

LinkedIn Profile Scraper + Email (No Cookies)

harvestapi/linkedin-profile-scraper

Extract detailed information from LinkedIn profiles in bulk, including complete work experience, education history, skills and more. No cookies or account required.

HarvestAPI

90K

4.6

(93)

Google Maps Scraper

compass/crawler-google-places

Extract data from thousands of Google Maps locations and businesses, including reviews, reviewer details, images, contact info, including full name, email, and job title, opening hours, prices & more. Export data, run via API, schedule and monitor runs, or integrate with other tools.

Compass

673K

4.7

(1,820)

Google Search Results Scraper

apify/google-search-scraper

Scrape Google Search Engine Results Pages (SERPs). Select the country or language and extract organic and paid results, AI Mode, AI overviews, ads, queries, People Also Ask, prices, reviews, like a Google SERP API. Export data, run the scraper via API, schedule runs, or integrate with other tools.

Apify

221K

4.6

(220)

Instagram Profile Scraper

apify/instagram-profile-scraper

Scrape all Instagram profile info. Just add Instagram usernames, IDs or URLs and extract name, join date, number of followers, location, bio, website, related profiles, video&post count, latest posts. Export scraped data, schedule scraper via API, and integrate with other tools or AI workflows.

Apify

268K

4.7

(175)

YouTube Scraper

streamers/youtube-scraper

YouTube crawler and video scraper. Alternative YouTube API with no limits or quotas. Extract and download channel name, likes, number of views, and number of subscribers.

Streamers

191K

4.8

(196)

Instagram Reel Scraper

apify/instagram-reel-scraper

Scrape or download Instagram reels. Just add Instagram profile or reel URL, and get caption, timestamp, transcript, hashtags, mentions, tagged users, comments, likes, shares, views, duration, and downloaded video. Export scraped data, schedule via API, and integrate with other tools or AI workflows.

Apify

179K

4.4

(102)

Tweet Scraper V2 - X Scraper / Twitter Scraper / X.com Scraper

apidojo/tweet-scraper

Lightning-fast search, URL, list, and profile scraping, with customizable filters. At $0.40 per 1000 tweets, and 30-80 tweets per second, it is ideal for researchers, entrepreneurs, and businesses! Get comprehensive insights from Twitter (X) now!