Website Content Crawler to Markdown for AI & LLMs
Pricing
from $0.12 / 1,000 pages
Website content crawler for AI, RAG and LLMs: give start URLs or a sitemap, get every page's clean Markdown and text with title, description, headings, author, dates, canonical URL and links. Fast HTTP crawling, URL globs, depth and page limits, sitemaps, llms.txt and PDFs. No login.
Pages to start from. The crawler follows links to their sub-pages (same path), e.g. https://docs.example.com/guide. A sitemap.xml or llms.txt URL crawls every page it lists.
Most pages to save (each one is charged once). Empty = the whole site.
Add the pages listed in each site's sitemap (from robots.txt or /sitemap.xml) that fall under your start URLs, to find pages no link points to.
How many links away from a start URL to go: 0 = only the start URLs, 1 = pages they link to, and so on.
Also crawl links that match, e.g. https://example.com/blog/** (** = anything, * = anything but /). Empty = only sub-pages of the start URLs.
[]Never crawl links that match, e.g. https://example.com/**/tag/** or **?page=*. Start URLs are always crawled.
[]Most URLs to request, including ones that fail or are skipped, as a safety stop on huge sites. Empty = no limit.
Also read each site's /llms.txt (an index for AI tools) and the pages it links to.
Save only pages updated after this date (from the sitemap, the page's own dates or the server), for refresh runs, e.g. 7 days. Empty = every page.
Skip URLs the site's robots.txt disallows for all crawlers (listed as free robotsTxtDisallowed rows in the errors dataset).
Also save linked documents: PDFs as rows with their text; Word, Excel, PowerPoint, CSV, RTF and EPUB files stored and linked in the fileUrl column (text files with their text). Off = files are skipped.
Save linked files of these content types, comma separated, wildcards allowed (image/*, text/csv). One row per file with its fileUrl, even when the file is outside your start URLs. Empty = Include files' documents.
Auto renders only JavaScript app shells in headless Chrome (renderedInBrowser). Browser compares each site's first pages with their HTML and reads the site over HTTP when the text is the same; screenshots render every page.
Saves a screenshot (JPEG, top of the page) of each page rendered in the browser, linked in the screenshotUrl column. With Browser for every page, that's every page.
The page's main content as Markdown (headings, lists, tables, code, links) in the markdown column.
Adds the html column: the main content's HTML after cleaning. Makes rows much larger.
Saves each page's cleaned HTML to the run's key-value store and links it in the htmlUrl column.
Treat example.com/page#a and example.com/page#b as two pages (single-page apps with # routes). Off = one page.
Off: pages that name the same canonical URL are saved once. On: every URL is its own page (for sites with a wrong canonical tag).
How to pick the content to keep after the elements below are removed.
Elements to drop before extracting, e.g. .ads, #comments. The default removes navigation, footers, scripts, banners and dialogs.
Extract only matching elements, e.g. article, .post-body. Empty = the main content.
Leave as is: Apify datacenter proxy, fresh IPs on refusals. Pick residential only for sites that block datacenter IPs.
{ "useApifyProxy": true}Kept for inputs saved from other crawlers: cheerio / jsdom = plain HTTP only; browser types work like Rendering: Auto.
Most requests at once (default: set from the run's memory).
Seconds per request.