Website Content Crawler to Markdown for AI & LLMs

Website Content Crawler: Markdown & Text for AI. Give it a site, a docs section or a sitemap, and get one row per page with its clean Markdown and text, title, description, headings, dates, canonical URL and links, ready for RAG, LLM knowledge bases, search indexes and content audits.

🏷️ Title	🔗 Page	🔢 Words	📑 Headings	🪜 Depth
Web scraping basics for JavaScript devs | Academy | Apify Documentation	https://docs.apify.com/academy/scraping-basics-javascript	981	# Web scraping basics for JavaScript devs, ## What we'll do	1
API scraping | Academy | Apify Documentation	https://docs.apify.com/academy/api-scraping	890	# API scraping, ## What's an API?	1
Expert scraping with Apify | Academy | Apify Documentation	https://docs.apify.com/academy/expert-scraping-with-apify	489	# Expert scraping with Apify, ## Preparations	1

Price: $0.15 per 1,000 pages on Starter ($0.18 Free, $0.13 Scale, $0.12 Business). Pages that need JavaScript and are rendered in headless Chrome add $3.09 per 1,000 (renderedInBrowser: true); most sites never need it. The prefilled run (50 pages) costs less than $0.01. You pay only for pages saved: URLs that fail, are blocked or repeat a page you already have are free.

What you get

• Clean content: the page's main content as Markdown (headings, lists, tables, code blocks with their language, lines and indentation as the page shows them, links, images) and as plain text. Menus, footers, cookie banners, pop-ups and button labels are removed.
• Metadata: title, meta description, author, keywords, language, canonical URL, Open Graph tags, JSON-LD (raw and its schema.org types), published and modified dates, meta robots.
• Structure and links: the headings outline, internal and external links, the page each URL was found on, its link depth and the start URL that led to it.
• Change tracking: a content hash of the text, the server's Last-Modified, and the sitemap's lastmod, so you can see what changed between runs.

How it crawls

1. Each start URL crawls itself and its sub-pages (same path), the way most site crawlers do. Add Include URL patterns to go wider and Exclude URL patterns to skip sections (** matches anything, * anything but /).
2. Turn on Also crawl the sitemap to add the pages a site lists in its sitemap (from robots.txt or /sitemap.xml), or give a sitemap.xml / llms.txt URL as a start URL to crawl exactly what it lists.
3. Max pages caps what you pay for, Max link depth how far links are followed, Max URLs to fetch the requests themselves.
4. Pages are read over fast plain HTTP with a real browser's network fingerprint through Apify Proxy, many at once (the first row is in the dataset within about a second on most sites).
5. JavaScript pages: a page whose HTML is an empty app shell (React, Vue, Swagger UI, ...) is rendered in headless Chrome automatically and saved like any other page (renderedInBrowser: true). When a site's app loads each page as JSON from an address built from the page's path, the crawler learns that address from 3 rendered pages and reads the site's other app pages straight from it, without the browser (plain rows, not charged as rendered; every 25th page is rendered again as a check). Set Rendering to HTTP only to never start a browser, or Browser for every page to render pages in the browser: the first 3 pages of each site are compared with their HTML, and when the HTML already holds the same text (server-rendered sites), that site's pages are read over HTTP, not charged as rendered, and re-checked every 20th page (with Save screenshots on, every page is rendered). For browser-heavy crawls give the run 2-4 GB of memory (more CPU).
6. A page is saved once per run, even when several URLs lead to it (redirects, http and https, tracking parameters, the same canonical URL).
7. Rows come out in crawl order (breadth-first, links in page order), so the same input gives the same pages in the same order, and a page cap keeps the pages closest to your start URL.

Refresh runs: set Only pages changed since (a date or e.g. 7 days) to save only pages updated since then, from the sitemap's lastmod, the page's own dates or the server's Last-Modified.

Files: turn on Include files to save linked PDFs as rows with their text (one Markdown section per PDF page), and Word, Excel, PowerPoint, OpenDocument, RTF, EPUB and CSV files as rows linking the downloaded file (fileUrl; CSV and other text files with their text). Or pick the content types in File types to save (saveContentTypes): image/* for images, application/*, text/csv for documents, or any list of types with wildcards. Linked files are saved even when they are outside your start URLs.

Screenshots: turn on Save screenshots for a JPEG of the first screen of every page rendered in the browser (screenshotUrl).

What isn't saved (and is free)

Every URL that doesn't become a row is listed in the run's ⚠️ URLs not saved dataset with the reason, never charged:

Reason	What it means
blocked	The site answered with a bot check or block page (Cloudflare, DataDome, PerimeterX, Akamai, ...) on several fresh IPs. Try the residential proxy in Advanced.
needsJavaScript	An app page whose content only appears in a browser, with Rendering set to HTTP only, or a page with no readable text even after rendering.
httpError, networkError	404s, 5xx errors, timeouts.
notAPage, tooLarge, emptyContent	Files that aren't pages (images, YAML, ...), responses over 15 MB, pages with no text.
robotsTxtDisallowed	Only with Respect robots.txt on.

Inputs from other crawlers work as they are

Inputs saved for Apify's Website Content Crawler run here unchanged: startUrls, includeUrlGlobs, excludeUrlGlobs, maxCrawlDepth, maxCrawlPages, maxResults, useSitemaps, useLlmsTxt, respectRobotsTxtFile, onlyCrawlPagesModifiedSince, keepUrlFragments, ignoreCanonicalUrl, removeElementsCssSelector, keepElementsCssSelector, htmlTransformer, saveMarkdown, saveHtml, saveHtmlAsFile, saveFiles / saveContentTypes, saveScreenshots. crawlerType: cheerio / jsdom = HTTP only; the browser types render only the pages that need JavaScript (the same content, much faster). Its nested crawl.* and metadata.* fields are flat columns here (loadedUrl, depth, httpStatusCode, canonicalUrl, title, languageCode, openGraph, jsonLd, responseHeaders...).

Output fields

One row per page, in this order: url, title, description, textSnippet, markdown, text, html (opt-in), htmlUrl (opt-in), screenshotUrl (opt-in), fileUrl (files), headings, wordCount, image (og:image), author, keywords, languageCode, publishedAt, modifiedAt, canonicalUrl, loadedUrl, requestedUrl, referrerUrl, depth, source (startUrl, link, sitemap, llmsTxt), startUrl, httpStatusCode, contentType, contentLengthBytes, renderedInBrowser, internalLinks, externalLinks, linksCount, ogTitle, ogDescription, ogType, ogSiteName, openGraph, jsonLdTypes, jsonLd, metaRobots, responseHeaders, sitemapLastModifiedAt, contentHash, loadMs, scrapedAt. Empty values are null. Dates are ISO 8601 UTC.

Dataset views: 📊 Overview, 📄 Content for AI (URL, title, Markdown), 🔎 SEO (title, description, canonical, robots, status, Open Graph, JSON-LD types), 🔗 Links, 📈 Stats and 📎 Screenshots & Files. The run's OUTPUT record summarizes pages, skipped URLs by reason and timings.

FAQ

Can it crawl sites that need a login? No. It reads public pages only.

Does it render JavaScript? Yes, when a page needs it: pages are read over fast HTTP first, and only app-shell pages are rendered in headless Chrome. Most documentation, blog, news and company sites serve their content in the HTML and never start the browser.

How fast is it? On the Apify platform at the default 512 MB: 50 documentation pages in 3.7-4.9 s, 100 heavy blog pages (650 KB each) in 12.5 s, 600 docs pages in 63 s (measured 2026-10-23).

Your feedback

Found a page it reads badly, or a field you need? Open an issue on the Actor's page. If it saved you time, a review on the Store helps a lot.