Website Text Extractor for AI, RAG and LLM

Give it a list of links or one address plus a crawl depth. Get back one row per page: the readable text with the menus and footers gone, the same content as Markdown, the page metadata, and — if you ask for it — the page already split into chunks with overlap, ready for a vector database.

No browser. No proxies. No account anywhere. Plain HTTP and the Python standard library, which is why it is cheap, and also why it has one honest limit — read the next section before you buy.

The number you should decide on: 73.1 %

This Actor reads what the server sends. It does not run JavaScript. So there is a part of the web it cannot read, and here is how big that part measured out.

52 live pages, seven kinds of site, fetched from Apify servers on 2026-09-17 (run 7fpElZkFZAnx8DObw, the same code that ships here):

outcome	pages	share	what it means
readable text	38	73.1 %	120 words or more. HTTP was enough.
refused (401/403)	6	11.5 %	apnews, reuters, etsy, ebay, nestle, tesla
empty shell	3	5.8 %	the JavaScript case — we got nothing
thin (under 120 words)	2	3.8 %	a real page, but mostly headlines
network error	2	3.8 %	amazon answered 500, bestbuy dropped the connection
disallowed by robots.txt	1	1.9 %	sap.com said no, so we did not download it

The same measurement from a laptop gave 73.1 % as well, so this is not an Apify-network effect.

It depends enormously on what you crawl:

kind of site	readable	pages
documentation	10 / 10	docs.python.org, MDN, Django, Kubernetes, React, GitHub, FastAPI, Apify, PostgreSQL, nginx
reference	4 / 4	Wikipedia, arXiv, GitHub repo, Hacker News
blogs	7 / 8	Simon Willison, Martin Fowler, Cloudflare, danluu, Paul Graham, Rust, jvns
news	7 / 10	BBC, Guardian, TechCrunch, Ars Technica, Wired, NPR, CNBC
government	3 / 4	IRS, gov.uk, European Commission
corporate	4 / 8	Stripe, IBM, Salesforce, Siemens
online shops	3 / 8	the worst case by a distance

So: for documentation sites, knowledge bases, blogs and news this Actor is a straight fit. For large retail, for sites behind Cloudflare's stricter modes, and for single-page apps that render everything client-side, use a browser-based Actor instead — that is what they are for, and we are not going to pretend otherwise.

Median text on a page that worked: 1480 words.

What one row contains

46 fields. Flat — no nested objects to unwrap before a CSV or a BigQuery load.

Address — url, final_url (after redirects), canonical_url, domain, depth, redirected, dedup_key, canonical_key.

Response — http_status, content_type, charset, response_bytes, response_time_ms, error, scraped_at.

Page card — title, description, author, published_at, modified_at, language, site_name, page_type, image_url, keywords, robots_meta, og (every og: and twitter: tag), jsonld_types, alternate_languages, feeds.

The content — text, markdown, word_count, char_count, reading_time_min, and html when you switch it on.

Structure — headings (level and text, in document order), heading_count, links (url, anchor text, rel), link_count, images (url, alt), image_count.

For RAG — chunks and chunk_count.

For monitoring — change_type (NEW / UPDATED / UNCHANGED) and content_hash.

Dates and authors, honestly

Measured on the same 52 pages: an author came back on 28.8 % and a publication date on 17.3 %. That is not a parsing failure — most of those 52 are section fronts and documentation pages, which carry no byline and no date. Where a page states them (article:published_time, JSON-LD, parsely-pub-date, or a <time datetime> inside the article) they are read and normalised to ISO-8601, including Unix-epoch values. What is deliberately not done is guessing: "2 days ago" and "March 3" stay empty, because a guessed date is worse than a missing one — it looks just as trustworthy.

Chunking, which is the point for a vector database

Turn on Split into chunks and every row gains a chunks array. The split follows the document, not a character counter:

• a heading always starts a new chunk;
• then paragraph boundaries, then sentence ends, and only as a last resort a space;
• a fenced code block is never cut while it fits in one chunk;
• chunkOverlap repeats the tail of the previous chunk, cut at a sentence boundary, so a thought that lands on the seam is still findable whole. It is capped at half the chunk size.

Each chunk carries index, text, chars, words, start_char, url, title and heading_path — the chain of headings above it. That last one is the difference between storing a nameless paragraph about limits and storing one you can cite as Pricing → Enterprise → Limits.

Monitor mode: re-index only what changed

Turn on Monitor mode and the Actor remembers a fingerprint of each page's text between runs, in a named key-value store. Every row gets change_type:

• NEW — never seen before;
• UPDATED — the text changed;
• UNCHANGED — identical, and by default not delivered and not charged.

The fingerprint covers the text only. A view counter, a fresh "related posts" block or a new ad slot is not a change — otherwise a monitor would report everything as changed every day, and quietly become a plain re-crawl.

For a scheduled pipeline this is money twice over: you do not pay us for the unchanged page, and you do not pay your embedding provider to vectorise it again.

Crawling a site

Leave Crawl depth at 0 to open exactly the addresses you gave. Raise it to walk a site breadth-first from the entry point: depth 1 adds the links on those pages, depth 2 one level further.

• Stay on the starting domain and Allow subdomains keep the crawl from wandering off a news site into the whole internet.
• Only crawl URLs matching / Never crawl URLs matching take wildcard patterns such as https://example.com/docs/*. They apply to links found while crawling, never to the addresses you supplied yourself.
• Maximum pages is the budget for the run, counted in requests actually sent. Maximum pages per domain stops one big site from eating it all.
• Seed the crawl from sitemap.xml queues the site's own sitemap. This is how you reach a site whose menu is drawn by JavaScript: the links are not in the HTML, but the sitemap lists every page anyway.

rel="nofollow" links are not followed, and duplicate addresses are collapsed before you are charged: ?utm_source=..., a trailing slash, www., http vs https and a declared canonical_url all resolve to one row.

robots.txt is always obeyed

There is no switch. Before a page is requested, the site's robots.txt is read once per host and matched by the RFC 9309 rules — longest matching rule wins, Allow beats Disallow on a tie, * and $ supported. A Crawl-delay is obeyed too.

This is not only the correct thing to do. The disallowed sections of a site are usually search results, carts and calendars — infinite URL spaces where a crawl budget burns away for nothing.

A page refused this way still gets a row, with error: robots-disallowed, so you can see which address was skipped and why.

Empty output always names its reason

A green run with nothing in it is the worst thing that can happen to a buyer. So the Actor never stays silent about it: the REPORT record carries an empty_reason and the log carries the same sentence in words. Eleven named reasons:

no-start-urls, all-urls-invalid, nothing-requested, patterns-exclude-start-urls, charge-limit-reached, all-pages-refused, all-pages-blocked-by-robots, no-text-found, filtered-out, nothing-changed, nothing-collected.

The first five are decided from your input alone, and are checked before a single byte is downloaded — you are not charged time for pages that the next line of code would have thrown away.

Compared with the three Apify Actors in this niche

Measured from the Apify API on 2026-09-17.

	this Actor	apify/web-fetch	apify/website-content-crawler	apify/rag-web-browser
price	see the store listing	$0.0015 per fetch (Free/Bronze plan)	you pay platform usage	you pay platform usage
runs JavaScript	no	yes	yes	yes
proxies, anti-blocking	no	yes	yes	yes
Google search built in	no	no	no	yes
crawls a whole site	yes	no, single URL	yes	no
chunking for a vector DB	yes, with overlap and heading path	no	no	no
change detection between runs	yes	no	no	no
publication date and author fields	yes	no	no	no
heading outline, link and image lists	yes	no	partly	no
default memory	1024 MB	—	8192 MB	—
output shape	flat, 46 fields	formats you pick	nested (crawl, metadata)	nested

Where they are ahead and we are not going to catch up: JavaScript rendering, proxy rotation and anti-blocking, screenshots, LLM summaries, file downloads, and full CSS-selector control. Those need a browser or a paid service; this Actor has neither, by design. On the 52-page measurement above that cost us 3 pages out of 52 to empty shells and 6 to outright refusals.

Where this Actor is ahead: chunking, change detection, dates and authors, a flat schema, named empty reasons, a per-domain budget, and a price you can calculate before the run instead of a compute bill you find out about after it.

removeElementsSelector and keepElementsSelector accept simple selectors only — a tag (aside), a class (.cookie-bar) or an id (#sidebar). Combinators and pseudo-classes are not supported. A half-implemented selector engine would fail silently and look like a broken Actor, so it is refused instead.

Cost and speed

Measured on Apify, 1024 MB, explicit memory on every run:

• the 52-page measurement run took 76 seconds for 52 pages across 52 different hosts, at 512 MB;
• pages are fetched in parallel (default 5 at a time), but requests to the same host are spaced out, and a Crawl-delay in robots.txt is respected.

Maximum rows caps what you are charged. It limits the output, not the crawl: rows removed by a filter do not consume it.

Notes for the fine print

• Non-HTML responses (PDF, images, archives) are recorded with their content_type and an error of not-text. They are never parsed as text.
• A 200 response with an empty body is treated as a failure, not a success — that is what a polite refusal looks like, and a buyer should not pay for a row with no words in it.
• Character encoding is taken from the header, then the meta tag, then tried as UTF-8 and cp1252. Expired or self-signed certificates do not stop a read: you supplied the address, and nothing is sent to it.
• Pages that fail still produce a row, with the status and the error, so a list of 1000 addresses comes back with 1000 answers. That row is a row like any other, so it is charged like any other — said plainly rather than buried. Switch on Drop pages that did not open and they are neither delivered nor charged. On the 52-page measurement the failures were 15.4 % of the list.