Turn up to 100 URLs into clean Markdown, text or HTML. It tries plain HTTP first, retries with browser headers, and opens a headless browser only when a page needs it. You pay $0.001 per page that comes back with content. Failed URLs are free and get a clear error code.
HTML is converted in a background thread with its own memory cap and time limit. A heavy page no longer makes other URLs time out, and a page too big to convert fails its own row with CONTENT_TOO_LARGE instead of taking the run down.
Rows with extraction: "large-page" now match the dataset schema. The large-page path starts at 3,000 HTML elements (was 5,000), and it's also used when Defuddle runs past the time limit.
Fewer false bot-check results. Cloudflare's detection script on normal pages, or words like "DDoS-Guard" in an article, no longer end as BLOCKED_BY_SITE. A literal <script> inside <title> no longer removes the page body.
Content-Encoding lists (such as gzip, identity) and raw deflate are decoded. Unknown encodings and binary bodies end as UNSUPPORTED_CONTENT_TYPE and aren't charged. An empty body is EMPTY_CONTENT (the browser gets a try in auto mode), and a redirect status without a Location header is HTTP_ERROR. Deeply nested pages end as EMPTY_CONTENT instead of INTERNAL_ERROR.
The Docker image no longer installs Defuddle's optional dependencies.
0.1.0 (2026-11-05)
First version.
Fetches up to 100 URLs per run and returns each one as Markdown, plain text or cleaned HTML, through a fallback chain of plain HTTP, then HTTP with Chrome-like headers, then headless Chromium only when a page is blocked, needs JavaScript or is empty. Each row lists the steps it tried.
Main content detection with Defuddle, a full-page mode, and a fast path for pages with more than 5,000 HTML elements.
16 per-URL error codes. Only rows with content are charged, at $0.001 per page.
SSRF guard on the URL, on every redirect hop and inside the connection's DNS lookup, plus a request guard in the browser.
Caps of 5 MB per page, 10 redirects, 1,000,000 characters of content and 120 seconds per step. The run summary goes to the OUTPUT record, and the run stops starting URLs at its maximum charge.
