# Changelog of PDF & Document to Markdown - PDF, DOCX & HTML for LLMs (`entranced_gelato/ai-document-reader`) Actor - **URL**: https://apify.com/entranced_gelato/ai-document-reader/changelog.md - **Full Actor documentation**: https://apify.com/entranced_gelato/ai-document-reader.md ## Changelog ### 2026-11-02: Pricing notes - README: a notice that the new prices start on 16 November 2026 at 14:41 UTC, and what applies until then. ### 2026-11-01 #### Added - **Several documents per run.** New input `urls` ("More document URLs", up to 100). Each document becomes its own row and is charged as one "Document read"; documents that can't be read are free. Page range, OCR and AI summary settings apply to every document, robots.txt and TDM are checked per document, `maxOcrPages` stays a limit for the whole run, and the run stops before a document the maximum total charge can't cover. One document behaves exactly as before. #### Changed - Store title: **PDF to Markdown & Text Extractor - OCR, Tables, DOCX**. ### 2026-10-17: Compliance update #### Added - **robots.txt check** (`respectRobotsTxt`, default `true`). Before a document link is downloaded, the Actor checks the site's robots.txt: the `AIDevsBot` group if the file has one, otherwise `*`, with RFC 9309 matching. Each redirect target is checked before it is followed. - Disallowed documents are not downloaded and are free. They get `errorCode` `ROBOTS_DISALLOWED`, or `ROBOTS_UNREACHABLE` when robots.txt answers 429 or 5xx or can't be loaded; 401/403 count as disallowed and 404 as "no rules". - Files uploaded to Apify storage are not affected. - A redirect to a non-web address gives `BAD_REDIRECT`. - **TDM reservation.** New output fields `tdmReserved` and `tdmPolicy`, from the TDMRep `tdm-reservation` / `tdm-policy` headers and, for HTML documents, the matching `` tags (the tag wins). Reserved documents are never sent to AI: no AI call and no AI charge, and `aiError` says why. Extraction still runs. - New AI label fields `aiGeneratedAt` (ISO 8601, UTC) and `aiDigitalSourceType` (IPTC `trainedAlgorithmicMedia`). The overview view now shows `tdmReserved`, `aiGenerated` and `aiNotice`. - `LICENSE` file (all rights reserved). #### Changed - Requests identify as `Mozilla/5.0 (compatible; AIDevsBot/1.0; +https://apify.com/entranced_gelato/ai-document-reader)` instead of a Chrome 126 User-Agent. This covers document downloads, robots.txt and AI calls. - `aiNotice` now reads "AI-generated; may contain errors. Check facts before relying on them." - Apify AI gateway: only `openai/...` and `anthropic/...` models are used, with no `:free` or stealth variants. Any other `aiModel` falls back to `openai/gpt-4o-mini`, and the status message says so. Gateway requests send `provider.data_collection: "deny"`. - Email addresses and phone numbers are masked in the text sent to AI, on both paths. Dataset fields keep the document as it is. - `package.json` is now `UNLICENSED` and `private` (closed source). - README: new "AI and your data" section; answers on robots.txt and TDM; a rewritten "Is it legal?"; data-protection and "Wrong charge?" lines; a not-affiliated line; the OCR price start wording; an arXiv bulk-data note. The input descriptions repeat the AI points. - Tests now get 20 s each instead of Vitest's 5 s default: a cold pdf.js load on a 2-CPU machine could take longer. ### 2026-10-11 #### Changed - **Apify's standard proxies only.** Residential proxy groups and the `GOOGLE_SERP` group are dropped from the proxy settings and custom proxy URLs are ignored. If nothing usable is left, the run uses Apify Proxy in automatic mode (datacenter proxies). When settings are dropped the run logs a warning and sets a status message; it never fails because of it. "No proxy" still means no proxy. ### 2026-10-10 #### Added - **OCR for scanned PDFs.** Pages without a text layer are rendered (pdf.js + @napi-rs/canvas) and read with tesseract.js using the bundled English model, fully offline. New inputs `ocrMode` (`auto` default, `off`, `force`) and `maxOcrPages` (default 20). OCR runs only once the new `ocr-page` event is priced (proposed $0.003 per page); until then such pages are returned with `needsOcr: true`, no text and an explanation in `ocrNote`. - **Tables to Markdown.** DOCX and HTML tables become Markdown tables (header row, merged cells, escaped pipes). PDF tables: best effort for simple grid tables. - **Page ranges** for PDFs (`pageRange`, e.g. `1-3,7` or `5-`), with per-page output (`pages`). - **File upload** (`documentFile`, file-upload editor) as an alternative to `url`; uploaded files are read from Apify storage. - **Built-in AI summary** through Apify's AI gateway ($0.02, only when the "AI summary" event is priced and a summary is produced; `aiModel` to override the model). AI output is labelled (`aiGenerated`, `aiModel`, `aiNotice`; `aiError` explains a missing summary). - New output fields: `source`, `fileName`, `pageRange`, `pagesProcessed`, `pages`, `ocrPageCount`, `needsOcr`, `ocrNote`, `tableCount`. - `.actor/store-listing.json`, dataset views (overview, Markdown, pages, status), output schema links, CI workflow, offline tests (unit + local end-to-end with real OCR) and a fixture generator. #### Changed - PDF engine: pdf-parse 1.1.1 (bundled pdf.js 1.10) replaced by pdfjs-dist 5.6. Text keeps the old line-joining rule; separate pieces of one line are no longer glued together. - DOCX/HTML plain text (`content`) keeps paragraph breaks instead of running paragraphs together. - With your own `openaiApiKey` the summary no longer triggers the "AI summary" charge. - "Document read" ($0.004, as low as $0.0028) is charged only for documents that produced text, after the record is saved. Failed downloads, blocked pages, unsupported, password-protected or empty documents are free. The run respects the max total charge, and records are kept under the 9 MB item limit. - `url` is no longer required in the input schema (an upload can replace it); one of the two must be given. - Dockerfile installs optional dependencies so the native canvas binary for the Alpine base image is present. - README rewritten with pricing, OCR, tables and page-range documentation.