Website Content Crawler | Bulk Text & Markdown
Pricing
from $3.00 / 1,000 delivered pages
Website Content Crawler | Bulk Text & Markdown
Fetch public website pages in bulk and export clean text, titles, Markdown and source evidence, with bounded concurrency, robots checks and explicit partial results.
Pricing
from $3.00 / 1,000 delivered pages
Rating
0.0
(0)
Developer
tingyou333 zhuang
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
13 days ago
Last modified
Categories
Share
Independent Bulk Website Content Crawler
Supply only startUrls, a string array of bare domains, internationalized domains or public HTTP(S) page URLs. The Actor fetches the supplied pages concurrently, cleans public HTML text with explicit hidden-element filtering and emits unique domain,text results progressively. Optional fields add final URL, title, a clearly labeled leading-text summary excerpt, markdown, headings, links, source hashes, pinned IP/DNS route, robots decisions and explicit truncation state. These output fields are this Actor's own contract. There is no browser, proxy, login or access-control bypass.
Supplied-page scope: a bare domain resolves to its HTTPS root page; a supplied URL with a path addresses that page. Only normalized supplied URLs are scheduled, subject to robots, public-network checks, redirects and the documented caps. Extracted links are output metadata; they are not recursively scheduled. This is not a whole-site coverage promise.
Primary public contract and comparison boundary
The fixed PRIMARY is 6sigmag/fast-website-content-crawler. Its complete public README, input definition and pricing were captured anonymously on 2026-10-08 UTC and are preserved in the project research actor_goal_20261009/research/content-primary/. The public README SHA256 is cbdf29f16db77a598757c38045a55157ed03f7524df4375a4150cd342de553b3. The earlier evidence/6sigmag--fast-website-content-crawler.json is a historical tool summary, not the complete README or a native output schema.
The primary explicitly advertises bulk processing, www-prefix and international-domain handling, high concurrency, cleaning and deduplication. Its native input has only startUrls, an array with a stringList editor and a bare-domain example; it does not specify items.type, empty-array policy, numeric limits or URL normalization rules. This Actor's strict string validation, supported ranges and HTTPS/IDNA choices are its own disclosed contract.
The primary's observed public default definition supplies no native output/Dataset schema or published example rows. The older tool inferred domain:string,text:string from recent runs; that inference remains the historical compatibility target, not a verified native declaration. The complete README describes quick insights into content, structure and key elements but does not promise native summary/title/headings/links/Markdown fields. This Actor retains these fields as additional value: summaryType:leading_text_excerpt labels a source-derived excerpt, not an AI summary.
The primary links to separate specific-page and deeper-crawl tools. Those recommendations do not establish its exact page selection, recursion depth, link scheduling or URL-path policy. We neither assert that the primary only fetches one page nor that it recursively crawls an entire site. Its private cleaning/deduplication algorithms, normalization rules, numerical concurrency/performance and failure/charging order remain unpublished or unmeasured; observable public capabilities can be assessed without copying private algorithms.
apify/website-content-crawler remains a secondary market comparator only; it does not replace or enlarge the primary contract. This Actor does not claim the official Actor's JavaScript rendering, login, PDF/files, sitemap traversal or recursive crawling. A historical primary invocation returned HTTP 403 without a run ID in the then-current account/call environment. It provides no direct functional comparison and does not prove global disablement. No comparator is executed by the offline checks below. Release acceptance is recorded separately against the exact build; historical source receipts alone do not establish it. Usage counters and public prices are not buyer/payment/revenue evidence. See COMPETITOR_MATRIX.md for all contract rows, source attribution and remaining gaps.
Examples
Minimal input (examples/default.json):
{"startUrls":["https://docs.python.org/3/library/urllib.error.html"]}
Bulk input (examples/bulk.json):
{"startUrls": ["docs.python.org/3/library/urllib.parse.html","https://docs.python.org/3/library/urllib.request.html","docs.python.org/3/library/urllib.error.html"],"maxConcurrency": 3,"maxRequests": 12,"runTimeoutSecs": 90}
examples/normalization.json illustrates www, bare domains, IDNs and URL fragments; those example domains are illustrative and are not an acceptance claim. examples/caps.json exercises partial output. examples/security.json demonstrates explicit unsafe-target failures. An empty startUrls: [] is an empty batch with no custom result event. startUrls remains the sole required input at runtime: omitting it or passing null fails with INVALID_INPUT; neither means an empty batch. To allow an explicit empty array through Apify's required-array validation, the native stringList definition retains required and uses nullable:true, default:null as a compatibility sentinel. The platform may therefore create a run for a missing/null value before runtime rejects it; the startup fee and runtime/report costs can still apply, with no Dataset row or custom result charge. The HTTPS prefill and examples/default.json remain the normal nonempty default input. Native/local validation alone does not establish exact-build cloud empty-batch acceptance. A started standard run incurs the automatic startup fee even when no page result is produced; at 256 MiB that fee is $0.001. FREE source-validation receipts and owner-private PPE receipts are distinguished below. See the pricing status below.
A result always retains this Actor's stable core keys (the old tool's inferred primary compatibility target, not a published primary native schema):
{"domain": "docs.python.org","text": "Clean visible page text ...","url": "https://docs.python.org/3/library/urllib.error.html","title": "Page title from the source","markdown": "# Source heading ...","status": "succeeded","caps": [],"textSha256": "SHA256 of delivered text","provenance": {"startUrl": "https://docs.python.org/3/library/urllib.error.html","inputIndex": 0,"redirects": [{"url":"https://docs.python.org/3/library/urllib.error.html","status":200,"ip":"validated pinned IP","dnsSource":"system or fixed_cloudflare_doh_fakeip_only"}],"robots": [{"url":"https://docs.python.org/robots.txt","status":200,"policy":"parsed","sha256":"body hash"}],"bodySha256": "source bytes hash","bodyBytes": 24667,"fetchedAt": "UTC timestamp"}}
This is a structural illustration. Current unabridged output in evidence/real-output.json, real-report.json and local-acceptance.json comes from the three saved real HTML documents replayed with the actual SDK and three concurrent fixture page workers, with network access prohibited. Provenance says frozen_fixture_no_network and does not invent a new pinned IP. Historical live transport evidence remains in evidence/history/0.1.0/; it does not bind the changed runtime. Output ordering follows completion; inputIndex records original order. domain is the final ASCII hostname with a leading www removed; full final hostname/path/query remain in url. Fragments and default ports normalize away. www and non-www are not assumed equivalent before fetching. Bare domains default to HTTPS; no automatic insecure fallback occurs.
Safety, resource bounds and result semantics
Every supplied URL, redirect and robots redirect is normalized. Credentials, private/loopback/link-local/reserved IPs, IPv4-mapped/tunnel IPv6, non-HTTP protocols, scoped IPv6, local hostnames and nonstandard ports are refused. All system DNS answers must be public; a mixed public/private or fake/public answer fails. The validated numeric IP is passed to a pinned httpcore.SyncBackend; TLS SNI and certificate verification use the original hostname. Pools are per request with no retry, environment-proxy inheritance or hostname reconnection. Checks also apply to each redirect destination and each root robots fetch.
Only when all OS answers are Clash fakeIP in 198.18.0.0/15 may the default fakeIpDohFallback:true query fixed Cloudflare DoH: TCP 1.1.1.1, HTTPS identity cloudflare-dns.com. Query/status, CNAME-to-address association and all returned addresses are validated. The target connection then pins the validated public answer with its original TLS hostname. A fallback for ordinary DNS failure, private answers or mixed answers is prohibited. Set the option false for explicit LOCAL_FAKE_IP_DNS. This fallback sends the requested hostname to Cloudflare and is recorded in provenance; it does not disable security.
The root /robots.txt is fetched before page access. 404/410 permits access, 401/403 denies, and transport errors/5xx/truncated or HTML robots responses fail closed. Robots redirects are bounded and must remain /robots.txt; cross-origin redirected robots retrieval still has public-DNS/pinning checks. Protego evaluates wildcard/end-anchor rules and longest allow/disallow precedence for each page and redirect target. Crawl-delay and request-rate directives conservatively pace per-origin starts within the same time budget; time-window rate extensions fail closed with an explicit code. Robots policy is cached per origin for this bounded run. Pages expose HTML or plain text only; unsupported files, compressed bodies and access challenges produce explicit failures. The client asks for identity encoding so source and decoded body size are bounded.
Character decoding uses a usable HTTP charset first, then a byte-order mark, then an HTML/XML encoding declaration within the first 4096 bytes for HTML, and finally UTF-8. Unknown or non-text codecs fall through; malformed bytes retain explicit replacement characters. Plain text ignores HTML meta declarations. No statistical encoding guesser is used. The same decoded text feeds title, clean text and Markdown extraction.
Defaults and hard supported ranges:
| Option | Default | Supported range |
|---|---|---|
| startUrls length | required | 0–100000 strings |
| maxConcurrency | 16 | 1–64 |
| maxPages | 10000 | 1–100000 |
| maxRequests | 30000 | 1–1000000 |
| maxBodyBytes | 2000000 | 1024–10000000 |
| maxTextChars | 500000 | 100–2000000 |
| maxRedirects | 5 | 0–10 |
| pageTimeoutSecs | 40 | 1–120 |
| runTimeoutSecs | 240 | 1–3600 |
maxPages caps attempted distinct supplied URLs, not discovered links; extracted links are metadata and are not scheduled. maxRequests includes robots, page, redirect and DoH HTTP requests. System DNS uses at most four concurrent killable subprocesses with bounded caller waits and a five-second maximum per lookup. Timeout kills and reaps the resolver child. Streaming reads/writes/TLS honor an absolute page/run deadline, including slow chunked bodies. HTML parsing operates only on the bounded body; its CPU time and final SDK/storage cleanup can add small overhead beyond the network deadline. No unbounded task queue or retries are used. Large inputs require appropriate memory/time/cap settings; limits report incomplete scope explicitly.
Clean text prefers main/article content; scripts/styles/navigation/footer/forms and explicit hidden elements are removed, whitespace normalized and repeated exact lines removed. Hidden filtering recognizes the HTML hidden attribute, case-insensitive aria-hidden=true and inline display:none, visibility:hidden/collapse and content-visibility:hidden. tinycss2 parses comments, escaped identifiers, declaration order and !important. Only the property-specific supported single-keyword values in INLINE_VALUES may update the inline declaration state; unknown or unsupported values are ignored even with !important. In particular, a later bogus display/visibility/content-visibility value cannot clear an earlier valid hidden declaration. CSS-wide keywords, multi-keyword values and computed styles are not evaluated. This is explicit declaration filtering, not browser computed CSS: external stylesheets, class rules, CSS variables, layout, media queries and descendants overriding inherited visibility are not evaluated. Markdown is a lightweight derivation, not a full fidelity converter; headings and normalized links are retained as separate bounded arrays (200 headings with at most 2000 characters each, 500 links); optional metadata truncation is reported in metadataCaps. Exact normalized input URLs, final URLs and delivered text hashes deduplicate results across the batch. Cross-domain identical text is one result, with duplicate diagnostics retaining input references. Near-duplicates are not falsely merged.
The already parsed DOM is reused for Markdown instead of parsing another complete DOM. Before Dataset delivery, a conservative ASCII-escaped JSON row bound of 8000000 bytes removes optional Markdown/links/headings first, retaining core text whenever it fits; if core text must be truncated, caps:dataset_row_bytes, partial state, a refreshed text hash and summary are explicit. UTF-8/non-BMP characters and JSON escaping count toward this row bound. Input limits and default concurrency are unchanged. This bounds serialized item size, not whole-process RSS, parse CPU or total protocol traffic. Bulk jobs/outcomes and checkpoint payloads still grow with input size; 100000 URLs/64 workers/10 MB bodies have no 256 MiB qualification.
Only nonempty results enter the default Dataset. Body/text caps yield a delivered partial result with status:partial and caps; hard time/request/page/spend caps and inaccessible sources yield failed/skipped outcomes, never successful empty placeholders. A 200 page with no visible content is empty. A source error is failed. Failures, empties, duplicates and skipped inputs appear in the progressively checkpointed RUN-REPORT KVS record and do not receive a result charge. Mixed batches are partial. An all-error batch fails the Actor; all empty content is an explicitly empty completed batch. An uncertain Dataset/charge write is recorded as delivery_uncertain; the run stops and never retries that write.
SDK delivery supports explicit FREE/source-validation mode and PAY_PER_EVENT. PPE requires a positive, finite result price. In cloud SDK mode, apify-default-dataset-item may be absent and contributes zero USD; the observed two-event price plan omits it. An explicitly configured finite nonnegative synthetic price is supported by the runtime for compatibility, but a positive price adds to each delivered row and is outside the observed two-event plan. Explicit zero remains an offline compatibility case, not a claim that the current platform accepts zero-price configuration. Missing synthetic pricing outside cloud SDK mode is rejected before writing because the pinned SDK uses a different local fallback. Optional apify-actor-start is platform managed; the Actor never charges it manually. Unknown event/synthetic configurations, other pricing models, missing prices or changes during a batch fail closed before a new write. Input URLs and concurrency are not reduced to accommodate bad pricing.
PPE uses SDK actor.push_data(item, charged_event_name="result") under the SDK reentrant charge lock, with combined custom/synthetic preflight. FREE uses uncharged SDK push. The 0.1.4 runtime configures one transport attempt for Dataset writes, custom charges and report writes using an Actor-local client and existing run-default storage. It pins the inspected SDK adapter versions, rejects alternate cloud storage IDs and does not mutate the installed SDK or its default retry policy for other Actors. Read failures can therefore fail sooner; unknown writes stop delivery and are not replayed. After the SDK returns, the Actor requires actual ChargeResult.charged_count to equal one for PPE or zero for FREE, both event-count deltas and total accounted amount to match the expected delivery, and the rest of ChargeResult to have the expected shape. An exception or mismatch after a possible write is terminal DELIVERY_UNCERTAIN, not a failed-empty row. Claiming work and checking/pushing completed work share one delivery lock. All completed workers recheck terminal state inside it; the first stop reason is retained, and no later worker pushes or charges after it.
Dataset, KVS and platform charging remain separate services without a cross-service transaction. A checkpoint failure halts new delivery and returns a failed in-memory report; the entry point logs terminal diagnostics. An unavailable KVS cannot promise a final stored report. No failed checkpoint or uncertain delivery is retried automatically; reconciliation must inspect Dataset and accounting before any new run. No exactly-once distributed guarantee is claimed.
Saved runtime-baseline receipts in evidence/review-fix-0.1.2/ record the actual local SDK lifecycle and real ChargeResult against saved docs with a three-page barrier. The positive/zero synthetic-price cases, exact simulated $0.004 combined cap, invalid-event configurations, corrupted returned charged_count, write-after-success exception and checkpoint failure are tested. These are local test prices/accounting, not real charges or revenue. No validation script deploys, builds remotely, starts competitors or changes cloud pricing. Platform event configuration and actual remote charge/readback still require separate parent-owned verification.
Pricing and dated validation evidence
The owner Console Monetization panel observed on 2026-10-09 UTC shows active pay-per-event pricing with User pays platform usage costs: No for standard runs. This is effective configuration evidence, not a paying-customer invoice. The observed event plan is:
- custom
result: $0.003 per unique delivered nonempty page result, including a delivered partial result; apify-default-dataset-item: omitted, with no additional Dataset event fee;- automatic
apify-actor-start: $0.001 per event, one event per GB of Actor memory with a minimum of one; - platform usage is included for standard runs. Standby has separate platform billing and is not included in these examples.
At 256 MiB, one delivered result plus startup is $0.004, three results $0.010 and eight results $0.025. Failed, empty, duplicate and skipped pages incur no custom result fee, but a started run still incurs its startup fee. A run failing after earlier delivery can retain those earlier result charges. Startup is platform managed and is never charged manually by this Actor. Larger-memory runs can incur multiple startup events.
The primary's captured public pricing, dated 2026-10-08 UTC, uses apify-default-dataset-item as its event key and result as that event's display title, at $0.003 per Dataset item. Its startup price is $0.0015 per event, one event per GB, minimum one, with platform usage included. For the same number of charged results and one startup event, our observed nominal event total is $0.0005 lower. This is a dated tariff comparison, not a claim of equivalent private cleaning/deduplication, measured speed, successful-only billing or current competitor output quality. The Store's current pricing is authoritative for a new run.
Runtime 0.1.4, build F78gaCG4fc3uTH8vp, passed a FREE three-Python-document source case as run PnEnKPfYjPNYNqhNI, Dataset IxjcNlSld7JLNTxAh, with three real rows and four requests. An owner-private PPE cap case on that build, run m03ca19BLa5eljTzU, Dataset wP7MpgH2CpZ0CGHZE, delivered one real urllib.error result and recorded one result and one startup event, nominal total $0.004. Its application report is partial with SPEND_CAP, one succeeded and two skipped inputs; one skipped input had already completed collection. It therefore does not mean only one page was fetched. Saved observed run usage was $0.0003879846433583233; a later read revised it slightly downward, and the conservative earlier amount remains in the ledger.
These are owned validation receipts, not customer revenue or full-workload profitability. Build, storage retention, readback/recovery, subscription, failed workloads and maintenance costs are separate. The owner-run USER usage attribution is not evidence of a different customer's invoice; current standard-run configuration is supported separately by the Console panel. The API pricing-history projection omitted the requested false flag, so that omission is not interpreted as Boolean false.
Current local real-source probes separately delivered IANA plus readable Japanese from 日本語.jp with duplicate handling, and eight distinct Python documentation pages with eight observed workers. Offline production-runner controls covered 100000 short URLs with maxPages=1 (one result, 99999 explicit PAGE_CAP outcomes) and 64 distinct small fixture jobs at a worker barrier. These controls do not qualify 100000 complete live pages, maximum-sized inputs/bodies, cloud memory limits or network throughput. The 100000-input report encoded by the pinned SDK is about 14.38 MB; repeated report checkpoints, failures and duplicates can consume resources without generating result revenue. Select memory, time and cost limits for the actual workload; a low event cap does not cap platform resource consumption for the developer.
Earlier 0.1.2/0.1.3 FREE runs R9hRzWNY8iZUR8mRz / h5qH7TKd6Kd47mRhb and HzTDL6hykWuxOQgQ4 / lAF9Vvyl7qjpa1Xlg remain historical evidence for their exact packages. Any final document-only build has a new complete source identity and its own exact-build release acceptance; the historical receipts above are not relabeled as that new build's tests.
Reproduction
uv sync --frozen.venv/bin/python -m pytest -qapify validate-schema.venv/bin/python scripts/offline_review_acceptance.py
The offline acceptance script runs the actual Actor SDK in children with inherited APIFY/ACTOR/CRAWLEE settings removed, isolated local storage and audit hooks rejecting DNS/connect/sendto. It replays saved real documents, records actual local Dataset rows and ChargeResult, and refuses to overwrite an existing storage-review-0.1.2/ receipt. It proves concurrent offline behavior, not fresh live transport/cloud parity. The earlier local_acceptance.py performs live public HTTP and is historical; it is not current 0.1.4 cloud acceptance. The commands above are reproduction entry points, not evidence that this author reran them. Approval and exact-package status must be read from the independently issued records, not inferred from these local tests.
uv.lock pins runtime/dev dependencies; requirements.lock pins all runtime transitives with hashes for Docker pip --require-hashes. Docker uses Python 3.12.14, runs as uid 10001 and includes only code/Actor metadata. The Python image version is pinned; an image digest and actual Docker build/run are not verified when Docker is unavailable. SVG artwork is provided at .actor/icon.svg; cloud upload/icon format conversion is parent release work.
Full requirement/evidence matrix
| ID | Public primary contract or own extension | Implementation / own promise | Evidence and remaining limit |
|---|---|---|---|
| P01 | primary: only required startUrls array, stringList UI; no native items.type or limits | own strict strings, sole required runtime input; native required/stringList retained with nullable/default-null compatibility sentinel | native validator and public-client replay plus runtime missing/null rejection; explicit [] support still requires exact-build cloud acceptance; 0–100000 is our range |
| P02 | primary: bare-domain example; exact page/URL policy unpublished | HTTPS bare-domain root and public supplied page URLs | normalization fixtures and bare docs URLs; full URLs are own support, primary page-set parity unmeasured |
| P03 | primary: www handling advertised; transformation unpublished | final domain strips leading www; URL preserved; hosts not pre-merged | fixtures; exact primary policy unmeasured |
| P04 | primary: international domains advertised; encoding rules unpublished | strict IDNA UTS46 ASCII; URL Unicode escaping; explicit character decoding | saved real www IANA and Unicode 日本語.jp source probes; Shift_JIS repair replay; current exact-build cloud IDN qualification remains pending |
| P05 | primary: default Dataset pricing; domain/text only old tool inference, no published native output schema | own stable domain,text string rows | saved docs/local SDK replay and 0.1.4 FREE three-doc cloud receipt; not primary native schema proof |
| P06 | primary: bulk/high concurrency advertised; no numerical SLA | bounded rolling thread workers, default 16 / max 64 | 2/3/8-worker barriers, eight real local pages, 64-worker small-fixture control and three-doc cloud receipt; no primary speed or all-max qualification |
| P07 | primary: cleaning advertised; exact DOM/CSS algorithm unpublished | main/article preference; noise/explicit hidden removal | cleaning fixtures, saved docs, 0.1.2 inline CSS subset; no full browser CSS or byte parity |
| P08 | primary: deduplication advertised; scope/threshold unpublished | exact lines/input/final URL/text hashes, cross-domain identical-text merge | line/cross-page tests; near-duplicates preserved; no private-algorithm replication required |
| P09 | primary: broad content/structure/key-elements insights/use cases, not native structured-field promises | own labeled leading-text excerpt, title, headings, links, Markdown | source-derived fields retained and tested; no invented AI summary, sitemap or primary native-field guarantee |
| E01 | own additional output | stable final URL/title/Markdown | extraction tests and saved SDK rows; lightweight conversion |
| E02 | own source provenance | body/text hashes, timestamp, IP/DNS route, robots/trace | saved-source replay explicitly labeled; historical live transport separate |
| S01 | own safety | public target checks on every page/robots redirect | private redirect and second-origin robots tests |
| S02 | own safety | numeric TCP pin, original TLS SNI/certificate identity | transport tests; historical live receipts, no new transport test in this doc repair |
| S03 | own environment handling | all-fakeIP-only fixed DoH, no ordinary failure/private/mixed fallback | fake/mixed/private/error tests; route/privacy limitation disclosed |
| S04 | own policy | per-origin robots/pacing and bounded redirects | denied/private/new-origin/503/truncated/pacing tests |
| S05 | own resource limits | bounded body/time/request/concurrency; deadline and rolling worker window | stream/deadline/request/active-worker tests; maximum bulk memory not qualified |
| S06 | own access scope | no browser/proxy/login/access-control bypass | direct httpcore and explicit challenge failure |
| R01 | own result/error contract; primary failure details unpublished | partial vs failed/empty/skipped with RUN-REPORT | mixed/empty/cap/denial tests; no successful empty placeholders |
| R02 | own progressive delivery; primary ordering unpublished | serialized SDK push after completion and checkpointing | local SDK/checkpoint evidence; cross-service transaction not guaranteed |
| B01 | own configured custom result $0.003; Dataset synthetic event omitted, distinct from primary event identity | unique delivered nonempty/partial result; no custom event for failed/empty/duplicate/skipped pages | offline SDK accounting plus actual 0.1.4 owner-private PPE receipt: one result/startup; no customer billing claim |
| B02 | own spend/uncertain-write guarantees and startup $0.001 per GB, minimum one | combined preflight, receipt deltas, terminal first-stop/no retries | local cap/config/fault/barrier cases and actual SPEND_CAP receipt; empty/failed work does not waive startup; broader case acceptance separate |
| Q01 | own packaging | locked runtime deps, three native schemas, Docker entry | saved local receipts; actual Docker build/run not established by doc repair |
| Q02 | own documentation/examples/artwork | README plus full COMPETITOR_MATRIX, unchanged examples/icon | 0.1.4 runtime promises retained; schema/docs compatibility package and new exact-build acceptance recorded separately |
| Q03 | own evidence | 268-test reviewed runtime baseline with actual local SDK scenarios | independent runtime review and offline receipts; document-only changes do not rerun unchanged tests or count them as remote billing |
| G01 | comparison gate | fixed primary; generic public capabilities remain in scope | full public README/input/pricing obtained; historical 403 yields no direct runtime parity |
| G02 | exact-build source gate | every cloud receipt binds its recorded exact package | 0.1.4 build F78gaCG4fc3uTH8vp / run PnEnKPfYjPNYNqhNI accepted three real pages; final document-only build requires its own exact acceptance |
| G03 | demand/release gate | no verified buyer/payment/revenue or publication claim | demand signals and configured pricing are not paid adoption; independent final contract/economics/publication acceptance is separate |
Evidence and release boundary
Current runtime version is 0.1.4. It adds cloud-only support for omitted synthetic Dataset pricing without weakening result-price validation, spend checks or internal event-count verification. It retains the 0.1.3 encoding repair. It adds explicit HTTP/BOM/HTML-declaration decoding using the already pinned Beautiful Soup dependency. Runtime dependency pins, output/dataset schemas, resource limits and examples are unchanged. The input schema now includes the explicit-empty compatibility fields described above. Historical archives and exact-source approvals retain their original identities and do not authorize this changed package.
The saved real response from https://日本語.jp/ (ASCII host xn--wgv71a119e.jp) is Shift_JIS HTML with no HTTP charset. Its saved source body has 8703 bytes and SHA256 bed87b619effb7434922592cbfd8e790212718db52971c2caa9d08f847c84676. Replaying those bytes through the repaired extractor changes 691 replacement characters in the old clean text to zero, with readable Japanese title and text. Saved IANA and three Python-document outputs retain their non-provenance values. This is a finite saved-source regression, not new live/cloud acceptance or a universal encoding guarantee. The separate Austria IDN probe failed and is not counted as a successful path.
The independently reviewed documentation repair is recorded in reviews/content-review-v4-documents.md in the project goal evidence. The 0.1.4 runtime has an independent review, 268 passing local tests and the dated cloud receipts above. This schema-and-document update preserves runtime, output/dataset schemas and dependency bytes; its complete source map and new exact-build acceptance are recorded separately.
Full primary contract acceptance, exact-build source/PPE acceptance, unit economics, public Store state and platform automatic QA are distinct evidence tiers. The dated bounded receipts above are not a claim that all tiers have passed. Unpublished primary private algorithms remain unknown; advertised bulk/concurrency/www/IDN/clean/dedup capabilities and our disclosed extensions remain in scope.