# Website Content Crawler | Bulk Text & Markdown (`peerless_columbine/independent-bulk-content-crawler`) Actor

Fetch public website pages in bulk and export clean text, titles, Markdown and source evidence, with bounded concurrency, robots checks and explicit partial results.

- **URL**: https://apify.com/peerless_columbine/independent-bulk-content-crawler.md
- **Developed by:** [tingyou333 zhuang](https://apify.com/peerless_columbine) (community)
- **Categories:** Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.00 / 1,000 delivered pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Independent Bulk Website Content Crawler

Supply **only `startUrls`**, a string array of bare domains, internationalized domains or public HTTP(S) page URLs. The Actor fetches the supplied pages concurrently, cleans public HTML text with explicit hidden-element filtering and emits unique `domain,text` results progressively. Optional fields add final URL, title, a clearly labeled leading-text summary excerpt, markdown, headings, links, source hashes, pinned IP/DNS route, robots decisions and explicit truncation state. These output fields are this Actor's own contract. There is no browser, proxy, login or access-control bypass.

**Supplied-page scope:** a bare domain resolves to its HTTPS root page; a supplied URL with a path addresses that page. Only normalized supplied URLs are scheduled, subject to robots, public-network checks, redirects and the documented caps. Extracted links are output metadata; they are not recursively scheduled. This is not a whole-site coverage promise.

### Primary public contract and comparison boundary

The fixed PRIMARY is **[6sigmag/fast-website-content-crawler](https://apify.com/6sigmag/fast-website-content-crawler)**. Its complete public README, input definition and pricing were captured anonymously on 2026-10-08 UTC and are preserved in the project research `actor_goal_20261009/research/content-primary/`. The public README SHA256 is `cbdf29f16db77a598757c38045a55157ed03f7524df4375a4150cd342de553b3`. The earlier `evidence/6sigmag--fast-website-content-crawler.json` is a historical tool summary, not the complete README or a native output schema.

The primary explicitly advertises bulk processing, www-prefix and international-domain handling, high concurrency, cleaning and deduplication. Its native input has only `startUrls`, an array with a `stringList` editor and a bare-domain example; it does not specify `items.type`, empty-array policy, numeric limits or URL normalization rules. This Actor's strict string validation, supported ranges and HTTPS/IDNA choices are its own disclosed contract.

The primary's observed public default definition supplies no native output/Dataset schema or published example rows. The older tool inferred `domain:string,text:string` from recent runs; that inference remains the historical compatibility target, not a verified native declaration. The complete README describes quick insights into content, structure and key elements but does not promise native summary/title/headings/links/Markdown fields. This Actor retains these fields as additional value: `summaryType:leading_text_excerpt` labels a source-derived excerpt, not an AI summary.

The primary links to separate specific-page and deeper-crawl tools. Those recommendations do not establish its exact page selection, recursion depth, link scheduling or URL-path policy. We neither assert that the primary only fetches one page nor that it recursively crawls an entire site. Its private cleaning/deduplication algorithms, normalization rules, numerical concurrency/performance and failure/charging order remain unpublished or unmeasured; observable public capabilities can be assessed without copying private algorithms.

**`apify/website-content-crawler` remains a secondary market comparator only**; it does not replace or enlarge the primary contract. This Actor does not claim the official Actor's JavaScript rendering, login, PDF/files, sitemap traversal or recursive crawling. A historical primary invocation returned HTTP 403 without a run ID in the then-current account/call environment. It provides no direct functional comparison and does not prove global disablement. No comparator is executed by the offline checks below. Release acceptance is recorded separately against the exact build; historical source receipts alone do not establish it. Usage counters and public prices are not buyer/payment/revenue evidence. See [COMPETITOR_MATRIX.md](COMPETITOR_MATRIX.md) for all contract rows, source attribution and remaining gaps.

### Examples

Minimal input (`examples/default.json`):

```json
{"startUrls":["https://docs.python.org/3/library/urllib.error.html"]}
```

Bulk input (`examples/bulk.json`):

```json
{
  "startUrls": [
    "docs.python.org/3/library/urllib.parse.html",
    "https://docs.python.org/3/library/urllib.request.html",
    "docs.python.org/3/library/urllib.error.html"
  ],
  "maxConcurrency": 3,
  "maxRequests": 12,
  "runTimeoutSecs": 90
}
```

`examples/normalization.json` illustrates www, bare domains, IDNs and URL fragments; those example domains are illustrative and are not an acceptance claim. `examples/caps.json` exercises partial output. `examples/security.json` demonstrates explicit unsafe-target failures. An empty `startUrls: []` is an empty batch with no custom `result` event. `startUrls` remains the sole required input at runtime: omitting it or passing `null` fails with `INVALID_INPUT`; neither means an empty batch. To allow an explicit empty array through Apify's required-array validation, the native `stringList` definition retains `required` and uses `nullable:true, default:null` as a compatibility sentinel. The platform may therefore create a run for a missing/null value before runtime rejects it; the startup fee and runtime/report costs can still apply, with no Dataset row or custom `result` charge. The HTTPS prefill and `examples/default.json` remain the normal nonempty default input. Native/local validation alone does not establish exact-build cloud empty-batch acceptance. A started standard run incurs the automatic startup fee even when no page result is produced; at 256 MiB that fee is $0.001. FREE source-validation receipts and owner-private PPE receipts are distinguished below. See the pricing status below.

A result always retains this Actor's stable core keys (the old tool's inferred primary compatibility target, not a published primary native schema):

```json
{
  "domain": "docs.python.org",
  "text": "Clean visible page text ...",
  "url": "https://docs.python.org/3/library/urllib.error.html",
  "title": "Page title from the source",
  "markdown": "# Source heading ...",
  "status": "succeeded",
  "caps": [],
  "textSha256": "SHA256 of delivered text",
  "provenance": {
    "startUrl": "https://docs.python.org/3/library/urllib.error.html",
    "inputIndex": 0,
    "redirects": [{"url":"https://docs.python.org/3/library/urllib.error.html","status":200,"ip":"validated pinned IP","dnsSource":"system or fixed_cloudflare_doh_fakeip_only"}],
    "robots": [{"url":"https://docs.python.org/robots.txt","status":200,"policy":"parsed","sha256":"body hash"}],
    "bodySha256": "source bytes hash",
    "bodyBytes": 24667,
    "fetchedAt": "UTC timestamp"
  }
}
```

This is a structural illustration. Current unabridged output in `evidence/real-output.json`, `real-report.json` and `local-acceptance.json` comes from the three saved real HTML documents replayed with the actual SDK and three concurrent fixture page workers, with network access prohibited. Provenance says `frozen_fixture_no_network` and does not invent a new pinned IP. Historical live transport evidence remains in `evidence/history/0.1.0/`; it does not bind the changed runtime. Output ordering follows completion; `inputIndex` records original order. `domain` is the final ASCII hostname with a leading www removed; full final hostname/path/query remain in `url`. Fragments and default ports normalize away. www and non-www are not assumed equivalent before fetching. Bare domains default to HTTPS; no automatic insecure fallback occurs.

### Safety, resource bounds and result semantics

Every supplied URL, redirect and robots redirect is normalized. Credentials, private/loopback/link-local/reserved IPs, IPv4-mapped/tunnel IPv6, non-HTTP protocols, scoped IPv6, local hostnames and nonstandard ports are refused. All system DNS answers must be public; a mixed public/private or fake/public answer fails. The validated numeric IP is passed to a pinned `httpcore.SyncBackend`; TLS SNI and certificate verification use the original hostname. Pools are per request with no retry, environment-proxy inheritance or hostname reconnection. Checks also apply to each redirect destination and each root robots fetch.

Only when **all** OS answers are Clash fakeIP in `198.18.0.0/15` may the default `fakeIpDohFallback:true` query fixed Cloudflare DoH: TCP `1.1.1.1`, HTTPS identity `cloudflare-dns.com`. Query/status, CNAME-to-address association and **all** returned addresses are validated. The target connection then pins the validated public answer with its original TLS hostname. A fallback for ordinary DNS failure, private answers or mixed answers is prohibited. Set the option false for explicit `LOCAL_FAKE_IP_DNS`. This fallback sends the requested hostname to Cloudflare and is recorded in provenance; it does not disable security.

The root `/robots.txt` is fetched before page access. 404/410 permits access, 401/403 denies, and transport errors/5xx/truncated or HTML robots responses fail closed. Robots redirects are bounded and must remain `/robots.txt`; cross-origin redirected robots retrieval still has public-DNS/pinning checks. Protego evaluates wildcard/end-anchor rules and longest allow/disallow precedence for each page and redirect target. Crawl-delay and request-rate directives conservatively pace per-origin starts within the same time budget; time-window rate extensions fail closed with an explicit code. Robots policy is cached per origin for this bounded run. Pages expose HTML or plain text only; unsupported files, compressed bodies and access challenges produce explicit failures. The client asks for identity encoding so source and decoded body size are bounded.

Character decoding uses a usable HTTP charset first, then a byte-order mark, then an HTML/XML encoding declaration within the first 4096 bytes for HTML, and finally UTF-8. Unknown or non-text codecs fall through; malformed bytes retain explicit replacement characters. Plain text ignores HTML meta declarations. No statistical encoding guesser is used. The same decoded text feeds title, clean text and Markdown extraction.

Defaults and hard supported ranges:

| Option | Default | Supported range |
|---|---:|---:|
| startUrls length | required | 0–100000 strings |
| maxConcurrency | 16 | 1–64 |
| maxPages | 10000 | 1–100000 |
| maxRequests | 30000 | 1–1000000 |
| maxBodyBytes | 2000000 | 1024–10000000 |
| maxTextChars | 500000 | 100–2000000 |
| maxRedirects | 5 | 0–10 |
| pageTimeoutSecs | 40 | 1–120 |
| runTimeoutSecs | 240 | 1–3600 |

`maxPages` caps attempted distinct supplied URLs, not discovered links; extracted links are metadata and are not scheduled. `maxRequests` includes robots, page, redirect and DoH HTTP requests. System DNS uses at most four concurrent killable subprocesses with bounded caller waits and a five-second maximum per lookup. Timeout kills and reaps the resolver child. Streaming reads/writes/TLS honor an absolute page/run deadline, including slow chunked bodies. HTML parsing operates only on the bounded body; its CPU time and final SDK/storage cleanup can add small overhead beyond the network deadline. No unbounded task queue or retries are used. Large inputs require appropriate memory/time/cap settings; limits report incomplete scope explicitly.

Clean text prefers main/article content; scripts/styles/navigation/footer/forms and explicit hidden elements are removed, whitespace normalized and repeated exact lines removed. Hidden filtering recognizes the HTML hidden attribute, case-insensitive aria-hidden=true and inline display:none, visibility:hidden/collapse and content-visibility:hidden. tinycss2 parses comments, escaped identifiers, declaration order and !important. Only the property-specific supported single-keyword values in `INLINE_VALUES` may update the inline declaration state; unknown or unsupported values are ignored even with !important. In particular, a later bogus display/visibility/content-visibility value cannot clear an earlier valid hidden declaration. CSS-wide keywords, multi-keyword values and computed styles are not evaluated. This is explicit declaration filtering, not browser computed CSS: external stylesheets, class rules, CSS variables, layout, media queries and descendants overriding inherited visibility are not evaluated. Markdown is a lightweight derivation, not a full fidelity converter; headings and normalized links are retained as separate bounded arrays (200 headings with at most 2000 characters each, 500 links); optional metadata truncation is reported in `metadataCaps`. Exact normalized input URLs, final URLs and delivered text hashes deduplicate results across the batch. Cross-domain identical text is one result, with duplicate diagnostics retaining input references. Near-duplicates are not falsely merged.

The already parsed DOM is reused for Markdown instead of parsing another complete DOM. Before Dataset delivery, a conservative ASCII-escaped JSON row bound of 8000000 bytes removes optional Markdown/links/headings first, retaining core text whenever it fits; if core text must be truncated, `caps:dataset_row_bytes`, partial state, a refreshed text hash and summary are explicit. UTF-8/non-BMP characters and JSON escaping count toward this row bound. Input limits and default concurrency are unchanged. This bounds serialized item size, not whole-process RSS, parse CPU or total protocol traffic. Bulk jobs/outcomes and checkpoint payloads still grow with input size; 100000 URLs/64 workers/10 MB bodies have no 256 MiB qualification.

Only nonempty results enter the default Dataset. Body/text caps yield a **delivered partial** result with `status:partial` and `caps`; hard time/request/page/spend caps and inaccessible sources yield failed/skipped outcomes, never successful empty placeholders. A 200 page with no visible content is `empty`. A source error is `failed`. Failures, empties, duplicates and skipped inputs appear in the progressively checkpointed `RUN-REPORT` KVS record and do not receive a `result` charge. Mixed batches are `partial`. An all-error batch fails the Actor; all empty content is an explicitly empty completed batch. An uncertain Dataset/charge write is recorded as `delivery_uncertain`; the run stops and never retries that write.

SDK delivery supports explicit FREE/source-validation mode and PAY_PER_EVENT. PPE requires a positive, finite `result` price. In cloud SDK mode, `apify-default-dataset-item` may be absent and contributes zero USD; the observed two-event price plan omits it. An explicitly configured finite nonnegative synthetic price is supported by the runtime for compatibility, but a positive price adds to each delivered row and is outside the observed two-event plan. Explicit zero remains an offline compatibility case, not a claim that the current platform accepts zero-price configuration. Missing synthetic pricing outside cloud SDK mode is rejected before writing because the pinned SDK uses a different local fallback. Optional `apify-actor-start` is platform managed; the Actor never charges it manually. Unknown event/synthetic configurations, other pricing models, missing prices or changes during a batch fail closed before a new write. Input URLs and concurrency are not reduced to accommodate bad pricing.

PPE uses SDK `actor.push_data(item, charged_event_name="result")` under the SDK reentrant charge lock, with combined custom/synthetic preflight. FREE uses uncharged SDK push. The 0.1.4 runtime configures one transport attempt for Dataset writes, custom charges and report writes using an Actor-local client and existing run-default storage. It pins the inspected SDK adapter versions, rejects alternate cloud storage IDs and does not mutate the installed SDK or its default retry policy for other Actors. Read failures can therefore fail sooner; unknown writes stop delivery and are not replayed. After the SDK returns, the Actor requires actual `ChargeResult.charged_count` to equal one for PPE or zero for FREE, both event-count deltas and total accounted amount to match the expected delivery, and the rest of ChargeResult to have the expected shape. An exception or mismatch after a possible write is terminal `DELIVERY_UNCERTAIN`, not a failed-empty row. Claiming work and checking/pushing completed work share one delivery lock. All completed workers recheck terminal state inside it; the first stop reason is retained, and no later worker pushes or charges after it.

Dataset, KVS and platform charging remain separate services without a cross-service transaction. A checkpoint failure halts new delivery and returns a failed in-memory report; the entry point logs terminal diagnostics. An unavailable KVS cannot promise a final stored report. No failed checkpoint or uncertain delivery is retried automatically; reconciliation must inspect Dataset and accounting before any new run. No exactly-once distributed guarantee is claimed.

Saved runtime-baseline receipts in `evidence/review-fix-0.1.2/` record the actual local SDK lifecycle and real ChargeResult against saved docs with a three-page barrier. The positive/zero synthetic-price cases, exact simulated $0.004 combined cap, invalid-event configurations, corrupted returned charged_count, write-after-success exception and checkpoint failure are tested. These are local test prices/accounting, not real charges or revenue. No validation script deploys, builds remotely, starts competitors or changes cloud pricing. Platform event configuration and actual remote charge/readback still require separate parent-owned verification.

### Pricing and dated validation evidence

The owner Console Monetization panel observed on **2026-10-09 UTC** shows active pay-per-event pricing with **User pays platform usage costs: No** for standard runs. This is effective configuration evidence, not a paying-customer invoice. The observed event plan is:

- custom `result`: **$0.003 per unique delivered nonempty page result**, including a delivered partial result;
- `apify-default-dataset-item`: **omitted**, with no additional Dataset event fee;
- automatic `apify-actor-start`: **$0.001 per event**, one event per GB of Actor memory with a minimum of one;
- platform usage is included for standard runs. Standby has separate platform billing and is not included in these examples.

At 256 MiB, one delivered result plus startup is $0.004, three results $0.010 and eight results $0.025. **Failed, empty, duplicate and skipped pages incur no custom result fee, but a started run still incurs its startup fee.** A run failing after earlier delivery can retain those earlier result charges. Startup is platform managed and is never charged manually by this Actor. Larger-memory runs can incur multiple startup events.

The primary's captured [public pricing](https://apify.com/6sigmag/fast-website-content-crawler/pricing), dated 2026-10-08 UTC, uses `apify-default-dataset-item` as its event key and `result` as that event's display title, at $0.003 per Dataset item. Its startup price is $0.0015 per event, one event per GB, minimum one, with platform usage included. For the same number of charged results and one startup event, our observed nominal event total is $0.0005 lower. This is a dated tariff comparison, not a claim of equivalent private cleaning/deduplication, measured speed, successful-only billing or current competitor output quality. The Store's current pricing is authoritative for a new run.

Runtime **0.1.4**, build `F78gaCG4fc3uTH8vp`, passed a FREE three-Python-document source case as run `PnEnKPfYjPNYNqhNI`, Dataset `IxjcNlSld7JLNTxAh`, with three real rows and four requests. An owner-private PPE cap case on that build, run `m03ca19BLa5eljTzU`, Dataset `wP7MpgH2CpZ0CGHZE`, delivered one real urllib.error result and recorded one `result` and one startup event, nominal total $0.004. Its application report is partial with `SPEND_CAP`, one succeeded and two skipped inputs; one skipped input had already completed collection. It therefore does not mean only one page was fetched. Saved observed run usage was $0.0003879846433583233; a later read revised it slightly downward, and the conservative earlier amount remains in the ledger.

These are owned validation receipts, **not customer revenue or full-workload profitability**. Build, storage retention, readback/recovery, subscription, failed workloads and maintenance costs are separate. The owner-run `USER` usage attribution is not evidence of a different customer's invoice; current standard-run configuration is supported separately by the Console panel. The API pricing-history projection omitted the requested false flag, so that omission is not interpreted as Boolean false.

Current local real-source probes separately delivered IANA plus readable Japanese from 日本語.jp with duplicate handling, and eight distinct Python documentation pages with eight observed workers. Offline production-runner controls covered 100000 short URLs with maxPages=1 (one result, 99999 explicit PAGE_CAP outcomes) and 64 distinct small fixture jobs at a worker barrier. These controls do not qualify 100000 complete live pages, maximum-sized inputs/bodies, cloud memory limits or network throughput. The 100000-input report encoded by the pinned SDK is about 14.38 MB; repeated report checkpoints, failures and duplicates can consume resources without generating result revenue. Select memory, time and cost limits for the actual workload; a low event cap does not cap platform resource consumption for the developer.

Earlier 0.1.2/0.1.3 FREE runs `R9hRzWNY8iZUR8mRz` / `h5qH7TKd6Kd47mRhb` and `HzTDL6hykWuxOQgQ4` / `lAF9Vvyl7qjpa1Xlg` remain historical evidence for their exact packages. Any final document-only build has a new complete source identity and its own exact-build release acceptance; the historical receipts above are not relabeled as that new build's tests.

### Reproduction

```sh
uv sync --frozen
.venv/bin/python -m pytest -q
apify validate-schema
.venv/bin/python scripts/offline_review_acceptance.py
```

The offline acceptance script runs the actual Actor SDK in children with inherited APIFY/ACTOR/CRAWLEE settings removed, isolated local storage and audit hooks rejecting DNS/connect/sendto. It replays saved real documents, records actual local Dataset rows and ChargeResult, and refuses to overwrite an existing `storage-review-0.1.2/` receipt. It proves concurrent offline behavior, not fresh live transport/cloud parity. The earlier `local_acceptance.py` performs live public HTTP and is historical; it is not current 0.1.4 cloud acceptance. The commands above are reproduction entry points, not evidence that this author reran them. Approval and exact-package status must be read from the independently issued records, not inferred from these local tests.

`uv.lock` pins runtime/dev dependencies; `requirements.lock` pins all runtime transitives with hashes for Docker `pip --require-hashes`. Docker uses Python 3.12.14, runs as uid 10001 and includes only code/Actor metadata. The Python image version is pinned; an image digest and actual Docker build/run are not verified when Docker is unavailable. SVG artwork is provided at `.actor/icon.svg`; cloud upload/icon format conversion is parent release work.

### Full requirement/evidence matrix

| ID | Public primary contract or own extension | Implementation / own promise | Evidence and remaining limit |
|---|---|---|---|
| P01 | primary: only required startUrls array, stringList UI; no native items.type or limits | own strict strings, sole required runtime input; native required/stringList retained with nullable/default-null compatibility sentinel | native validator and public-client replay plus runtime missing/null rejection; explicit \[] support still requires exact-build cloud acceptance; 0–100000 is our range |
| P02 | primary: bare-domain example; exact page/URL policy unpublished | HTTPS bare-domain root and public supplied page URLs | normalization fixtures and bare docs URLs; full URLs are own support, primary page-set parity unmeasured |
| P03 | primary: www handling advertised; transformation unpublished | final domain strips leading www; URL preserved; hosts not pre-merged | fixtures; exact primary policy unmeasured |
| P04 | primary: international domains advertised; encoding rules unpublished | strict IDNA UTS46 ASCII; URL Unicode escaping; explicit character decoding | saved real www IANA and Unicode 日本語.jp source probes; Shift_JIS repair replay; current exact-build cloud IDN qualification remains pending |
| P05 | primary: default Dataset pricing; domain/text only old tool inference, no published native output schema | own stable domain,text string rows | saved docs/local SDK replay and 0.1.4 FREE three-doc cloud receipt; not primary native schema proof |
| P06 | primary: bulk/high concurrency advertised; no numerical SLA | bounded rolling thread workers, default 16 / max 64 | 2/3/8-worker barriers, eight real local pages, 64-worker small-fixture control and three-doc cloud receipt; no primary speed or all-max qualification |
| P07 | primary: cleaning advertised; exact DOM/CSS algorithm unpublished | main/article preference; noise/explicit hidden removal | cleaning fixtures, saved docs, 0.1.2 inline CSS subset; no full browser CSS or byte parity |
| P08 | primary: deduplication advertised; scope/threshold unpublished | exact lines/input/final URL/text hashes, cross-domain identical-text merge | line/cross-page tests; near-duplicates preserved; no private-algorithm replication required |
| P09 | primary: broad content/structure/key-elements insights/use cases, not native structured-field promises | own labeled leading-text excerpt, title, headings, links, Markdown | source-derived fields retained and tested; no invented AI summary, sitemap or primary native-field guarantee |
| E01 | own additional output | stable final URL/title/Markdown | extraction tests and saved SDK rows; lightweight conversion |
| E02 | own source provenance | body/text hashes, timestamp, IP/DNS route, robots/trace | saved-source replay explicitly labeled; historical live transport separate |
| S01 | own safety | public target checks on every page/robots redirect | private redirect and second-origin robots tests |
| S02 | own safety | numeric TCP pin, original TLS SNI/certificate identity | transport tests; historical live receipts, no new transport test in this doc repair |
| S03 | own environment handling | all-fakeIP-only fixed DoH, no ordinary failure/private/mixed fallback | fake/mixed/private/error tests; route/privacy limitation disclosed |
| S04 | own policy | per-origin robots/pacing and bounded redirects | denied/private/new-origin/503/truncated/pacing tests |
| S05 | own resource limits | bounded body/time/request/concurrency; deadline and rolling worker window | stream/deadline/request/active-worker tests; maximum bulk memory not qualified |
| S06 | own access scope | no browser/proxy/login/access-control bypass | direct httpcore and explicit challenge failure |
| R01 | own result/error contract; primary failure details unpublished | partial vs failed/empty/skipped with RUN-REPORT | mixed/empty/cap/denial tests; no successful empty placeholders |
| R02 | own progressive delivery; primary ordering unpublished | serialized SDK push after completion and checkpointing | local SDK/checkpoint evidence; cross-service transaction not guaranteed |
| B01 | own configured custom result $0.003; Dataset synthetic event omitted, distinct from primary event identity | unique delivered nonempty/partial result; no custom event for failed/empty/duplicate/skipped pages | offline SDK accounting plus actual 0.1.4 owner-private PPE receipt: one result/startup; no customer billing claim |
| B02 | own spend/uncertain-write guarantees and startup $0.001 per GB, minimum one | combined preflight, receipt deltas, terminal first-stop/no retries | local cap/config/fault/barrier cases and actual SPEND_CAP receipt; empty/failed work does not waive startup; broader case acceptance separate |
| Q01 | own packaging | locked runtime deps, three native schemas, Docker entry | saved local receipts; actual Docker build/run not established by doc repair |
| Q02 | own documentation/examples/artwork | README plus full COMPETITOR_MATRIX, unchanged examples/icon | 0.1.4 runtime promises retained; schema/docs compatibility package and new exact-build acceptance recorded separately |
| Q03 | own evidence | 268-test reviewed runtime baseline with actual local SDK scenarios | independent runtime review and offline receipts; document-only changes do not rerun unchanged tests or count them as remote billing |
| G01 | comparison gate | fixed primary; generic public capabilities remain in scope | full public README/input/pricing obtained; historical 403 yields no direct runtime parity |
| G02 | exact-build source gate | every cloud receipt binds its recorded exact package | 0.1.4 build F78gaCG4fc3uTH8vp / run PnEnKPfYjPNYNqhNI accepted three real pages; final document-only build requires its own exact acceptance |
| G03 | demand/release gate | no verified buyer/payment/revenue or publication claim | demand signals and configured pricing are not paid adoption; independent final contract/economics/publication acceptance is separate |

### Evidence and release boundary

Current runtime version is **0.1.4**. It adds cloud-only support for omitted synthetic Dataset pricing without weakening result-price validation, spend checks or internal event-count verification. It retains the 0.1.3 encoding repair. It adds explicit HTTP/BOM/HTML-declaration decoding using the already pinned Beautiful Soup dependency. Runtime dependency pins, output/dataset schemas, resource limits and examples are unchanged. The input schema now includes the explicit-empty compatibility fields described above. Historical archives and exact-source approvals retain their original identities and do not authorize this changed package.

The saved real response from `https://日本語.jp/` (ASCII host `xn--wgv71a119e.jp`) is Shift_JIS HTML with no HTTP charset. Its saved source body has 8703 bytes and SHA256 `bed87b619effb7434922592cbfd8e790212718db52971c2caa9d08f847c84676`. Replaying those bytes through the repaired extractor changes 691 replacement characters in the old clean text to zero, with readable Japanese title and text. Saved IANA and three Python-document outputs retain their non-provenance values. This is a finite saved-source regression, not new live/cloud acceptance or a universal encoding guarantee. The separate Austria IDN probe failed and is not counted as a successful path.

The independently reviewed documentation repair is recorded in `reviews/content-review-v4-documents.md` in the project goal evidence. The 0.1.4 runtime has an independent review, 268 passing local tests and the dated cloud receipts above. This schema-and-document update preserves runtime, output/dataset schemas and dependency bytes; its complete source map and new exact-build acceptance are recorded separately.

Full primary contract acceptance, exact-build source/PPE acceptance, unit economics, public Store state and platform automatic QA are distinct evidence tiers. The dated bounded receipts above are not a claim that all tiers have passed. Unpublished primary private algorithms remain unknown; advertised bulk/concurrency/www/IDN/clean/dedup capabilities and our disclosed extensions remain in scope.

# Actor input Schema

## `startUrls` (type: `array`):

Bare domains, IDNs, or full public HTTP(S) URLs. One supplied page per entry.

## `maxConcurrency` (type: `integer`):

Resource safety limit; reached limits are reported explicitly.

## `maxRequests` (type: `integer`):

Resource safety limit; reached limits are reported explicitly.

## `maxPages` (type: `integer`):

Resource safety limit; reached limits are reported explicitly.

## `maxBodyBytes` (type: `integer`):

Resource safety limit; reached limits are reported explicitly.

## `maxTextChars` (type: `integer`):

Resource safety limit; reached limits are reported explicitly.

## `maxRedirects` (type: `integer`):

Resource safety limit; reached limits are reported explicitly.

## `pageTimeoutSecs` (type: `integer`):

Resource safety limit; reached limits are reported explicitly.

## `runTimeoutSecs` (type: `integer`):

Resource safety limit; reached limits are reported explicitly.

## `fakeIpDohFallback` (type: `boolean`):

Only all-fakeIP OS DNS uses fixed public Cloudflare DoH, validated answers, pinned TCP and hostname TLS. No fallback for other DNS failures.

## Actor input object example

```json
{
  "startUrls": [
    "https://docs.python.org/3/library/urllib.error.html"
  ],
  "maxConcurrency": 16,
  "maxRequests": 30000,
  "maxPages": 10000,
  "maxBodyBytes": 2000000,
  "maxTextChars": 500000,
  "maxRedirects": 5,
  "pageTimeoutSecs": 40,
  "runTimeoutSecs": 240,
  "fakeIpDohFallback": true
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `report` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://docs.python.org/3/library/urllib.error.html"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("peerless_columbine/independent-bulk-content-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": ["https://docs.python.org/3/library/urllib.error.html"] }

# Run the Actor and wait for it to finish
run = client.actor("peerless_columbine/independent-bulk-content-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://docs.python.org/3/library/urllib.error.html"
  ]
}' |
apify call peerless_columbine/independent-bulk-content-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,peerless_columbine/independent-bulk-content-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/sbFeVNhnEGCPordaA/builds/DfksG9ppOUHeO5CME/openapi.json