Website Document Index

Turn a small list of public resource pages into a document-link manifest. Each row contains one observed target URL, its declared format hints, and bounded anchor evidence from the pages that link to it. Optional HEAD checks add response metadata without reading document bodies.

Use it to export the document links actually observed on the supplied public pages. It does not claim to find every document on a website.

First run

Use the default input to index the GNU license resource page:

{
  "pageUrls": ["https://www.gnu.org/licenses/"],
  "maxCandidates": 100,
  "verifyMaxCandidates": 0,
  "verificationHosts": []
}

An API request with no input body and an explicit {} use the same defaults. An empty pageUrls array is invalid. The default public control produced eight ODT/RTF candidates on September 9, 2026; public pages can change.

To check response metadata for at most one candidate:

{
  "pageUrls": ["https://www.gnu.org/licenses/"],
  "maxCandidates": 10,
  "verifyMaxCandidates": 1,
  "verificationHosts": ["www.gnu.org"]
}

The source returns a redirect for the first GPL document. HEAD stays on the same hostname, follows at most two redirects, and reads response headers only. A redirect uses another request attempt. A 405 response stays unknown; there is no GET or Range fallback.

What becomes a candidate

An HTML <a href> becomes a candidate when its URL path has a supported extension, its download attribute is present, or its type attribute names a supported document media type. A filename in download can provide a format hint. An empty download attribute can produce a candidate with no format hint. Ordinary extensionless links with neither attribute are outside this version.

Supported hints: PDF, DOC, DOCX, XLS, XLSX, PPT, PPTX, ODT, ODS, ODP, ODG, RTF, CSV and TSV. Extensions are case insensitive. Query text and anchor text are not guessed to be filenames or formats. Archive links such as ZIP files are not document candidates unless an explicit download attribute makes them a candidate of unknown format.

Relative links use the fetched page URL and the first HTML base href, when valid. Script, style and template content is excluded. No JavaScript is executed. Unclosed anchors can be retained; malformed or blocked candidate links produce source diagnostics without logging their unsafe URL values.

Reading the output

The default dataset contains candidate rows only. The key-value store's OUTPUT record contains source outcomes, limits, network counters and delivery/charging facts. Dataset CSV export may encode nested occurrences as JSON; use JSON to preserve all evidence.

A shortened row from the saved GNU source control:

{
  "url": "https://www.gnu.org/licenses/gpl.odt",
  "formatHints": ["odt"],
  "discoveryMethods": ["url_extension"],
  "occurrenceCountSeen": 1,
  "occurrencesTruncated": false,
  "occurrences": [{
    "pageIndex": 0,
    "sourceUrl": "https://www.gnu.org/licenses/",
    "anchorIndex": 61,
    "observedUrl": "https://www.gnu.org/licenses/gpl.odt",
    "href": "/licenses/gpl.odt",
    "anchorText": "ODF"
  }],
  "verification": {
    "state": "NOT_REQUESTED",
    "observedMediaType": null,
    "observedFormat": null,
    "formatConfirmedFromBytes": false,
    "bodyBytesRead": 0
  }
}

Full rows also contain a SHA256 candidateId, discovery time, per-occurrence format/type/download hints, and text truncation flags. candidate_schema.json and output_record_schema.json describe the complete records.

Identity uses the resolved target URL without its fragment. Scheme/host spelling and explicit default ports are normalized. URL path/query spelling, percent-escape case, query order, repeated query parameters and an empty query delimiter remain distinct. Fragments remain on each observedUrl. Two queries are never assumed to identify the same document. Redirect destinations do not merge candidate identities. Repeated links merge into one candidate with multiple occurrences; duplicate input pages are fetched once and point to the first input index.

Each candidate retains up to 20 occurrences, with a 1,000-occurrence run total. The total counts every observed occurrence for retained candidates even when stored evidence fills. New candidates are not admitted without room for at least one occurrence. droppedCandidateOccurrences counts discarded link occurrences, not unknown distinct documents. droppedEvidenceOccurrences counts observations beyond the evidence cap. Anchor text is limited to 300 characters, download filenames to 256 and declared type text to 128, with truncation flags. These are bounded observed-page counts, not complete-site claims.

Metadata means server declarations

formatHints and occurrence declaredMediaType describe the HTML or URL. observedMediaType is the final HEAD response's Content-Type. Neither proves the underlying file contents.

Verification state	Meaning
NOT_REQUESTED	Metadata verification was disabled
NOT_CHECKED	The explicit verification count was exhausted
BLOCKED	The observed link's hostname was outside the verification policy
UNKNOWN	Missing/generic media type, non-200 response, timeout, network or safety failure
MEDIA_TYPE_MATCH	HTTP 200 has a recognized document media type matching every declared format hint
MEDIA_TYPE_OBSERVED	HTTP 200 declares a recognized document media type and there were no format hints
MISMATCH	HTTP 200 declares a media type inconsistent with the hint, including HTML at a PDF-looking URL
NON_DOCUMENT_MEDIA_TYPE	HTTP 200 declares an unrecognized media type and there was no format hint

Conflicting PDF and DOCX hints remain a mismatch even when one matches the response. Generic application/octet-stream remains unknown. Content-Length and a bounded Content-Disposition string are response declarations, not verified size or format. formatConfirmedFromBytes is always false.

Host and network policy

Provide public pages you are authorized to inspect. HTTP and HTTPS use default ports only. No login, user credentials, custom headers, cookies, proxies, personal drives, search engines, guessed paths, crawl frontier or remote document extraction are supported. Known credential query keys and URL userinfo are rejected; do not place secrets in any input URL or query value, including opaque parameters the Actor cannot recognize as credentials.

External links are reported as observed. Verification requires an explicit exact lowercase hostname in verificationHosts, and each selected host must occur in pageUrls. There is no wildcard, suffix, subdomain or automatic external-host permission. Adding a source-host entry means supplying and fetching a real authorized index page on that host, not using a fabricated allowlist placeholder. Each source and HEAD redirect must remain on its initial exact hostname, with no HTTPS downgrade.

Every network attempt resolves DNS afresh, rejects any nonpublic result, connects to one validated address, and checks the connected peer. There is no reconnect or retry. DNS uses a child resolver with a killable deadline. Proxy environment variables and Set-Cookie responses do not change source routing. No browser or stored authentication is used.

Document verification stops at the final header delimiter even if a server sends an unsolicited body, advertises an enormous length, uses chunking, redirects, or rejects HEAD. The zero-body promise is about application reads: an uncooperative server can still send bytes into OS/TLS buffers. Explicit index GETs require HTML response metadata before any body read. A document falsely served as HTML can consume up to the bounded index response limit before its signature is rejected. Do not submit document URLs as index pages. The Actor intentionally fetches HTML index bodies and never intentionally fetches document bodies.

Limits and outcomes

Resource	Hard bound
Input index pages	10, sequential; duplicate URLs fetched once
Candidate rows	200 total; default 100
Retained anchor occurrences	20 per candidate and 1,000 total
Optional HEAD candidates	20; default 0
Redirects	2 per source/HEAD operation
Network attempts	90 total, including DNS-rejected attempts
Time for source collection and verification	60 seconds shared, up to 8 seconds per request including DNS/connect/headers/body
Response headers	16 KiB each; bounded header count
HTML wire bytes	256 KiB per response, including chunk framing; an unknown-length overflow can read one sentinel byte
Decoded HTML	512 KiB per response
HTML nodes and anchors	10,000 parser events and 5,000 anchors per page
Dataset delivery	One append, one transport attempt, less than 4 MiB serialized total and 256 KiB per row

HTML supports UTF-8, ASCII, ISO-8859-1 and Windows-1252. Identity, a single gzip member, and zlib-wrapped deflate are accepted with complete-stream and decoded-size checks. Brotli, raw deflate, concatenated members and trailing compressed data are rejected explicitly. Strict HTTP framing rejects ambiguous length/transfer headers, duplicate non-cookie headers, chunk extensions and trailers. These conservative limits can reject otherwise usable sites.

If long URL/provenance fields make the serialized dataset exceed its one-append limit, the run fails with DATASET_PAYLOAD_LIMIT before any append or event charge. The individual row and occurrence limits do not guarantee that every combination fits the batch limit.

Top-level USEFUL means candidate rows were selected without a source/collection limit, EMPTY means all processed pages had no candidates, and PARTIAL means at least one source, evidence or candidate budget was limited. FAILED means no usable source or a runtime/delivery error. INVALID means rejected input, before source access. Metadata unknowns and capped checks stay on individual rows; a useful manifest does not mean every candidate was verified. Process exit is 10 for invalid input, 11 for failure, and 0 for useful/empty/partial. Check OUTPUT, not just exit status.

requestAttempts counts operations admitted before DNS; socketAttempts counts actual connect attempts. Blocked redirects do not become request attempts. metadataAttempts counts admitted candidate checks, so a check stopped at the shared deadline can have no additional network attempt. htmlWireBytesRead excludes headers; headerBytesRead is separate. Counts describe this implementation, not independent packet telemetry. The 90-request count is a ceiling, not a completion guarantee. The shared deadline can leave later metadata checks UNKNOWN while preserving the discovered rows.

Delivery, restarts and metering

Paid runs require the single event apify-default-dataset-item at exactly $0.001 per delivered candidate, with no other positive event. Check the current Store pricing before starting. A finite run event cap is required; a different price, missing event or extra fee fails before source reads.

The output records the effective event ceiling received from the platform. At $0.0025, at most two candidate rows can be delivered and charged ($0.002). The locally tested runtime admits no source reads or rows when it receives a zero or sub-event allowance, but API launch parameters of $0 or less than one event are not a no-spend promise: the platform may reject them or apply different effective options. Use a supported positive cap and inspect the actual run options and OUTPUT. With the hard 200-row limit, event charges cannot exceed $0.200. Compute, storage, operations and other platform costs are separate; an event cap does not prove a total operating-cost ceiling.

A persisted journal precedes collection and append. The dataset adapter makes one POST attempt with no redirects or automatic retry. If append outcome is uncertain, delivery and charging counts are null. If transport confirmed delivery but subsequent SDK charge bookkeeping failed, the delivered count remains known and charging is null. Confirmed counts survive later journal or OUTPUT save faults. No unknown count is presented as zero.

Restart behavior is covered by local tests only; hosted same-storage recovery has not been validated. Keep automatic restart disabled. In local tests, a completed journal reuses the saved receipt without source reads or another append, while incomplete or corrupt journals fail closed and preserve known facts. This is not an exactly-once service guarantee. Investigate uncertain delivery and charging before starting a replacement run. A new independent run uses new default storage.

Local development and qualification

Install the hash-locked binary wheels into an isolated Python 3.12 environment. Run python run_tests.py from this Actor directory after installing the matching benchmark test dependencies. The local suite also uses the saved original benchmark fixtures and real GNU source control in the repository.

The package uses Apify SDK 4.0.0 and apify-client 3.2.0 with a pinned internal dataset seam. SDK upgrades require reviewing that seam and repeating delivery, charge and restart tests. Official input-schema validation uses @apify/input_schema 3.28.21 and AJV 8.17.1. The Docker base is the official apify/actor-python:3.12 tag.

This Actor runs with Limited permissions and is deliberately small. It offers no document text, document download, deep crawl, dynamic rendering, site-completeness, amendment tracking, legal reuse rights or market-demand guarantee.