# Website Document Index (`sapph1re/website-document-index`) Actor

Build a finite, source-linked manifest of document-link candidates with optional HEAD response metadata.

- **URL**: https://apify.com/sapph1re/website-document-index.md
- **Developed by:** [Roman V](https://apify.com/sapph1re) (community)
- **Stats:** 1 total users, 1 monthly users, 80.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$1.00 / 1,000 document candidates

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Website Document Index

Turn a small list of public resource pages into a document-link manifest. Each row contains one observed target URL, its declared format hints, and bounded anchor evidence from the pages that link to it. Optional HEAD checks add response metadata without reading document bodies.

Use it to export the document links actually observed on the supplied public pages. It does not claim to find every document on a website.

### First run

Use the default input to index the GNU license resource page:

```json
{
  "pageUrls": ["https://www.gnu.org/licenses/"],
  "maxCandidates": 100,
  "verifyMaxCandidates": 0,
  "verificationHosts": []
}
```

An API request with no input body and an explicit `{}` use the same defaults. An empty `pageUrls` array is invalid. The default public control produced eight ODT/RTF candidates on September 9, 2026; public pages can change.

To check response metadata for at most one candidate:

```json
{
  "pageUrls": ["https://www.gnu.org/licenses/"],
  "maxCandidates": 10,
  "verifyMaxCandidates": 1,
  "verificationHosts": ["www.gnu.org"]
}
```

The source returns a redirect for the first GPL document. HEAD stays on the same hostname, follows at most two redirects, and reads response headers only. A redirect uses another request attempt. A 405 response stays unknown; there is no GET or Range fallback.

### What becomes a candidate

An HTML `<a href>` becomes a candidate when its URL path has a supported extension, its `download` attribute is present, or its `type` attribute names a supported document media type. A filename in `download` can provide a format hint. An empty download attribute can produce a candidate with no format hint. Ordinary extensionless links with neither attribute are outside this version.

Supported hints: PDF, DOC, DOCX, XLS, XLSX, PPT, PPTX, ODT, ODS, ODP, ODG, RTF, CSV and TSV. Extensions are case insensitive. Query text and anchor text are not guessed to be filenames or formats. Archive links such as ZIP files are not document candidates unless an explicit download attribute makes them a candidate of unknown format.

Relative links use the fetched page URL and the first HTML base href, when valid. Script, style and template content is excluded. No JavaScript is executed. Unclosed anchors can be retained; malformed or blocked candidate links produce source diagnostics without logging their unsafe URL values.

### Reading the output

The default dataset contains candidate rows only. The key-value store's `OUTPUT` record contains source outcomes, limits, network counters and delivery/charging facts. Dataset CSV export may encode nested occurrences as JSON; use JSON to preserve all evidence.

A shortened row from the saved GNU source control:

```json
{
  "url": "https://www.gnu.org/licenses/gpl.odt",
  "formatHints": ["odt"],
  "discoveryMethods": ["url_extension"],
  "occurrenceCountSeen": 1,
  "occurrencesTruncated": false,
  "occurrences": [{
    "pageIndex": 0,
    "sourceUrl": "https://www.gnu.org/licenses/",
    "anchorIndex": 61,
    "observedUrl": "https://www.gnu.org/licenses/gpl.odt",
    "href": "/licenses/gpl.odt",
    "anchorText": "ODF"
  }],
  "verification": {
    "state": "NOT_REQUESTED",
    "observedMediaType": null,
    "observedFormat": null,
    "formatConfirmedFromBytes": false,
    "bodyBytesRead": 0
  }
}
```

Full rows also contain a SHA256 `candidateId`, discovery time, per-occurrence format/type/download hints, and text truncation flags. `candidate_schema.json` and `output_record_schema.json` describe the complete records.

Identity uses the resolved target URL without its fragment. Scheme/host spelling and explicit default ports are normalized. URL path/query spelling, percent-escape case, query order, repeated query parameters and an empty query delimiter remain distinct. Fragments remain on each `observedUrl`. Two queries are never assumed to identify the same document. Redirect destinations do not merge candidate identities. Repeated links merge into one candidate with multiple occurrences; duplicate input pages are fetched once and point to the first input index.

Each candidate retains up to 20 occurrences, with a 1,000-occurrence run total. The total counts every observed occurrence for retained candidates even when stored evidence fills. New candidates are not admitted without room for at least one occurrence. `droppedCandidateOccurrences` counts discarded link occurrences, not unknown distinct documents. `droppedEvidenceOccurrences` counts observations beyond the evidence cap. Anchor text is limited to 300 characters, download filenames to 256 and declared type text to 128, with truncation flags. These are bounded observed-page counts, not complete-site claims.

### Metadata means server declarations

`formatHints` and occurrence `declaredMediaType` describe the HTML or URL. `observedMediaType` is the final HEAD response's Content-Type. Neither proves the underlying file contents.

| Verification state | Meaning |
| --- | --- |
| NOT\_REQUESTED | Metadata verification was disabled |
| NOT\_CHECKED | The explicit verification count was exhausted |
| BLOCKED | The observed link's hostname was outside the verification policy |
| UNKNOWN | Missing/generic media type, non-200 response, timeout, network or safety failure |
| MEDIA\_TYPE\_MATCH | HTTP 200 has a recognized document media type matching every declared format hint |
| MEDIA\_TYPE\_OBSERVED | HTTP 200 declares a recognized document media type and there were no format hints |
| MISMATCH | HTTP 200 declares a media type inconsistent with the hint, including HTML at a PDF-looking URL |
| NON\_DOCUMENT\_MEDIA\_TYPE | HTTP 200 declares an unrecognized media type and there was no format hint |

Conflicting PDF and DOCX hints remain a mismatch even when one matches the response. Generic `application/octet-stream` remains unknown. Content-Length and a bounded Content-Disposition string are response declarations, not verified size or format. `formatConfirmedFromBytes` is always false.

### Host and network policy

Provide public pages you are authorized to inspect. HTTP and HTTPS use default ports only. No login, user credentials, custom headers, cookies, proxies, personal drives, search engines, guessed paths, crawl frontier or remote document extraction are supported. Known credential query keys and URL userinfo are rejected; do not place secrets in any input URL or query value, including opaque parameters the Actor cannot recognize as credentials.

External links are reported as observed. Verification requires an explicit exact lowercase hostname in `verificationHosts`, and each selected host must occur in `pageUrls`. There is no wildcard, suffix, subdomain or automatic external-host permission. Adding a source-host entry means supplying and fetching a real authorized index page on that host, not using a fabricated allowlist placeholder. Each source and HEAD redirect must remain on its initial exact hostname, with no HTTPS downgrade.

Every network attempt resolves DNS afresh, rejects any nonpublic result, connects to one validated address, and checks the connected peer. There is no reconnect or retry. DNS uses a child resolver with a killable deadline. Proxy environment variables and Set-Cookie responses do not change source routing. No browser or stored authentication is used.

Document verification stops at the final header delimiter even if a server sends an unsolicited body, advertises an enormous length, uses chunking, redirects, or rejects HEAD. The zero-body promise is about application reads: an uncooperative server can still send bytes into OS/TLS buffers. Explicit index GETs require HTML response metadata before any body read. A document falsely served as HTML can consume up to the bounded index response limit before its signature is rejected. Do not submit document URLs as index pages. The Actor intentionally fetches HTML index bodies and never intentionally fetches document bodies.

### Limits and outcomes

| Resource | Hard bound |
| --- | --- |
| Input index pages | 10, sequential; duplicate URLs fetched once |
| Candidate rows | 200 total; default 100 |
| Retained anchor occurrences | 20 per candidate and 1,000 total |
| Optional HEAD candidates | 20; default 0 |
| Redirects | 2 per source/HEAD operation |
| Network attempts | 90 total, including DNS-rejected attempts |
| Time for source collection and verification | 60 seconds shared, up to 8 seconds per request including DNS/connect/headers/body |
| Response headers | 16 KiB each; bounded header count |
| HTML wire bytes | 256 KiB per response, including chunk framing; an unknown-length overflow can read one sentinel byte |
| Decoded HTML | 512 KiB per response |
| HTML nodes and anchors | 10,000 parser events and 5,000 anchors per page |
| Dataset delivery | One append, one transport attempt, less than 4 MiB serialized total and 256 KiB per row |

HTML supports UTF-8, ASCII, ISO-8859-1 and Windows-1252. Identity, a single gzip member, and zlib-wrapped deflate are accepted with complete-stream and decoded-size checks. Brotli, raw deflate, concatenated members and trailing compressed data are rejected explicitly. Strict HTTP framing rejects ambiguous length/transfer headers, duplicate non-cookie headers, chunk extensions and trailers. These conservative limits can reject otherwise usable sites.

If long URL/provenance fields make the serialized dataset exceed its one-append limit, the run fails with `DATASET_PAYLOAD_LIMIT` before any append or event charge. The individual row and occurrence limits do not guarantee that every combination fits the batch limit.

Top-level `USEFUL` means candidate rows were selected without a source/collection limit, `EMPTY` means all processed pages had no candidates, and `PARTIAL` means at least one source, evidence or candidate budget was limited. `FAILED` means no usable source or a runtime/delivery error. `INVALID` means rejected input, before source access. Metadata unknowns and capped checks stay on individual rows; a useful manifest does not mean every candidate was verified. Process exit is 10 for invalid input, 11 for failure, and 0 for useful/empty/partial. Check `OUTPUT`, not just exit status.

`requestAttempts` counts operations admitted before DNS; `socketAttempts` counts actual connect attempts. Blocked redirects do not become request attempts. `metadataAttempts` counts admitted candidate checks, so a check stopped at the shared deadline can have no additional network attempt. `htmlWireBytesRead` excludes headers; `headerBytesRead` is separate. Counts describe this implementation, not independent packet telemetry. The 90-request count is a ceiling, not a completion guarantee. The shared deadline can leave later metadata checks UNKNOWN while preserving the discovered rows.

### Delivery, restarts and metering

Paid runs require the single event `apify-default-dataset-item` at exactly $0.001 per delivered candidate, with no other positive event. Check the current Store pricing before starting. A finite run event cap is required; a different price, missing event or extra fee fails before source reads.

The output records the effective event ceiling received from the platform. At $0.0025, at most two candidate rows can be delivered and charged ($0.002). The locally tested runtime admits no source reads or rows when it receives a zero or sub-event allowance, but API launch parameters of $0 or less than one event are not a no-spend promise: the platform may reject them or apply different effective options. Use a supported positive cap and inspect the actual run options and OUTPUT. With the hard 200-row limit, event charges cannot exceed $0.200. Compute, storage, operations and other platform costs are separate; an event cap does not prove a total operating-cost ceiling.

A persisted journal precedes collection and append. The dataset adapter makes one POST attempt with no redirects or automatic retry. If append outcome is uncertain, delivery and charging counts are null. If transport confirmed delivery but subsequent SDK charge bookkeeping failed, the delivered count remains known and charging is null. Confirmed counts survive later journal or OUTPUT save faults. No unknown count is presented as zero.

Restart behavior is covered by local tests only; hosted same-storage recovery has not been validated. Keep automatic restart disabled. In local tests, a completed journal reuses the saved receipt without source reads or another append, while incomplete or corrupt journals fail closed and preserve known facts. This is not an exactly-once service guarantee. Investigate uncertain delivery and charging before starting a replacement run. A new independent run uses new default storage.

### Local development and qualification

Install the hash-locked binary wheels into an isolated Python 3.12 environment. Run `python run_tests.py` from this Actor directory after installing the matching benchmark test dependencies. The local suite also uses the saved original benchmark fixtures and real GNU source control in the repository.

The package uses Apify SDK 4.0.0 and apify-client 3.2.0 with a pinned internal dataset seam. SDK upgrades require reviewing that seam and repeating delivery, charge and restart tests. Official input-schema validation uses `@apify/input_schema` 3.28.21 and AJV 8.17.1. The Docker base is the official `apify/actor-python:3.12` tag.

This Actor runs with Limited permissions and is deliberately small. It offers no document text, document download, deep crawl, dynamic rendering, site-completeness, amendment tracking, legal reuse rights or market-demand guarantee.

# Actor input Schema

## `pageUrls` (type: `array`):

One to ten exact public HTTP(S) index/resource pages on default ports. Omission uses the GNU license index. No crawl, login or document bodies. Duplicate input pages are fetched once.

## `maxCandidates` (type: `integer`):

Retain at most this many distinct document-link targets across the page list; a lower event budget reduces the limit. Queries stay distinct.

## `verifyMaxCandidates` (type: `integer`):

Zero leaves all candidates as hints. A positive value performs up to 20 HEAD-only checks, with no GET fallback or document body reads. Requires verificationHosts.

## `verificationHosts` (type: `array`):

Exact lowercase hostnames selected from pageUrls, for example www.gnu.org. Empty blocks all verification. No implicit subdomains or external-host checks. Verification redirects must remain on the candidate host.

## Actor input object example

```json
{
  "pageUrls": [
    "https://www.gnu.org/licenses/"
  ],
  "maxCandidates": 100,
  "verifyMaxCandidates": 0,
  "verificationHosts": []
}
```

# Actor output Schema

## `candidates` (type: `string`):

No description

## `outcomes` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "pageUrls": [
        "https://www.gnu.org/licenses/"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("sapph1re/website-document-index").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "pageUrls": ["https://www.gnu.org/licenses/"] }

# Run the Actor and wait for it to finish
run = client.actor("sapph1re/website-document-index").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "pageUrls": [
    "https://www.gnu.org/licenses/"
  ]
}' |
apify call sapph1re/website-document-index --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,sapph1re/website-document-index"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/YvewXa5VavCzr8I5F/builds/sYz7G4Oh8wbP96Tui/openapi.json