# Web & PDF to Markdown API for AI Agents, Cited and Verified (`lmhsolutions/web-page-to-markdown-pdf-to-text`) Actor

Returns mainText, mainMarkdown, title, links, metadata, sourceSpans (citable paragraph offsets + SHA-256), contentSha256, fetch timestamp and an Ed25519-signed receipt per URL. Web pages, PDF, .docx. Batch or sync HTTP API. Failed URLs are free.

- **URL**: https://apify.com/lmhsolutions/web-page-to-markdown-pdf-to-text.md
- **Developed by:** [LMH Solutions](https://apify.com/lmhsolutions) (community)
- **Categories:** AI, Developer tools, Integrations
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 web page fetcheds

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Web & PDF to Markdown API for AI Agents, Cited and Verified

**What you get per URL:** `mainText`, `mainMarkdown`, `title`, `links`, `metadata`, **`sourceSpans`** (citable paragraph ids with offsets and a SHA-256 each), **`receipt.contentSha256`** (hash of the raw bytes fetched), **`receipt.fetchedAt`** (timestamp), `receipt.textSha256` and an **Ed25519 `signature`** over all of it. Use it as a batch Actor or as a **synchronous HTTP API** (Standby).

Convert any **web page to Markdown**, **PDF to text** and Word **.docx to text**: paste a list of URLs and get back **one flat JSON item per URL** (URL to JSON for LLMs) with clean text, Markdown, metadata and links, ready for an LLM, AI agent or RAG index. **Failed URLs are free.** Also usable as an MCP tool via Apify's hosted MCP server: `https://mcp.apify.com/?actors=lmhsolutions/web-page-to-markdown-pdf-to-text`.

Optional **signed provenance**: a completeness report (what the extractor dropped), a prompt-injection scan and an **Ed25519-signed receipt** proving what was fetched, when and how. As far as we know, no other top Store extractor signs its output.

### Output (one item per URL, trimmed)

```json
{
  "url": "https://en.wikipedia.org/wiki/Sourdough",
  "finalUrl": "https://en.wikipedia.org/wiki/Sourdough",
  "httpStatus": 200,
  "docType": "html",
  "title": "Sourdough",
  "lang": "en",
  "mainText": "Sourdough is a type of bread made using ...",
  "mainMarkdown": "Sourdough is a type of bread made using ...",
  "links": [{ "url": "https://en.wikipedia.org/wiki/Bread", "text": "bread", "inMain": true }],
  "metadata": { "description": "...", "canonical": "https://en.wikipedia.org/wiki/Sourdough" },
  "billing": { "event": "page-fetched", "events": ["page-fetched", "signed-provenance"], "totalUsd": 0.033 },
  "completeness": { "coverageScore": 0.995, "droppedSectionsCount": 2, "warnings": [] },
  "security": { "riskLevel": "none", "injectionFindings": [] },
  "sourceSpans": { "count": 14, "spans": [{ "id": "s1", "start": 0, "end": 212, "chars": 212, "sha256": "…", "kind": "paragraph", "preview": "Sourdough is a type of bread…" }] },
  "receipt": { "contentSha256": "…", "outputSha256": "…", "textSha256": "…", "alg": "Ed25519", "keyId": "p2j-JX8i4uxGCZqSSDrd", "signature": "…" }
}
```

`completeness`, `security`, `sourceSpans` and `receipt` are only present with `signedProvenance: true` (the default). A failed URL is `{ "url": "...", "error": { "code": "...", "message": "..." } }` and costs nothing.

### Input

```json
{ "urls": ["https://en.wikipedia.org/wiki/Sourdough", "https://arxiv.org/pdf/1706.03762"] }
```

Optional: `signedProvenance` (default `true`), `sourcePackManifest` (default `false`), `markdown`, `links`, `maxLinks`, `includeDroppedText`, `jsonLd`, `sourceSpans` (default `true`), `injection` (`flag`/`strip`), `useApifyProxy`, `maxConcurrency`.
Verify mode: `{"mode":"verify","receipts":[...],"responses":[...optional full items...],"manifest":{...optional...}}`.

### Pricing (pay per event)

| Event | When | Price (Apify FREE plan) | Top tier (DIAMOND) |
|---|---|---|---|
| `page-fetched` | one HTML page fetched and cleaned | $0.003 | $0.0015 |
| `document-extracted` | one PDF or .docx extracted | $0.006 | $0.003 |
| `signed-provenance` (add-on) | per extracted item, when `signedProvenance` is on (default) | $0.03 | $0.015 |
| `source-pack-manifest` (add-on) | once per run, when `sourcePackManifest` is on and at least one URL succeeded | $0.10 | $0.05 |
| `receipt-verified` | per receipt or manifest checked in verify mode | $0.001 | $0.001 |
| Failed URL, 4xx/5xx page, cookie/login wall or bot challenge | | **free** | **free** |

Bronze, Silver, Gold and Platinum plans get prices in between. Plus Apify's standard tiny Actor-start fee. Examples (FREE plan): 100 web pages with provenance = $3.30; 100 pages with `signedProvenance: false` = $0.30; 20 PDFs plus a manifest = $0.82. The Actor respects your **maximum cost per run**: it reserves budget before each URL, stops cleanly when the next item would exceed it, and lists skipped URLs in the run summary (key-value store `RUN_SUMMARY`).

### Synchronous API (Standby mode)

No run to start and no dataset to poll: send one HTTP request, get one JSON back. Same events and prices as batch mode (cheap base fetch, plus the verified/cited add-on unless you turn it off). Failed URLs are free.

```bash
## verified + cited (page-fetched + signed-provenance)
curl -H "Authorization: Bearer $APIFY_TOKEN" \
  "https://lmhsolutions--web-page-to-markdown-pdf-to-text.apify.actor/extract?url=https://en.wikipedia.org/wiki/Sourdough"

## cheap base fetch only (page-fetched)
curl -H "Authorization: Bearer $APIFY_TOKEN" \
  "https://lmhsolutions--web-page-to-markdown-pdf-to-text.apify.actor/extract?url=https://arxiv.org/pdf/1706.03762&signedProvenance=false"

## JSON body, and receipt verification
curl -X POST -H "Authorization: Bearer $APIFY_TOKEN" -H "content-type: application/json" \
  -d '{"url":"https://example.com","links":false}' https://lmhsolutions--web-page-to-markdown-pdf-to-text.apify.actor/extract
curl -X POST -H "Authorization: Bearer $APIFY_TOKEN" -H "content-type: application/json" \
  -d '{"receipt":{...},"response":{...optional full item...}}' https://lmhsolutions--web-page-to-markdown-pdf-to-text.apify.actor/verify
```

Query/body options: `url` (required), `signedProvenance`, `markdown`, `links`, `maxLinks`, `jsonLd`, `sourceSpans`, `includeDroppedText`, `injection`. `GET /` lists endpoints and prices for free. HTTP 200 = delivered and charged; 4xx/5xx = not charged (422 means the URL loaded but had no extractable text, an error page or a wall). The first request after idle can take a few seconds (cold start). Standby runs also use some platform compute while waiting for requests, per Apify's Standby billing rules.

### Citing with source spans

`sourceSpans.spans[i]` gives `[start, end)` offsets into `normalizeText(mainText)`, the exact text hashed into the signed `receipt.textSha256`. To check a citation: slice the normalized text, SHA-256 it, compare with `span.sha256`; then verify the receipt signature. An agent can quote `s7` and a reviewer can prove that passage came from that URL at that time.

### Compared with apify/rag-web-browser (features only)

RAG Web Browser is Apify's own, widely used tool and the better choice if you need **Google search plus fetch in one call** or a **full headless browser** for JavaScript-heavy pages and collapsed sections. This Actor does not search and does not run a browser. Where it differs:

| | This Actor | apify/rag-web-browser |
|---|---|---|
| Web search built in | No (you pass URLs) | Yes (Google) |
| Headless browser for JS pages | No (plain HTTP; flags `js_rendered_likely`) | Yes, or plain HTTP mode |
| PDF and .docx to text | Yes | Not listed in its README |
| Synchronous HTTP API (Standby) | Yes | Yes |
| Raw-content hash + timestamp per page | Yes, signed | Load time in output; no hash listed |
| Citable source spans with per-span hashes | Yes | No |
| Ed25519-signed receipt + verify endpoint | Yes | No |
| Completeness score + dropped-section report | Yes | No |
| Prompt-injection scan / strip | Yes | No |
| Open source | No | Yes |

Comparison based on each Actor's public README as of November 2026; check their page for changes.

### Why signed provenance

- **Completeness report.** `coverageScore` (0–1), dropped and uncertain sections (with `clause` flags for payment, liability and deadline text), and warnings such as `js_rendered_likely`, `needs_ocr`, `headers_footers_dropped`.
- **Prompt-injection scan.** Hidden text and "ignore previous instructions" patterns are flagged or stripped. `riskLevel` is none, low, medium or high.
- **Signed receipt.** Covers URL, final URL, time, HTTP status, raw-bytes SHA-256, output SHA-256, normalized-text SHA-256, extractor versions and fetch provenance (redirect chain, direct or proxy, detected walls).
- **Source-pack manifest.** One signed list of every item's URL, `outputSha256`, `textSha256` and receipt id, so a whole RAG corpus or evidence pack can be checked in one step. Saved as dataset row `kind: "source-pack-manifest"` and key-value record `SOURCE_PACK_MANIFEST`.
- **Source spans.** Paragraph-level citation ids with offsets and per-span SHA-256, bound to the signed text hash.
- **Verify mode.** Re-check receipts, full items and manifests later at $0.001 per check.

### Features

- HTML via Readability, PDF via pdf.js (unpdf), Word via mammoth. Markdown, links, JSON-LD and metadata.
- Cookie/consent walls, login walls and bot challenges (Cloudflare, DataDome, AWS WAF, …) are detected and **not charged**.
- Blocks private and metadata IPs (SSRF), with size and time limits. Failed URLs return an error item instead of crashing the run.

### How completeness works

`coverageScore` is the share of meaningful visible text (by words) whose word trigrams appear in the extracted title, byline and main text. Navigation, cookie banners and footers are excluded and reported. A score below about 0.8, or `js_rendered_likely`, means you should not trust the text as the whole page.

### Verifying a receipt yourself

- **Public key** (Ed25519, JWK): `{"kty":"OKP","crv":"Ed25519","x":"W7fo3MOr-WnJe__hpWGTHlhucxLhhp_LCKGvTLBlAMc"}`, keyId `p2j-JX8i4uxGCZqSSDrd`.
- **Steps:**
  1. Take `receipt` and remove `signature`.
  2. Serialize to canonical JSON (recursively sorted keys, no whitespace).
  3. Prepend `page-to-json-receipt-v1\n`.
  4. Verify the base64url signature with the key above.
- **Check the output:** compute `outputSha256` as the SHA-256 of the canonical JSON of the item without `receipt`. Compute `textSha256` from `mainText` after the documented normalization (`receipt.textCanonicalization`).
- **Or skip the steps:** run this Actor with `mode: "verify"`.

### Verifying a manifest yourself

Remove `signature` from the manifest, serialize the rest to canonical JSON, prepend `page-to-json-source-pack-manifest-v1\n` and verify the base64url Ed25519 signature with the same public key. A `receiptId` is the SHA-256 hex of the canonical JSON of the full receipt (signature included).

### Limitations

- No JavaScript rendering. Single-page apps are flagged `js_rendered_likely`, and coverage is capped at 0.1.
- No OCR. Scanned PDF pages are flagged, not read.
- No login, no cookies, no CAPTCHA solving. Walls are detected and reported, not bypassed.
- Public URLs only. Private and internal addresses are blocked.
- Size and time limits apply per URL. Truncation is flagged in the receipt.

### FAQ

- **Do I need the signed provenance?** No. Set `signedProvenance: false` for plain text, Markdown, metadata and links at the base price only.
- **Why was a URL free?** It failed, returned 4xx/5xx, or hit a wall or bot challenge. `billing.notChargedReason` says which.
- **Can I use a proxy?** Yes, set `useApifyProxy: true` (charged as normal Apify proxy usage). The receipt records `method: "proxy"`.
- **Is the receipt proof the site said this?** It proves that this Actor fetched these bytes from this URL at this time and produced this output. It does not prove the site's content is true.
- **Standby mode?** Not in v1. For low-latency single calls, use the HTTP API version of Page to JSON.

# Actor input Schema

## `mode` (type: `string`):

extract: fetch each URL and return clean JSON (with a signed receipt when signedProvenance is on). verify: check receipts and/or a source-pack manifest from earlier runs.

## `urls` (type: `array`):

Public http(s) URLs of web pages, PDFs or Word .docx files. One dataset item per URL. Required in extract mode.

## `markdown` (type: `boolean`):

Add mainMarkdown next to mainText.

## `links` (type: `boolean`):

Return the links found in the page or document.

## `maxLinks` (type: `integer`):

Cap on returned links per URL (server cap 300).

## `includeDroppedText` (type: `boolean`):

By default each dropped section has a 160-character preview. Turn on to get its full text.

## `jsonLd` (type: `boolean`):

Return parsed schema.org JSON-LD blocks in metadata.

## `sourceSpans` (type: `boolean`):

With signed provenance: paragraph-level spans (id, start/end offsets into the hashed text, SHA-256 per span) so an agent can cite and anyone can re-check a quote.

## `injection` (type: `string`):

flag: report suspected prompt injection in `security`. strip: also remove medium/high-risk spans from mainText and mainMarkdown.

## `useApifyProxy` (type: `boolean`):

Fetch through Apify Proxy (datacenter by default). Recorded in each receipt's fetchProvenance. Does not unlock login walls or solve bot challenges.

## `proxyConfiguration` (type: `object`):

Optional Apify Proxy groups or country, used when Use Apify Proxy is on.

## `maxConcurrency` (type: `integer`):

URLs processed in parallel.

## `signedProvenance` (type: `boolean`):

On (default): each extracted item also gets a completeness report (coverage score, dropped sections), a prompt-injection scan and an Ed25519-signed receipt you can verify later. Charged $0.03 per successfully extracted page or document (FREE-plan price), on top of the base price. Off: you get clean text, Markdown, metadata and links only, without the completeness, security and receipt sections, and this add-on is not charged.

## `sourcePackManifest` (type: `boolean`):

On: when at least one URL succeeds, add one extra signed manifest item (also saved as SOURCE_PACK_MANIFEST in the key-value store) listing every extracted item's URL, outputSha256, textSha256 and receipt id, signed with the same Ed25519 key. Verify it later with mode verify. Charged once per run ($0.10, FREE-plan price); not charged if every URL failed.

## `receipts` (type: `array`):

Verify mode: receipt objects (the `receipt` field of earlier dataset items).

## `responses` (type: `array`):

Verify mode: the full dataset items, in the same order as receipts (or alone; the receipt is taken from each item). Also checks outputSha256 and textSha256.

## `manifest` (type: `object`):

Verify mode: a source-pack manifest (the `manifest` field of the manifest item, or the SOURCE_PACK_MANIFEST record). If receipts are also given, each must be listed in the manifest.

## Actor input object example

```json
{
  "mode": "extract",
  "urls": [
    "https://en.wikipedia.org/wiki/Sourdough",
    "https://arxiv.org/pdf/1706.03762"
  ],
  "markdown": true,
  "links": true,
  "maxLinks": 300,
  "includeDroppedText": false,
  "jsonLd": true,
  "sourceSpans": true,
  "injection": "flag",
  "useApifyProxy": false,
  "maxConcurrency": 5,
  "signedProvenance": true,
  "sourcePackManifest": false
}
```

# Actor output Schema

## `results` (type: `string`):

Dataset items, one per URL (or per verified receipt in verify mode).

## `runSummary` (type: `string`):

JSON summary of the run: version, mode, counts, timings, charged events.

## `sourcePackManifest` (type: `string`):

Signed manifest covering all extracted items (only when a source pack was requested).

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://en.wikipedia.org/wiki/Sourdough",
        "https://arxiv.org/pdf/1706.03762"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("lmhsolutions/web-page-to-markdown-pdf-to-text").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [
        "https://en.wikipedia.org/wiki/Sourdough",
        "https://arxiv.org/pdf/1706.03762",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("lmhsolutions/web-page-to-markdown-pdf-to-text").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://en.wikipedia.org/wiki/Sourdough",
    "https://arxiv.org/pdf/1706.03762"
  ]
}' |
apify call lmhsolutions/web-page-to-markdown-pdf-to-text --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,lmhsolutions/web-page-to-markdown-pdf-to-text"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/hPiBC9agNkTWeFsV1/builds/Or1Gw3Ainr46p4zcq/openapi.json