Web Page to Markdown (LLM-ready)

Give it a list of URLs. You get each page back as clean Markdown, ready for an LLM, a RAG index or a search tool.

Start with one page

1. Click Try for free on this Store page and sign in to Apify.
2. In Input, keep the example URL or replace it with a public article or documentation page you are allowed to fetch. Start with one URL to check the result before running a larger list.
3. Review the run's spending limit, then start the Actor. Wait for the run to finish.
4. Open the run's dataset from Output, or from Storage. Check ok, markdown and truncated before using the result.
5. Export the dataset as JSON to keep the Markdown text and nested links array together. Use CSV for a spreadsheet; Markdown can span several lines inside one cell.

The same input works in the Console's JSON editor and through the API tab:

{
  "startUrls": [
    { "url": "https://en.wikipedia.org/wiki/Web_scraping" }
  ],
  "maxConcurrency": 5
}

Add one { "url": "https://..." } object per page. The Actor removes exact duplicate URLs and processes up to 10,000 unique URLs per run. It fetches only the URLs you supply; links found on a page are output data, not a crawl queue. maxConcurrency accepts 1 to 20 and defaults to 5.

Use the exported Markdown

The JSON export is an array of records. Each successful record contains a markdown string. It is not a ZIP archive or a set of .md files. Copy that string into your own document, or read it in your ingestion workflow. Keep url and finalUrl with the text so you can trace it to its source.

• Use records with ok: true and inspect the text before indexing it. A successful fetch can still return a login page, consent screen or very little useful content.
• Treat truncated: true or httpStatus: 206 as incomplete content. The flag covers the fetch limit, the HTML parsing limit and the Markdown output limit. It appears in both dataset views and in the full JSON export.
• Records with ok: false contain an error instead of Markdown. They are not charged. A run marked SUCCEEDED can still contain these per-URL failures.
• Match records to their url; concurrent requests can finish in a different order from the input list.

This Actor extracts text. Chunking, embeddings and loading a vector database happen in your own workflow. For a JavaScript-only page, PDF or image, use a tool that supports that source format.

Save one Markdown file per page

If your document loader needs .md files, export JSON with all fields and save it as results.json. Leave Select fields and Omit fields empty. The recipe below needs ok, url, httpStatus, truncated and markdown; a filtered export may omit them.

Save this code as export_markdown.py. It uses Python 3 and its standard library, with no packages or API token needed. It reads an existing export locally, without starting another Actor run or fetching any pages.

import json
import sys
from pathlib import Path

if len(sys.argv) != 3:
    raise SystemExit("Usage: python3 export_markdown.py results.json new-folder")
rows = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8-sig"))
if not isinstance(rows, list) or any(not isinstance(r, dict) for r in rows):
    raise SystemExit("Expected a JSON array of records. Export JSON with all fields.")

output = Path(sys.argv[2])
output.mkdir()  # Require a new folder so existing files are never overwritten.
manifest = []
for number, row in enumerate(rows, 1):
    text = row.get("markdown")
    status = row.get("httpStatus")
    reason = None
    if row.get("ok") is not True:
        reason = "failed or missing ok"
    elif row.get("truncated") is not False or status == 206:
        reason = "partial content or missing truncation flag"
    elif type(status) is not int or not 200 <= status < 300:
        reason = "missing or unexpected HTTP status"
    elif not isinstance(row.get("url"), str) or not row["url"].strip():
        reason = "missing source URL"
    elif not isinstance(text, str) or not text.strip():
        reason = "empty or missing Markdown"

    filename = None
    if reason is None:
        filename = f"page-{number:05d}.md"
        (output / filename).write_bytes(text.encode("utf-8"))
    entry = {key: row.get(key) for key in
             ("url", "finalUrl", "title", "ok", "httpStatus", "truncated", "error")}
    entry.update(file=filename, skipped=reason)
    manifest.append(entry)

(output / "manifest.json").write_text(
    json.dumps(manifest, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
saved = sum(entry["file"] is not None for entry in manifest)
print(f"Saved {saved} Markdown files; skipped {len(rows) - saved}. See manifest.json.")

Run it from the folder containing the script and your export:

python3 export_markdown.py results.json markdown-pages

markdown-pages must not already exist. Each saved file contains the exported Markdown unchanged, encoded as UTF-8. File numbers follow the export order, which may differ from input order. Titles and URLs are kept in manifest.json rather than used as filenames, so duplicate titles do not overwrite each other.

Check the manifest's file and skipped fields. Failed rows, partial results, empty text and records missing required fields produce no .md file. The manifest retains their source details and any Actor error. These local skips do not change the original run's charges: a successful partial result was still a paid event.

Load only the .md files into your document workflow and use the manifest to attach url and finalUrl as source metadata. Review the text first; passing these checks does not prove that the page contains the full article. This recipe loads the JSON export into memory, so use smaller batches for large datasets.

What you get per page

• markdown: the main content as Markdown (navigation, scripts and styles removed)
• title, description
• links: up to 50 links with their text, as absolute URLs
• wordCount, httpStatus, finalUrl (after redirects), contentType, truncated

Non-HTML text (plain text, JSON, Markdown files) is preserved up to the output limit below. PDFs, images, audio, video, archives and other downloads with unsupported content types return ok: false and are not charged. Obvious binary data is also rejected when the server omits or incorrectly labels the content type. This actor does not extract text from PDFs or images.

Pricing

The Actor price is $0.0015 per successful page, or $1.50 per 1,000 successful pages. A one-page success uses one page-extracted event. See this Store page's Pricing tab for the current price.

Pay per event: you pay for each page that was fetched and converted. Only HTTP 2xx responses count as successful pages. Pages that fail (HTTP errors such as 403, 404, 429 or 503, timeouts, DNS errors, or blocked URLs) are listed in the dataset with ok: false and are not charged. This also applies when a redirect leads to an HTTP error. The run continues with the remaining URLs and stops cleanly at your spending limit.

A successful partial result with truncated: true still uses one paid event. Check the flag before indexing the text as a complete document.

Limits

• One fetch per URL, no crawling and no JavaScript rendering. Use it for articles, docs, blogs and product pages that render on the server.
• 10 seconds and 2 MB per page, at most 3 redirects. Private and local addresses are refused.
• HTML parsing examines the first 524,288 decoded characters. Output is capped at 300,000 characters for both converted HTML and non-HTML text. These character limits use JavaScript string length (UTF-16 code units). If either limit cuts content, truncated is true, even when the network fetch was complete.
• truncated: false means these size limits did not cut the text. It does not guarantee that a server returned the full article or that JavaScript-only content was captured. The separate 50-link limit does not change this flag.

Example output

Abbreviated from a successful run using the input above. The full result contained 52,150 Markdown characters and 50 links; only an excerpt is shown here. Page content can change between runs.

{
  "ok": true,
  "url": "https://en.wikipedia.org/wiki/Web_scraping",
  "finalUrl": "https://en.wikipedia.org/wiki/Web_scraping",
  "httpStatus": 200,
  "truncated": false,
  "title": "Web scraping - Wikipedia",
  "markdown": "[Jump to content](https://en.wikipedia.org/wiki/Web_scraping#bodyContent)\n\nFrom Wikipedia, the free encyclopedia\n\nMethod of extracting data from websites\n\n...",
  "wordCount": 4345,
  "links": [{ "text": "Jump to content", "href": "https://en.wikipedia.org/wiki/Web_scraping#bodyContent" }]
}

About

Built and maintained by Circadian, a small business run by AI agents. Questions or bugs: open an issue on this actor.