Website to Markdown for AI

Turn web pages, PDFs or whole websites into clean Markdown for LLMs, RAG pipelines, vector databases and AI agents.

You give it page URLs, start URLs to crawl, or both. You get one row per page with:

• the main content as Markdown: headings, lists, tables, code blocks with their language, links and image alt text
• menus, footers, cookie banners, share bars, "related stories" and popups removed
• metadata: title, description, canonical URL, language, author, published and modified dates, site name, preview image
• word count and token count (o200k tokenizer, used by GPT-4o and newer)
• optional chunks for RAG, with chunk ids, token counts and the heading path of each chunk

Pages are fetched with plain HTTP, which is fast and cheap. A headless browser is used only for pages that need JavaScript to show their content, and those pages are priced separately.

Who it is for

• Teams that feed websites, documentation or news into a chatbot, search index or RAG system
• AI agent builders who need "read this URL" as a clean, short Markdown text
• Anyone who wants to keep a knowledge base in sync with a website (monitor mode)

Works with any language. Arabic, Chinese and other non-Latin sites are decoded correctly, including old charsets such as windows-1256, and link URLs stay readable in the Markdown.

Input

Single pages (no crawling):

{
    "urls": [
        "https://www.bbc.com/news/articles/cwzd6q630vjeo",
        "https://arxiv.org/pdf/1706.03762"
    ]
}

Crawl a documentation section into 512-token chunks:

{
    "startUrls": [{ "url": "https://docs.python.org/3/tutorial/index.html" }],
    "maxPages": 100,
    "maxCrawlDepth": 3,
    "chunkSize": 512,
    "chunkOverlap": 64
}

Main options:

Option	What it does
Page URLs	Pages to convert. Links are not followed.
Start URLs	Sites or sections to crawl. Same site only, inside the start URL's folder by default.
Max pages per start URL	Stops each crawl after this many saved pages.
Max link depth	How many clicks from the start URL.
Use sitemaps	Also finds pages from sitemap.xml. They are used after the linked pages.
Only crawl / Skip URLs matching	URL patterns such as https://example.com/blog/** or **/tag/**.
Render JavaScript	auto (browser only when needed), never, always.
Chunk size / overlap	Token-based chunks for RAG. 0 = off.
Main content CSS selector	Force the content area, for example article.
Keep images / Keep link URLs	Turn off to save tokens.
Respect robots.txt	On by default, including Crawl-delay.
Only new or changed pages	Monitor mode for scheduled runs.

Output

A saved page (real run, Markdown shortened):

{
    "url": "https://www.bbc.com/news/articles/cwzd6q630vjeo",
    "loadedUrl": "https://www.bbc.com/news/articles/cwzd6q630vjeo",
    "status": "ok",
    "httpStatus": 200,
    "title": "Eric Bauza, voice of Bugs Bunny and Daffy Duck, dies aged 46",
    "description": "Bauza had voiced a number of characters in this summer's animated hit Coyote vs. Acme.",
    "canonicalUrl": "https://www.bbc.com/news/articles/cwzd6q630vjeo",
    "language": "en-GB",
    "author": "Roland Hughes",
    "publishedAt": "2026-10-26T06:16:15.893Z",
    "modifiedAt": "2026-10-26T06:16:15.893Z",
    "siteName": "BBC News",
    "imageUrl": "https://ichef.bbci.co.uk/news/1024/branded_news/2ba1/live/0257f500-d100-11f1-9d25-c3fafb2ab4de.jpg",
    "wordCount": 294,
    "tokenCount": 579,
    "charCount": 2259,
    "markdown": "# Eric Bauza, voice of Bugs Bunny and Daffy Duck, dies aged 46\n\n![AFP via Getty Images A man in a dark suit with a close-cropped goatee smiles at a premiere](https://ichef.bbci.co.uk/news/480/cpsprodpb/2ba1/live/0257f500-d100-11f1-9d25-c3fafb2ab4de.jpg.webp)AFP via Getty Images\n\nEric Bauza, the voice artist behind Looney Tunes characters including Bugs Bunny, Daffy Duck and Porky Pig, has died aged 46. ...",
    "fetchedWith": "http",
    "extraction": "readability",
    "depth": 0,
    "startUrl": null,
    "referrerUrl": null,
    "contentHash": "2d6a1e37aeadcad01d2aa4e8311688c9e7da4ec1",
    "crawledAt": "2026-10-26T12:10:27.654Z",
    "chunks": [
        {
            "chunkId": "b82a8d99b288-0000",
            "chunkIndex": 0,
            "text": "# Eric Bauza, voice of Bugs Bunny and Daffy Duck, dies aged 46\n\n...",
            "tokenCount": 168,
            "headingPath": "Eric Bauza, voice of Bugs Bunny and Daffy Duck, dies aged 46"
        }
    ]
}

fetchedWith is http, browser, pdf or text. chunks appears only when Chunk size is above 0. Chunk ids are stable for the same URL, so you can upsert them into a vector database.

A page that could not be saved gets a free status row:

{
    "url": "https://docs.python.org/3/nope.html",
    "status": "not_found",
    "error": "HTTP 404",
    "httpStatus": 404,
    "loadedUrl": "https://docs.python.org/3/nope.html",
    "depth": 0,
    "startUrl": null,
    "referrerUrl": null,
    "charged": false,
    "crawledAt": "2026-10-26T12:10:27.657Z"
}

Status values: ok (paid), and free ones: not_found, http_error, login_required, blocked, network_error, fetch_failed, robots_disallowed, needs_javascript, empty, duplicate_content, pdf_no_text, pdf_failed, skipped_pdf, skipped_not_html, too_large, render_failed, invalid_input, skipped_cost_limit.

The run always ends as SUCCEEDED. The key-value store record RUN_SUMMARY lists pages saved, status counts, each crawl's scope, timings, requests and bytes.

How JavaScript pages are handled

With Render JavaScript = auto, every page is first fetched with plain HTTP. The browser is used only when:

• the page is an empty app shell (an empty #root, #app or #__next, a "please enable JavaScript" notice, or almost no text and many scripts), or
• the site refuses plain requests but may serve a browser.

The browser version is used, and charged as a browser-rendered page, only when it shows clearly more text than the plain version. Images, fonts, video and styles are not downloaded. Captchas are never solved: a page that still shows a bot check gets a free blocked row.

PDF links

PDF files (up to 8 MB, first 300 pages) are converted to Markdown text with <!-- page N --> markers. Scanned PDFs have no text layer and get a free pdf_no_text row. OCR is not included.

Monitor mode (only new or changed pages)

Turn on "Only new or changed pages" and schedule the actor (Apify Console > Schedules, for example daily). Each run compares the Markdown of every page with the previous run and saves only new or changed pages, with changeStatus set to new or changed. Unchanged pages are not saved and not charged. Use a different Monitor name for each schedule. In monitor mode, unchanged pages do not count toward Max pages, so each run can also discover pages it has not seen before.

Pricing

Pay per saved page. Platform usage is included in the price.

Event	Price
Page (plain HTTP, PDF or text file)	$1.00 per 1,000 pages
Browser-rendered page (only pages that need JavaScript)	$3.00 per 1,000 pages

Status rows (errors, blocked, skipped, duplicates, robots.txt) are free. Unchanged pages in monitor mode are free. Set a maximum cost per run in the run options and the actor stops cleanly when it is reached.

Limits

• Public pages only. No login, no cookies, no captcha solving.
• Sites that block automated visitors with captchas return blocked rows.
• Pages larger than 8 MB are skipped.
• Token counts use the o200k tokenizer. Other model families count slightly differently.
• Main content detection is automatic. For unusual layouts, set Main content CSS selector.
• Default memory is 1024 MB. Plain HTTP pages work at 512 MB; JavaScript pages need 1024 MB or more (one browser tab below 2 GB, two at 2 GB, three at 4 GB).

Tips

• Skip old documentation versions with patterns such as **/docs/3.*/**.
• For the fewest tokens, turn off Keep images and Keep link URLs.
• Put Page URLs and Start URLs in the same run; each crawl has its own page limit.