Web Page to Markdown

Turn any web page URL into clean, LLM-ready Markdown, plain text and metadata in one call, with an optional AI summary. Built for AI engineers, RAG pipelines, AI agents and no-code automations that need to "read this page" reliably.

Quick start

1. Paste a page into Page URL (an article, a docs page, a blog post). To read a list of pages in one run, add them under More page URLs (up to 100; one row and one charge per page read).
2. Optionally switch on Generate AI summary for a TL;DR and key points (no API key needed; see AI and your data).
3. Click Start, then open the Output tab or download JSON, CSV or Excel.

Sample output

{
  "ok": true,
  "url": "https://journal.example.com/guides/widgets",
  "title": "How Example Widgets Work",
  "byline": "Alex Example",
  "siteName": "Example Journal",
  "publishedTime": "2026-09-01T08:00:00Z",
  "lang": "en",
  "wordCount": 83,
  "content": "How Example Widgets Work\n\nExample widgets turn raw input into useful output...",
  "markdown": "# How Example Widgets Work\n\nExample widgets turn **raw input** into *useful output*...",
  "tdmReserved": false,
  "tdmPolicy": null,
  "summary": "Widgets work in three stages: collect, transform and publish.",
  "keyPoints": ["Collect the input", "Transform it", "Publish the result"],
  "aiGenerated": true,
  "aiModel": "openai/gpt-4o-mini",
  "aiNotice": "AI-generated; may contain errors. Check facts before relying on them.",
  "aiGeneratedAt": "2026-10-17T09:00:02.000Z",
  "aiDigitalSourceType": "http://cv.iptc.org/newscodes/digitalsourcetype/trainedAlgorithmicMedia",
  "fetchedAt": "2026-10-17T09:00:02.000Z"
}

Without the AI summary, summary and keyPoints are null, aiGenerated is false and the other AI labels are null.

How much does it cost?

Event	Price
Page read	$0.004 per page on the free plan, $0.0036 on Bronze, $0.0032 on Silver, $0.0028 on Gold and above
AI summary (optional)	$0.02 per summary through Apify's AI gateway; no AI charge with your own OpenAI key

• 100 pages: $0.40 ($0.28 on Gold). With AI summaries: $2.40.
• 1,000 pages: $4.00 ($2.80 on Gold). With AI summaries: $24.00.

Platform usage is included in the price. You are charged only for pages that were read successfully. Pages that fail, are blocked, are disallowed by robots.txt, have no readable text or turn out to be documents (PDF, DOCX) are reported for free. The AI summary is charged only when a summary is produced. AI model usage on Apify's gateway can be billed separately; see AI and your data.

Pricing update on 16 November 2026. The prices above take effect on 16 November 2026 at 14:41 UTC (Apify's 14-day notice). Until then the previous pricing applies: $0.004 for every row in the results (less on higher plans), including rows for pages that could not be read. The AI summary price stays $0.02. The Pricing tab always shows what applies today.

AI and your data

• AI is off by default. When you switch on Generate AI summary, the page title and text go either to Apify's AI gateway (Apify's OpenRouter Actor, then OpenRouter, then the model provider; openai/gpt-4o-mini by default) or, with your own OpenAI key, straight to OpenAI.
• Apify may bill the AI model's usage to your Apify account separately through its OpenRouter Actor, at OpenRouter's rates; free-plan accounts pay more and may be blocked. That model usage isn't part of the prices above.
• Before the text is sent, email addresses and phone numbers in it are masked. Requests to the gateway ask OpenRouter to use only providers that don't collect your data, and only openai/... and anthropic/... models are used there (no :free or stealth variants; other model IDs fall back to openai/gpt-4o-mini). Pages that reserve text-and-data-mining rights are never sent to AI.
• If you choose a model, you also accept that model provider's terms and acceptable-use policy.
• Own OpenAI key: Use a dedicated OpenAI project key with a spending limit. It's used only inside your run, sent only to OpenAI, and never stored or logged. You're responsible for following OpenAI's terms. With your own key there is no AI charge.
• EU AI Act: AIDevs is the provider of this AI feature; you're the deployer. AI fields are marked with aiGenerated, aiModel, aiGeneratedAt, aiDigitalSourceType and aiNotice. Keep the label if you publish AI text.

Web Page to Markdown or Website to Markdown Crawler?

	Web Page to Markdown (this Actor)	Website to Markdown Crawler
Best for	Specific URLs you already know	Whole websites or sections, cheaply
Price	$0.004 per page	$0.001 per page
Cleaning	Richer: main-content detection, removes navigation, ads, cookie banners, newsletter and share widgets	Main-content extraction with standard cleaning
Metadata	Title, author, site name, published date, language, description, OG image, TDM reservation	Title, description, word count, links found, depth
AI	Optional AI summary and key points	None
Crawling	One URL per run, robots.txt respected by default	Follows links and sitemaps, URL patterns, respects robots.txt

Use this reader when an agent or workflow needs the best version of one page. Use the crawler to turn a whole documentation site or blog into Markdown.

FAQ

Is it legal? The Actor reads one public page that you choose: no logins, no cookies, robots.txt is respected by default, and it stops at CAPTCHAs, bot challenges and login walls instead of trying to get around them. What you may do with the content depends on the site and on copyright. Page text stays under its owner's copyright or licence; for example Wikipedia text is CC BY-SA and needs attribution when you reuse it. Each site's own terms apply as well (for Wikipedia pages, the Wikimedia Foundation's Terms of Use and its robot policy). The output can contain personal data that's on the page, such as author bylines (byline) or names, email addresses and phone numbers in the text. You are the data controller for it and need a lawful basis (for example under the GDPR) to store or use it, so keep only what you need. This is general information, not legal advice.

Does it respect robots.txt? Yes, by default (Respect robots.txt). Before reading a page it fetches the site's robots.txt once and follows the rules for AIDevsBot if the file has them, otherwise the rules for all bots (*); the longest matching rule wins, and redirect targets are checked too. If robots.txt answers 401 or 403, the site is treated as off-limits; if there is none (404), the page is read; if it can't be loaded (429, server error, timeout), the page isn't read in that run. Pages that aren't read for this reason are reported for free (ROBOTS_DISALLOWED or ROBOTS_UNREACHABLE). Turn the option off only for sites you own or have permission to read.

Requests identify as AIDevsBot; site owners can opt out with User-agent: AIDevsBot / Disallow: / in robots.txt.

What does tdmReserved mean? Under EU copyright law (Article 4(3) of the DSM Directive) a site can reserve its text-and-data-mining rights in machine-readable form. The Actor reads the TDMRep signals of each page, the tdm-reservation HTTP header and the <meta name="tdm-reservation"> tag (the tag wins), and reports them as tdmReserved (true or false) and tdmPolicy (the site's policy link, or null). A reserved page is still converted to text and Markdown, but it is never sent to AI (no AI call, no AI charge). The general EU text-and-data-mining exception (Article 4) doesn't cover such pages, so make sure you have the rightholder's permission before you use their text for data mining or AI training.

Do I need an OpenAI key for the summary? No. Summaries go through Apify's AI gateway for $0.02 per summary (Apify may also bill the model usage separately; see AI and your data). If you prefer, add your own OpenAI key: there is then no AI charge and OpenAI bills you for the tokens.

What happens when a page blocks bots? The Actor stops at CAPTCHAs, bot challenges and login walls, reports it in the output and the status message, and charges nothing. It never tries to get around them.

What happens to my data? AIDevs processes your input and results only to run the Actor. Nothing is kept or shared beyond your own Apify storages and the AI service you switch on.

Wrong charge? Report it in the Issues tab; we reply within 14 days.

Markdown or plain text? Both. markdown keeps formatting for LLMs, content is clean plain text. Set includeMarkdown: false if you only need text.

Limitations

• Up to 100 pages per run, listed by URL (use the crawler to discover the pages of a whole site).
• No JavaScript rendering: pages that build their content in the browser may come back empty (reported as EMPTY_CONTENT, free).
• Public pages only; no logins or cookies.
• Pages that robots.txt disallows, and sites whose robots.txt can't be loaded, aren't read (reported for free).
• Documents such as PDF or DOCX are not converted here; use PDF to Markdown Converter.
• AI summaries can contain mistakes; they are labelled aiGenerated: true with an aiNotice. Apify's AI gateway may not work on free-plan accounts; the page is then still read, without a summary and without an AI charge.
• Uses Apify's standard datacenter proxies only; residential and custom proxies aren't supported.

Works with AI agents through Apify's MCP server.

More tools from AIDevs

• PDF to Markdown Converter: PDF, DOCX and HTML documents to clean Markdown for LLMs and RAG.
• Website to Markdown Crawler: whole websites to Markdown at $0.001 per page.
• Competitor Website Analyzer: competitive, SEO and sales briefs from any website.
• SaaS Competitor Battlecards: sales battlecards from public pricing and feature pages.
• PricePulse AI: cross-marketplace product and price intelligence.
• Google Maps Lead Engine: local business leads with AI scoring.
• Nordic Jobs Scraper: Swedish and Norwegian job postings from official open-data APIs.
• LinkedIn Jobs Scraper Pro: public LinkedIn job listings, no login or cookies.

Not affiliated with, endorsed by or sponsored by the Wikimedia Foundation (Wikipedia), OpenAI, Anthropic, OpenRouter, Google, LinkedIn, LangChain, n8n, Make, Zapier or Slack.

---

How it works

1. Check robots.txt (unless you switch it off): one request per site, rules for AIDevsBot first, otherwise *.
2. Fetch the page at url as AIDevsBot, following up to 5 redirects; each redirect target is checked against its own robots.txt first.
3. Extract metadata: title, description, author/byline, site name, published time, language, OG image and the TDMRep reservation.
4. Clean: remove scripts, styles, navigation, header, footer, ads, cookie, newsletter and share widgets.
5. Isolate the main content: prefer <article>, <main> and common content containers, otherwise the densest text block.
6. Convert to Markdown (headings, lists, links, bold/italic, code, blockquotes, images) and derive well-spaced plain text.
7. Summarize (optional) with AI, with emails and phone numbers masked; skipped for pages that reserve text-and-data-mining rights.
8. Output one record to the dataset.

Input reference

Field	Type	Default	Description
url	string	(required)	The public web page to read.
urls	array	[]	More pages to read in the same run (up to 100 in total). One row and one "Page read" charge per page read.
includeMarkdown	boolean	true	Also return a clean Markdown version.
summarize	boolean	false	AI TL;DR and key points ($0.02 via Apify's AI gateway, no AI charge with your own key).
openaiApiKey	string (secret)	(empty)	Optional own OpenAI key for the summary.
model	enum	gpt-4o-mini	Model for the summary.
aiModel	string	(empty)	Advanced: model ID such as openai/gpt-4o-mini (on the gateway only openai/... and anthropic/...).
maxChars	integer	0	Cap returned text/Markdown length (0 = no limit).
respectRobotsTxt	boolean	true	Check robots.txt before reading. Turn it off only for sites you own or have permission to read.
proxyConfiguration	object	Apify Proxy	Proxy settings. Apify's standard datacenter proxies only; residential and custom proxies are ignored.

Example input: a list of pages in one run

{
  "url": "https://journal.example.com/guides/widgets",
  "urls": [
    "https://journal.example.com/guides/gadgets",
    "https://docs.example.org/getting-started"
  ],
  "includeMarkdown": true,
  "maxChars": 20000
}

Each page becomes its own row with its own "Page read" charge; pages that can't be read are free.

Output reference

Field	Description
url	The page URL that was read.
title, byline, siteName, publishedTime, lang, description, image	Page metadata (null when absent). byline is usually a person's name.
wordCount	Word count of the extracted text.
content	Clean plain text.
markdown	LLM-ready Markdown (null when includeMarkdown is off).
tdmReserved, tdmPolicy	Whether the site reserves text-and-data-mining rights for the page (TDMRep), and its policy link (null when none).
summary, keyPoints	AI summary (only when requested and produced).
aiGenerated, aiModel, aiNotice, aiGeneratedAt, aiDigitalSourceType	AI transparency labels: the model used, a notice, when the text was generated (ISO 8601, UTC) and the standard digital-source-type code for content made by a trained AI model.
aiError	Why a requested summary was not produced (otherwise null).
fetchedAt	ISO timestamp.

Failed reads have ok: false, error and errorCode (e.g. HTTP_404, BLOCKED, LOGIN_REQUIRED, ROBOTS_DISALLOWED, ROBOTS_UNREACHABLE, EMPTY_CONTENT, UNSUPPORTED_CONTENT_TYPE) and are free.

API and integrations

curl "https://api.apify.com/v2/acts/entranced_gelato~ai-web-page-reader/run-sync-get-dataset-items?token=<APIFY_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://en.wikipedia.org/wiki/Web_scraping", "includeMarkdown": true }'

Several pages in one call:

curl "https://api.apify.com/v2/acts/entranced_gelato~ai-web-page-reader/run-sync-get-dataset-items?token=<APIFY_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://example.com/article-1", "urls": ["https://example.com/article-2", "https://example.com/article-3"] }'

from apify_client import ApifyClient

client = ApifyClient("<APIFY_TOKEN>")
run = client.actor("entranced_gelato/ai-web-page-reader").call(
    run_input={"url": "https://example.com/article", "includeMarkdown": True}
)
item = next(client.dataset(run["defaultDatasetId"]).iterate_items())
print(item["title"], "->", item["wordCount"], "words")
print(item["markdown"][:500])

LangChain: load markdown (or content) into documents with ApifyWrapper.call_actor(...) and a dataset_mapping_function.

n8n, Make and Zapier: add the Apify node (or an HTTP request to the run-sync-get-dataset-items endpoint above) and map markdown, content and the metadata fields into the next step. Typical flows:

• URL to Slack summary: a webhook receives a URL, this Actor runs with summarize: true, and Slack posts the TL;DR.
• RAG ingestion: URLs from a sheet go through this Actor, the markdown field is chunked, and a vector database node stores the embeddings.
• Competitor blog monitor: an RSS trigger sends new posts here, and an LLM node classifies relevance.