Web Scraper & Crawler for RAG (Markdown)

Give your AI the web as clean, ready-to-use text

Turn any page, or an entire website, into clean Markdown or structured JSON that your LLM can actually use: no menus, no cookie banners, no boilerplate. $1.50 per 1,000 pages, and pages that come back empty are free.

What you get

• Clean content for RAG and chatbots. Only the main text, with links, images and page details when you want them.
• Whole sites in one run. Crawl with depth and path rules, or list every URL of a domain in seconds.
• Data, not just text. Describe the fields you need in plain words ("every job with title, location and link") and get JSON back, on your own Anthropic key at no extra charge from us.
• Pages that need JavaScript. A real browser is used only where a page needs one, so the rest stays fast and cheap.
• Know when a page changes. Watches re-check pages on a schedule and return only what changed.
• Plug into any agent. HTTP API and MCP server, ready for Claude, Cursor or your own code.

Built for

• AI engineers feeding RAG pipelines, chatbots and vector databases
• Growth and sales teams turning websites into lead and price lists
• Researchers and analysts collecting and monitoring web content
• Agent builders who need the live web as a tool

What data can SiteReader extract?

Field	Description
url	The page that was read
markdown	Main content as Markdown
text	The same content as plain text, ready for search indexes and vector databases
html / raw_html	Cleaned or original HTML
links / images	Every link and image on the page, as absolute URLs
screenshot	Full-page screenshot (PNG, base64)
summary	A short AI summary of the page
fields	The fields you asked for, extracted by AI
structured	What the page publishes for machines: schema.org items (jobs, products, articles), Open Graph, tables, headings - no AI cost
info	Title, description, language, status code, engine used, archive date when an archive copy was used

How to read a website as Markdown

1. Open the Input tab and choose What to do: Read pages, Explore a site, Discover a site's URLs, AI fields or Find.
2. Add one or more URLs (or a search query).
3. Pick the Output you need. For AI fields, choose AI fields and describe what to extract.
4. Click Start. Results appear in the Output tab; failed URLs are listed in the ERRORS record.

How much does it cost?

SiteReader uses pay per event: you pay only for what is done, and your Apify free monthly credit covers trying it.

Event	When it is charged
Page (fetch)	each page read with a fast HTTP fetch, a PDF/DOCX, or an archive copy
Page (browser)	each page that needs a real browser (JavaScript, steps, screenshots)
AI page	each page with an AI summary or AI fields, using Claude Haiku through SiteReader
AI page (premium)	the same with Claude Opus
Proxy page	each page read through a country proxy
Site map	each site whose URLs are discovered
Search query	each web search

AI pages are not charged when you use your own Anthropic key. You can set a maximum cost for every run.

Cost examples

Job	Cost
1,000 simple pages as Markdown	$1.50
1,000 JavaScript-heavy pages (real browser)	$4.00
1,000 pages with AI fields, your own Anthropic key	$1.50 (AI billed to your key)
A site map of one domain	$0.01
A page that comes back empty	free

Input

See the Input tab for every option. A few notes:

• Browser steps use a JSON list, for example [{"do": "click", "selector": "#load-more"}, {"do": "wait", "ms": 1000}].
• AI fields: end a URL with /* to let SiteReader pick the most relevant pages of that site.
• robots.txt is ignored by default; choose Respect to follow each site's rules.
• Discover also reads a site's /llms.txt (a list of pages written for AI) when it has one.
• Headers and cookies are sent only to the URLs of your run; pages read with them are never cached.

Output

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel.

{
  "url": "https://books.toscrape.com/",
  "markdown": "# All products\n\n* [A Light in the Attic](https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html) £51.77 ...",
  "fields": {"books": [{"title": "A Light in the Attic", "price": "£51.77"}]},
  "info": {"title": "All products | Books to Scrape - Sandbox", "status_code": 200, "engine": "fetch"}
}

Use it as an API

Run the Actor in standby mode and call it over HTTP:

curl -X POST "https://<your-standby-url>/v1/read" \
  -H "Authorization: Bearer $APIFY_TOKEN" -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "output": ["markdown", "links"]}'

Every answer has the same shape: {"ok": true, ..., "cost": {"units": 1}}, or {"ok": false, "error": "..."}. Long tasks (/v1/explore, /v1/read/batch, /v1/fields, /v1/lookup) return a job; follow it at /v1/jobs/{id}, or add deliver to have the results sent to your webhook or storage when done. Also: /v1/watches, /v1/recipes, /v1/feed and the one-line GET /v1/read?url=.... The full API description is on the Server tab.

Connect your AI agent (MCP)

SiteReader is also an MCP server, so Claude, Cursor, VS Code and any MCP client can use it as a tool. Add this to your client's MCP settings (your Apify token goes in the header):

{
  "mcpServers": {
    "sitereader": {
      "url": "https://<your-standby-url>/mcp",
      "headers": {"Authorization": "Bearer <APIFY_TOKEN>"}
    }
  }
}

Tools: read, read_batch (up to 10 pages), discover, explore, fields, find, find_batch (up to 5 searches) and usage. Pages that contain text trying to give orders to an AI are flagged with content_warning, so your agent treats them as data.

Tips

• Keep Engine on Auto: the browser is used only where it is needed, which keeps runs cheap.
• Use Main content and Remove these parts to keep the Markdown short for AI.
• For job boards and widgets that load late, set Wait after load to 1000-2000 ms.

FAQ, disclaimers and support

Is it legal to scrape websites? Our Actors are ethical and do not extract any private user data, such as email addresses, gender, or location. They only extract what the user has chosen to share publicly. We therefore believe that our Actors, when used for ethical purposes by Apify users, are safe. However, you should be aware that your results could contain personal data. Personal data is protected by the GDPR in the European Union and by other regulations around the world. You should not scrape personal data unless you have a legitimate reason to do so. If you're unsure whether your reason is legitimate, consult your lawyers.

A site returned an error or blocked the request. SiteReader stops reading that site for the run. Choose Use the archive copy to get its latest archived version instead.

Found a bug or need a feature? Open an issue in the Issues tab. For programmatic access, see the API tab.

byteblisk actors

Built and supported by byteblisk. Same record conventions, API style and MCP design across all three, so an AI agent can use them together:

• 📄 SiteReader: any website to clean Markdown or JSON for AI, with AI fields, site maps and watches.
• ⭐ App Reviews & Insights: Google Play and App Store reviews in one format, with AI insights and alerts.
• 🔎 Google Play & Search Toolkit: Play apps, reviews and search, top-100 search rank, and exports to your warehouse, in one actor.

Something missing or broken? Open an issue in the Issues tab: we answer every one.