Website Crawler to Markdown — Same-Site Pages for LLM and RAG
Pricing
from $0.40 / 1,000 static page converteds
Website Crawler to Markdown — Same-Site Pages for LLM and RAG
Website crawler: Same-host Markdown crawls with sitemaps and RAG provenance. $0.0005 per static page; $0.003 per rendered page instead of a static page; $0.0025 per crawl with results; start extra; usage included; failed results uncharged.
Pricing
from $0.40 / 1,000 static page converteds
Rating
0.0
(0)
Developer
Logan Parker
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
3 days ago
Last modified
Categories
Features
Share
Website crawler returns Markdown from permitted same-host pages for LLM and RAG pipelines, with sitemap discovery, page and depth bounds, optional rendering and source provenance; robots and crawl-delay are enforced. AI agents can call and pay through Apify MCP. List event rates are $0.0005 per static page; $0.003 per rendered page instead of a static page; $0.0025 per crawl with results; start is extra, usage is included and failed results are uncharged.
Example result
Example record for the minimal input under How to use it, as this Actor returned it on 2026-10-28 in a run of its production image; the record is kept unchanged with the release.
{"url": "https://docs.apify.com/actors/monetize","finalUrl": "https://docs.apify.com/actors/monetize","ok": true,"seedUrl": "https://docs.apify.com/actors/monetize","discoveredFrom": null,"discovery": "seed","depth": 0,"status": 200,"title": "Monetize Actors","description": null,"lang": null,"markdown": "> ## Documentation index\n> Fetch the complete documentation index at: https://docs.apify.com/llms.txt\n> Use this file to discover all available pages before exploring further.\n\n# Monetize Actors\n\nYou can monetize your automation, AI agent, and web scraping projects by publishing them as paid Actors on [Apify Store](https://apify.com/store).\n\n## Pricing models\n\nYou can publish Actors on Apify Store under one of the following pricing models:\n\n* [Pay per event (PPE)](https://docs.apify.com/actors/monetize/pay-per-event.md): Users pay for specific events that are programmatically triggered from the Actor's source code. You can define these events and include actions such as generating a single result or starting an Actor. You can also choose whether to pass the platform usage costs on to users.\n* Pay per usage: Users can run the Actor without any additional charges beyond the platform usage costs generated by the Actor.\n\nFor a detailed comparison of pricing models from the perspective of your users, see [Actors in Store](https://docs.apify.com/actors/running/actors-in-store.md).\n\n## Fees and costs\n\nYour payout is your Actor's revenue minus Apify's commission and platform usage costs. For details, see [Pricing and costs](https://docs.apify.com/actors/monetize/pricing-and-costs.md).\n\n### Platform usage\n\nRunning Actors generates platform usage costs for infrastructure, proxies, and storage. Who covers these costs depends on the pricing model and the mode in which the Actor runs.\n\nFor pay-per-event Actors, Apify deducts platform usage costs from your payout. If your Actor pricing doesn't cover these costs, you might run into a [negative profit](https://docs.apify.com/actors/monetize/pay-per-event.md#avoid-negative-profit). However, you'll never have to pay Apify for other users running your Actors.\n\n### Testing costs\n\nWhen you test your Actor on the Apify platform, costs for running the Actor are deducted from your prepaid usage.\n\nTry Creator plan\n\nWith the [Creator plan](https://apify.com/pricing/creator-plan), you get $500 of prepaid usage to test your Actors during the first 6 months.\n\n### Apify's commission\n\nApify keeps 20% of your Actor's revenue as a commission. Apify will never charge you if your Actor doesn't earn anything.\n\n## Make your Actor eligible for agentic payments\n\nAgentic payments let AI agents run and pay for your Actor without an Apify account. Which Actors an agent can run depends on how it pays:\n\n* Agents that buy a prepaid token from [Apify AGI](https://docs.apify.com/agents/payments.md) over [x402](https://docs.apify.com/integrations/x402.md) or [MPP](https://docs.apify.com/integrations/mpp.md) can run any Actor that uses [limited permissions](https://docs.apify.com/actors/development/permissions.md). The pricing model doesn't matter.\n* Agents that pay per request, such as with [Skyfire](https://docs.apify.com/integrations/skyfire.md), can run only Actors that meet the requirements below.\n\nTo be eligible for per-request agentic payments, an Actor must:\n\n* Use the [pay-per-event](https://docs.apify.com/actors/monetize/pay-per-event.md) pricing model. Pay-per-usage Actors are not supported.\n* Charge only for events, not for [platform usage](https://docs.apify.com/actors/monetize/pay-per-event.md#platform-usage-costs). Actors with the **Pay per event + usage** option switched on are excluded.\n* Run with [limited permissions](https://docs.apify.com/actors/development/permissions.md). Actors that request full permissions are excluded.\n* Not use [Standby](https://docs.apify.com/actors/running/standby.md) mode.\n\nThe Actor's developer must also have completed [identity verification (KYC)](https://docs.apify.com/actors/monetize/monthly-payouts.md#verify-your-identity). Until they do, none of their Actors are eligible.\n\nApify checks these requirements automatically, with no separate opt-in.\n\n## Handle free users\n\nWhen monetizing your Actor, you might want to limit features or usage for users on the Apify free plan. If you choose to do this, you *must* handle it transparently:\n\n* Communicate upfront: Clearly state any limitations in your Actor's `README` and input schema. Users should know about restrictions *before* they run the Actor.\n* Graceful exits: If a free user hits a limit, don't crash the Actor or return a system error. Instead, exit gracefully with a clear [status message](https://docs.apify.com/actors/development/programming-interface/status-messages.md#communicating-limitations) explaining the limit (e.g., \"Free tier limit reached\").\n* Avoid confusion: Never make a policy restriction look like a bug or platform error.\n\n## Actor analytics\n\nMonitor your Actors' performance through the [Actor Analytics](https://console.apify.com/actors/insights/analytics) dashboard under **Development > Insights > Analytics**.\n\nThe analytics dashboard allows you to select specific Actors and view key metrics aggregated across all user runs:\n\n* Revenue, costs and profit trends over time\n* User growth metrics (both paid and free users)\n* Cost per 1,000 results to optimize pricing\n* Run success rate statistics\n* User acquisition funnel analytics\n* Shared debug runs from users\n\nAll metrics can be exported as JSON for custom analysis and reporting.\n\n## Promote your Actor\n\nCreate search-engine-optimized descriptions and README files to improve search engine visibility. Share your Actor on multiple channels:\n\n* Post on Reddit, Quora, and social media platforms\n* Create tutorial videos demonstrating key features\n* Publish articles about your Actor on relevant websites\n* Consider creating a product showcase on platforms like Product Hunt\n\nRemember to tag Apify in your social media posts for additional exposure. Effective promotion can significantly impact your Actor's success, differentiating between those with many paid users and those with few to none.\n\nLearn more about promoting your Actor with [Apify's marketing checklist](https://docs.apify.com/academy/actor-marketing-playbook/promote-your-actor/checklist.md).","wordCount": 760,"rendered": false,"contentFormat": "markdown","canonicalUrl": null,"fetchedAt": "2026-10-28T13:13:51.045176+00:00","contentSha256": "7e01ba0a0beeacc0496e6c6dd25cf456eed9491b78901c02105d9ada181c3b33","originalChars": 6017,"returnedChars": 6017,"truncated": false,"chunks": [],"warnings": []}
How to use it
Start a run in Apify Console, by API or from an agent with an input like this one — it is the input Apify runs every day to test this Actor:
{"startUrls": ["https://docs.apify.com/actors/monetize"], "maxPages": 1, "maxDepth": 0, "useSitemaps": false, "render": "never", "maxDurationSecs": 600}
Start with the minimal JSON example above, then adjust the fields in the table below. Read the Example result and Output fields sections for the returned record, Errors for failure codes, and Events and prices for result events and the separate platform start.
| Field | Type | Meaning |
|---|---|---|
startUrls | array | 1-10 public HTTP(S) seeds, one exact hostname and compatible port scope per run; no login, cookies or evasion. Produces one record per page attempt, not per seed. |
maxPages | integer | Maximum unique page attempts for the whole run, including failed pages and seeds; not per seed. |
maxDepth | integer | Seeds have depth 0; sitemap discoveries depth 1; link edges add 1. No discovery at depth 0. |
maxDurationSecs | integer | Work deadline in seconds, including policy waits, navigation and extraction; stop early for output and charging. |
useSitemaps | boolean | Discover same-host sitemaps through robots and conventional paths; listed pages still pass scope, depth and policy. |
render | string | auto renders short HTML; never uses static extraction; always renders eligible HTML. Native Markdown is never rendered. |
includeUrlGlobs | string | Up to 20 newline-separated case-sensitive whole-URL fnmatch patterns; empty allows same-host URLs. Applies to seeds too. |
excludeUrlGlobs | string | Up to 20 newline-separated whole-URL fnmatch patterns; excludes win, including seeds. No brace or extglob syntax. |
maxChars | integer | Maximum returned Markdown characters per page; original length, hash and truncation are explicit. |
chunkChars | integer | 0 disables chunks; otherwise 1000-20000 character spans with offsets, within the page record; no extra fee. No token or semantic guarantee. |
Events and prices
| Event | Title | Charged when | FREE / BRONZE | SILVER | GOLD / PLATINUM / DIAMOND |
|---|---|---|---|---|---|
page-converted | Static page converted | One readable non-browser page record, after write. Never stacked with page-rendered. | $0.0005 | $0.00045 | $0.0004 |
page-rendered | Rendered page converted | One readable browser page record, after write, replacing page-converted. | $0.003 | $0.0027 | $0.0024 |
crawl-opened | Crawl opened | Once after the first readable successful page, in addition to that page event; absent if no page succeeds. | $0.0025 | $0.00225 | $0.002 |
Platform usage is not passed through. Apify's automatic actor-start event remains $0.00005 per started GB (one GB or less counts once). The synthetic dataset-item event is absent. Charge limits stop further paying work; unprocessed inputs receive uncharged errors. Completed readable pending events are settled on abort.
Worked cost example at 1 GB and list: ten readable static pages cost $0.005, plus $0.0025 once after the first success and $0.00005 platform start, totaling $0.00755. One static page is $0.00305; 1,000 are $0.50255 if limits permit completion. One rendered page is $0.00555. Failed pages and chunks have no conversion fee; platform start applies even with no results.
Call it from an agent
Through Apify's MCP server (https://mcp.apify.com) use call-actor:
{"actor": "hermeskd/website-to-markdown","input": {"startUrls": ["https://docs.apify.com/actors/monetize"],"maxPages": 1,"maxDepth": 0,"useSitemaps": false,"render": "never","maxDurationSecs": 600},"callOptions": {"maxTotalChargeUsd": 0.1,"timeout": 660},"waitSecs": 30}
Then call get-dataset-items with the returned defaultDatasetId. Read the free CRAWL_SUMMARY key-value record to see coverage, stop reasons, attempted/successful/failed counts and charged events; a capped crawl is explicitly partial. The summary distinguishes produced events, confirmed chargedEvents, actual pendingEvents and all unconfirmedEvents; pending or unconfirmed counts are not payment receipts. A durable readable result stays successful through abort even when settlement remains pending. Dedicated tool URL: https://mcp.apify.com?tools=hermeskd/website-to-markdown
Output fields
| Field | Meaning |
|---|---|
url | Normalized requested page URL, retained on refusals; one result per attempted page. |
ok | True for a returned result; a failure is never charged. |
finalUrl | Actual final fetched URL; canonical metadata never changes crawl identity. |
seedUrl | First input seed whose traversal admitted this page. |
discoveredFrom | Parent page or sitemap URL; null for seeds. |
discovery | seed, link or sitemap. |
depth | Shortest discovered admitted depth; seeds 0, sitemap pages 1. |
status | HTTP response status; null when refused before navigation. |
title | Page title; no author or contact metadata. |
description | Page meta description. |
lang | Document language. |
markdown | Readable main content, truncated at maxChars. |
wordCount | Words in returned Markdown. |
rendered | True only for successful browser extraction. |
contentFormat | html, markdown or text; documents are refused with a converter pointer. |
canonicalUrl | Declared canonical metadata only; never fetched or used to drop a page. |
fetchedAt | ISO fetch completion timestamp. |
contentSha256 | SHA-256 of complete extracted Markdown before truncation; not an authenticity proof. |
originalChars | Complete extracted Markdown character count. |
returnedChars | Returned Markdown character count. |
truncated | Whether the page was shortened. |
chunks | Optional ordered character spans of returned Markdown; each has index, startChar, endChar, text; no extra records or charges. |
warnings | Bounded machine-readable extraction, render-fallback or discovery warnings. |
error | Failure, always uncharged. |
Limits and run time
1-10 seeds on one exact hostname; global maxPages 1-1000 (100 default), maxDepth 0-20 (3 default), maxDurationSecs 30-600 (300 default); partial crawls reported. Ten MiB per HTTP response (pages, browser resources and sitemaps; robots is capped at 500 KiB), at most 20 sitemap documents/10 MiB expanded per document, 50 MiB expanded in aggregate and 50 MiB sitemap bytes total, 10000 candidate URLs, 100 MiB returned UTF-8 and 200 MiB fetched page and browser-resource bodies per run. Two request starts per second per host maximum, further reduced by crawl-delay. No files, login, cookies, evasion or subdomain traversal. The platform start event is $0.00005 per GB (minimum one event), including on runs with no results; no custom event is charged without readable content. HTML is limited to 100,000 tags; browser images, fonts, media, WebSockets and non-GET/HEAD requests are blocked. Browser requests, including scripts and frames, obey the same per-host schedule. This can limit JavaScript applications requiring cross-host assets or state-changing requests. HTTP gzip/deflate bodies are decoded in bounded chunks; other unsolicited content encodings are refused. Generic and native Markdown bodies must prove UTF-8 text before filename/content sniffing.
The default run timeout is 660 seconds. Container and cold-cost measurements are recorded in the build evidence; no unmeasured speed or accuracy comparison is claimed.
Errors
| Code | Meaning |
|---|---|
http_error | The source answered an unsuccessful HTTP status. |
download_failed | The request or connection failed within its bounded deadline. |
out_of_scope | URL, redirect or document frame leaves the exact hostname/allowed port or downgrade policy; no out-of-scope document fetched. |
unsupported_content_type | Non-page file; no file conversion; message names the Hermes converter when available. |
login_required | Sign-in wall; no login or consent action attempted. |
empty_content | No readable content; uncharged. |
crawl_delay_exceeds_budget | The required host wait cannot finish before the work deadline; nothing fetched early. |
rate_limited | HTTP 429/503; stop this host rather than evade; partial results retained. |
resource_limit_reached | A bounded body, sitemap, output or frontier resource cap stopped work. |
duplicate_url | A redirect alias of an already returned final URL; not charged. |
source_schema_changed | A sitemap or structured source is malformed or unsupported. |
robots_disallowed | robots.txt forbids fetching the URL; no request was made to it. |
blocked_host | The host is on the deny-list or resolves to a private or local address; no request was made. |
too_large | The response exceeded the size cap. |
timeout | The input did not finish within the run's time budget. |
invalid_input | The run input passed the input schema but cannot be used; the message names the field. |
internal_error | Unexpected failure; the message carries the error class. |
charge_limit_reached | The run reached its maximum total charge; this input was not processed. |
No failure record is charged. A run fails only when all input outcomes are Actor-side failures; caller errors and a mixture of caller/Actor failures do not fail the run.
Sources and terms
Only process content you have the right to process. URL fetches obey HermesBot robots-group selection, the shared deny-list, public-address checks on connections and redirects, honest identity, deadlines and rate limits. No caller credentials, cookies, CAPTCHA solving, residential proxies or evasion. Only content the caller has the right to process; the shared deny-list, robots rules and private-address guard apply to seeds, discoveries, redirects and browser documents. Sitemap URLs never grant permission; canonical URLs are metadata only. Apify documentation is the permissively licensed example source. Native Markdown discovery uses CommonMark link tokens; raw HTML links inside Markdown are not expanded.
Not affiliated
Independent Hermes Actor, not affiliated with operators of caller-selected websites.
Related Hermes Actors
- AI Crawler Checker: Which AI Bots robots.txt Allows per URL — check declared robots.txt permissions for six AI crawlers at URL paths.
- Article Extractor — News & Blog Text, Date and Markdown by URL — an article's clean text, title and publish date, without the rest of the page.
- 100K+ indexed jobs — Career Site Jobs API | 2026-11-03 — search open jobs in the Hermes directory of Greenhouse and Lever boards, each confirmed live.
- Speech to Text — Transcribe Audio & Video URLs, Speaker Labels — speech to text with SRT and VTT subtitles.
- Video Converter — MP4 to MP3, Trim, GIF and Audio Extraction — audio and video conversion: MP4 to MP3, trim, GIF.
- Broken Links Checker — Links, Assets, Redirects and Their Pages — broken links, 404s and redirects on a site.
- 100K+ indexed jobs — Company ATS Finder API | 2026-11-03 — which hiring system and job board a company uses.
- SEC EDGAR Filings — Recent Company Forms, Dates and XBRL Flags — Read recent company financial filing metadata from the official SEC JSON API
- DOI Lookup — Crossref Titles, Venues, Dates and Citation Counts — crossref doi metadata.
- CVE Vulnerability Lookup — NVD CVSS Scores and Exploited Flags — look up NVD CVE dates, status, numeric CVSS scores and severity.
- Merge JSON Datasets — Deduplicate Rows, Export JSON and CSV — json dataset cleaner.
- DNS Lookup — Records, SPF Lookup Counts, DKIM & DMARC by Domain — DNS records and SPF, DKIM and DMARC checks.
- DOCX to Markdown — Word, Excel, PowerPoint and EPUB Converter — Word, Excel, PowerPoint and other office files to Markdown.
- Domain Availability Checker — Bulk RDAP Status and Registrar — whether domain names are available, from registry RDAP.
- USGS Earthquake Lookup — Event Magnitude, Depth, Coordinates — Read numeric USGS earthquake metadata for known event identifiers.
- Currency Exchange Rates — ECB Daily Euro Rates and Cross Pairs — Read latest ECB daily currency reference rates and calculated cross pairs.
- Bulk File Downloader — Save URL Lists to Storage With SHA-256 — files and images from URLs, saved with a public link each.
- GitHub Repo Stats — Stars, Forks, Subscribers and Open Issues — Read numeric metrics for known public GitHub repositories.
- 47K+ Greenhouse jobs — Live API | 2026-11-03 — every open job on a company's Greenhouse board.
- Hacker News Story Metrics — Scores, Comment Counts, Timestamps — read numeric Hacker News story scores, timestamps and comment counts by ID.
- HTTP GET Request — Bulk Response Bodies, Status and SHA-256 — public http get.
- Security Headers Checker — HSTS, CSP, Framing, Nosniff by URL — Check seven declared HTTP security header families without returning raw values.
- Image Background Remover — U2NetP Cutouts for JPEG, PNG, WebP — Remove image backgrounds with the small U2NetP model and return transparent PNG pixels as base64
- Image Converter — JPEG, PNG and WebP With Resize and Quality — image format converter.
- Image Metadata Remover — EXIF and GPS Scrub, Pixels Verified — GPS location, camera serial numbers and other private metadata removed from photos, losslessly.
- Image to Text OCR — Text With Word Positions and Confidence — OCR: the text in images.
- Image Upscaler — Real-ESRGAN 2x/4x for Photos and Graphics — image upscaling 2x or 4x, and photo restoration.
- 25K+ Lever jobs — Live API | 2026-11-03 — every published job on a company's Lever site, global or EU.
- Lighthouse Audit — Lab Web Vitals and Scores, Mobile & Desktop — Lighthouse scores and Core Web Vitals.
- npm Package Lookup — Latest Version, License, Dependency Count — read latest npm package versions, license tokens and dependency counts.
- Website Change Monitor — Page Text Diffs Against Your Snapshots — what changed on a page since the last check.
- PDF to Markdown — PDF Parser & Text Extractor With OCR — PDF to Markdown and text, with OCR for scanned pages.
- Public Holidays Calendar — Holiday Dates for Five Countries — build offline public-holiday calendars for five countries and bounded years.
- 99 Remote OK jobs — Remote OK Feed API | 2026-11-02 — current remote jobs from Remote OK, with salary and listing links.
- Research Paper Search — OpenAlex Papers, DOIs, Citation Counts — Search OpenAlex for work-level paper metadata without authors, affiliations, abstracts or full text
- RSS Feed Reader — Atom and JSON Feed Parser With Enclosures — every item of an RSS, Atom or JSON feed.
- SEO Audit — On-Page Score, Meta Tags, Schema & Fix Guidance — an on-page SEO audit: meta tags, schema, headings and links.
- Sitemap Extractor — Unique URLs From XML, Gzip & Index Sitemaps — the URLs in a site's sitemaps.
- Text to Speech — Offline English, German, Spanish WAV Voices — Convert short nonpersonal texts to offline formant WAV speech for agents
- Time Zone Converter API — Convert one offset-qualified instant to named IANA time zones using pinned data.
- URL to Markdown — Clean Text for LLM & RAG, With Selectors — a public job page as clean Markdown where the site allows it (deny-listed job sites are refused).
- Extract Video Frames — Timed Stills by Interval or Scene Cut — frames, thumbnails and scenes from a video.
- Website Color Palette — CSS Colors With Alpha and Usage Counts — Extract declared website CSS color literals and counts from public HTML and bounded linked stylesheets
- Favicon Extractor — Icons, Touch Icons, Manifests, Theme Color — declared website icons, theme colors and manifest links.
- Website Screenshot — Full Pages, Elements, Dark Mode & PDF — screenshots and PDF prints of web pages.
- Tech Stack Detector — Website Technologies With Match Evidence — the technologies a website runs, detected live.
Changelog
- 1.0 (2026-10-28) — first bounded crawler implementation; publication held pending an eligible day and QA.