Website Crawler to Markdown — Same-Site Pages for LLM and RAG avatar

Website Crawler to Markdown — Same-Site Pages for LLM and RAG

Pricing

from $0.40 / 1,000 static page converteds

Go to Apify Store
Website Crawler to Markdown — Same-Site Pages for LLM and RAG

Website Crawler to Markdown — Same-Site Pages for LLM and RAG

Website crawler: Same-host Markdown crawls with sitemaps and RAG provenance. $0.0005 per static page; $0.003 per rendered page instead of a static page; $0.0025 per crawl with results; start extra; usage included; failed results uncharged.

Pricing

from $0.40 / 1,000 static page converteds

Rating

0.0

(0)

Developer

Logan Parker

Logan Parker

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

3 days ago

Last modified

Features

Limited permissions

Share

Website crawler returns Markdown from permitted same-host pages for LLM and RAG pipelines, with sitemap discovery, page and depth bounds, optional rendering and source provenance; robots and crawl-delay are enforced. AI agents can call and pay through Apify MCP. List event rates are $0.0005 per static page; $0.003 per rendered page instead of a static page; $0.0025 per crawl with results; start is extra, usage is included and failed results are uncharged.

Example result

Example record for the minimal input under How to use it, as this Actor returned it on 2026-10-28 in a run of its production image; the record is kept unchanged with the release.

{
"url": "https://docs.apify.com/actors/monetize",
"finalUrl": "https://docs.apify.com/actors/monetize",
"ok": true,
"seedUrl": "https://docs.apify.com/actors/monetize",
"discoveredFrom": null,
"discovery": "seed",
"depth": 0,
"status": 200,
"title": "Monetize Actors",
"description": null,
"lang": null,
"markdown": "> ## Documentation index\n> Fetch the complete documentation index at: https://docs.apify.com/llms.txt\n> Use this file to discover all available pages before exploring further.\n\n# Monetize Actors\n\nYou can monetize your automation, AI agent, and web scraping projects by publishing them as paid Actors on [Apify Store](https://apify.com/store).\n\n## Pricing models\n\nYou can publish Actors on Apify Store under one of the following pricing models:\n\n* [Pay per event (PPE)](https://docs.apify.com/actors/monetize/pay-per-event.md): Users pay for specific events that are programmatically triggered from the Actor's source code. You can define these events and include actions such as generating a single result or starting an Actor. You can also choose whether to pass the platform usage costs on to users.\n* Pay per usage: Users can run the Actor without any additional charges beyond the platform usage costs generated by the Actor.\n\nFor a detailed comparison of pricing models from the perspective of your users, see [Actors in Store](https://docs.apify.com/actors/running/actors-in-store.md).\n\n## Fees and costs\n\nYour payout is your Actor's revenue minus Apify's commission and platform usage costs. For details, see [Pricing and costs](https://docs.apify.com/actors/monetize/pricing-and-costs.md).\n\n### Platform usage\n\nRunning Actors generates platform usage costs for infrastructure, proxies, and storage. Who covers these costs depends on the pricing model and the mode in which the Actor runs.\n\nFor pay-per-event Actors, Apify deducts platform usage costs from your payout. If your Actor pricing doesn't cover these costs, you might run into a [negative profit](https://docs.apify.com/actors/monetize/pay-per-event.md#avoid-negative-profit). However, you'll never have to pay Apify for other users running your Actors.\n\n### Testing costs\n\nWhen you test your Actor on the Apify platform, costs for running the Actor are deducted from your prepaid usage.\n\nTry Creator plan\n\nWith the [Creator plan](https://apify.com/pricing/creator-plan), you get $500 of prepaid usage to test your Actors during the first 6 months.\n\n### Apify's commission\n\nApify keeps 20% of your Actor's revenue as a commission. Apify will never charge you if your Actor doesn't earn anything.\n\n## Make your Actor eligible for agentic payments\n\nAgentic payments let AI agents run and pay for your Actor without an Apify account. Which Actors an agent can run depends on how it pays:\n\n* Agents that buy a prepaid token from [Apify AGI](https://docs.apify.com/agents/payments.md) over [x402](https://docs.apify.com/integrations/x402.md) or [MPP](https://docs.apify.com/integrations/mpp.md) can run any Actor that uses [limited permissions](https://docs.apify.com/actors/development/permissions.md). The pricing model doesn't matter.\n* Agents that pay per request, such as with [Skyfire](https://docs.apify.com/integrations/skyfire.md), can run only Actors that meet the requirements below.\n\nTo be eligible for per-request agentic payments, an Actor must:\n\n* Use the [pay-per-event](https://docs.apify.com/actors/monetize/pay-per-event.md) pricing model. Pay-per-usage Actors are not supported.\n* Charge only for events, not for [platform usage](https://docs.apify.com/actors/monetize/pay-per-event.md#platform-usage-costs). Actors with the **Pay per event + usage** option switched on are excluded.\n* Run with [limited permissions](https://docs.apify.com/actors/development/permissions.md). Actors that request full permissions are excluded.\n* Not use [Standby](https://docs.apify.com/actors/running/standby.md) mode.\n\nThe Actor's developer must also have completed [identity verification (KYC)](https://docs.apify.com/actors/monetize/monthly-payouts.md#verify-your-identity). Until they do, none of their Actors are eligible.\n\nApify checks these requirements automatically, with no separate opt-in.\n\n## Handle free users\n\nWhen monetizing your Actor, you might want to limit features or usage for users on the Apify free plan. If you choose to do this, you *must* handle it transparently:\n\n* Communicate upfront: Clearly state any limitations in your Actor's `README` and input schema. Users should know about restrictions *before* they run the Actor.\n* Graceful exits: If a free user hits a limit, don't crash the Actor or return a system error. Instead, exit gracefully with a clear [status message](https://docs.apify.com/actors/development/programming-interface/status-messages.md#communicating-limitations) explaining the limit (e.g., \"Free tier limit reached\").\n* Avoid confusion: Never make a policy restriction look like a bug or platform error.\n\n## Actor analytics\n\nMonitor your Actors' performance through the [Actor Analytics](https://console.apify.com/actors/insights/analytics) dashboard under **Development > Insights > Analytics**.\n\nThe analytics dashboard allows you to select specific Actors and view key metrics aggregated across all user runs:\n\n* Revenue, costs and profit trends over time\n* User growth metrics (both paid and free users)\n* Cost per 1,000 results to optimize pricing\n* Run success rate statistics\n* User acquisition funnel analytics\n* Shared debug runs from users\n\nAll metrics can be exported as JSON for custom analysis and reporting.\n\n## Promote your Actor\n\nCreate search-engine-optimized descriptions and README files to improve search engine visibility. Share your Actor on multiple channels:\n\n* Post on Reddit, Quora, and social media platforms\n* Create tutorial videos demonstrating key features\n* Publish articles about your Actor on relevant websites\n* Consider creating a product showcase on platforms like Product Hunt\n\nRemember to tag Apify in your social media posts for additional exposure. Effective promotion can significantly impact your Actor's success, differentiating between those with many paid users and those with few to none.\n\nLearn more about promoting your Actor with [Apify's marketing checklist](https://docs.apify.com/academy/actor-marketing-playbook/promote-your-actor/checklist.md).",
"wordCount": 760,
"rendered": false,
"contentFormat": "markdown",
"canonicalUrl": null,
"fetchedAt": "2026-10-28T13:13:51.045176+00:00",
"contentSha256": "7e01ba0a0beeacc0496e6c6dd25cf456eed9491b78901c02105d9ada181c3b33",
"originalChars": 6017,
"returnedChars": 6017,
"truncated": false,
"chunks": [],
"warnings": []
}

How to use it

Start a run in Apify Console, by API or from an agent with an input like this one — it is the input Apify runs every day to test this Actor:

{"startUrls": ["https://docs.apify.com/actors/monetize"], "maxPages": 1, "maxDepth": 0, "useSitemaps": false, "render": "never", "maxDurationSecs": 600}

Start with the minimal JSON example above, then adjust the fields in the table below. Read the Example result and Output fields sections for the returned record, Errors for failure codes, and Events and prices for result events and the separate platform start.

FieldTypeMeaning
startUrlsarray1-10 public HTTP(S) seeds, one exact hostname and compatible port scope per run; no login, cookies or evasion. Produces one record per page attempt, not per seed.
maxPagesintegerMaximum unique page attempts for the whole run, including failed pages and seeds; not per seed.
maxDepthintegerSeeds have depth 0; sitemap discoveries depth 1; link edges add 1. No discovery at depth 0.
maxDurationSecsintegerWork deadline in seconds, including policy waits, navigation and extraction; stop early for output and charging.
useSitemapsbooleanDiscover same-host sitemaps through robots and conventional paths; listed pages still pass scope, depth and policy.
renderstringauto renders short HTML; never uses static extraction; always renders eligible HTML. Native Markdown is never rendered.
includeUrlGlobsstringUp to 20 newline-separated case-sensitive whole-URL fnmatch patterns; empty allows same-host URLs. Applies to seeds too.
excludeUrlGlobsstringUp to 20 newline-separated whole-URL fnmatch patterns; excludes win, including seeds. No brace or extglob syntax.
maxCharsintegerMaximum returned Markdown characters per page; original length, hash and truncation are explicit.
chunkCharsinteger0 disables chunks; otherwise 1000-20000 character spans with offsets, within the page record; no extra fee. No token or semantic guarantee.

Events and prices

EventTitleCharged whenFREE / BRONZESILVERGOLD / PLATINUM / DIAMOND
page-convertedStatic page convertedOne readable non-browser page record, after write. Never stacked with page-rendered.$0.0005$0.00045$0.0004
page-renderedRendered page convertedOne readable browser page record, after write, replacing page-converted.$0.003$0.0027$0.0024
crawl-openedCrawl openedOnce after the first readable successful page, in addition to that page event; absent if no page succeeds.$0.0025$0.00225$0.002

Platform usage is not passed through. Apify's automatic actor-start event remains $0.00005 per started GB (one GB or less counts once). The synthetic dataset-item event is absent. Charge limits stop further paying work; unprocessed inputs receive uncharged errors. Completed readable pending events are settled on abort.

Worked cost example at 1 GB and list: ten readable static pages cost $0.005, plus $0.0025 once after the first success and $0.00005 platform start, totaling $0.00755. One static page is $0.00305; 1,000 are $0.50255 if limits permit completion. One rendered page is $0.00555. Failed pages and chunks have no conversion fee; platform start applies even with no results.

Call it from an agent

Through Apify's MCP server (https://mcp.apify.com) use call-actor:

{
"actor": "hermeskd/website-to-markdown",
"input": {
"startUrls": [
"https://docs.apify.com/actors/monetize"
],
"maxPages": 1,
"maxDepth": 0,
"useSitemaps": false,
"render": "never",
"maxDurationSecs": 600
},
"callOptions": {
"maxTotalChargeUsd": 0.1,
"timeout": 660
},
"waitSecs": 30
}

Then call get-dataset-items with the returned defaultDatasetId. Read the free CRAWL_SUMMARY key-value record to see coverage, stop reasons, attempted/successful/failed counts and charged events; a capped crawl is explicitly partial. The summary distinguishes produced events, confirmed chargedEvents, actual pendingEvents and all unconfirmedEvents; pending or unconfirmed counts are not payment receipts. A durable readable result stays successful through abort even when settlement remains pending. Dedicated tool URL: https://mcp.apify.com?tools=hermeskd/website-to-markdown

Output fields

FieldMeaning
urlNormalized requested page URL, retained on refusals; one result per attempted page.
okTrue for a returned result; a failure is never charged.
finalUrlActual final fetched URL; canonical metadata never changes crawl identity.
seedUrlFirst input seed whose traversal admitted this page.
discoveredFromParent page or sitemap URL; null for seeds.
discoveryseed, link or sitemap.
depthShortest discovered admitted depth; seeds 0, sitemap pages 1.
statusHTTP response status; null when refused before navigation.
titlePage title; no author or contact metadata.
descriptionPage meta description.
langDocument language.
markdownReadable main content, truncated at maxChars.
wordCountWords in returned Markdown.
renderedTrue only for successful browser extraction.
contentFormathtml, markdown or text; documents are refused with a converter pointer.
canonicalUrlDeclared canonical metadata only; never fetched or used to drop a page.
fetchedAtISO fetch completion timestamp.
contentSha256SHA-256 of complete extracted Markdown before truncation; not an authenticity proof.
originalCharsComplete extracted Markdown character count.
returnedCharsReturned Markdown character count.
truncatedWhether the page was shortened.
chunksOptional ordered character spans of returned Markdown; each has index, startChar, endChar, text; no extra records or charges.
warningsBounded machine-readable extraction, render-fallback or discovery warnings.
errorFailure, always uncharged.

Limits and run time

1-10 seeds on one exact hostname; global maxPages 1-1000 (100 default), maxDepth 0-20 (3 default), maxDurationSecs 30-600 (300 default); partial crawls reported. Ten MiB per HTTP response (pages, browser resources and sitemaps; robots is capped at 500 KiB), at most 20 sitemap documents/10 MiB expanded per document, 50 MiB expanded in aggregate and 50 MiB sitemap bytes total, 10000 candidate URLs, 100 MiB returned UTF-8 and 200 MiB fetched page and browser-resource bodies per run. Two request starts per second per host maximum, further reduced by crawl-delay. No files, login, cookies, evasion or subdomain traversal. The platform start event is $0.00005 per GB (minimum one event), including on runs with no results; no custom event is charged without readable content. HTML is limited to 100,000 tags; browser images, fonts, media, WebSockets and non-GET/HEAD requests are blocked. Browser requests, including scripts and frames, obey the same per-host schedule. This can limit JavaScript applications requiring cross-host assets or state-changing requests. HTTP gzip/deflate bodies are decoded in bounded chunks; other unsolicited content encodings are refused. Generic and native Markdown bodies must prove UTF-8 text before filename/content sniffing.

The default run timeout is 660 seconds. Container and cold-cost measurements are recorded in the build evidence; no unmeasured speed or accuracy comparison is claimed.

Errors

CodeMeaning
http_errorThe source answered an unsuccessful HTTP status.
download_failedThe request or connection failed within its bounded deadline.
out_of_scopeURL, redirect or document frame leaves the exact hostname/allowed port or downgrade policy; no out-of-scope document fetched.
unsupported_content_typeNon-page file; no file conversion; message names the Hermes converter when available.
login_requiredSign-in wall; no login or consent action attempted.
empty_contentNo readable content; uncharged.
crawl_delay_exceeds_budgetThe required host wait cannot finish before the work deadline; nothing fetched early.
rate_limitedHTTP 429/503; stop this host rather than evade; partial results retained.
resource_limit_reachedA bounded body, sitemap, output or frontier resource cap stopped work.
duplicate_urlA redirect alias of an already returned final URL; not charged.
source_schema_changedA sitemap or structured source is malformed or unsupported.
robots_disallowedrobots.txt forbids fetching the URL; no request was made to it.
blocked_hostThe host is on the deny-list or resolves to a private or local address; no request was made.
too_largeThe response exceeded the size cap.
timeoutThe input did not finish within the run's time budget.
invalid_inputThe run input passed the input schema but cannot be used; the message names the field.
internal_errorUnexpected failure; the message carries the error class.
charge_limit_reachedThe run reached its maximum total charge; this input was not processed.

No failure record is charged. A run fails only when all input outcomes are Actor-side failures; caller errors and a mixture of caller/Actor failures do not fail the run.

Sources and terms

Only process content you have the right to process. URL fetches obey HermesBot robots-group selection, the shared deny-list, public-address checks on connections and redirects, honest identity, deadlines and rate limits. No caller credentials, cookies, CAPTCHA solving, residential proxies or evasion. Only content the caller has the right to process; the shared deny-list, robots rules and private-address guard apply to seeds, discoveries, redirects and browser documents. Sitemap URLs never grant permission; canonical URLs are metadata only. Apify documentation is the permissively licensed example source. Native Markdown discovery uses CommonMark link tokens; raw HTML links inside Markdown are not expanded.

Not affiliated

Independent Hermes Actor, not affiliated with operators of caller-selected websites.

Changelog

  • 1.0 (2026-10-28) — first bounded crawler implementation; publication held pending an eligible day and QA.