Pricing

from $5.00 / 1,000 page extracteds

Website to Markdown for LLMs & RAG — Content Extractor

Turn any URL or whole site into clean, LLM-ready Markdown, text, or JSON. Strips nav/ads/boilerplate; keeps headings, links, tables, code. Sitemap-aware concurrent crawl, URL filters, robots.txt respected, rich metadata. Charged only per page extracted — no startup fee.

Pricing

from $5.00 / 1,000 page extracteds

Rating

0.0

(0)

Developer

Runlayer

Actor stats

Bookmarked

Total users

Monthly active users

17 hours ago

Last modified

Categories

Automation

Developer tools

You can access the Website to Markdown for LLMs & RAG — Content Extractor programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in Integrations settings in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

{
  "openapi": "3.0.1",
  "info": {
    "version": "0.1",
    "x-build-id": "01NkHOmISRezFghaO"
  },
  "servers": [
    {
      "url": "https://api.apify.com/v2"
    }
  ],
  "paths": {
    "/acts/runlayer~website-to-markdown/run-sync-get-dataset-items": {
      "post": {
        "operationId": "run-sync-get-dataset-items-runlayer-website-to-markdown",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK"
          }
        }
      }
    },
    "/acts/runlayer~website-to-markdown/runs": {
      "post": {
        "operationId": "runs-sync-runlayer-website-to-markdown",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor and returns information about the initiated run in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK",
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/runsResponseSchema"
                }
              }
            }
          }
        }
      }
    },
    "/acts/runlayer~website-to-markdown/run-sync": {
      "post": {
        "operationId": "run-sync-runlayer-website-to-markdown",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK"
          }
        }
      }
    }
  },
  "components": {
    "schemas": {
      "inputSchema": {
        "type": "object",
        "required": [
          "urls"
        ],
        "properties": {
          "urls": {
            "title": "URLs",
            "type": "array",
            "description": "One or more URLs to convert. Each extracted page produces one dataset item and one billable event. You may omit the https:// prefix.",
            "items": {
              "type": "string"
            }
          },
          "outputFormat": {
            "title": "Output format",
            "enum": [
              "markdown",
              "text",
              "html",
              "json"
            ],
            "type": "string",
            "description": "markdown (best for LLMs/RAG), text (plain), html (cleaned main-content HTML), or json (everything: markdown + text + metadata + links).",
            "default": "markdown"
          },
          "onlyMainContent": {
            "title": "Extract main content only",
            "type": "boolean",
            "description": "Use readability to keep only the main article and strip navigation, sidebars, ads, footers and boilerplate. Turn off to convert the whole page body.",
            "default": true
          },
          "jsRendering": {
            "title": "Render JavaScript",
            "type": "boolean",
            "description": "Load the page in a real browser so JS-rendered content (SPAs, React/Vue sites) is captured. Turn off for a much faster fetch on static/server-rendered pages.",
            "default": true
          },
          "includeLinks": {
            "title": "Include links",
            "type": "boolean",
            "description": "Include the list of links found on each page in the output (also used as crawl targets when crawling is on).",
            "default": true
          },
          "crawl": {
            "title": "Crawl the site (follow links)",
            "type": "boolean",
            "description": "Beyond the URLs you provide, follow links to discover and convert more pages, up to the limits below.",
            "default": false
          },
          "maxPages": {
            "title": "Max pages",
            "minimum": 1,
            "maximum": 5000,
            "type": "integer",
            "description": "Hard cap on the total number of pages converted (across all seeds). Protects you from runaway crawls and cost.",
            "default": 10
          },
          "maxDepth": {
            "title": "Max crawl depth",
            "minimum": 0,
            "maximum": 10,
            "type": "integer",
            "description": "How many link-hops away from your seed URLs to follow. 0 = only the URLs you gave. 1 = them plus pages they link to, etc.",
            "default": 1
          },
          "sameDomainOnly": {
            "title": "Stay on the same domain",
            "type": "boolean",
            "description": "Only follow links that stay on the seed page's domain.",
            "default": true
          },
          "respectRobotsTxt": {
            "title": "Respect robots.txt",
            "type": "boolean",
            "description": "Skip pages disallowed by the site's robots.txt. Recommended — polite crawling.",
            "default": true
          },
          "useSitemap": {
            "title": "Discover URLs from sitemap.xml",
            "type": "boolean",
            "description": "When crawling, also seed from the site's own sitemap.xml (and any Sitemap: entries in robots.txt) — the fastest, most complete way to reach every page. Subject to the URL filters below.",
            "default": false
          },
          "includeUrlPatterns": {
            "title": "Include URL patterns",
            "type": "array",
            "description": "Only crawl/keep discovered URLs matching these glob patterns (* = wildcard), e.g. https://site.com/blog/* or */docs/*. Empty = allow all. Does not filter the URLs you explicitly provide.",
            "default": [],
            "items": {
              "type": "string"
            }
          },
          "excludeUrlPatterns": {
            "title": "Exclude URL patterns",
            "type": "array",
            "description": "Skip discovered URLs matching these glob patterns (* = wildcard), e.g. */tag/*, */login*, *.pdf.",
            "default": [],
            "items": {
              "type": "string"
            }
          },
          "maxConcurrency": {
            "title": "Max concurrency",
            "minimum": 1,
            "maximum": 15,
            "type": "integer",
            "description": "How many pages to fetch in parallel. Higher = faster crawls, but uses more memory — be considerate of the target site.",
            "default": 5
          },
          "removeSelectors": {
            "title": "Remove elements (CSS selectors)",
            "type": "array",
            "description": "Extra CSS selectors to delete before extraction, e.g. '.related-articles' or '#comments'.",
            "default": [],
            "items": {
              "type": "string"
            }
          },
          "waitUntil": {
            "title": "Wait until",
            "enum": [
              "load",
              "domcontentloaded",
              "networkidle"
            ],
            "type": "string",
            "description": "When to consider a page loaded (only applies when Render JavaScript is on).",
            "default": "load"
          },
          "timeoutMs": {
            "title": "Timeout per page (ms)",
            "minimum": 5000,
            "maximum": 120000,
            "type": "integer",
            "description": "Max time to wait for a page before giving up on it.",
            "default": 45000
          },
          "proxyConfiguration": {
            "title": "Proxy",
            "type": "object",
            "description": "Off by default (fast, works on most sites). Enable Apify Proxy — including residential — for bot-protected sites that block datacenter IPs.",
            "default": {
              "useApifyProxy": false
            }
          }
        }
      },
      "runsResponseSchema": {
        "type": "object",
        "properties": {
          "data": {
            "type": "object",
            "properties": {
              "id": {
                "type": "string"
              },
              "actId": {
                "type": "string"
              },
              "userId": {
                "type": "string"
              },
              "startedAt": {
                "type": "string",
                "format": "date-time",
                "example": "2025-01-08T00:00:00.000Z"
              },
              "finishedAt": {
                "type": "string",
                "format": "date-time",
                "example": "2025-01-08T00:00:00.000Z"
              },
              "status": {
                "type": "string",
                "example": "READY"
              },
              "meta": {
                "type": "object",
                "properties": {
                  "origin": {
                    "type": "string",
                    "example": "API"
                  },
                  "userAgent": {
                    "type": "string"
                  }
                }
              },
              "stats": {
                "type": "object",
                "properties": {
                  "inputBodyLen": {
                    "type": "integer",
                    "example": 2000
                  },
                  "rebootCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "restartCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "resurrectCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "computeUnits": {
                    "type": "integer",
                    "example": 0
                  }
                }
              },
              "options": {
                "type": "object",
                "properties": {
                  "build": {
                    "type": "string",
                    "example": "latest"
                  },
                  "timeoutSecs": {
                    "type": "integer",
                    "example": 300
                  },
                  "memoryMbytes": {
                    "type": "integer",
                    "example": 1024
                  },
                  "diskMbytes": {
                    "type": "integer",
                    "example": 2048
                  }
                }
              },
              "buildId": {
                "type": "string"
              },
              "defaultKeyValueStoreId": {
                "type": "string"
              },
              "defaultDatasetId": {
                "type": "string"
              },
              "defaultRequestQueueId": {
                "type": "string"
              },
              "buildNumber": {
                "type": "string",
                "example": "1.0.0"
              },
              "containerUrl": {
                "type": "string"
              },
              "usage": {
                "type": "object",
                "properties": {
                  "ACTOR_COMPUTE_UNITS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_WRITES": {
                    "type": "integer",
                    "example": 1
                  },
                  "KEY_VALUE_STORE_LISTS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_INTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_EXTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_SERPS": {
                    "type": "integer",
                    "example": 0
                  }
                }
              },
              "usageTotalUsd": {
                "type": "number",
                "example": 0.00005
              },
              "usageUsd": {
                "type": "object",
                "properties": {
                  "ACTOR_COMPUTE_UNITS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_WRITES": {
                    "type": "number",
                    "example": 0.00005
                  },
                  "KEY_VALUE_STORE_LISTS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_INTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_EXTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_SERPS": {
                    "type": "integer",
                    "example": 0
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Website to Markdown Converter API for LLMs & RAG OpenAPI definition

OpenAPI is a standard for designing and describing RESTful APIs, allowing developers to define API structure, endpoints, and data formats in a machine-readable way. It simplifies API development, integration, and documentation.

OpenAPI is effective when used with AI agents and GPTs by standardizing how these systems interact with various APIs, for reliable integrations and efficient communication.

By defining machine-readable API specifications, OpenAPI allows AI models like GPTs to understand and use varied data sources, improving accuracy. This accelerates development, reduces errors, and provides context-aware responses, making OpenAPI a core component for AI applications.

You can download the OpenAPI definitions for Website to Markdown for LLMs & RAG — Content Extractor from the options below:

OpenAPI.json

If you’d like to learn more about how OpenAPI powers GPTs, read our blog post.

You can also check out our other API clients:

Website to Markdown for LLMs & RAG — Content Extractor API in Python

Website to Markdown for LLMs & RAG — Content Extractor API in JavaScript

Website to Markdown for LLMs & RAG — Content Extractor API through CLI

Website to Markdown for LLMs & RAG — Content Extractor API

Website to Markdown – Clean LLM & RAG Content Extractor

dataquarry/website-to-markdown

Convert any public web page to clean, LLM-ready Markdown with metadata — by URL, a list of URLs, or a whole-site crawl. Strips nav/ads/boilerplate, keeps headings/lists/tables/code. Respects robots.txt. No API key.

Daniel Brenner

RAG Web Extractor — Clean Markdown, HTML & Chunks

junipr/rag-web-extractor

Extract clean website content for RAG and AI search. Crawl pages, remove boilerplate, preserve structure, and export markdown, HTML, text, JSON, and chunks.

junipr

Website to Markdown — LLM & RAG Content Exporter

sturdydata/website-markdown-exporter

Crawl any website and get one clean Markdown document per page — ready for RAG pipelines, vector databases, LLM fine-tuning, or docs migration. Boilerplate (nav, footers, cookie banners) stripped, main content auto-detected, sitemap-seeded crawling, robots.txt respected. HARD page caps and flat p...

Sturdy Data

Website to Markdown Crawler for LLM & RAG

logiover/website-text-markdown-crawler

Crawl any website to clean Markdown and plain text for LLM training and RAG. HTML to Markdown, no API or login. Export website text to CSV or JSON.

Logiover

Website To Markdown

smart_api/website-to-markdown

Convert any webpage into clean, LLM-ready Markdown in seconds — perfect for AI training data, RAG pipelines, and content archiving.

SmartApi

5.0

(1)

Web to LLM Spanish — RAG-ready content extractor

fanciful_geode/web-to-llm-es

Crawls Spanish-language websites and extracts clean markdown content optimized for LLM RAG pipelines. Handles Spanish character encoding, removes boilerplate, and outputs structured chunks ready for embedding.

Ernesto de Armas

Website Content Crawler - Markdown & Text for LLM / RAG

pear_fight/website-content-crawler-markdown-text-for-llm-rag

Crawl any website and extract clean article text and Markdown, ready to feed into LLMs, ChatGPT, vector databases and RAG pipelines. Removes navigation, ads and boilerplate. Configurable crawl depth and page limits. Export to JSON, CSV, Excel.

Harald

Website Content Pipeline for AI: Markdown, Tokens, RAG Chunks

scrapemint/website-content-crawler

Crawl any website and ship clean Markdown, plain text, and HTML for AI, LLM, and RAG pipelines. Each row carries token estimates, JSON LD metadata, link graph, and optional auto chunk splitting for vector databases. Pay per page.

Ken M

Website Content Crawler for AI — Clean Markdown, 4x Cheaper

joyouscam35875/website-content-crawler

Crawl any website and extract clean text/markdown for LLMs, RAG pipelines, vector databases. BFS crawl with depth control, robots.txt support, boilerplate removal. Perfect for feeding AI models. $0.001/page — 4x cheaper than the official Apify crawler.

Ken Digital

Website to Text & Markdown — AI / RAG Content Crawler

inexhaustible_glass/rag-website-crawler

Scrape any website into clean text & Markdown with RAG-ready chunks and token counts for LLMs, vector databases (Pinecone, Qdrant) and AI chatbots. Also extracts linked PDF/Word/Excel. Anti-block, robots.txt-aware. Website-to-text for beginners, full RAG pipeline for pros. CPU only, no API key.