Pricing

from $1.50 / 1,000 pages

Try for free

Go to Apify Store

Website to Markdown for AI & RAG: Fast Crawler

Try for free

Crawl any website and convert every page into clean, LLM-ready Markdown for RAG, AI agents and fine-tuning. Main content extraction, sitemap discovery, chunking with overlap, token estimates. Fast HTTP crawler, pay per page.

Pricing

from $1.50 / 1,000 pages

Rating

0.0

(0)

Developer

Seungki Min

Actor stats

Bookmarked

Total users

Monthly active users

5 hours ago

Last modified

Categories

Developer tools

Automation

You can access the Website to Markdown for AI & RAG: Fast Crawler programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

{
  "openapi": "3.0.1",
  "info": {
    "version": "0.1",
    "x-build-id": "RDtxaTFgwgVPnfS3z"
  },
  "servers": [
    {
      "url": "https://api.apify.com/v2"
    }
  ],
  "paths": {
    "/acts/tindacloud~website-to-markdown/run-sync-get-dataset-items": {
      "post": {
        "operationId": "run-sync-get-dataset-items-tindacloud-website-to-markdown",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK"
          }
        }
      }
    },
    "/acts/tindacloud~website-to-markdown/runs": {
      "post": {
        "operationId": "runs-sync-tindacloud-website-to-markdown",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor and returns information about the initiated run in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK",
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/runsResponseSchema"
                }
              }
            }
          }
        }
      }
    },
    "/acts/tindacloud~website-to-markdown/run-sync": {
      "post": {
        "operationId": "run-sync-tindacloud-website-to-markdown",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK"
          }
        }
      }
    }
  },
  "components": {
    "schemas": {
      "inputSchema": {
        "type": "object",
        "required": [
          "startUrls"
        ],
        "properties": {
          "startUrls": {
            "title": "Start URLs",
            "type": "array",
            "description": "Websites or sections to convert, e.g. https://docs.example.com/guide/. Pages under the same path are crawled.",
            "items": {
              "type": "object",
              "required": [
                "url"
              ],
              "properties": {
                "url": {
                  "type": "string",
                  "title": "URL of a web page",
                  "format": "uri"
                }
              }
            }
          },
          "maxPages": {
            "title": "Max pages",
            "minimum": 1,
            "type": "integer",
            "description": "Total pages to convert. You are charged per page.",
            "default": 100
          },
          "maxCrawlDepth": {
            "title": "Max link depth",
            "minimum": 0,
            "type": "integer",
            "description": "How many clicks away from the start URL to follow links (0 = only start URLs and sitemap pages).",
            "default": 5
          },
          "stayWithinStartPath": {
            "title": "Stay within start path",
            "type": "boolean",
            "description": "Only crawl URLs under the start URL's folder (e.g. /docs/). Turn off to crawl the whole domain.",
            "default": true
          },
          "useSitemaps": {
            "title": "Use sitemaps",
            "type": "boolean",
            "description": "Discover pages from robots.txt and sitemap.xml, including last-modified dates.",
            "default": true
          },
          "includeUrlGlobs": {
            "title": "Include URL patterns",
            "type": "array",
            "description": "Only convert URLs matching these globs, e.g. https://example.com/blog/**",
            "items": {
              "type": "string"
            }
          },
          "excludeUrlGlobs": {
            "title": "Exclude URL patterns",
            "type": "array",
            "description": "Skip URLs matching these globs, e.g. **/tag/**, **/page/*",
            "items": {
              "type": "string"
            }
          },
          "extractionMode": {
            "title": "Content extraction",
            "enum": [
              "article",
              "fullPage"
            ],
            "type": "string",
            "description": "How much of each page to keep.",
            "default": "article"
          },
          "includeLinks": {
            "title": "Keep links in Markdown",
            "type": "boolean",
            "description": "Keep [text](url) links. Turn off for plain text-like Markdown.",
            "default": true
          },
          "includeImages": {
            "title": "Keep images",
            "type": "boolean",
            "description": "Keep ![alt](src) image references.",
            "default": false
          },
          "includePlainText": {
            "title": "Also output plain text",
            "type": "boolean",
            "description": "Adds a text field without Markdown syntax.",
            "default": false
          },
          "chunkSize": {
            "title": "Chunk size (characters)",
            "minimum": 0,
            "type": "integer",
            "description": "Split each page into chunks of about this many characters for embeddings (0 = no chunking). ~4 characters ≈ 1 token.",
            "default": 0
          },
          "chunkOverlap": {
            "title": "Chunk overlap (characters)",
            "minimum": 0,
            "type": "integer",
            "description": "Characters repeated between consecutive chunks.",
            "default": 200
          },
          "minWordCount": {
            "title": "Skip pages shorter than (words)",
            "minimum": 0,
            "type": "integer",
            "description": "Pages with less content are not returned or charged.",
            "default": 20
          },
          "respectRobotsTxt": {
            "title": "Respect robots.txt",
            "type": "boolean",
            "description": "Skip URLs disallowed for all crawlers.",
            "default": true
          },
          "proxy": {
            "title": "Proxy",
            "type": "object",
            "description": "Usually not needed. Enable Apify Proxy if a site blocks requests."
          }
        }
      },
      "runsResponseSchema": {
        "type": "object",
        "properties": {
          "data": {
            "type": "object",
            "properties": {
              "id": {
                "type": "string"
              },
              "actId": {
                "type": "string"
              },
              "userId": {
                "type": "string"
              },
              "startedAt": {
                "type": "string",
                "format": "date-time",
                "example": "2025-01-08T00:00:00.000Z"
              },
              "finishedAt": {
                "type": "string",
                "format": "date-time",
                "example": "2025-01-08T00:00:00.000Z"
              },
              "status": {
                "type": "string",
                "example": "READY"
              },
              "meta": {
                "type": "object",
                "properties": {
                  "origin": {
                    "type": "string",
                    "example": "API"
                  },
                  "userAgent": {
                    "type": "string"
                  }
                }
              },
              "stats": {
                "type": "object",
                "properties": {
                  "inputBodyLen": {
                    "type": "integer",
                    "example": 2000
                  },
                  "rebootCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "restartCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "resurrectCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "computeUnits": {
                    "type": "integer",
                    "example": 0
                  }
                }
              },
              "options": {
                "type": "object",
                "properties": {
                  "build": {
                    "type": "string",
                    "example": "latest"
                  },
                  "timeoutSecs": {
                    "type": "integer",
                    "example": 300
                  },
                  "memoryMbytes": {
                    "type": "integer",
                    "example": 1024
                  },
                  "diskMbytes": {
                    "type": "integer",
                    "example": 2048
                  }
                }
              },
              "buildId": {
                "type": "string"
              },
              "defaultKeyValueStoreId": {
                "type": "string"
              },
              "defaultDatasetId": {
                "type": "string"
              },
              "defaultRequestQueueId": {
                "type": "string"
              },
              "buildNumber": {
                "type": "string",
                "example": "1.0.0"
              },
              "containerUrl": {
                "type": "string"
              },
              "usage": {
                "type": "object",
                "properties": {
                  "ACTOR_COMPUTE_UNITS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_WRITES": {
                    "type": "integer",
                    "example": 1
                  },
                  "KEY_VALUE_STORE_LISTS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_INTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_EXTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_SERPS": {
                    "type": "integer",
                    "example": 0
                  }
                }
              },
              "usageTotalUsd": {
                "type": "number",
                "example": 0.00005
              },
              "usageUsd": {
                "type": "object",
                "properties": {
                  "ACTOR_COMPUTE_UNITS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_WRITES": {
                    "type": "number",
                    "example": 0.00005
                  },
                  "KEY_VALUE_STORE_LISTS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_INTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_EXTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_SERPS": {
                    "type": "integer",
                    "example": 0
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Website to Markdown API - LLM & RAG Ready Crawler OpenAPI definition

OpenAPI is a standard for designing and describing RESTful APIs, allowing developers to define API structure, endpoints, and data formats in a machine-readable way. It simplifies API development, integration, and documentation.

OpenAPI is effective when used with AI agents and GPTs by standardizing how these systems interact with various APIs, for reliable integrations and efficient communication.

By defining machine-readable API specifications, OpenAPI allows AI models like GPTs to understand and use varied data sources, improving accuracy. This accelerates development, reduces errors, and provides context-aware responses, making OpenAPI a core component for AI applications.

You can download the OpenAPI definitions for Website to Markdown for AI & RAG: Fast Crawler from the options below:

OpenAPI.json

If you’d like to learn more about how OpenAPI powers GPTs, read our blog post.

You can also check out our other API clients:

Website to Markdown for AI & RAG: Fast Crawler API in Python

Website to Markdown for AI & RAG: Fast Crawler API in JavaScript

Website to Markdown for AI & RAG: Fast Crawler API through CLI

Website to Markdown for AI & RAG: Fast Crawler API

YouTube Channel Scraper — Fast & Cheap, Official API

joyouscam35875/youtube-channel-scraper

Scrape YouTube channel data and videos using the official YouTube Data API. Get channel stats, video titles, views, likes, descriptions, thumbnails. 1000 videos in 30 seconds. Bring your own API key. $0.001/video — 3x cheaper than alternatives.

Ken Digital

Website to Markdown Crawler for LLM & RAG (fast, no browser)

webdatatools/website-to-markdown

Website to Markdown Crawler turns any site into clean Markdown per page for LLMs, RAG pipelines and vector DBs — no headless browser, $1 per 1,000 pages.

Murat Uzun

Website Crawler API — Markdown for RAG

tugelbay/website-content-crawler

Website crawler API for public pages and clean Markdown, text, or HTML output for RAG pipelines, AI agents, documentation indexing, and monitoring. Guide: https://konabayev.com/tools/website-content-crawler/?utm_source=apify_info&utm_medium=referral&utm_campaign=website-content-crawler

Tugelbay Konabayev

AI / RAG Web Crawler

groupoject/ai-rag-web-crawler

Crawl any website and extract clean, LLM-ready Markdown chunks to feed AI agents, chatbots, and RAG pipelines. One row per embeddable chunk.

Group Oject

Website Content Crawler — Text, Markdown & HTML for AI/LLM

hichemdev/website-content-crawler

Crawl any website and extract clean text, Markdown, and HTML from every page — ready for LLM, RAG, and AI ingestion.

Hichem Ben Moussa

Website to Clean Markdown (AI & RAG Ready)

ahmed_jasarevic/website-to-clean-markdown-ai-rag-ready

Convert any website into clean, noise-free Markdown. Perfect for training LLMs, building Custom GPTs, and RAG pipelines. Save 80% on OpenAI tokens by stripping HTML junk. From $2.80 per 1k results

Ahmed Jasarevic

Dark Web Scraper – Onion Search & Page Extractor

lofomachines/tor-scraper

Search the dark web by keyword and convert any .onion page into clean Markdown, JSON, HTML or text with rich metadata.

Lofomachines

AI-Ready Website Crawler

optimus-fulcria/ai-ready-website-crawler

Crawl websites and convert to clean markdown for AI/RAG, LLM fine-tuning, and document pipelines.

Fulcria Labs

Website Content Crawler Lite

fetch_cat/website-content-crawler-lite

Fast, reliable HTTP website crawler for clean text, Markdown, HTML, metadata, and links. Built for AI/RAG, SEO audits, monitoring, and automation with robots.txt, retries, proxy support, and bounded exports.

Hanna Nosova

Website Content Scraper: Clean Markdown for AI and RAG

scrapemint/website-content-scraper

Crawl any website and get clean markdown, text, or HTML per page, ready for RAG pipelines, chatbots, and LLM fine tuning. Plain HTTP, no browser, no API key. Pay per page.