Pricing

from $1.00 / 1,000 page crawleds

Try for free

Go to Apify Store

Website to Markdown Crawler for LLM & RAG (llms.txt)

Try for free

Crawl any website and get every page as clean Markdown for LLMs, RAG and AI agents. Main-content extraction, sitemap discovery, RAG chunks, llms.txt and llms-full.txt. Fast HTTP crawler with a flat price per page: no browser, no compute surprises.

Pricing

from $1.00 / 1,000 page crawleds

Rating

0.0

(0)

Developer

Catalin Friptu

Actor stats

Bookmarked

Total users

Monthly active users

a day ago

Last modified

Categories

Developer tools

SEO tools

You can access the Website to Markdown Crawler for LLM & RAG (llms.txt) programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

{
  "openapi": "3.0.1",
  "info": {
    "version": "0.1",
    "x-build-id": "Bh5o0PQ0Q7G4fFYRm"
  },
  "servers": [
    {
      "url": "https://api.apify.com/v2"
    }
  ],
  "paths": {
    "/acts/malmurk~website-to-markdown-crawler/run-sync-get-dataset-items": {
      "post": {
        "operationId": "run-sync-get-dataset-items-malmurk-website-to-markdown-crawler",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK"
          }
        }
      }
    },
    "/acts/malmurk~website-to-markdown-crawler/runs": {
      "post": {
        "operationId": "runs-sync-malmurk-website-to-markdown-crawler",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor and returns information about the initiated run in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK",
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/runsResponseSchema"
                }
              }
            }
          }
        }
      }
    },
    "/acts/malmurk~website-to-markdown-crawler/run-sync": {
      "post": {
        "operationId": "run-sync-malmurk-website-to-markdown-crawler",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK"
          }
        }
      }
    }
  },
  "components": {
    "schemas": {
      "inputSchema": {
        "type": "object",
        "properties": {
          "startUrls": {
            "title": "Start URLs",
            "type": "array",
            "description": "Where the crawl starts. Only links under each start URL's path are followed: https://example.com/docs/ crawls the docs section, https://example.com/ crawls the whole site.",
            "items": {
              "type": "object",
              "required": [
                "url"
              ],
              "properties": {
                "url": {
                  "type": "string",
                  "title": "URL of a web page",
                  "format": "uri"
                }
              }
            }
          },
          "maxPages": {
            "title": "Max pages",
            "minimum": 1,
            "maximum": 1000000,
            "type": "integer",
            "description": "Stop after saving this many pages. You pay only for saved pages, so this also caps the cost of a run.",
            "default": 100
          },
          "maxCrawlDepth": {
            "title": "Max crawl depth",
            "minimum": 0,
            "maximum": 1000,
            "type": "integer",
            "description": "How many links away from a start URL the crawler may go. 0 = only the start URLs themselves. Pages found in sitemaps count as depth 1.",
            "default": 20
          },
          "useSitemaps": {
            "title": "Use sitemaps",
            "type": "boolean",
            "description": "Also queue every in-scope URL listed in the site's sitemap.xml (found through robots.txt). Finds pages that no menu links to.",
            "default": false
          },
          "includeUrlGlobs": {
            "title": "Include URLs (globs)",
            "type": "array",
            "description": "Crawl only URLs that match one of these patterns, instead of the default 'under the start URL' rule. Example: https://example.com/{blog,docs}/** (** matches anything, * matches anything except a slash).",
            "items": {
              "type": "object",
              "required": [
                "glob"
              ],
              "properties": {
                "glob": {
                  "type": "string",
                  "title": "Glob of a web page"
                }
              }
            }
          },
          "excludeUrlGlobs": {
            "title": "Exclude URLs (globs)",
            "type": "array",
            "description": "Never crawl URLs that match one of these patterns. Example: **/tag/** or https://example.com/blog/page/*",
            "items": {
              "type": "object",
              "required": [
                "glob"
              ],
              "properties": {
                "glob": {
                  "type": "string",
                  "title": "Glob of a web page"
                }
              }
            }
          },
          "respectRobotsTxt": {
            "title": "Respect robots.txt",
            "type": "boolean",
            "description": "Skip pages the site's robots.txt disallows. Turn off only for your own site.",
            "default": true
          },
          "maxConcurrency": {
            "title": "Max concurrency",
            "minimum": 1,
            "maximum": 50,
            "type": "integer",
            "description": "Pages fetched at the same time. Lower it for small or slow sites.",
            "default": 10
          },
          "contentMode": {
            "title": "Content",
            "enum": [
              "main",
              "full"
            ],
            "type": "string",
            "description": "Main content: the article or documentation text only, without menus, cookie banners, sidebars and footers (best for RAG). Full page: everything in the page body.",
            "default": "main"
          },
          "includeLinks": {
            "title": "Keep links in Markdown",
            "type": "boolean",
            "description": "Keep hyperlinks as [text](url). Off gives cleaner text for embeddings.",
            "default": false
          },
          "includeImages": {
            "title": "Keep images in Markdown",
            "type": "boolean",
            "description": "Keep images as ![alt](src).",
            "default": false
          },
          "saveText": {
            "title": "Also save plain text",
            "type": "boolean",
            "description": "Add a 'text' field with the content as plain text (no Markdown syntax).",
            "default": false
          },
          "saveHtml": {
            "title": "Also save raw HTML",
            "type": "boolean",
            "description": "Add an 'html' field with the page's original HTML. Makes the dataset much bigger.",
            "default": false
          },
          "generateLlmsTxt": {
            "title": "Generate llms.txt",
            "type": "boolean",
            "description": "Write llms.txt (an index of the crawled pages in the llmstxt.org format) and llms-full.txt (every page's Markdown in one file) to the run's key-value store. Free.",
            "default": true
          },
          "detectLanguage": {
            "title": "Detect language",
            "type": "boolean",
            "description": "Fill 'language' from the text when the page doesn't declare one in its html lang attribute.",
            "default": true
          },
          "chunkSize": {
            "title": "Chunk size (characters)",
            "minimum": 0,
            "type": "integer",
            "description": "Also split each page's Markdown into chunks of about this many characters, ready for embeddings (a 'chunks' field). 0 turns chunking off. 1000-2000 suits most embedding models.",
            "default": 0
          },
          "chunkOverlap": {
            "title": "Chunk overlap (characters)",
            "minimum": 0,
            "type": "integer",
            "description": "How many characters neighbouring chunks share, so a sentence cut at a boundary still appears whole in one chunk.",
            "default": 200
          }
        }
      },
      "runsResponseSchema": {
        "type": "object",
        "properties": {
          "data": {
            "type": "object",
            "properties": {
              "id": {
                "type": "string"
              },
              "actId": {
                "type": "string"
              },
              "userId": {
                "type": "string"
              },
              "startedAt": {
                "type": "string",
                "format": "date-time",
                "example": "2025-01-08T00:00:00.000Z"
              },
              "finishedAt": {
                "type": "string",
                "format": "date-time",
                "example": "2025-01-08T00:00:00.000Z"
              },
              "status": {
                "type": "string",
                "example": "READY"
              },
              "meta": {
                "type": "object",
                "properties": {
                  "origin": {
                    "type": "string",
                    "example": "API"
                  },
                  "userAgent": {
                    "type": "string"
                  }
                }
              },
              "stats": {
                "type": "object",
                "properties": {
                  "inputBodyLen": {
                    "type": "integer",
                    "example": 2000
                  },
                  "rebootCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "restartCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "resurrectCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "computeUnits": {
                    "type": "integer",
                    "example": 0
                  }
                }
              },
              "options": {
                "type": "object",
                "properties": {
                  "build": {
                    "type": "string",
                    "example": "latest"
                  },
                  "timeoutSecs": {
                    "type": "integer",
                    "example": 300
                  },
                  "memoryMbytes": {
                    "type": "integer",
                    "example": 1024
                  },
                  "diskMbytes": {
                    "type": "integer",
                    "example": 2048
                  }
                }
              },
              "buildId": {
                "type": "string"
              },
              "defaultKeyValueStoreId": {
                "type": "string"
              },
              "defaultDatasetId": {
                "type": "string"
              },
              "defaultRequestQueueId": {
                "type": "string"
              },
              "buildNumber": {
                "type": "string",
                "example": "1.0.0"
              },
              "containerUrl": {
                "type": "string"
              },
              "usage": {
                "type": "object",
                "properties": {
                  "ACTOR_COMPUTE_UNITS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_WRITES": {
                    "type": "integer",
                    "example": 1
                  },
                  "KEY_VALUE_STORE_LISTS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_INTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_EXTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_SERPS": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_UNBLOCKER_UNITS": {
                    "type": "integer",
                    "example": 0
                  }
                }
              },
              "usageTotalUsd": {
                "type": "number",
                "example": 0.00005
              },
              "usageUsd": {
                "type": "object",
                "properties": {
                  "ACTOR_COMPUTE_UNITS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_WRITES": {
                    "type": "number",
                    "example": 0.00005
                  },
                  "KEY_VALUE_STORE_LISTS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_INTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_EXTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_SERPS": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_UNBLOCKER_UNITS": {
                    "type": "integer",
                    "example": 0
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Website to Markdown Crawler for LLM, RAG & llms.txt OpenAPI definition

OpenAPI is a standard for designing and describing RESTful APIs, allowing developers to define API structure, endpoints, and data formats in a machine-readable way. It simplifies API development, integration, and documentation.

OpenAPI is effective when used with AI agents and GPTs by standardizing how these systems interact with various APIs, for reliable integrations and efficient communication.

By defining machine-readable API specifications, OpenAPI allows AI models like GPTs to understand and use varied data sources, improving accuracy. This accelerates development, reduces errors, and provides context-aware responses, making OpenAPI a core component for AI applications.

You can download the OpenAPI definitions for Website to Markdown Crawler for LLM & RAG (llms.txt) from the options below:

OpenAPI.json

If you’d like to learn more about how OpenAPI powers GPTs, read our blog post.

You can also check out our other API clients:

Website to Markdown Crawler for LLM & RAG (llms.txt) API in Python

Website to Markdown Crawler for LLM & RAG (llms.txt) API in JavaScript

Website to Markdown Crawler for LLM & RAG (llms.txt) API through CLI

Website to Markdown Crawler for LLM & RAG (llms.txt) API

Tweet Scraper V2 - X Scraper / Twitter Scraper / X.com Scraper

apidojo/tweet-scraper

Lightning-fast search, URL, list, and profile scraping, with customizable filters. At $0.40 per 1000 tweets, and 30-80 tweets per second, it is ideal for researchers, entrepreneurs, and businesses! Get comprehensive insights from Twitter (X) now!

API Dojo

113K

3.9

(196)

Facebook Search Scraper — People, Pages, Posts & More

memo23/facebook-search-scraper

The most complete Facebook search: people, pages, places, events, posts, videos and top — in one actor, no login and no cookies. Get names, URLs, profile pictures, verified badges, event dates, and post text with authors. Auto-paginates across results. Export to JSON, CSV or Excel.

Muhamed Didovic

5.6K

5.0

(2)

Facebook Search Posts

scrapeforge/facebook-search-posts

Search Facebook posts, pages, groups, people, videos and events by keyword. Returns detailed data including reactions breakdown, media URLs, and engagement metrics.

ScrapeForge

68K

3.6

(10)

TikTok Scraper

clockworks/tiktok-scraper

Extract data from TikTok videos, hashtags, and users. Use URLs or search queries to scrape TikTok profiles, hashtags, posts, URLs, shares, followers, hearts, names, video, and music-related data. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Clockworks

354K

4.7

(370)

Youtube Scraper (Pay Per Result)

apidojo/youtube-scraper

Experience unparalleled efficiency and depth with its ultra-fast capabilities in searching, URL, playlist, channel, and profile scraping. It comes equipped with a variety of filters to customize your data collection precisely. Plus, it's incredibly affordable at only $0.50 per 1000 videos!

API Dojo

35K

3.3

(21)

TikTok Search Scraper: Keywords, Hashtags, Creators

xmolodtsov/tiktok-search-scraper

Scrape TikTok search results by keyword or hashtag and discover creators. Pay only per result. Views, likes, comments, shares, bookmarks, channel stats, music, subtitles and direct video URLs as JSON. Output compatible with popular TikTok scrapers. No proxies or login needed. Works via API and MCP.

GoFetch

66K

5.0

(1)

Search Threads By Keywords

watcher.data/search-threads-by-keywords

Fast and reliable scraper for searching posts on Threads by Meta. Perfect for social media monitoring, research, and business intelligence.

Giichi Arai

8.4K

3.4

(10)

Twitter (X.com) Scraper Unlimited: No Limits

apidojo/twitter-scraper-lite

Introducing Twitter Scraper Unlimited, the most comprehensive Twitter data extraction solution available. Our enterprise-grade scraper offers unmatched capabilities with a transparent event-based pricing model, making it perfect for both small-scale and large-scale data extraction needs.

API Dojo

40K

4.8

(100)

Threads Search Scraper

igview-owner/threads-search-scraper

Powerful Meta Threads search scraper for posts and users. Filter by keywords or usernames, export structured data for analytics, research, monitoring and reporting.

Sachin Kumar Yadav

12K

3.5

(6)

Facebook Search Scraper

igview-owner/facebook-old-posts-search

Search Facebook posts in seconds. Find viral posts, photos and videos by keyword, date range and engagement – perfect for research, content ideas, brand monitoring and sentiment analysis.

Sachin Kumar Yadav

3.2K

4.8

(3)