Pricing

from $2.00 / 1,000 page extracteds

Try for free

Go to Apify Store

Website Crawler - Site Pages to Markdown and Text

Try for free

Crawl one public website within page, depth and time limits: clean text, Markdown, links and metadata per page, plus a coverage report. Private build under development.

Pricing

from $2.00 / 1,000 page extracteds

Rating

0.0

(0)

Developer

Lead Proof

Actor stats

Bookmarked

Total users

Monthly active users

5 hours ago

Last modified

Categories

Developer tools

You can access the Website Crawler - Site Pages to Markdown and Text programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

{
  "openapi": "3.0.1",
  "info": {
    "version": "0.0",
    "x-build-id": "Ute69lfhjtO3nQ8bu"
  },
  "servers": [
    {
      "url": "https://api.apify.com/v2"
    }
  ],
  "paths": {
    "/acts/leadproof~website-crawler/run-sync-get-dataset-items": {
      "post": {
        "operationId": "run-sync-get-dataset-items-leadproof-website-crawler",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK"
          }
        }
      }
    },
    "/acts/leadproof~website-crawler/runs": {
      "post": {
        "operationId": "runs-sync-leadproof-website-crawler",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor and returns information about the initiated run in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK",
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/runsResponseSchema"
                }
              }
            }
          }
        }
      }
    },
    "/acts/leadproof~website-crawler/run-sync": {
      "post": {
        "operationId": "run-sync-leadproof-website-crawler",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK"
          }
        }
      }
    }
  },
  "components": {
    "schemas": {
      "inputSchema": {
        "type": "object",
        "required": [
          "startUrls"
        ],
        "properties": {
          "startUrls": {
            "title": "Start URLs",
            "minItems": 1,
            "maxItems": 100,
            "type": "array",
            "description": "Where the crawl starts. The crawl stays on these hosts (www and the bare domain count as the same site). Add an optional id per URL to find its row in the output.",
            "items": {
              "type": "object",
              "required": [
                "url"
              ],
              "properties": {
                "url": {
                  "type": "string",
                  "title": "URL of a web page",
                  "format": "uri"
                }
              }
            }
          },
          "maxPages": {
            "title": "Maximum pages",
            "minimum": 1,
            "maximum": 10000,
            "type": "integer",
            "description": "Maximum page reads, including failed ones. The crawl stops scheduling at this number and reports how many discovered URLs were left.",
            "default": 50
          },
          "maxDepth": {
            "title": "Maximum link depth",
            "minimum": 0,
            "maximum": 50,
            "type": "integer",
            "description": "Link hops from a start URL (start = 0, pages from the sitemap = 1). 0 reads only the start URLs.",
            "default": 5
          },
          "includeSubdomains": {
            "title": "Include subdomains",
            "type": "boolean",
            "description": "Also crawl subdomains of each start host (for example blog.example.com when starting at example.com). Parent and sibling domains are never crawled.",
            "default": false
          },
          "useSitemaps": {
            "title": "Use sitemaps",
            "type": "boolean",
            "description": "Also discover pages from the site's sitemaps (robots.txt Sitemap lines, else /sitemap.xml). Nested sitemap indexes are followed up to 3 levels.",
            "default": true
          },
          "respectRobotsTxt": {
            "title": "Respect robots.txt",
            "type": "boolean",
            "description": "Follow the site's robots.txt rules and Crawl-delay (RFC 9309). Turn off only for sites you own or may crawl regardless.",
            "default": true
          },
          "includeUrlPatterns": {
            "title": "Only crawl URLs matching",
            "maxItems": 50,
            "type": "array",
            "description": "Glob patterns matched against full URLs. ** matches anything, * matches anything except /. Example: https://example.com/blog/**. Empty = every URL in scope. Start URLs are always read.",
            "default": [],
            "items": {
              "type": "string"
            }
          },
          "excludeUrlPatterns": {
            "title": "Never crawl URLs matching",
            "maxItems": 50,
            "type": "array",
            "description": "Glob patterns for URLs to skip, same syntax. Example: https://example.com/tag/**.",
            "default": [],
            "items": {
              "type": "string"
            }
          },
          "renderMode": {
            "title": "Rendering",
            "enum": [
              "auto",
              "http",
              "browser"
            ],
            "type": "string",
            "description": "auto: read over HTTP and open a headless browser only when the page is a JavaScript shell, has almost no text or is a bot-challenge page. http: never use a browser (cheapest). browser: render every page (slowest, needs more memory).",
            "default": "auto"
          },
          "maxConcurrency": {
            "title": "Concurrent page reads",
            "minimum": 1,
            "maximum": 20,
            "type": "integer",
            "description": "Pages read at the same time across all hosts.",
            "default": 5
          },
          "maxRequestsPerMinute": {
            "title": "Requests per minute per host",
            "minimum": 1,
            "maximum": 600,
            "type": "integer",
            "description": "Politeness limit per host (robots.txt, sitemaps and pages). A longer robots.txt Crawl-delay wins.",
            "default": 60
          },
          "maxCrawlSeconds": {
            "title": "Maximum crawl time (seconds)",
            "minimum": 30,
            "maximum": 86400,
            "type": "integer",
            "description": "Wall-clock budget. When it runs out no new page starts; pages already loading finish and the partial crawl is reported.",
            "default": 3600
          },
          "pageTimeoutSeconds": {
            "title": "Page timeout (seconds)",
            "minimum": 5,
            "maximum": 120,
            "type": "integer",
            "description": "Maximum time for one page read.",
            "default": 30
          },
          "maxPageSizeKb": {
            "title": "Maximum page size (KB)",
            "minimum": 100,
            "maximum": 20000,
            "type": "integer",
            "description": "Larger pages are cut at this size and marked partial.",
            "default": 5000
          },
          "maxUrlsPerPattern": {
            "title": "Maximum URLs per URL pattern",
            "minimum": 1,
            "maximum": 10000,
            "type": "integer",
            "description": "Protects against calendars and filter combinations: URLs that differ only in numbers, dates, IDs or query values share a pattern, and each pattern gets at most this many URLs.",
            "default": 200
          },
          "maxDiscoveredUrls": {
            "title": "Maximum tracked URLs",
            "minimum": 100,
            "maximum": 20000,
            "type": "integer",
            "description": "Cap on distinct in-scope URLs the crawl keeps track of (queued, read or skipped). Further URLs are counted but not tracked.",
            "default": 5000
          },
          "maxSitemapUrls": {
            "title": "Maximum URLs taken from sitemaps",
            "minimum": 0,
            "maximum": 20000,
            "type": "integer",
            "description": "Cap on page URLs taken from sitemaps. 0 disables sitemaps.",
            "default": 5000
          },
          "maxSitemapFiles": {
            "title": "Maximum sitemap files",
            "minimum": 1,
            "maximum": 200,
            "type": "integer",
            "description": "Cap on sitemap files downloaded, indexes included.",
            "default": 20
          },
          "includeHtml": {
            "title": "Include raw HTML",
            "type": "boolean",
            "description": "Add the page HTML (up to 1 MB per page) to each item. Makes the dataset much larger.",
            "default": false
          }
        }
      },
      "runsResponseSchema": {
        "type": "object",
        "properties": {
          "data": {
            "type": "object",
            "properties": {
              "id": {
                "type": "string"
              },
              "actId": {
                "type": "string"
              },
              "userId": {
                "type": "string"
              },
              "startedAt": {
                "type": "string",
                "format": "date-time",
                "example": "2025-01-08T00:00:00.000Z"
              },
              "finishedAt": {
                "type": "string",
                "format": "date-time",
                "example": "2025-01-08T00:00:00.000Z"
              },
              "status": {
                "type": "string",
                "example": "READY"
              },
              "meta": {
                "type": "object",
                "properties": {
                  "origin": {
                    "type": "string",
                    "example": "API"
                  },
                  "userAgent": {
                    "type": "string"
                  }
                }
              },
              "stats": {
                "type": "object",
                "properties": {
                  "inputBodyLen": {
                    "type": "integer",
                    "example": 2000
                  },
                  "rebootCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "restartCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "resurrectCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "computeUnits": {
                    "type": "integer",
                    "example": 0
                  }
                }
              },
              "options": {
                "type": "object",
                "properties": {
                  "build": {
                    "type": "string",
                    "example": "latest"
                  },
                  "timeoutSecs": {
                    "type": "integer",
                    "example": 300
                  },
                  "memoryMbytes": {
                    "type": "integer",
                    "example": 1024
                  },
                  "diskMbytes": {
                    "type": "integer",
                    "example": 2048
                  }
                }
              },
              "buildId": {
                "type": "string"
              },
              "defaultKeyValueStoreId": {
                "type": "string"
              },
              "defaultDatasetId": {
                "type": "string"
              },
              "defaultRequestQueueId": {
                "type": "string"
              },
              "buildNumber": {
                "type": "string",
                "example": "1.0.0"
              },
              "containerUrl": {
                "type": "string"
              },
              "usage": {
                "type": "object",
                "properties": {
                  "ACTOR_COMPUTE_UNITS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_WRITES": {
                    "type": "integer",
                    "example": 1
                  },
                  "KEY_VALUE_STORE_LISTS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_INTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_EXTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_SERPS": {
                    "type": "integer",
                    "example": 0
                  }
                }
              },
              "usageTotalUsd": {
                "type": "number",
                "example": 0.00005
              },
              "usageUsd": {
                "type": "object",
                "properties": {
                  "ACTOR_COMPUTE_UNITS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_WRITES": {
                    "type": "number",
                    "example": 0.00005
                  },
                  "KEY_VALUE_STORE_LISTS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_INTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_EXTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_SERPS": {
                    "type": "integer",
                    "example": 0
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Website Crawler: Crawl a Site to Markdown, Text and JSON OpenAPI definition

OpenAPI is a standard for designing and describing RESTful APIs, allowing developers to define API structure, endpoints, and data formats in a machine-readable way. It simplifies API development, integration, and documentation.

OpenAPI is effective when used with AI agents and GPTs by standardizing how these systems interact with various APIs, for reliable integrations and efficient communication.

By defining machine-readable API specifications, OpenAPI allows AI models like GPTs to understand and use varied data sources, improving accuracy. This accelerates development, reduces errors, and provides context-aware responses, making OpenAPI a core component for AI applications.

You can download the OpenAPI definitions for Website Crawler - Site Pages to Markdown and Text from the options below:

OpenAPI.json

If you’d like to learn more about how OpenAPI powers GPTs, read our blog post.

You can also check out our other API clients:

Website Crawler - Site Pages to Markdown and Text API in Python

Website Crawler - Site Pages to Markdown and Text API in JavaScript

Website Crawler - Site Pages to Markdown and Text API through CLI

Website Crawler - Site Pages to Markdown and Text API

Website Markdown Crawler - Content & Change Tracking

benthepythondev/website-markdown-crawler

Crawl public HTML within a website section. Export Markdown, text, source URLs and change hashes, with page limits and a URL-level coverage report.

Ben

Website to Markdown Crawler - Full-Site Text for LLMs & RAG

entranced_gelato/website-to-markdown-crawler

Crawl any website from a start URL and get every page as clean text + Markdown for LLMs, RAG, and AI agents. Follows internal links with depth and page limits, strips nav and ads, and returns one structured record per page. A fast, no-config site-to-Markdown crawler.

AIDevs

Website Content Crawler — Clean Markdown for AI

dallel-data/website-content-to-markdown

Crawl public HTML websites into clean Markdown and text for AI, RAG and content research. Export titles, metadata, canonical URLs and links with depth and page limits.

GUIR Dallel

Website to Markdown Crawler for LLM & RAG

logiover/website-text-markdown-crawler

Crawl any website to clean Markdown and plain text for LLM training and RAG. HTML to Markdown, no API or login. Export website text to CSV or JSON.

Logiover

Website Content Crawler - Markdown & Text for LLM / RAG

pear_fight/website-content-crawler-markdown-text-for-llm-rag

Crawl any website and extract clean article text and Markdown, ready to feed into LLMs, ChatGPT, vector databases and RAG pipelines. Removes navigation, ads and boilerplate. Configurable crawl depth and page limits. Export to JSON, CSV, Excel.

Harald

Website Content Crawler — Text, Markdown & HTML for AI/LLM

hichemdev/website-content-crawler

Crawl any website and extract clean text, Markdown, and HTML from every page — ready for LLM, RAG, and AI ingestion.

Hichem Ben Moussa

Website to Markdown Scraper

receptional_blender/website-to-markdown-scraper

Crawl any website and turn its pages into clean Markdown — plus optional plain text, raw HTML and full-page screenshots. Built for LLM, RAG and AI training datasets.

Assia Fadli

Website to Markdown — Clean Page Text for LLMs

sourcedirect/website-to-markdown

Fetch any URL and return clean markdown/text plus links. No headless browser for most pages. Free demo: https://vaytric.com/page-to-markdown/ — bulk URL lists here.

Kurt Landman

Website Crawler Scraper

automation-lab/website-content-crawler

Crawl public websites into page-level metadata, links, clean text, and Markdown for search indexes and LLM knowledge bases.

Automation Lab

Website to Markdown Crawler for LLM & RAG (fast, no browser)

webdatatools/website-to-markdown

Website to Markdown Crawler turns any site into clean Markdown per page for LLMs, RAG pipelines and vector DBs — no headless browser, $1 per 1,000 pages.