Pricing

from $0.50 / 1,000 page extracteds

Try for free

Go to Apify Store

Sitemap Content Extractor — Website to Clean Markdown

Try for free

Turn any website's sitemap into clean, LLM-ready Markdown. Auto-discovers sitemaps via robots.txt, extracts article text, tables, headings, links and images while stripping navigation, ads and cookie banners. Filter pages by lastmod date.

Pricing

from $0.50 / 1,000 page extracteds

Rating

0.0

(0)

Developer

XiaoZhi DataTools

Actor stats

Bookmarked

Total users

Monthly active users

4 hours ago

Last modified

Categories

Developer tools

Automation

You can access the Sitemap Content Extractor — Website to Clean Markdown programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

{
  "openapi": "3.0.1",
  "info": {
    "version": "0.5",
    "x-build-id": "jNEtsvcEWVI03ej5w"
  },
  "servers": [
    {
      "url": "https://api.apify.com/v2"
    }
  ],
  "paths": {
    "/acts/celebrated-quadraphonic~sitemap-content-extractor/run-sync-get-dataset-items": {
      "post": {
        "operationId": "run-sync-get-dataset-items-celebrated-quadraphonic-sitemap-content-extractor",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK"
          }
        }
      }
    },
    "/acts/celebrated-quadraphonic~sitemap-content-extractor/runs": {
      "post": {
        "operationId": "runs-sync-celebrated-quadraphonic-sitemap-content-extractor",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor and returns information about the initiated run in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK",
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/runsResponseSchema"
                }
              }
            }
          }
        }
      }
    },
    "/acts/celebrated-quadraphonic~sitemap-content-extractor/run-sync": {
      "post": {
        "operationId": "run-sync-celebrated-quadraphonic-sitemap-content-extractor",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK"
          }
        }
      }
    }
  },
  "components": {
    "schemas": {
      "inputSchema": {
        "type": "object",
        "properties": {
          "includeMetadata": {
            "title": "Include metadata",
            "type": "boolean",
            "description": "Include title, meta description, language, headings, outbound links and image counts.",
            "default": true
          },
          "limitToStartDomain": {
            "title": "Limit to start domain",
            "type": "boolean",
            "description": "Only fetch URLs on the same domain as the start URL.",
            "default": true
          },
          "maxContentChars": {
            "title": "Max content chars",
            "minimum": 0,
            "type": "integer",
            "description": "Truncate extracted content per page to this many characters. 0 = no truncation.",
            "default": 0
          },
          "maxPages": {
            "title": "Max pages",
            "minimum": 1,
            "maximum": 50000,
            "type": "integer",
            "description": "Maximum number of pages to fetch and extract.",
            "default": 100
          },
          "maxSitemapDepth": {
            "title": "Max sitemap depth",
            "minimum": 1,
            "maximum": 10,
            "type": "integer",
            "description": "How many nested sitemap-index levels to expand.",
            "default": 3
          },
          "outputFormat": {
            "title": "Output format",
            "enum": [
              "markdown",
              "text",
              "html"
            ],
            "type": "string",
            "description": "Content format: clean Markdown (best for LLMs), plain text, or cleaned HTML.",
            "default": "markdown"
          },
          "proxyConfiguration": {
            "title": "Proxy configuration",
            "type": "object",
            "description": "Optional. Sitemaps and pages are fetched over plain HTTP; proxies are not needed for typical runs.",
            "default": {
              "useApifyProxy": false
            }
          },
          "requestConcurrency": {
            "title": "Request concurrency",
            "minimum": 1,
            "maximum": 30,
            "type": "integer",
            "description": "How many pages to fetch in parallel.",
            "default": 10
          },
          "respectRobotsTxt": {
            "title": "Respect robots.txt",
            "type": "boolean",
            "description": "Honor robots.txt Disallow rules when fetching page content.",
            "default": true
          },
          "startUrls": {
            "title": "Start URLs / domains",
            "type": "array",
            "description": "Domains or sitemap URLs, one per line. Examples: apify.com, https://example.com/sitemap.xml",
            "items": {
              "type": "string"
            }
          },
          "urlPatternInclude": {
            "title": "URL include patterns",
            "type": "array",
            "description": "Only keep URLs matching at least one of these regular expressions, one per line. Example: /docs/",
            "items": {
              "type": "string"
            }
          },
          "urlPatternExclude": {
            "title": "URL exclude patterns",
            "type": "array",
            "description": "Skip URLs matching any of these regular expressions, one per line. Example: /blog/draft",
            "items": {
              "type": "string"
            }
          },
          "minLastmod": {
            "title": "Min lastmod date",
            "type": "string",
            "description": "Only keep URLs whose sitemap lastmod is on/after this date (YYYY-MM-DD). URLs without lastmod are kept.",
            "default": ""
          },
          "maxLastmod": {
            "title": "Max lastmod date",
            "type": "string",
            "description": "Only keep URLs whose sitemap lastmod is on/before this date (YYYY-MM-DD). URLs without lastmod are kept.",
            "default": ""
          }
        }
      },
      "runsResponseSchema": {
        "type": "object",
        "properties": {
          "data": {
            "type": "object",
            "properties": {
              "id": {
                "type": "string"
              },
              "actId": {
                "type": "string"
              },
              "userId": {
                "type": "string"
              },
              "startedAt": {
                "type": "string",
                "format": "date-time",
                "example": "2025-01-08T00:00:00.000Z"
              },
              "finishedAt": {
                "type": "string",
                "format": "date-time",
                "example": "2025-01-08T00:00:00.000Z"
              },
              "status": {
                "type": "string",
                "example": "READY"
              },
              "meta": {
                "type": "object",
                "properties": {
                  "origin": {
                    "type": "string",
                    "example": "API"
                  },
                  "userAgent": {
                    "type": "string"
                  }
                }
              },
              "stats": {
                "type": "object",
                "properties": {
                  "inputBodyLen": {
                    "type": "integer",
                    "example": 2000
                  },
                  "rebootCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "restartCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "resurrectCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "computeUnits": {
                    "type": "integer",
                    "example": 0
                  }
                }
              },
              "options": {
                "type": "object",
                "properties": {
                  "build": {
                    "type": "string",
                    "example": "latest"
                  },
                  "timeoutSecs": {
                    "type": "integer",
                    "example": 300
                  },
                  "memoryMbytes": {
                    "type": "integer",
                    "example": 1024
                  },
                  "diskMbytes": {
                    "type": "integer",
                    "example": 2048
                  }
                }
              },
              "buildId": {
                "type": "string"
              },
              "defaultKeyValueStoreId": {
                "type": "string"
              },
              "defaultDatasetId": {
                "type": "string"
              },
              "defaultRequestQueueId": {
                "type": "string"
              },
              "buildNumber": {
                "type": "string",
                "example": "1.0.0"
              },
              "containerUrl": {
                "type": "string"
              },
              "usage": {
                "type": "object",
                "properties": {
                  "ACTOR_COMPUTE_UNITS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_WRITES": {
                    "type": "integer",
                    "example": 1
                  },
                  "KEY_VALUE_STORE_LISTS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_INTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_EXTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_SERPS": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_UNBLOCKER_UNITS": {
                    "type": "integer",
                    "example": 0
                  }
                }
              },
              "usageTotalUsd": {
                "type": "number",
                "example": 0.00005
              },
              "usageUsd": {
                "type": "object",
                "properties": {
                  "ACTOR_COMPUTE_UNITS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_WRITES": {
                    "type": "number",
                    "example": 0.00005
                  },
                  "KEY_VALUE_STORE_LISTS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_INTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_EXTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_SERPS": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_UNBLOCKER_UNITS": {
                    "type": "integer",
                    "example": 0
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Sitemap Content Extractor — Website to Clean Markdown OpenAPI definition

OpenAPI is a standard for designing and describing RESTful APIs, allowing developers to define API structure, endpoints, and data formats in a machine-readable way. It simplifies API development, integration, and documentation.

OpenAPI is effective when used with AI agents and GPTs by standardizing how these systems interact with various APIs, for reliable integrations and efficient communication.

By defining machine-readable API specifications, OpenAPI allows AI models like GPTs to understand and use varied data sources, improving accuracy. This accelerates development, reduces errors, and provides context-aware responses, making OpenAPI a core component for AI applications.

You can download the OpenAPI definitions for Sitemap Content Extractor — Website to Clean Markdown from the options below:

OpenAPI.json

If you’d like to learn more about how OpenAPI powers GPTs, read our blog post.

You can also check out our other API clients:

Sitemap Content Extractor — Website to Clean Markdown API in Python

Sitemap Content Extractor — Website to Clean Markdown API in JavaScript

Sitemap Content Extractor — Website to Clean Markdown API through CLI

Sitemap Content Extractor — Website to Clean Markdown API

Website Content Extractor - Markdown for LLM, RAG, Vector DB

snow_leo_data/website-to-markdown

Readable text from 73.1% of 52 live pages over plain HTTP, including 10 of 10 documentation sites. Website to LLM text and a website to markdown converter in one run: clean Markdown, chunks with overlap for a vector database, 46 flat fields, and what changed since the last run.

Snow Leo Data

Sitemap Extractor

smorgi_apps/sitemap-extractor

Smorgi Apps

Article Record Extractor

nexgenwatch/article-record-extractor

Turns a list of article URLs into normalized records: title, author, dates, canonical, heading outline, main text, word count, language, an extraction-confidence score, and per-field provenance.

NexGen Watch

Clean Content Extractor for RAG & AI Agents

hadster/clean-content-extractor-for-rag-ai-agents

Turn any article into clean markdown for LLMs. Renders JS-heavy pages, strips ads and navigation, pay only for successful extractions. Best for blog posts, news articles, and documentation pages — not homepage or listing pages.

Hadi Fadlallah

Sitemap URL Extractor

lost.fany/sitemap-url-extractor

Extract every URL from a website's sitemap.xml, sitemap index and robots.txt in seconds — with lastmod, changefreq, priority and hreflang. No crawling, no browser: fast, cheap, reliable.

Natalia Ivanova

Podcast RSS Scraper & Transcriber

celebrated-quadraphonic/podcast-rss-scraper

Scrape any podcast RSS feed for episodes with metadata, durations and chapters. Get FREE publisher transcripts (Podcasting 2.0) or fall back to Whisper with your own key. Bulk-import subscriptions via OPML, filter by date/duration/title, and monitor shows for new episodes.

XiaoZhi DataTools

Sitemap URL Extractor — robots.txt, Sitemap Index, gzip

chorelet/sitemap-url-extractor

Every URL of a website from its sitemaps: robots.txt discovery, sitemap indexes, gzip, lastmod, changefreq, priority, hreflang alternates, image/video/news extensions. Include/exclude patterns and date filter, CSV/JSON export and API.

Chorelet

Website Content Extractor for LLM and RAG

martellosystems/website-content-extractor

Crawl any website and extract the main readable content of each page as clean markdown - nav, footers, scripts and ads stripped. Built for RAG pipelines and LLM ingestion. One record per page {url, title, markdown, word_count} plus a final site inventory.

Martello Systems

Sitemap URLs: Exact File, No Page Crawl

jmspwr/sitemap-url-extractor

Extract the exact sitemap file you name, or discover a domain's sitemaps. Deduplicated URL, lastmod and source rows with explicit partial/error reports. No page crawl or proxy; empty runs are free.

James Power

SEC EDGAR Scraper — 10-K, 10-Q, 8-K, Form 4, 13F, Form D

celebrated-quadraphonic/sec-edgar-filings-scraper

Scrape SEC EDGAR filings with zero anti-bot friction: 10-K, 10-Q, 8-K, Form 3/4/5 insider trades, Form D funding, 13F holdings, 13D/G, S-1, proxy statements. XBRL financials, full-text search, monitor mode. Pure HTTP, no browser, no proxies needed.

XiaoZhi DataTools