Pricing

from $2.10 / 1,000 page scrapeds

Try for free

Go to Apify Store

Website to RAG Chunks: Vector-DB Ready Markdown

Try for free

Crawl a website, docs portal or blog and get LLM-ready chunks for RAG: clean Markdown split by headings to your token budget, with overlap, heading path, chunk IDs, content hashes and page metadata. Drop straight into Pinecone, Qdrant, Weaviate, pgvector or LangChain. Pay per page.

Pricing

from $2.10 / 1,000 page scrapeds

Rating

0.0

(0)

Developer

Torix

Actor stats

Bookmarked

Total users

Monthly active users

2 days ago

Last modified

Categories

Developer tools

Automation

You can access the Website to RAG Chunks: Vector-DB Ready Markdown programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

{
  "openapi": "3.0.1",
  "info": {
    "version": "1.0",
    "x-build-id": "EX5KP0BKnd2taGoDh"
  },
  "servers": [
    {
      "url": "https://api.apify.com/v2"
    }
  ],
  "paths": {
    "/acts/torix33~website-to-rag-chunks/run-sync-get-dataset-items": {
      "post": {
        "operationId": "run-sync-get-dataset-items-torix33-website-to-rag-chunks",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK"
          }
        }
      }
    },
    "/acts/torix33~website-to-rag-chunks/runs": {
      "post": {
        "operationId": "runs-sync-torix33-website-to-rag-chunks",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor and returns information about the initiated run in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK",
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/runsResponseSchema"
                }
              }
            }
          }
        }
      }
    },
    "/acts/torix33~website-to-rag-chunks/run-sync": {
      "post": {
        "operationId": "run-sync-torix33-website-to-rag-chunks",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK"
          }
        }
      }
    }
  },
  "components": {
    "schemas": {
      "inputSchema": {
        "type": "object",
        "required": [
          "startUrls"
        ],
        "properties": {
          "startUrls": {
            "title": "URLs",
            "type": "array",
            "description": "Start pages of the websites / docs to turn into chunks. One per line.",
            "items": {
              "type": "string"
            }
          },
          "crawl": {
            "title": "Crawl the website",
            "type": "boolean",
            "description": "Follow links on the same site from each start URL (breadth-first).",
            "default": true
          },
          "maxDepth": {
            "title": "Max crawl depth",
            "minimum": 0,
            "maximum": 10,
            "type": "integer",
            "description": "0 = start pages only, 1 = pages linked from them, etc.",
            "default": 2
          },
          "maxPagesPerSite": {
            "title": "Max pages per site",
            "minimum": 1,
            "type": "integer",
            "description": "Cap per website when crawling.",
            "default": 100
          },
          "useSitemap": {
            "title": "Also use sitemap.xml",
            "type": "boolean",
            "description": "Add the site's sitemap URLs to the crawl (fast way to cover a whole blog or docs site).",
            "default": false
          },
          "includeUrlPattern": {
            "title": "Include URLs matching (regex)",
            "type": "string",
            "description": "Only crawl URLs matching this regular expression, e.g. <code>/blog/</code>."
          },
          "excludeUrlPattern": {
            "title": "Exclude URLs matching (regex)",
            "type": "string",
            "description": "Skip URLs matching this regular expression, e.g. <code>/tag/|/page/\\d+</code>."
          },
          "extractionMode": {
            "title": "Content extraction",
            "enum": [
              "article",
              "full"
            ],
            "type": "string",
            "description": "<b>Article</b>: main content only via Mozilla Readability (best for blogs, news, docs). <b>Full page</b>: whole body minus navigation, header, footer and sidebars.",
            "default": "article"
          },
          "maxTokens": {
            "title": "Max tokens per chunk",
            "minimum": 50,
            "maximum": 8000,
            "type": "integer",
            "description": "Chunk size budget (≈4 characters per token). 300–800 suits most embedding models.",
            "default": 500
          },
          "overlapTokens": {
            "title": "Overlap tokens",
            "minimum": 0,
            "maximum": 2000,
            "type": "integer",
            "description": "Repeat the tail of the previous chunk (same section) at the start of the next one.",
            "default": 50
          },
          "splitOnHeadings": {
            "title": "Start a new chunk at every heading",
            "type": "boolean",
            "description": "Keeps each chunk inside one section (better retrieval). Off = pack sections together up to the token budget.",
            "default": true
          },
          "prependContext": {
            "title": "Prepend page title & heading path",
            "type": "boolean",
            "description": "Start each chunk's text with \"Page title > Section > Subsection\" so the embedding knows the context.",
            "default": true
          },
          "minWordCount": {
            "title": "Min word count",
            "minimum": 0,
            "type": "integer",
            "description": "Skip (and don't charge for) pages with fewer words, e.g. tag or listing pages.",
            "default": 50
          },
          "maxResults": {
            "title": "Max results (cost cap)",
            "minimum": 0,
            "type": "integer",
            "description": "Stop after this many pages in total. <code>0</code> = no limit.",
            "default": 0
          },
          "maxConcurrency": {
            "title": "Max concurrency",
            "minimum": 1,
            "maximum": 100,
            "type": "integer",
            "description": "Pages fetched in parallel.",
            "default": 20
          },
          "requestTimeoutSecs": {
            "title": "Page timeout (seconds)",
            "minimum": 5,
            "maximum": 120,
            "type": "integer",
            "description": "Give up on a page after this long.",
            "default": 30
          },
          "proxyConfiguration": {
            "title": "Proxy configuration",
            "type": "object",
            "description": "Not needed for most sites. Enable Apify Proxy for sites that block cloud servers.",
            "default": {
              "useApifyProxy": false
            }
          }
        }
      },
      "runsResponseSchema": {
        "type": "object",
        "properties": {
          "data": {
            "type": "object",
            "properties": {
              "id": {
                "type": "string"
              },
              "actId": {
                "type": "string"
              },
              "userId": {
                "type": "string"
              },
              "startedAt": {
                "type": "string",
                "format": "date-time",
                "example": "2025-01-08T00:00:00.000Z"
              },
              "finishedAt": {
                "type": "string",
                "format": "date-time",
                "example": "2025-01-08T00:00:00.000Z"
              },
              "status": {
                "type": "string",
                "example": "READY"
              },
              "meta": {
                "type": "object",
                "properties": {
                  "origin": {
                    "type": "string",
                    "example": "API"
                  },
                  "userAgent": {
                    "type": "string"
                  }
                }
              },
              "stats": {
                "type": "object",
                "properties": {
                  "inputBodyLen": {
                    "type": "integer",
                    "example": 2000
                  },
                  "rebootCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "restartCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "resurrectCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "computeUnits": {
                    "type": "integer",
                    "example": 0
                  }
                }
              },
              "options": {
                "type": "object",
                "properties": {
                  "build": {
                    "type": "string",
                    "example": "latest"
                  },
                  "timeoutSecs": {
                    "type": "integer",
                    "example": 300
                  },
                  "memoryMbytes": {
                    "type": "integer",
                    "example": 1024
                  },
                  "diskMbytes": {
                    "type": "integer",
                    "example": 2048
                  }
                }
              },
              "buildId": {
                "type": "string"
              },
              "defaultKeyValueStoreId": {
                "type": "string"
              },
              "defaultDatasetId": {
                "type": "string"
              },
              "defaultRequestQueueId": {
                "type": "string"
              },
              "buildNumber": {
                "type": "string",
                "example": "1.0.0"
              },
              "containerUrl": {
                "type": "string"
              },
              "usage": {
                "type": "object",
                "properties": {
                  "ACTOR_COMPUTE_UNITS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_WRITES": {
                    "type": "integer",
                    "example": 1
                  },
                  "KEY_VALUE_STORE_LISTS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_INTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_EXTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_SERPS": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_UNBLOCKER_UNITS": {
                    "type": "integer",
                    "example": 0
                  }
                }
              },
              "usageTotalUsd": {
                "type": "number",
                "example": 0.00005
              },
              "usageUsd": {
                "type": "object",
                "properties": {
                  "ACTOR_COMPUTE_UNITS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_WRITES": {
                    "type": "number",
                    "example": 0.00005
                  },
                  "KEY_VALUE_STORE_LISTS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_INTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_EXTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_SERPS": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_UNBLOCKER_UNITS": {
                    "type": "integer",
                    "example": 0
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Website to RAG Chunks – Vector-DB Ready Markdown OpenAPI definition

OpenAPI is a standard for designing and describing RESTful APIs, allowing developers to define API structure, endpoints, and data formats in a machine-readable way. It simplifies API development, integration, and documentation.

OpenAPI is effective when used with AI agents and GPTs by standardizing how these systems interact with various APIs, for reliable integrations and efficient communication.

By defining machine-readable API specifications, OpenAPI allows AI models like GPTs to understand and use varied data sources, improving accuracy. This accelerates development, reduces errors, and provides context-aware responses, making OpenAPI a core component for AI applications.

You can download the OpenAPI definitions for Website to RAG Chunks: Vector-DB Ready Markdown from the options below:

OpenAPI.json

If you’d like to learn more about how OpenAPI powers GPTs, read our blog post.

You can also check out our other API clients:

Website to RAG Chunks: Vector-DB Ready Markdown API in Python

Website to RAG Chunks: Vector-DB Ready Markdown API in JavaScript

Website to RAG Chunks: Vector-DB Ready Markdown API through CLI

Website to RAG Chunks: Vector-DB Ready Markdown API

Instagram Scraper

apify/instagram-scraper

Extract Instagram posts, reels, profiles, places, hashtags, carousels, and comments. Get data from Instagram using one or more Instagram URLs or search queries: content, context, metrics, metadata. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Apify

446K

4.7

(603)

Instagram Profile Scraper

apify/instagram-profile-scraper

Scrape all Instagram profile info. Just add Instagram usernames, IDs or URLs and extract name, join date, number of followers, location, bio, website, related profiles, video&post count, latest posts. Export scraped data, schedule scraper via API, and integrate with other tools or AI workflows.

Apify

261K

4.7

(175)

Google Maps Scraper

compass/crawler-google-places

Extract data from thousands of Google Maps locations and businesses, including reviews, reviewer details, images, contact info, including full name, email, and job title, opening hours, prices & more. Export data, run via API, schedule and monitor runs, or integrate with other tools.

Compass

660K

4.7

(1,818)

TikTok Scraper

clockworks/tiktok-scraper

Extract data from TikTok videos, hashtags, and users. Use URLs or search queries to scrape TikTok profiles, hashtags, posts, URLs, shares, followers, hearts, names, video, and music-related data. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Clockworks

354K

4.7

(370)

Google Search Results Scraper

apify/google-search-scraper

Scrape Google Search Engine Results Pages (SERPs). Select the country or language and extract organic and paid results, AI Mode, AI overviews, ads, queries, People Also Ask, prices, reviews, like a Google SERP API. Export data, run the scraper via API, schedule runs, or integrate with other tools.

Apify

213K

4.6

(220)

Instagram Reel Scraper

apify/instagram-reel-scraper

Scrape or download Instagram reels. Just add Instagram profile or reel URL, and get caption, timestamp, transcript, hashtags, mentions, tagged users, comments, likes, shares, views, duration, and downloaded video. Export scraped data, schedule via API, and integrate with other tools or AI workflows.

Apify

173K

4.4

(103)

1688 Wholesale Products Scraper - 50+ Fields from 1688.com

zen-studio/1688-wholesale-scraper

Extract 50+ fields per product from 1688.com at 250+ products/minute. Tiered wholesale pricing, supplier verification, factory stats, dropship channels, certifications, trade service scores. Search by keyword (Chinese/English) or offer IDs. JSON output, no login required.

Zen Studio

4.9K

5.0

(4)

LinkedIn Profile Posts Scraper (No Cookies)

harvestapi/linkedin-profile-posts

Extract posts from LinkedIn profiles, including content, media, engagement, reactions, comments, and more. No cookies or account required.

HarvestAPI

44K

4.4

(55)

LinkedIn Profile Scraper + Email (No Cookies)

harvestapi/linkedin-profile-scraper

Extract detailed information from LinkedIn profiles in bulk, including complete work experience, education history, skills and more. No cookies or account required.

HarvestAPI

86K

4.6

(93)

Web Fetch

apify/web-fetch

Download any web page by URL and convert it to clean Markdown, plain text, HTML, and other formats. Automatically renders JavaScript and works around bot blocking. The Actor provides a real-time API for quick response times.

Apify

37K

5.0

(3)