Under maintenance

Pricing

from $5.00 / 1,000 url crawleds

Try for free

Go to Apify Store

LLM-Ready Web Scraper – RAG & Vertical Data Extraction

Under maintenance

Try for free

Scrapes any URL and returns clean LLM-ready content. Strips ads, nav, and boilerplate. Returns markdown, chunked text, token estimates, and metadata. Vertical modes for Legal, Medical, Property, E-commerce, Research, and News. Firecrawl alternative at $0.005 per URL.

Pricing

from $5.00 / 1,000 url crawleds

Rating

0.0

(0)

Developer

joseph fadero

Actor stats

Bookmarked

Total users

Monthly active users

14 hours ago

Last modified

Categories

Developer tools

Automation

You can access the LLM-Ready Web Scraper – RAG & Vertical Data Extraction programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in Integrations settings in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

{
  "openapi": "3.0.1",
  "info": {
    "version": "1.0",
    "x-build-id": "mSO6S1SydcYYoBEdx"
  },
  "servers": [
    {
      "url": "https://api.apify.com/v2"
    }
  ],
  "paths": {
    "/acts/conceivable_extension~llm-ready-web-scraper/run-sync-get-dataset-items": {
      "post": {
        "operationId": "run-sync-get-dataset-items-conceivable_extension-llm-ready-web-scraper",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK"
          }
        }
      }
    },
    "/acts/conceivable_extension~llm-ready-web-scraper/runs": {
      "post": {
        "operationId": "runs-sync-conceivable_extension-llm-ready-web-scraper",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor and returns information about the initiated run in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK",
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/runsResponseSchema"
                }
              }
            }
          }
        }
      }
    },
    "/acts/conceivable_extension~llm-ready-web-scraper/run-sync": {
      "post": {
        "operationId": "run-sync-conceivable_extension-llm-ready-web-scraper",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK"
          }
        }
      }
    }
  },
  "components": {
    "schemas": {
      "inputSchema": {
        "type": "object",
        "required": [
          "urls"
        ],
        "properties": {
          "urls": {
            "title": "URLs to Scrape",
            "minItems": 1,
            "type": "array",
            "description": "List of URLs to scrape and convert to LLM-ready content. Supports any public web page.",
            "items": {
              "type": "string"
            }
          },
          "outputFormat": {
            "title": "Output Format",
            "enum": [
              "markdown",
              "plaintext",
              "json"
            ],
            "type": "string",
            "description": "Format for the extracted content. Markdown is recommended for LLMs.",
            "default": "markdown"
          },
          "vertical": {
            "title": "Content Vertical",
            "enum": [
              "general",
              "legal",
              "medical",
              "property",
              "ecommerce",
              "research",
              "news"
            ],
            "type": "string",
            "description": "Optimises extraction rules for the content type. General works for all pages.",
            "default": "general"
          },
          "chunkContent": {
            "title": "Chunk Content for RAG",
            "type": "boolean",
            "description": "Split content into chunks suitable for embedding. Set chunk size with chunkTokenSize.",
            "default": false
          },
          "chunkTokenSize": {
            "title": "Chunk Token Size",
            "minimum": 128,
            "maximum": 4096,
            "type": "integer",
            "description": "Approximate token size per chunk when chunking is enabled. 512 is standard for most embedding models.",
            "default": 512
          },
          "includeMetadata": {
            "title": "Include Page Metadata",
            "type": "boolean",
            "description": "Include title, description, author, publish date, word count, and estimated token count in output.",
            "default": true
          },
          "removeElements": {
            "title": "Additional Elements to Remove",
            "type": "array",
            "description": "CSS selectors for elements to strip from the page before extraction. Example: '.sidebar', '#comments'",
            "items": {
              "type": "string"
            }
          },
          "followLinks": {
            "title": "Follow Internal Links",
            "type": "boolean",
            "description": "If enabled, follows internal links from each starting URL and scrapes linked pages too. Use maxDepth to control depth.",
            "default": false
          },
          "maxDepth": {
            "title": "Max Link Depth",
            "minimum": 1,
            "maximum": 3,
            "type": "integer",
            "description": "How many levels of internal links to follow. Only used when followLinks is true.",
            "default": 1
          },
          "maxPagesPerUrl": {
            "title": "Max Pages Per Starting URL",
            "minimum": 1,
            "maximum": 100,
            "type": "integer",
            "description": "Maximum pages to scrape per starting URL when following links.",
            "default": 10
          },
          "proxyConfiguration": {
            "title": "Proxy Configuration",
            "type": "object",
            "description": "Proxy settings. Residential proxy recommended for protected sites."
          }
        }
      },
      "runsResponseSchema": {
        "type": "object",
        "properties": {
          "data": {
            "type": "object",
            "properties": {
              "id": {
                "type": "string"
              },
              "actId": {
                "type": "string"
              },
              "userId": {
                "type": "string"
              },
              "startedAt": {
                "type": "string",
                "format": "date-time",
                "example": "2025-01-08T00:00:00.000Z"
              },
              "finishedAt": {
                "type": "string",
                "format": "date-time",
                "example": "2025-01-08T00:00:00.000Z"
              },
              "status": {
                "type": "string",
                "example": "READY"
              },
              "meta": {
                "type": "object",
                "properties": {
                  "origin": {
                    "type": "string",
                    "example": "API"
                  },
                  "userAgent": {
                    "type": "string"
                  }
                }
              },
              "stats": {
                "type": "object",
                "properties": {
                  "inputBodyLen": {
                    "type": "integer",
                    "example": 2000
                  },
                  "rebootCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "restartCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "resurrectCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "computeUnits": {
                    "type": "integer",
                    "example": 0
                  }
                }
              },
              "options": {
                "type": "object",
                "properties": {
                  "build": {
                    "type": "string",
                    "example": "latest"
                  },
                  "timeoutSecs": {
                    "type": "integer",
                    "example": 300
                  },
                  "memoryMbytes": {
                    "type": "integer",
                    "example": 1024
                  },
                  "diskMbytes": {
                    "type": "integer",
                    "example": 2048
                  }
                }
              },
              "buildId": {
                "type": "string"
              },
              "defaultKeyValueStoreId": {
                "type": "string"
              },
              "defaultDatasetId": {
                "type": "string"
              },
              "defaultRequestQueueId": {
                "type": "string"
              },
              "buildNumber": {
                "type": "string",
                "example": "1.0.0"
              },
              "containerUrl": {
                "type": "string"
              },
              "usage": {
                "type": "object",
                "properties": {
                  "ACTOR_COMPUTE_UNITS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_WRITES": {
                    "type": "integer",
                    "example": 1
                  },
                  "KEY_VALUE_STORE_LISTS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_INTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_EXTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_SERPS": {
                    "type": "integer",
                    "example": 0
                  }
                }
              },
              "usageTotalUsd": {
                "type": "number",
                "example": 0.00005
              },
              "usageUsd": {
                "type": "object",
                "properties": {
                  "ACTOR_COMPUTE_UNITS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_WRITES": {
                    "type": "number",
                    "example": 0.00005
                  },
                  "KEY_VALUE_STORE_LISTS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_INTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_EXTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_SERPS": {
                    "type": "integer",
                    "example": 0
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

LLM-Ready Web Scraper – RAG & Vertical Data Extraction OpenAPI definition

OpenAPI is a standard for designing and describing RESTful APIs, allowing developers to define API structure, endpoints, and data formats in a machine-readable way. It simplifies API development, integration, and documentation.

OpenAPI is effective when used with AI agents and GPTs by standardizing how these systems interact with various APIs, for reliable integrations and efficient communication.

By defining machine-readable API specifications, OpenAPI allows AI models like GPTs to understand and use varied data sources, improving accuracy. This accelerates development, reduces errors, and provides context-aware responses, making OpenAPI a core component for AI applications.

You can download the OpenAPI definitions for LLM-Ready Web Scraper – RAG & Vertical Data Extraction from the options below:

OpenAPI.json

If you’d like to learn more about how OpenAPI powers GPTs, read our blog post.

You can also check out our other API clients:

LLM-Ready Web Scraper – RAG & Vertical Data Extraction API in Python

LLM-Ready Web Scraper – RAG & Vertical Data Extraction API in JavaScript

LLM-Ready Web Scraper – RAG & Vertical Data Extraction API through CLI

LLM-Ready Web Scraper – RAG & Vertical Data Extraction API

YouTube Transcript Scraper & Captions Scraper

harvestlab/youtube-scraper

YouTube transcript scraper and captions scraper for RAG datasets, AI training, content research, creator analytics, and video monitoring. Scrape videos, channels, comments, timestamped transcript segments, and AI chapters without a YouTube API key. PPE, x402-ready, Skyfire bundle.

Nick

Meta Ads Library Scraper - Competitor Ad Intel

harvestlab/facebook-ads-library-scraper

Meta Ads Library scraper for competitor ad intelligence: Facebook and Instagram creatives, spend and impression ranges, demographics, active status, and AI creative analysis. Use Graph API input when available, with web fallback diagnostics.

Nick

Google Maps Reviews Scraper - AI Reputation Monitor

harvestlab/google-maps-reviews-scraper

Google Maps reviews scraper for reputation monitoring: place search, review text, ratings, owner replies, AI sentiment/topics, and webhook-ready datasets. Fast-fails with diagnostics when Google blocks a run. PPE at $0.0003/review.

Nick

UK Companies House Lead Scraper - Directors & PSC

harvestlab/companies-house-scraper

UK Companies House scraper for B2B lead enrichment: company search, directors, PSC beneficial owners, SIC codes, filing history, and AI company summaries. Extract clean UK company records for sales, compliance, and market research.

Nick

Reddit Answers API "Ask Reddit" - AI Insights for n8n Pipelines

clearpath/reddit-answers-api

Extract AI-powered answers in 6 languages from Reddit discussions at scale. Structured JSON + markdown for n8n, Make, and LLM pipelines. Includes full post/comment context, quotes with citations, and subreddit metadata. 6 languages supported. No login required. Pay per successful answer.

ClearPath

LLM-Ready Web Scraper

devoted_helix/llm-web-scraper

Convert web pages to clean, LLM-friendly text. Perfect for RAG pipelines, AI chatbot training, and fine-tuning datasets. Removes ads,menus, and clutter automatically.

batuhan senavci

AI Training Data Scraper - LLM and RAG-Ready

george.the.developer/ai-training-data-scraper

Extract web content formatted for LLM fine-tuning and RAG pipelines. Output in OpenAI JSONL, Claude JSONL, Markdown, or raw text.

George Kioko

AI Web Crawler

hounderd/ai-web-crawler

Crawl websites and extract clean, LLM-ready markdown content with stealth browser rendering, anti-bot hardening, smart content filtering, and structured metadata extraction. Built for RAG pipelines, AI agents, and data workflows.

Hounderd

AI Training Data Curator

ryanclinton/ai-training-data-curator

Crawl any website and extract clean, structured text data ready for LLM fine-tuning, RAG pipelines, and AI model training.

Ryan Clinton

Article Extraction API

tugelbay/article-extractor

Convert article URLs to clean Markdown, text, or HTML for RAG and LLM pipelines. Extract title, author, date, images, links, word count, canonical URL, Open Graph, and JSON-LD metadata while removing ads and boilerplate.