Pricing

Pay per usage

TheCrawler — Scrape Everything from Any Page

Scrape any webpage with JS rendering (Playwright) or fast HTTP (Cheerio). Extract text, links, images, meta, headings, tables, JSON-LD, emails, phones, OG/Twitter cards, social links. LLM-ready markdown with heading-aware chunking. CSS selectors, recursive crawling, URL filtering. $0.003/page.

Pricing

Pay per usage

Rating

0.0

(0)

Developer

Manchitt Sanan

Actor stats

Bookmarked

Total users

Monthly active users

2 hours ago

Last modified

Categories

Developer tools

You can access the TheCrawler — Scrape Everything from Any Page programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in Integrations settings in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

{
  "openapi": "3.0.1",
  "info": {
    "version": "0.1",
    "x-build-id": "2bhTgRkcX6PCgzLsH"
  },
  "servers": [
    {
      "url": "https://api.apify.com/v2"
    }
  ],
  "paths": {
    "/acts/accurate_pouch~the-crawler/run-sync-get-dataset-items": {
      "post": {
        "operationId": "run-sync-get-dataset-items-accurate_pouch-the-crawler",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK"
          }
        }
      }
    },
    "/acts/accurate_pouch~the-crawler/runs": {
      "post": {
        "operationId": "runs-sync-accurate_pouch-the-crawler",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor and returns information about the initiated run in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK",
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/runsResponseSchema"
                }
              }
            }
          }
        }
      }
    },
    "/acts/accurate_pouch~the-crawler/run-sync": {
      "post": {
        "operationId": "run-sync-accurate_pouch-the-crawler",
        "x-openai-isConsequential": false,
        "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
        "tags": [
          "Run Actor"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/inputSchema"
              }
            }
          }
        },
        "parameters": [
          {
            "name": "token",
            "in": "query",
            "required": true,
            "schema": {
              "type": "string"
            },
            "description": "Enter your Apify token here"
          }
        ],
        "responses": {
          "200": {
            "description": "OK"
          }
        }
      }
    }
  },
  "components": {
    "schemas": {
      "inputSchema": {
        "type": "object",
        "required": [
          "urls"
        ],
        "properties": {
          "urls": {
            "title": "URLs to Scrape",
            "type": "array",
            "description": "List of URLs to scrape.",
            "items": {
              "type": "string"
            }
          },
          "extractText": {
            "title": "Extract Text",
            "type": "boolean",
            "description": "Extract visible text content (scripts/styles removed).",
            "default": true
          },
          "extractLinks": {
            "title": "Extract Links",
            "type": "boolean",
            "description": "Extract all links with href, anchor text, and internal/external flag.",
            "default": true
          },
          "extractImages": {
            "title": "Extract Images",
            "type": "boolean",
            "description": "Extract all images with src, alt text, dimensions.",
            "default": true
          },
          "extractMeta": {
            "title": "Extract Meta Tags",
            "type": "boolean",
            "description": "Extract all meta tags (description, og:title, keywords, etc).",
            "default": true
          },
          "extractHeadings": {
            "title": "Extract Headings",
            "type": "boolean",
            "description": "Extract all h1-h6 headings with their level.",
            "default": true
          },
          "extractTables": {
            "title": "Extract Tables",
            "type": "boolean",
            "description": "Extract HTML tables as structured data (headers + rows).",
            "default": true
          },
          "extractStructuredData": {
            "title": "Extract JSON-LD",
            "type": "boolean",
            "description": "Extract JSON-LD structured data (schema.org).",
            "default": true
          },
          "extractEmails": {
            "title": "Extract Emails",
            "type": "boolean",
            "description": "Find email addresses in the page HTML.",
            "default": true
          },
          "extractPhones": {
            "title": "Extract Phone Numbers",
            "type": "boolean",
            "description": "Find phone numbers in the page HTML.",
            "default": true
          },
          "cssSelector": {
            "title": "CSS Selector",
            "type": "string",
            "description": "Target specific content. Returns matching text in selectedContent field. Example: .product-price, #main-content, article"
          },
          "maxDepth": {
            "title": "Crawl Depth",
            "minimum": 0,
            "maximum": 5,
            "type": "integer",
            "description": "0 = scrape listed URLs only. 1+ = follow internal links to that depth.",
            "default": 0
          },
          "maxPages": {
            "title": "Max Pages",
            "minimum": 1,
            "maximum": 10000,
            "type": "integer",
            "description": "Maximum total pages to scrape (including followed links).",
            "default": 100
          },
          "usePlaywright": {
            "title": "Enable JS Rendering (Playwright)",
            "type": "boolean",
            "description": "Render JavaScript with a real browser. Required for SPAs (React, Vue, Angular), dynamic content, and sites that load data via JS. Uses more compute than default Cheerio mode.",
            "default": false
          },
          "waitForSelector": {
            "title": "Wait for Selector",
            "type": "string",
            "description": "CSS selector to wait for before extracting. Only works with Playwright enabled. Example: .product-list, #content-loaded"
          },
          "waitForMs": {
            "title": "Wait (ms)",
            "minimum": 0,
            "maximum": 30000,
            "type": "integer",
            "description": "Milliseconds to wait after page load before extracting. Only works with Playwright enabled. Useful for lazy-loading content.",
            "default": 0
          },
          "extractMarkdown": {
            "title": "Extract Markdown",
            "type": "boolean",
            "description": "Convert page content to clean Markdown. Strips navigation, footers, and boilerplate when stripBoilerplate is enabled. Ideal for LLM/RAG pipelines.",
            "default": false
          },
          "stripBoilerplate": {
            "title": "Strip Boilerplate",
            "type": "boolean",
            "description": "Remove nav, header, footer, sidebar, and ads before markdown conversion. Keeps only main content.",
            "default": true
          },
          "chunkSize": {
            "title": "Chunk Size (chars)",
            "minimum": 0,
            "maximum": 50000,
            "type": "integer",
            "description": "Split markdown into chunks of this size. Each chunk includes section heading, index, and content hash. Set to 0 to disable chunking.",
            "default": 0
          },
          "chunkOverlap": {
            "title": "Chunk Overlap (chars)",
            "minimum": 0,
            "maximum": 5000,
            "type": "integer",
            "description": "Character overlap between consecutive chunks. Prevents losing context at chunk boundaries. Typical: 10-20% of chunk size.",
            "default": 200
          },
          "includeGlobs": {
            "title": "Include URL Patterns",
            "type": "array",
            "description": "Only follow links matching these glob patterns when crawling. Example: https://example.com/blog/**",
            "items": {
              "type": "string"
            }
          },
          "excludeGlobs": {
            "title": "Exclude URL Patterns",
            "type": "array",
            "description": "Skip links matching these glob patterns. Example: **/login/**, **/admin/**",
            "items": {
              "type": "string"
            }
          },
          "proxyUrl": {
            "title": "Proxy URL",
            "type": "string",
            "description": "HTTP/SOCKS proxy URL for requests. Format: http://user:pass@host:port or socks5://user:pass@host:port. Use Apify Proxy or your own."
          },
          "searchQuery": {
            "title": "Search Query",
            "type": "string",
            "description": "Search Google and scrape the top results. Leave URLs empty to use search mode. Example: 'best project management tools 2026'"
          },
          "searchLimit": {
            "title": "Search Results Limit",
            "minimum": 1,
            "maximum": 50,
            "type": "integer",
            "description": "Maximum number of search results to scrape (default: 10).",
            "default": 10
          },
          "actions": {
            "title": "Browser Actions (Playwright only)",
            "type": "array",
            "description": "Actions to perform before extraction. Requires Playwright enabled. JSON array. Types: click (selector), fill (selector + value), scroll (pixels), wait (ms), screenshot. Example: [{\"type\":\"click\",\"selector\":\".load-more\"},{\"type\":\"scroll\",\"pixels\":2000}]"
          },
          "dryRun": {
            "title": "Dry Run",
            "type": "boolean",
            "description": "Scrape pages but don't charge.",
            "default": false
          }
        }
      },
      "runsResponseSchema": {
        "type": "object",
        "properties": {
          "data": {
            "type": "object",
            "properties": {
              "id": {
                "type": "string"
              },
              "actId": {
                "type": "string"
              },
              "userId": {
                "type": "string"
              },
              "startedAt": {
                "type": "string",
                "format": "date-time",
                "example": "2025-01-08T00:00:00.000Z"
              },
              "finishedAt": {
                "type": "string",
                "format": "date-time",
                "example": "2025-01-08T00:00:00.000Z"
              },
              "status": {
                "type": "string",
                "example": "READY"
              },
              "meta": {
                "type": "object",
                "properties": {
                  "origin": {
                    "type": "string",
                    "example": "API"
                  },
                  "userAgent": {
                    "type": "string"
                  }
                }
              },
              "stats": {
                "type": "object",
                "properties": {
                  "inputBodyLen": {
                    "type": "integer",
                    "example": 2000
                  },
                  "rebootCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "restartCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "resurrectCount": {
                    "type": "integer",
                    "example": 0
                  },
                  "computeUnits": {
                    "type": "integer",
                    "example": 0
                  }
                }
              },
              "options": {
                "type": "object",
                "properties": {
                  "build": {
                    "type": "string",
                    "example": "latest"
                  },
                  "timeoutSecs": {
                    "type": "integer",
                    "example": 300
                  },
                  "memoryMbytes": {
                    "type": "integer",
                    "example": 1024
                  },
                  "diskMbytes": {
                    "type": "integer",
                    "example": 2048
                  }
                }
              },
              "buildId": {
                "type": "string"
              },
              "defaultKeyValueStoreId": {
                "type": "string"
              },
              "defaultDatasetId": {
                "type": "string"
              },
              "defaultRequestQueueId": {
                "type": "string"
              },
              "buildNumber": {
                "type": "string",
                "example": "1.0.0"
              },
              "containerUrl": {
                "type": "string"
              },
              "usage": {
                "type": "object",
                "properties": {
                  "ACTOR_COMPUTE_UNITS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_WRITES": {
                    "type": "integer",
                    "example": 1
                  },
                  "KEY_VALUE_STORE_LISTS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_INTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_EXTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_SERPS": {
                    "type": "integer",
                    "example": 0
                  }
                }
              },
              "usageTotalUsd": {
                "type": "number",
                "example": 0.00005
              },
              "usageUsd": {
                "type": "object",
                "properties": {
                  "ACTOR_COMPUTE_UNITS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATASET_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "KEY_VALUE_STORE_WRITES": {
                    "type": "number",
                    "example": 0.00005
                  },
                  "KEY_VALUE_STORE_LISTS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_READS": {
                    "type": "integer",
                    "example": 0
                  },
                  "REQUEST_QUEUE_WRITES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_INTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "DATA_TRANSFER_EXTERNAL_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                    "type": "integer",
                    "example": 0
                  },
                  "PROXY_SERPS": {
                    "type": "integer",
                    "example": 0
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

TheCrawler — Scrape Everything from Any Page OpenAPI definition

OpenAPI is a standard for designing and describing RESTful APIs, allowing developers to define API structure, endpoints, and data formats in a machine-readable way. It simplifies API development, integration, and documentation.

OpenAPI is effective when used with AI agents and GPTs by standardizing how these systems interact with various APIs, for reliable integrations and efficient communication.

By defining machine-readable API specifications, OpenAPI allows AI models like GPTs to understand and use varied data sources, improving accuracy. This accelerates development, reduces errors, and provides context-aware responses, making OpenAPI a core component for AI applications.

You can download the OpenAPI definitions for TheCrawler — Scrape Everything from Any Page from the options below:

OpenAPI.json

If you’d like to learn more about how OpenAPI powers GPTs, read our blog post.

You can also check out our other API clients:

TheCrawler — Scrape Everything from Any Page API in Python

TheCrawler — Scrape Everything from Any Page API in JavaScript

TheCrawler — Scrape Everything from Any Page API through CLI

TheCrawler — Scrape Everything from Any Page API

Salesforce AppExchange Scraper

jungle_synthesizer/salesforce-appexchange-scraper

Scrape app listings from Salesforce AppExchange including developer info, ratings, categories, and supported products

BowTiedRaccoon

5.0

(1)

Linkedin Profile Scraper

crawlerbros/linkedin-profile-scraper

Scrape LinkedIn public profiles without login. Get name, headline, experience, education, followers, and recent articles.

Crawler Bros

213

5.0

(11)

Google Maps Lead Extractor with Email Finder

electrifying_haircut/google-map-scrapper-neo

Extract outreach-ready leads from Google Maps — with emails. Get business names, phone numbers, websites, ratings, reviews, GPS coordinates AND business emails scraped directly from their websites. Skip the scrape-then-enrich pipeline. One actor, complete leads.

Gagan

175

Web Crawler & Semantic Schema-Enhanced Extractor

devil_port369-owner/web-crawler

Depth-controlled web crawler that transforms websites into structured analytics-ready data. Starting from one or more URLs, it crawls internal links up to a configurable depth and outputs detailed JSON records per page

DataFusionX

5.0

(7)

Zillow Property Scraper

crawlerbros/zillow-scraper

Scrape property listings from Zillow. Get prices, descriptions, photos, agent details, schools, tax history, price history, and 30+ fields per property.

Crawler Bros

5.0

(15)

Dubizzle Product Search Scraper

crawlerbros/dubizzle-scraper

Scrape product listings from Dubizzle UAE for motors, real estate, jobs, electronics, and more. Extract prices, photos, seller info, location, and 70+ fields per listing.

Crawler Bros

5.0

(14)

ZonaProp Scraper

crawlerbros/zonaprop-scraper

Scrape real estate listings from ZonaProp.com.ar, Argentina's leading property portal. Extract prices, locations, features, images, and publisher info.

Crawler Bros

5.0

(15)

Cromur Profile Scraper

getdataforme/cromur-profile-scraper

Extract detailed product data from Cromur marketplace with this robust Apify Actor. Scrapes names, descriptions, prices, images, and availability for e-commerce analysis, market research, and competitive intelligence....

GetDataForMe

My Actor

dynamic_hospital/my-actor

Connect apify with everything you build.

Presley

Email ✉️ & Phone ☎️ Extractor

anchor/email-phone-extractor

Extract emails, phone numbers, and other contact information like Twitter, LinkedIn, Instagram... from websites you provide. Best for lead generation and data enrichment. Export data in structured formats and dominate your outreach game. Capture your leads almost for free, fast, and without limits.