🌐 Best Web Scraper API (via CrawlKit)

Scrape any website and get clean markdown, HTML, metadata and links. Powered by CrawlKit - supports stealth mode for anti-bot protection.

Features

• Clean Content: Get markdown, HTML, and raw HTML formats
• Metadata Extraction: Title, description, OpenGraph data, and more
• Link Extraction: Internal and external links
• Stealth Mode: Bypass anti-bot protection (+2 credits)
• Browser Actions: Click, type, scroll, wait for elements
• Custom Selectors: Extract specific content sections

Input

Field	Type	Required	Description
url	string	Yes	URL to scrape
options.timeout	number	No	Request timeout in ms (default: 30000, max: 600000)
options.stealth	boolean	No	Enable stealth mode for anti-bot sites
options.waitFor	string/number	No	CSS selector or milliseconds to wait
options.contentSelector	string	No	CSS selector for specific content
options.headers	object	No	Additional HTTP headers

Input Example

{
  "url": "https://news.ycombinator.com",
  "options": {
    "timeout": 60000,
    "stealth": false
  }
}

Output

Returns scraped content in multiple formats:

• markdown: Clean markdown content
• html: Cleaned HTML (main content only)
• rawHtml: Original raw HTML
• metadata: Page title, description, OG data
• links: Internal and external links
• stats: Processing time statistics

Output Example

{
  "url": "https://news.ycombinator.com",
  "finalUrl": "https://news.ycombinator.com/",
  "markdown": "# Hacker News\n\n1. Article Title...",
  "html": "<h1>Hacker News</h1>...",
  "metadata": {
    "title": "Hacker News",
    "description": "Links for the intellectually curious"
  },
  "links": {
    "internal": ["https://news.ycombinator.com/newest"],
    "external": ["https://example.com/article"]
  },
  "stats": {
    "fetchTime": 1234,
    "totalTime": 2345
  },
  "creditsUsed": 1,
  "creditsRemaining": 99999
}

Pricing

• Base Cost: 1 credit per request
• Stealth Mode: +2 credits per request
• Rate Limit: 100 requests per minute

Use Cases

• Content Aggregation: Collect articles and news
• SEO Analysis: Extract metadata and links
• Price Monitoring: Track product information
• Research: Gather data from multiple sources

Links

• CrawlKit Documentation
• API Reference