Website Content Scraping — MCP Server

Extract structured data from any website via Firecrawl's API. This MCP server provides 20 tools for scraping, crawling, search, and data extraction — built for AI agents that need reliable web data without anti-bot complexity.

Features

• 20 tools covering scrape, crawl, search, extract, and utilities
• Batch scraping — process multiple URLs in a single call
• Deep crawl — configurable depth and scope for full-site extraction
• LLM-optimized output — generate llms.txt for AI context
• Structured data extraction — use CSS selectors to pull specific content
• Site mapping — generate complete URL maps for any website
• Deep research — start research jobs that synthesize information across pages
• No anti-bot complexity — API-based approach sidesteps proxy rotation, browser fingerprinting, and CAPTCHAs
• Per-tool pricing — pay only for what you use via Apify PAY_PER_EVENT
• MCP protocol ready — works as standalone batch processor or MCP server

Quick Start

# Push to Apify
apify push firecrawl-mcp

# Run via API with batch input
curl -X POST https://api.apify.com/v2/acts/your-actor-id/runs \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -d '{"tool": "scrape_and_extract_from_url", "params": {"url": "https://example.com"}}'

Input

Parameter	Type	Description
tool	string	Tool name (e.g., scrape_and_extract_from_url)
params	object	Tool-specific parameters
FIRECRAWL_BEARER_AUTH	string	Firecrawl API bearer token (set in Apify secrets)

Standalone Batch Input

{
  "tool": "scrape_and_extract_from_url",
  "params": {
    "url": "https://example.com"
  }
}

MCP Protocol (Standby Mode)

# Initialize
curl -X POST http://localhost:3000/mcp \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"test","version":"1.0"}}}'

# List tools
curl -X POST http://localhost:3000/mcp \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}'

# Call scrape tool
curl -X POST http://localhost:3000/mcp \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"scrape_and_extract_from_url","arguments":{"url":"https://example.com"}}}'

Tools

Scraping

Tool	Description	Price
scrape_and_extract_from_url	Scrape a single URL and return structured data	$0.10
batch_scrape_and_extract_from_urls	Batch scrape multiple URLs	$0.10

Crawling

Tool	Description	Price
crawl_urls	Crawl a website with configurable depth and scope	$0.15
crawl_get_status	Get crawl job status	free
crawl_cancel	Cancel an active crawl job	free
crawl_errors_get_crawl	Get crawl errors	free
crawl_get_active	List all active crawls	free

Search

Tool	Description	Price
search	Search for URLs matching a query	$0.08
firecrawl_search_search_and_scrape	Combined search and scrape	$0.08

Extraction

Tool	Description	Price
extract_data	Extract structured data using CSS selectors	$0.10
extract_get_status	Get extraction job status	free

Deep Research

Tool	Description	Price
deep-research_start	Start a deep research job	$0.12
deep-research_get_status	Get research job status	free

Maps

Tool	Description	Price
map_urls	Generate a URL map for a website	$0.05

LLM TXT

Tool	Description	Price
llmstxt_generate_llms_txt	Generate llms.txt for AI context	free
llmstxt_get_llms_txt_status	Check llms.txt generation status	free

Team Utilities

Tool	Description	Price
team_get_credit_usage	Get team credit usage	free
team_get_token_usage	Get team token usage	free
context	Get API domain context	$0.01
sync	Sync operation	$0.02
export	Export data	$0.03
import	Import data	$0.03
sql	SQL query	$0.05
workflow_status	Get workflow status	$0.02
workflow_archive	Archive workflow	$0.03

Output

{
  "success": true,
  "data": {
    "content": "Extracted content...",
    "metadata": {
      "title": "Page Title",
      "description": "Page description",
      "url": "https://example.com"
    }
  }
}

Pricing

PPE (Pay Per Event) pricing via Apify. Each tool call costs a fixed amount:

Tool	Price (USD)
scrape_and_extract_from_url	$0.10
batch_scrape_and_extract_from_urls	$0.10
crawl_urls	$0.15
map_urls	$0.05
search	$0.08
extract_data	$0.10
deep-research_start	$0.12
firecrawl_search_search_and_scrape	$0.08
context	$0.01
sync	$0.02
export	$0.03
import	$0.03
sql	$0.05
workflow_status	$0.02
workflow_archive	$0.03

Set FIRECRAWL_BEARER_AUTH in Apify secrets to enable billing.

Troubleshooting

"Authentication failed" error Verify FIRECRAWL_BEARER_AUTH is set in Apify secrets. The actor uses bearer token authentication — no API key format required.

Crawl job stuck Use crawl_cancel to stop the job, then restart with a smaller depth setting. Deep crawls on large sites can take hours.

No data returned Check the URL is publicly accessible. Some sites block bots even through API calls. Try map_urls first to verify the site is reachable.

Batch scrape incomplete The batch tool processes URLs sequentially. If one URL fails, the others continue. Check the response for partial results.

Rate limiting Firecrawl's API has rate limits. Space out batch requests or upgrade your Firecrawl plan for higher throughput.

Architecture

Apify Actor (handleRequest)
    |
    v
MCPProxy (Node.js child_process)
    |
    v
firecrawl-pp-mcp binary (stdio)
    |
    v
Firecrawl API

• Actor spawns firecrawl-pp-mcp as a subprocess with stdio transport
• JSON-RPC requests are proxied through stdin/stdout
• PPE charges applied via Actor.charge() before tool calls
• Standby HTTP server handles MCP protocol over HTTP

Authentication

Firecrawl uses bearer token authentication. Set FIRECRAWL_BEARER_AUTH in Apify secrets:

$apify secrets add FIRECRAWL_BEARER_AUTH "your_token_here"

GitHub Topics

firecrawl web-scraping ai-agents no-api-key-required mcp apify

Related

• Firecrawl Documentation
• Apify MCP Documentation
• Printing Press Library

License

MIT