Firecrawl Agent

"AI That Navigates Websites Like a Human, Extracts Data Like a Pro" by John Rippy | johnrippy.link

---

Stop Writing Custom Scrapers for Every Website

You're currently doing: Building different scrapers for each website, handling pagination manually, dealing with JavaScript-rendered content, writing complex CSS selectors.

What if you could just describe what you want?

Firecrawl Agent uses Firecrawl's autonomous Agent API to navigate websites intelligently and extract exactly what you need:

• Natural Language Goals - Describe extraction in plain English
• Autonomous Navigation - AI decides which links to click
• Smart Pagination - Automatically handles "Load More" and infinite scroll
• JavaScript Rendering - Full browser rendering for SPAs
• Backtracking - Recovers from dead ends automatically

Powered by Firecrawl's Agent API - the same technology behind production web scraping systems.

---

How It Works

URL + Natural Language Goal
          ↓
    ┌─────────────┐
    │  GOAL       │ Parse user objective
    │  ANALYSIS   │ Identify data requirements
    └─────────────┘
          ↓
    ┌─────────────┐
    │  STRATEGIC  │ Find likely paths to data
    │  NAVIGATION │ Click links, fill forms
    └─────────────┘
          ↓
    ┌─────────────┐
    │  CONTENT    │ Render JavaScript
    │  EXTRACTION │ Scrape clean content
    └─────────────┘
          ↓
    ┌─────────────┐
    │  DATA       │ Apply optional schema
    │  STRUCTURING│ Format as JSON
    └─────────────┘
          ↓
    ┌─────────────┐
    │  VALIDATION │ Verify data meets goal
    │             │ Backtrack if needed
    └─────────────┘
          ↓
       Clean JSON Output

---

Use Cases

1. Competitive Pricing Research

{
  "url": "https://competitor.com",
  "objective": "Find all pricing plans, their monthly and annual costs, and included features"
}

Returns:

{
  "plans": [
    {
      "name": "Starter",
      "monthlyPrice": 29,
      "annualPrice": 290,
      "features": ["5 users", "10GB storage", "Email support"]
    },
    {
      "name": "Pro",
      "monthlyPrice": 99,
      "annualPrice": 990,
      "features": ["Unlimited users", "100GB storage", "Priority support"]
    }
  ]
}

2. Job Listings Extraction

{
  "url": "https://company.com/careers",
  "objective": "Extract all software engineering job openings with title, location, and requirements"
}

3. Product Catalog Scraping

{
  "url": "https://store.com/products",
  "objective": "Get all products with name, price, description, and availability",
  "schema": {
    "type": "array",
    "items": {
      "type": "object",
      "properties": {
        "name": { "type": "string" },
        "price": { "type": "number" },
        "inStock": { "type": "boolean" }
      }
    }
  }
}

4. Company Research

{
  "url": "https://company.com",
  "objective": "Find the leadership team, their titles, and LinkedIn profiles"
}

5. Event/Conference Extraction

{
  "url": "https://conference.com",
  "objective": "Extract the conference schedule with session titles, speakers, times, and room locations"
}

6. Documentation Scraping

{
  "url": "https://docs.example.com",
  "objective": "Extract all API endpoints with their methods, parameters, and descriptions"
}

---

Input Parameters

Parameter	Type	Required	Default	Description
url	string	Yes*	-	Starting URL for extraction
objective	string	Yes*	-	Natural language description of what to extract
schema	object	No	-	JSON schema for structured output
maxSteps	integer	No	10	Maximum navigation actions
allowBacktracking	boolean	No	true	Allow agent to try alternative paths
firecrawlApiKey	string	Yes*	-	Your Firecrawl API key
demoMode	boolean	No	false	Test with sample data

*Not required in demo mode

---

Output Format

{
  "success": true,
  "url": "https://example.com",
  "objective": "Find pricing plans",
  "data": {
    "plans": [
      {
        "name": "Starter",
        "price": 29,
        "features": ["5 users", "10GB storage"]
      },
      {
        "name": "Pro",
        "price": 99,
        "features": ["Unlimited users", "100GB storage", "Priority support"]
      }
    ]
  },
  "stepsUsed": 4,
  "pagesVisited": [
    "https://example.com",
    "https://example.com/pricing"
  ],
  "extractedAt": "2024-12-23T10:30:00Z"
}

---

Pricing

BYOK (Bring Your Own Key)

Provider	Cost	Purpose
Firecrawl	~$0.01-0.05/extraction	Agent API usage
Apify	~$0.10-0.25/run	Compute

Cost Per Extraction

Complexity	Estimated Cost
Single page	~$0.01-0.02
2-5 pages	~$0.02-0.03
Complex multi-page	~$0.03-0.05

Cost Comparison

Task	Firecrawl Agent	Custom Scraper Development
Extract pricing from 1 site	~$0.03	2-4 hours dev time
Scrape 100 product listings	~$0.50	1-2 days dev time
Monthly extraction (100 sites)	~$5	Maintenance + hosting

No development time. No maintenance. Pay per use.

---

API Integration

Using the Apify API

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });

const run = await client.actor('localhowl/firecrawl-agent').call({
  url: 'https://competitor.com/pricing',
  objective: 'Extract all pricing plans with features and costs',
  firecrawlApiKey: 'fc-xxx'
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items[0].data);

Using cURL

curl -X POST "https://api.apify.com/v2/acts/localhowl~firecrawl-agent/runs?token=YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/pricing",
    "objective": "Find all pricing plans with their features",
    "firecrawlApiKey": "fc-xxx"
  }'

---

Demo Mode

Test the actor without an API key:

{
  "demoMode": true,
  "url": "https://example-saas.com",
  "objective": "Find the pricing information"
}

Returns sample pricing data extraction.

---

Firecrawl Agent vs Fire-Enrich

Feature	Firecrawl Agent	Fire-Enrich
Purpose	General web extraction	Lead/company enrichment
Input	URL + natural language goal	Company name/website
Agents	1 autonomous agent	5 specialized agents
Output	Custom based on goal	Structured company data
Best For	Flexible extraction tasks	B2B lead enrichment

Use Firecrawl Agent when you need to extract arbitrary data from any website.

Use Fire-Enrich when you specifically need to enrich company/lead information.

---

Perfect For

Sales Teams

• Extract competitor pricing
• Gather prospect information
• Build target account lists

Product Managers

• Competitive analysis
• Feature comparison research
• Market research

Developers

• API documentation scraping
• Data migration projects
• Integration research

Researchers

• Content aggregation
• Data collection for analysis
• Automated research tasks

---

Advanced Features

Schema Enforcement

Force output to match a specific structure:

{
  "url": "https://store.com/products",
  "objective": "Get all products",
  "schema": {
    "type": "array",
    "items": {
      "type": "object",
      "properties": {
        "name": { "type": "string" },
        "price": { "type": "number" },
        "sku": { "type": "string" },
        "inStock": { "type": "boolean" }
      },
      "required": ["name", "price"]
    }
  }
}

Backtracking

When allowBacktracking: true (default), the agent will:

1. Try the most likely path to find data
2. If unsuccessful, backtrack and try alternative paths
3. Continue until data is found or maxSteps is reached

---

Limitations

• Authentication: Cannot access login-protected content
• Rate Limits: Respects Firecrawl's API rate limits
• Captchas: Cannot bypass CAPTCHA challenges
• Heavy JavaScript: Very complex SPAs may require more steps
• Dynamic Content: Real-time updating content may vary between runs

---

Support

For issues or feature requests, contact support@localhowl.com

Get your Firecrawl API key at firecrawl.dev

---

Built by John Rippy | johnrippy.link

---

Keywords

firecrawl, web scraping agent, ai web scraper, autonomous web extraction, natural language scraper, javascript scraper, spa scraper, web data extraction, pricing scraper, product catalog scraper, competitor research, no-code scraper, firecrawl api, intelligent web crawler