Firecrawl Site Mapper

Discover all URLs on a website using Firecrawl's Map endpoint. Perfect for competitor analysis, site audits, and content gap discovery. Built by John Rippy (https://www.linkedin.com/in/johnrippy/).

---

What is Firecrawl?

Firecrawl is a web scraping API that handles JavaScript-heavy websites and anti-bot protection. This actor uses Firecrawl's Map endpoint to quickly discover all URLs on a website without crawling each page.

Why use this instead of a regular crawler?

• Much faster than crawling - discovers URLs from sitemaps and link analysis
• Handles JavaScript-rendered navigation
• Works on sites with anti-bot protection

---

How to Get Your Firecrawl API Key

1. Go to firecrawl.dev (or firecrawl.link/john-rippy for 10% off)
2. Sign up for an account (free tier: 500 credits/month)
3. Go to your dashboard → API Keys
4. Copy your API key
5. Paste it in the firecrawlApiKey field below

---

Features

• Fast URL Discovery - Quickly map entire websites using Firecrawl's Map API
• Search-Based Filtering - Find specific pages by topic or keyword (e.g., "pricing", "api", "blog")
• Sitemap Integration - Include URLs from XML sitemaps
• Geo-Targeting - Crawl from specific countries for localized content
• Multi-Language Support - Specify preferred languages for discovery
• Demo Mode - Test without an API key using sample data

Quick Start

Try it first (Free - Demo Mode)

{
  "demoMode": true
}

This returns sample URL data so you can see the output format without charges or API key.

Map All URLs on a Website

{
  "firecrawlApiKey": "fc-XXXXXXXXXXXXXXXX",
  "url": "https://example.com",
  "limit": 200,
  "demoMode": false
}

Find Specific Pages (e.g., Pricing)

{
  "firecrawlApiKey": "fc-XXXXXXXXXXXXXXXX",
  "url": "https://competitor.com",
  "search": "pricing",
  "limit": 50,
  "demoMode": false
}

Include Sitemap URLs

{
  "firecrawlApiKey": "fc-XXXXXXXXXXXXXXXX",
  "url": "https://example.com",
  "includeSitemap": true,
  "limit": 500,
  "demoMode": false
}

Demo Mode

Set demoMode: true to test with sample data (no charges). When you're ready for real results, set demoMode: false or omit it.

{
  "demoMode": true,
  ...
}

Input Parameters

Field	Type	Description	Default
url	string	Website URL to map (e.g., https://example.com)	Required
search	string	Filter results by keyword/topic	-
limit	number	Maximum URLs to discover	100
includeSitemap	boolean	Include URLs from sitemap	true
country	string	Country code for geo-targeting (US, GB, etc.)	-
languages	array	Preferred languages	-
firecrawlApiKey	string	Your Firecrawl API key	-
webhookUrl	string	URL for completion notification	-
demoMode	boolean	Run with sample data	false

Get Your Firecrawl API Key

Get 10% off at firecrawl.link/john-rippy - Sign up for free tier (500 credits/month) or paid plans.

Output Format

{
  "success": true,
  "url": "https://example.com",
  "search": "pricing",
  "totalUrls": 5,
  "urls": [
    {
      "url": "https://example.com/pricing",
      "title": "Pricing Plans",
      "description": "View our flexible pricing options"
    }
  ],
  "categories": {
    "pricing": 2,
    "products": 3
  },
  "mappedAt": "2024-01-15T10:30:00Z"
}

Common Problems & Solutions

"Invalid API key" error

Cause: Your Firecrawl API key is wrong or expired. Fix: Get your API key from firecrawl.dev dashboard and copy it exactly.

"Rate limit exceeded" error

Cause: You've hit Firecrawl's rate limits or used all your credits. Fix: Check your usage at firecrawl.dev dashboard. Reduce limit to discover fewer URLs.

Fewer URLs than expected

Cause: Some pages may be JavaScript-only navigation or blocked. Fix:

• Enable includeSitemap to include sitemap URLs
• Try a higher limit
• Some dynamic sites may have fewer discoverable URLs

Demo data showing instead of real results

Cause: demoMode is still set to true. Fix: Set demoMode: false and provide your firecrawlApiKey.

Firecrawl Actors Comparison

Actor	Best For	Use When...
Firecrawl Site Mapper (this one)	URL discovery	You need a list of all URLs on a site
Firecrawl Website Crawler	Full site crawling	You need page content, not just URLs
Firecrawl Scrape	Single pages	You have specific URLs to scrape
Firecrawl Agent	Complex extractions	You need AI to navigate and extract

Pricing

This actor uses pay-per-event billing:

Event	Description	Price
Site Mapped	Charged when a site is mapped	$0.01
URLs Discovered (per 10)	Charged per 10 URLs discovered	$0.001

Use Cases

• Competitor Analysis - Find all pricing, features, and blog pages on competitor sites
• Site Audits - Discover all pages on a website for SEO audits
• Content Gap Analysis - Map competitor content to identify opportunities
• Migration Planning - Get a complete URL inventory before site redesigns
• Link Building - Find relevant pages for outreach campaigns

---

Built by John Rippy | Actor Arsenal