Firecrawl Site Mapper - Fast URL Discovery with Search & Sitemap Integration

Discover all URLs on a website using Firecrawl's Map endpoint. Much faster than traditional crawling - discovers URLs from sitemaps and link analysis without loading every page. Filter by keyword to find specific pages (pricing, blog, API docs). Perfect for competitor analysis, site audits, content gap discovery, and migration planning.

Features

• Fast URL Discovery - Map entire websites in seconds using Firecrawl's Map API
• Search-Based Filtering - Find specific pages by keyword (e.g., "pricing", "api", "blog")
• Sitemap Integration - Include URLs from XML sitemaps automatically
• Page Metadata - Returns title and description for each discovered URL
• Category Analysis - Auto-categorizes URLs by content type
• Geo-Targeting - Discover localized content from specific countries
• Multi-Language Support - Specify preferred languages for discovery
• Webhook Support - Async delivery for automation pipelines
• Demo Mode - Test with sample data before going live

Who Should Use This Actor?

SEO Agencies

Map client websites to understand site structure before audits. Find all pages that need optimization, identify orphaned content, and build comprehensive URL inventories.

Competitive Intelligence Teams

Discover all competitor pages to understand their content strategy. Find pricing pages, feature lists, and blog content without manual exploration.

Web Developers

Get complete URL inventories before site redesigns or migrations. Map all existing pages to plan redirects and content migration.

Content Strategists

Identify content gaps by mapping competitor sites. Find topics they cover that you don't, and discover content organization patterns.

Link Building Teams

Find relevant pages for outreach campaigns. Map competitor blogs, resource pages, and partnership directories.

Site Migration Teams

Create comprehensive URL lists before platform migrations. Ensure no pages are lost during redesigns.

Quick Start

Demo Mode (Free Test)

{
  "demoMode": true
}

Map All URLs on a Website

{
  "firecrawlApiKey": "fc-XXXXXXXXXXXXXXXX",
  "url": "https://example.com",
  "limit": 200,
  "demoMode": false
}

Find Specific Pages (Pricing)

{
  "firecrawlApiKey": "fc-XXXXXXXXXXXXXXXX",
  "url": "https://competitor.com",
  "search": "pricing",
  "limit": 50,
  "demoMode": false
}

Include Sitemap URLs

{
  "firecrawlApiKey": "fc-XXXXXXXXXXXXXXXX",
  "url": "https://example.com",
  "includeSitemap": true,
  "limit": 500,
  "demoMode": false
}

Geo-Targeted Discovery

{
  "firecrawlApiKey": "fc-XXXXXXXXXXXXXXXX",
  "url": "https://global-site.com",
  "country": "GB",
  "languages": ["en"],
  "limit": 300,
  "demoMode": false
}

Input Parameters

Parameter	Type	Default	Description
url	string	-	Website URL to map (required unless demoMode)
search	string	-	Filter results by keyword/topic
limit	number	100	Maximum URLs to discover
includeSitemap	boolean	true	Include URLs from XML sitemap
country	string	-	Country code for geo-targeting (US, GB, DE, etc.)
languages	array	-	Preferred languages for discovery
firecrawlApiKey	string	-	Your Firecrawl API key (BYOK)
demoMode	boolean	true	Return sample data for testing
webhookUrl	string	-	Webhook URL for results delivery

Get Your Firecrawl API Key

1. Go to firecrawl.dev (or firecrawl.link/john-rippy for 10% off)
2. Sign up for free tier (500 credits/month) or paid plans
3. Go to Dashboard → API Keys
4. Copy your API key

Output Format

{
  "success": true,
  "url": "https://example.com",
  "search": "pricing",
  "totalUrls": 156,
  "urls": [
    {
      "url": "https://example.com/pricing",
      "title": "Pricing Plans",
      "description": "View our flexible pricing options for teams of all sizes"
    },
    {
      "url": "https://example.com/pricing/enterprise",
      "title": "Enterprise Pricing",
      "description": "Custom solutions for large organizations"
    }
  ],
  "categories": {
    "pricing": 3,
    "products": 45,
    "blog": 89,
    "docs": 19
  },
  "mappedAt": "2026-01-28T10:30:00.000Z"
}

Pricing (Pay-Per-Event)

Event	Description	Price
site_mapped	Per site mapping operation	$0.01
urls_discovered	Per 10 URLs discovered	$0.001

Example costs:

• Map 100 URLs: $0.01 + (10 x $0.001) = $0.02
• Map 500 URLs: $0.01 + (50 x $0.001) = $0.06
• Map 1,000 URLs: $0.01 + (100 x $0.001) = $0.11
• Demo mode: $0.00

Note: Firecrawl API usage is billed separately through your Firecrawl account (BYOK).

Common Scenarios

Scenario 1: Pre-Migration URL Inventory

{
  "firecrawlApiKey": "fc-XXXXXXXXXXXXXXXX",
  "url": "https://old-site.com",
  "includeSitemap": true,
  "limit": 1000,
  "webhookUrl": "https://hooks.zapier.com/...",
  "demoMode": false
}

Get complete URL list before site migration for redirect planning.

Scenario 2: Competitor Content Analysis

{
  "firecrawlApiKey": "fc-XXXXXXXXXXXXXXXX",
  "url": "https://competitor.com",
  "search": "blog",
  "limit": 500,
  "demoMode": false
}

Map competitor blog content to identify topics and content gaps.

Scenario 3: Find Pricing Pages Across Competitors

{
  "firecrawlApiKey": "fc-XXXXXXXXXXXXXXXX",
  "url": "https://competitor.com",
  "search": "pricing plans cost",
  "limit": 50,
  "demoMode": false
}

Discover all pricing-related pages for competitive intelligence.

Webhook & Automation Integration

Zapier / Make.com / n8n

1. Create a webhook trigger in your automation platform
2. Copy the webhook URL to webhookUrl
3. Route discovered URLs to spreadsheets or databases

Popular automations:

• URL list -> Google Sheets (site structure database)
• New URLs -> Screaming Frog (detailed SEO audit)
• Categories -> Airtable (content inventory)
• Mapping complete -> Slack alert (notification)

Apify Scheduled Runs

Schedule monthly site mapping to track content changes over time.

Firecrawl Actors Comparison

Actor	Best For	Use When...
Firecrawl Site Mapper (this one)	URL discovery	You need a list of all URLs on a site
Firecrawl Website Crawler	Full site crawling	You need page content, not just URLs
Firecrawl Pro	Advanced scraping	You need actions, screenshots, or stealth mode
Firecrawl Agent	Complex extractions	You need AI to navigate and extract

FAQ

Q: How is this faster than crawling?

A: Traditional crawlers load every page to find links. Site Mapper uses Firecrawl's Map API which analyzes sitemaps and link structures without loading full pages.

Q: Does it find JavaScript-rendered pages?

A: Yes. Firecrawl handles JavaScript-rendered navigation and discovers pages that might be missed by traditional crawlers.

Q: What's the difference between this and the Website Crawler?

A: Site Mapper discovers URLs only (fast, cheap). Website Crawler scrapes content from pages (slower, more detailed).

Q: Can I filter to specific sections?

A: Yes. Use the search parameter to filter results (e.g., "blog" for blog posts, "docs" for documentation).

Q: Does it include subdomains?

A: By default, mapping is limited to the primary domain. Subdomains need separate mapping runs.

Common Problems & Solutions

"Invalid API key" error

• Get your API key from firecrawl.dev dashboard
• Copy it exactly without extra spaces

Fewer URLs than expected

• Enable includeSitemap to include sitemap URLs
• Increase the limit parameter
• Some dynamic sites have fewer discoverable URLs

Missing specific pages

• Use search parameter to filter for specific content types
• Pages without internal links may not be discovered

"Demo data showing"

• Set demoMode: false and provide your firecrawlApiKey

---

📞 Support

• Actor Arsenal: Full Actor Catalog
• Developer: John Rippy

---

Built by John Rippy | Actor Arsenal