Pricing Radar

Apify Actor that detects pricing pages on any website and extracts structured plan data using an LLM (Claude Haiku or GPT-4o-mini).

How it works

1. HTTP Pre-flight (5s)     ─ Cheap fetch: detect platform, PSPs, pricing links
2. Sitemap Scan             ─ /sitemap.xml: discover pricing URLs
3. Browser Crawl            ─ Playwright + stealth: visit pages, detect pricing signals
4. LLM Extraction           ─ On pricing page: HTML → sanitize → Turndown → Markdown → LLM → JSON

Extraction Pipeline

pricingSectionHtml
      │
      ▼
sanitizeHtml()          ─ Strip scripts, styles, SVGs, hash-class attrs
      │
      ▼
Turndown + GFM plugin   ─ Convert to clean Markdown (headers, lists, tables)
      │
      ▼
MAX_MD_CHARS = 8000     ─ Markdown is ~50% more compact than HTML
      │
      ▼
LLM (Haiku / GPT-4o-mini)
      │
      ▼
JSON structured output  ─ Per-plan: name, price, type, features, confidence

Input

Field	Type	Required	Default	Description
url	string	yes	-	Website URL to analyze
llmApiKey	string	yes	-	API key for the LLM provider
llmProvider	enum	no	anthropic	anthropic or openai
maxPages	integer	no	5	Max pages to crawl (1-10)
timeoutSeconds	integer	no	45	Max execution time (10-120)

Example Input (Clay / API)

{
  "url": "https://liftos.app",
  "llmProvider": "anthropic",
  "llmApiKey": "sk-ant-..."
}

Output

{
  "domain": "liftos.app",
  "has_pricing_page": true,
  "pricing_urls": ["https://liftos.app/pricing"],
  "has_free_tier": true,
  "need_contact_for_price": false,
  "has_monthly_annual_toggle": false,
  "plans": [
    {
      "name": "Free forever",
      "price_text": "€0",
      "price_value": 0,
      "currency": "EUR",
      "interval": "none",
      "price_type": "free",
      "cta_text": "Use for free",
      "features": ["10 users", "Unlimited integrations", "20 App uses"],
      "confidence": 0.95
    },
    {
      "name": "Business",
      "price_text": "€9",
      "price_value": 9,
      "currency": "EUR",
      "interval": "month",
      "price_type": "fixed",
      "cta_text": "Get started",
      "features": ["Unlimited App uses", "1TB Workspace storage"],
      "confidence": 0.91
    }
  ],
  "confidence": 0.91,
  "scraping_status": {
    "phase_completed": "browser_scan",
    "pages_visited": 2,
    "time_elapsed_ms": 8500,
    "error": null,
    "blocked": false,
    "early_stopped": false
  }
}

Output Fields

Field	Type	Description
domain	string	Root domain analyzed
has_pricing_page	boolean	Whether a pricing page was found
pricing_urls	string[]	All discovered pricing URLs
has_free_tier	boolean	At least one free plan detected
need_contact_for_price	boolean	At least one plan requires contacting sales
has_monthly_annual_toggle	boolean	Monthly/annual billing toggle detected
plans	PlanData[] or null	Extracted plans (null if LLM fails or no pricing page)
confidence	number	Global confidence score (0.0 - 1.0)
scraping_status	object	Crawl metadata (phase, pages, timing, errors)

Plan Fields

Field	Type	Values
name	string	Plan name as displayed
price_text	string	Exact price text as shown
price_value	number or null	Numeric price value
currency	string or null	ISO currency code
interval	string	month, year, one_time, none, unknown
price_type	string	free, fixed, starting_from, contact_sales, custom, unknown
cta_text	string or null	Call-to-action button text
features	string[]	List of features for this plan
confidence	number	Per-plan confidence (0.0 - 1.0)

Deployment

cd pricing-detector
apify push brave_zygantrum/pricing-radar --force

After pushing, update the Build version in Clay.

Architecture

src/
  main.ts              ← Orchestrator: preflight, sitemap, browser crawl, LLM, output
  pricingExtractor.ts  ← HTML sanitize → Turndown → LLM → JSON parse/validate
  patterns.ts          ← Regex patterns, platform/PSP detection, URL hints
  stealth.ts           ← Anti-bot evasion, resource blocking
  utils.ts             ← URL helpers, HTTP preflight, platform detection
  scorer.ts            ← Weighted scoring for URL discovery confidence

Tech Stack

• Crawlee + Playwright (rebrowser-playwright) — Browser crawling with stealth
• Turndown + GFM — HTML to Markdown conversion (tables, lists, headings)
• Claude Haiku / GPT-4o-mini — Structured data extraction from Markdown
• Zod — Input validation
• Apify — Actor framework, dataset storage