BYO xAI (Grok) via the OpenAI-compatible API. Default model is grok-build-0.1.
Public-ready heuristic packs. No LLM key required for a usable row.
- Browser-render pricing / careers / contact when static HTML lacks prices or roles.
- Classify by the first path segment; ignore leftover nav CTAs.
- Extract
businessSignals.pricing (plan amounts or isCustomOrSalesLed).
- Clean company names, drop mascot/security emails, HQ phrases, broader funding lines.
- Heuristic summary is body text plus those signals, not the meta description.
Initial design spec: input schema, output/dataset schema, key-value store schema, extraction
strategy, error-handling matrix, pricing proposal, and worked example input/output. No src/
implementation yet.
Deployed: private GitHub repo + private Apify Actor, connected via a push webhook for
automatic builds.
Cap pages per category (2 products, 1 blog, 1 other) so Pricing / Careers / Contact
still get fetched after a /solutions flood.
Stay on the marketing host. www and the bare domain are the same site; developer.,
community., and off-site redirects (Loom → Atlassian support) are not. Duplicate
redirect targets are stored once.
Quality pass after the first platform run (6G1kNGYOKL8E27qX2, 10 companies, 7 ok /
3 false blocked).
- Do not treat the word "captcha" on a 200 page as a bot block (Intercom/Zapier/Webflow).
- Fetch pages in category-priority order so a sitemap of blog posts cannot starve
About / Pricing / Careers (HubSpot, ClickUp).
- Classify by path segment, not substring (
/company-news is no longer about).
- Canonicalize URLs so
/ and `` are one page; skip PDFs and tracking params.
- Ignore empty
mailto:, image-filename "emails", share buttons, and personal
LinkedIn /in/ profiles. Read sameAs from JSON-LD.
- 4xx/5xx HTML is a failed fetch, not a research page. 403/429 get one browser retry.
- Playwright waits for
domcontentloaded, not networkidle. Plain Chrome UA.
- Input cap of 100 companies, default concurrency 5, safer prefill, LICENSE, COST.md.