AI Sales Research Pack Generator

Turn a list of company websites into structured, AI-ready sales research packs — no manual digging through About pages, no fragile per-site scrapers.

Give it URLs. With no LLM key it still returns a usable pack: a body-text summary (not just the meta description), public prices or an honest “sales-led” flag, cleaned contacts, hiring and funding lines when the site states them, and an evidence URL behind every non-trivial claim. A BYO Anthropic/OpenAI/xAI key is an optional upgrade for a more natural value proposition and outreach angles.

Implemented as a Python Actor (apify + httpx + Playwright, no site-specific selectors). Run pip install -r requirements-dev.txt && pytest in this folder to run the test suite locally.

What you get

One dataset row per input URL, always the same shape, regardless of how much of the site cooperated:

{
  "inputUrl": "https://www.acme-widgets.com",
  "companyName": "Acme Widgets Inc.",
  "status": "ok",
  "summary": {
    "shortSummary": "Acme Widgets makes modular packaging hardware for e-commerce brands...",
    "outreachAngles": [
      "They list 3 open ops/fulfillment roles — likely scaling shipping volume right now.",
      "Pricing is gated behind 'Request a quote', suggesting custom/enterprise deals are the norm."
    ]
  },
  "contact": {
    "emails": ["sales@acme-widgets.com"],
    "socialLinks": { "linkedin": "https://www.linkedin.com/company/acme-widgets" }
  },
  "businessSignals": {
    "technologiesDetected": ["Webflow", "HubSpot", "Intercom"],
    "hiring": { "isHiringSignalFound": true, "openRolesMentioned": 3 },
    "fundingMentions": [{ "text": "raised a $6M Series A", "sourceUrl": "https://www.acme-widgets.com/about" }],
    "pricing": {
      "plans": [{ "name": "Standard", "price": "$16", "period": "month" }],
      "isCustomOrSalesLed": false,
      "pricingPageUrl": "https://www.acme-widgets.com/pricing"
    }
  },
  "evidence": [
    { "claim": "Raised a $6M Series A.", "sourceUrl": "https://www.acme-widgets.com/about" }
  ]
}

Full row shape: .actor/dataset_schema.json. Worked examples (including a blocked-site row): data/example_dataset.json.

Use with AI agents

Connected to the Apify MCP server? Ask for this Actor by name: zerrouki-samir/ai-sales-research-pack-generator

I want company research packs from the Apify Actor
zerrouki-samir/ai-sales-research-pack-generator.

Use it when I have a list of company websites and need a structured brief: what they do,
pricing (public amounts or sales-led), contacts, hiring/funding signals, and a source URL
for each claim. An LLM key is optional — without one I still get a heuristic pack.

How to call it: pass companyUrls. Defaults are fine for a first run (enableAiSummary can
stay on; leave llmApiKey blank for heuristic-only). Each distinct domain is one dataset
row with status ok | partial | blocked | failed.

Start with this input:
{
  "companyUrls": ["https://linear.app", "https://calendly.com", "https://www.loom.com"],
  "enableAiSummary": false,
  "maxPagesPerCompany": 12
}

Why no site-specific selectors

A list of 200 companies is 200 different page layouts. Any actor built on CSS selectors (.hero h1, div.about-text) breaks the moment a handful of those sites redesign, and it never worked on the other 190 to begin with. Everything here reads from structure and text that is effectively universal, so it degrades gracefully instead of returning nothing:

Signal	Universal source used	Never used
Company name	JSON-LD Organization, og:site_name, <title>, domain as last resort	A specific div/class
Main content	Readability-style extraction (largest coherent text block, <main>/<article> preference, boilerplate stripped)	Hand-picked selectors
Page purpose	URL path + link anchor text keyword matching (/pricing, "Pricing", "Plans")	Per-site nav maps
Contact info	mailto:/tel: href scanning, known social-domain regex	Contact-page HTML structure
Tech stack	Passive fingerprinting: script src/meta generator/cookie-name patterns (Wappalyzer-style signatures)	Reading rendered UI
Company facts	Regex/keyword pass over extracted text ("Series A", "we're hiring", "founded in", employee-count phrases) + JSON-LD Organization/LocalBusiness fields	LLM guessing without a source page

Extraction strategy

1. Normalize & dedupe. Each companyUrls entry is lowercased, forced to https:// if no scheme is given, and reduced to a root domain for deduplication (www.acme.com and acme.com merge). Malformed entries are rejected up front with a clear per-URL error, not a crawl failure.

2. Discover pages, per domain, cheaply first.

• Try robots.txt → sitemap.xml (and sitemap indexes) first. A sitemap is the cheapest, most complete map of a site and needs zero HTML parsing.
• If there's no usable sitemap, crawl from the homepage: fetch it, extract internal links, and classify each by URL path + anchor text keywords into categories (home, about, product, pricing, team, contact, careers, blog, press, case-studies, legal, other). Follow links up to crawlDepth hops.
• Stop discovery for a domain once maxPagesPerCompany is reached, keeping the categories in pageCategoriesToPrioritize first — an "About" page is worth more to a sales pack than the 15th blog post.

3. Fetch adaptively. Default renderingMode: auto: try a plain HTTP GET (httpx, with retry/backoff) first — fast, cheap, works for the majority of marketing sites. A page is escalated to a headless browser (Playwright) only when the static HTML looks JS-rendered (near-empty <body> text, an SPA root div with no children, a known framework shell with no server-rendered content) — and never for a page that already failed (a 404/500 needs fixing, not a heavier renderer). The escalation is per-domain and sticky: once one page on a domain needs a browser, the rest of that domain's fetches use it too, avoiding a state-dependent flip mid-crawl. A domain that never needs a browser never pays to launch one.

4. Extract per page, without selectors.

• Metadata: <title>, meta description, Open Graph tags, and — most valuable — JSON-LD (schema.org/Organization, LocalBusiness, WebSite). A large share of modern marketing sites (Webflow, Squarespace, most CMS platforms) ship this automatically; when present, it's a structured, site-author-verified source for name, address, logo, and social profiles.
• Main text: a Readability-style pass (largest-density text block, <main>/<article> preferred, boilerplate/nav/footer stripped) rather than a fixed selector. This is the same class of technique browsers' "reader mode" uses, and it is layout-agnostic by design.
• Links: every <a> on the page is scanned once for mailto:, tel:, and known social-platform domains (linkedin.com/company, x.com/twitter.com, facebook.com, instagram.com, youtube.com) — regex on the href, not a DOM position.
• Tech signals: script src hosts, meta name="generator", and a handful of known cookie/script-name fingerprints are matched against a signature table (Shopify, WordPress, Webflow, HubSpot, Segment, Intercom, Stripe, etc.) — response-structure based, immune to visual redesigns.

5. Aggregate signals across the domain's pages, not per-page: hiring language + a reachable /careers page → hiring.isHiringSignalFound; funding-keyword sentences ("raised", "Series A/B/C", "$_M/_B") → fundingMentions, each tagged with the page it came from; employee-count phrases and JSON-LD numberOfEmployees → estimatedCompanySizeHint; JSON-LD address or detected postal patterns → locations.

6. Synthesize the summary (optional AI step). With enableAiSummary: true and a key supplied, the actor sends the extracted text from the top-priority pages (home, about, product, pricing — never the raw HTML) plus the already-extracted structured signals to the chosen LLM, with a system prompt that:

• requires every sentence in outreachAngles/valueProposition to be traceable to the supplied text, and instructs the model to omit a claim rather than infer beyond the source;
• asks for strict JSON output matching the summary schema (via tool calling / structured output on the provider side), so the pack is always machine-parseable;
• is capped to a small, cheap model by default (claude-haiku-4-5-20251001 / gpt-4o-mini / grok-build-0.1) since this is synthesis over already-extracted text, not open-ended research.

Without a key (or llmProvider: none), the actor still ships a heuristic summary — the opening sentences of the About/home page text plus a templated sentence built from the structured signals — so the AI step is a quality upgrade, never a hard dependency. generatedBy on the summary always says which path produced it.

7. Every claim gets an evidence entry. evidence is populated both by the heuristic pass (directly, since it already knows the source page for each regex match) and by the LLM step (which is asked to cite the source URL for each claim it makes). Anything without a traceable source is dropped rather than included unsourced.

Error handling & stability

Designed so one bad site never sinks the run, and every row is equally shaped whether it came from a cooperative site or a hostile one.

Failure mode	Handling
Invalid/malformed input URL	Rejected at input-validation time with a clear reason; never enters the crawl queue.
DNS failure / connection refused / TLS error	Row is written with status: "failed" and a plain-language errorReason; does not retry indefinitely (2 retries with exponential backoff, then gives up).
HTTP 403/429, or a body matching known bot-check patterns (Cloudflare interstitial, "verify you are a human")	Two HTTP retries with backoff, then one headless-browser attempt on 403/429. If still blocked, status: "blocked" with the reason — never spammed with retries that would look like an attack. A recaptcha script on an otherwise-normal 200 page is not treated as a block.
robots.txt disallows a path	Skipped silently when respectRobotsTxt is true (default); does not count as a failure.
Site has some pages but others 404/timeout	status: "partial" — the pack ships with whatever was extracted, pagesFound shows per-page httpStatus, and nothing downstream (contact/signals/summary) is left in a half-written state.
A domain hangs or is unusually large	Bounded by maxCrawlBudgetPerCompanySecs (hard wall-clock cap) independent of maxPagesPerCompany, so a slow site degrades to partial instead of stalling the batch.
JS-only site under renderingMode: static	Extraction proceeds on whatever HTML is available; if that yields near-nothing, status: "partial" with errorReason noting the page appears to require JavaScript — a nudge to rerun with renderingMode: auto.
LLM call fails, times out, or returns invalid JSON	Falls back to the heuristic summary for that company only; the run is not failed and no other company is affected. summary.generatedBy reflects the fallback.
Duplicate input URLs (same root domain)	Merged before crawling; one row in, no duplicate charge.

No row is ever silently dropped for a distinct company: every distinct root domain produces exactly one dataset row, always in the status ∈ {ok, partial, blocked, failed} shape. Duplicate URLs for the same domain are merged into that one row. Malformed entries write a failed row.

Scalability

• Each company is a fully self-contained, independent pipeline (its own httpx client, its own proxy session, its own browser context if it needs one) run as an asyncio task, bounded by a single asyncio.Semaphore(maxConcurrency) — so maxConcurrency companies are being researched in parallel at any time, and one company's slowness or crash can never block another's.
• Per-domain, not global, budgets (maxPagesPerCompany, maxCrawlBudgetPerCompanySecs) mean a batch of 1 company and a batch of 10,000 behave the same way per-company; only maxConcurrency changes overall run wall-clock time.
• Static-first, browser-on-demand fetching keeps the common case (plain marketing site) cheap: a single Playwright browser process is shared and only launched lazily the first time any domain actually needs it, reserving the expensive path for domains that need it.
• Apify Proxy (datacenter by default, switchable to residential) with a sticky session per domain keeps well-behaved crawling from tripping basic bot defenses at scale.

Input

Only companyUrls is required. Full field list, defaults, and descriptions: .actor/input_schema.json.

{
  "companyUrls": [
    "https://linear.app",
    "https://calendly.com",
    "https://www.loom.com"
  ],
  "maxPagesPerCompany": 15,
  "crawlDepth": 2,
  "renderingMode": "auto",
  "respectRobotsTxt": true,
  "proxyConfiguration": { "useApifyProxy": true },
  "enableAiSummary": true,
  "llmProvider": "anthropic",
  "llmApiKey": "sk-ant-...",
  "includeRawText": false,
  "outputLanguage": "en"
}

Field	Type	Default	Notes
companyUrls	array of strings	— (required)	Homepage or any page on the domain; scheme optional.
maxPagesPerCompany	integer	15	Cap on pages fetched per domain.
crawlDepth	integer	2	Link-hops followed when no sitemap exists.
pageCategoriesToPrioritize	array of strings	see schema	Crawl order when the page cap is hit first.
renderingMode	auto|static|browser	auto	Headless-browser escalation strategy.
respectRobotsTxt	boolean	true
requestTimeoutSecs	integer	30	Per-page timeout.
maxCrawlBudgetPerCompanySecs	integer	90	Hard per-company wall-clock cap.
maxConcurrency	integer	5	Companies researched in parallel (each fetches its own pages one at a time).
proxyConfiguration	object	Apify Proxy on	Standard Apify proxy input.
enableAiSummary	boolean	true	Turns the LLM synthesis step on/off.
llmProvider	anthropic|openai|xai|none	anthropic
llmApiKey	string (secret)	—	Bring your own key; blank forces heuristic summaries.
llmModel	string	provider default	Optional override.
includeRawText	boolean	false	Adds cleaned full page text to each row.
outputLanguage	string	en	Summary language only.

Output

Dataset — one row per input URL, always present regardless of status: .actor/dataset_schema.json · data/example_dataset.json.

Field group	Contents
inputUrl, normalizedDomain, companyName (+ companyNameSource)	Identity
status, errorReason	ok | partial | blocked | failed, with a plain-language reason when not fully ok
summary	shortSummary, valueProposition, targetCustomer, outreachAngles[], generatedBy (ai/heuristic), confidence
pagesFound[]	Every page crawled: url, category, title, httpStatus, renderMode
contact	emails[], phones[], contactFormUrl, socialLinks (linkedin/twitter/facebook/instagram/youtube)
businessSignals	industryGuess, technologiesDetected[], hiring, fundingMentions[], estimatedCompanySizeHint, locations[], languagesDetected[]
evidence[]	claim, sourceUrl, snippet — one entry per non-trivial claim above
rawText[]	Only when includeRawText: true
metadata	crawledAt, page counts, renderModeUsed, processingTimeMs, llmUsed

Pricing (proposed)

Pay-per-event, no start fee — see .actor/actor.json:

Event	Charged when
company-researched (~$0.014–0.02)	A row ships with status: ok or partial
ai-summary-generated (~$0.007–0.01)	The LLM synthesis step actually ran and succeeded

Rows with status: blocked/failed, and merged duplicates, are never charged.

Limitations

• Sites that require login, or block all automated traffic outright, will return blocked — this actor doesn't attempt to defeat access controls.
• Signals are text-and-structure heuristics, not verified facts; treat businessSignals and summary as leads to check, which is why every claim carries a sourceUrl.
• Social profile links are captured as evidence, not enriched — this actor does not scrape LinkedIn/Twitter/Facebook content itself.
• maxPagesPerCompany/crawlDepth are cost/completeness dials, not a guarantee every relevant page on a large site is found.

Development

python -m venv .venv && source .venv/bin/activate
pip install -r requirements-dev.txt
playwright install chromium   # only needed to actually run a crawl, not for the test suite
ruff check src tests
pytest

Module layout mirrors this document: urls.py (normalize/dedupe), robots.py (robots.txt + sitemap), classify.py (page-purpose keywords), fetch.py (static/browser fetch, blocked/thin detection), extract.py (JSON-LD, meta, main text, links, contact, tech), signals.py (cross-page aggregation + evidence), summarize.py (heuristic + optional LLM summary), company.py (per-company pipeline), main.py (Actor entry point). tests/ runs entirely against local HTML fixtures and a mocked HTTP layer (respx) — no live network calls, no live sites.

To run it as an Actor locally: apify run (with the Apify CLI) or python -m src with an APIFY_INPUT_JSON / storage/key_value_stores/default/INPUT.json set up per the Apify SDK docs.

License

MIT