Product Hunt Scraper (with Email Enrichment)
Pricing
from $2.00 / 1,000 results
Extract product names, upvotes, makers, taglines, and website URLs from ProductHunt MCP-ready. $2.00/1K.
Pricing
from $2.00 / 1,000 results
Rating
0.0
(0)
Developer
Khadin Akbar
Maintained by CommunityActor stats
2
Bookmarked
72
Total users
13
Monthly active users
5 days ago
Last modified
Categories
Share
node_modules/dist/storage/.env.env.**.log.DS_StorePROVIDER_PRICING.jsonnode_modules/dist/storage/.env.env.**.log.DS_StoreFROM apify/actor-node-playwright-chrome:20
# Detect and save the image's pre-installed playwright version BEFORE any COPY.# The image ships playwright + matching Chromium. We must install the exact same# playwright version so browser executable paths always match.RUN node -e "try{process.stdout.write(require('./node_modules/playwright/package.json').version)}catch(e){process.stdout.write('unknown')}" > /tmp/IMAGE_PW_VERSION && \ echo "Image playwright: $(cat /tmp/IMAGE_PW_VERSION)"
# Copy ALL source (including package.json, package-lock.json, src/, .actor/)COPY . .
# Patch package.json + regenerate lock file to use the image's playwright versionRUN IMAGE_PW=$(cat /tmp/IMAGE_PW_VERSION) && \ npm pkg set dependencies.playwright="$IMAGE_PW" && \ npm install --package-lock-only --ignore-scripts 2>/dev/null && \ echo "Playwright pinned to $IMAGE_PW (matches image browsers)"
# Install ALL deps (devDependencies included so tsc is available)RUN NODE_ENV=development npm install --audit=false
# Compile TypeScriptRUN npm run build
# Remove devDependencies from the final imageRUN npm prune --omit=dev
CMD ["node", "dist/main.js"]1# Product Hunt lead-engine reliability graph — v1.42
3This graph is both the implementation model and the release gate.4
5```text6Product Hunt date / period / URL / search / topic input7 -> validate and normalize8 -> official Product Hunt GraphQL API9 -> bounded timelines use stable NEWEST ordering; ranks remain output fields10 -> resumable run: lightweight index pagination11 -> skip normalized Product Hunt launch URLs supplied by a prior capped run12 -> when a saved cursor exists, resume directly after the capped page13 -> otherwise, at first unseen launch, re-fetch that page in full and continue with full pages14 -> persist a safe RUN_CHECKPOINT periodically and after each enriched page15 -> for each unique product16 -> official external ProductLink or approved override?17 -> yes: accept with provenance18 -> no: Product Hunt redirect metadata19 -> no verified match: maker website identity evidence (up to 2)20 -> no verified match: product links published on maker websites21 -> protected foundation-model launch: identity-check vendor-owned pages22 -> no verified match: DataForSEO full-name query23 -> no verified match: one core-brand query24 (or one same-query retry for documented transient failure)25 -> no verified match: bounded direct brand-domain hypotheses26 -> no verified match/failure: SerpApi fallback query (1)27 -> every candidate homepage or official product page must pass identity and ownership scoring28 -> protected foundation-model names must use vendor-owned domains29 -> verified external homepage?30 -> no: persist unresolved provenance31 -> yes: static homepage assessment32 -> homepage-linked same-origin page plan33 contact -> about -> team -> privacy -> terms34 -> cap at 5 pages total35 -> merge browser-rendered footer/header links into the static plan36 -> recover plain, linked, structured, obfuscated, escaped, and Cloudflare-protected public emails37 -> keep up to 6 ranked candidates for verification; cap final output at 338 -> verify public emails with EmailListVerify39 -> below deliverable target: public maker homepages (up to 2)40 -> verify maker-homepage email candidates41 -> still below target: maker-name finder (5 credits)42 -> still empty: domain finder (10 credits)43 -> keep only same-domain, deliverable, high/medium matches44 -> merge/rank/cap final contacts at 345 -> calculate status/score/tier/signals46 -> validate complete row47 -> persist row with primary event48 -> charge one email-found event per persisted selected contact49 -> OUTPUT + RUN_SUMMARY50```51
52## Local graph loop53
54`npm test` covers:55
56- post, product-launch, daily/monthly/weekly/yearly URL normalization;57- resumable launch-URL normalization and rejection of non-launch exclusion values;58- Product Hunt's documented full reset-window delay on HTTP 429 responses and stable newest-first bounded timeline pagination;59- opaque-cursor continuation, safe interruption checkpoints, plus lightweight legacy indexing that switches to full pages at the first unseen launch;60- current ISO-week archive routing;61- invalid Product Hunt hosts, redirect seeds, malformed dates, and private targets;62- homepage plus four linked-page cap and contact/about/team/privacy/terms priority;63- public email dedupe, placeholder filtering, same-domain ranking, six-candidate verification reserve, encoded/obfuscated/Cloudflare recovery, and three-email output cap;64- static plus browser-rendered fallback-link merging without exceeding the five-page cap;65- documented EmailListVerify status normalization;66- transient-only EmailListVerify HTTP retry classification;67- DataForSEO and SerpApi result normalization, conservative core-brand queries, and transient-only retry classification;68- search-directory rejection, positive homepage identity confirmation, ambiguous single-word false-positive rejection, and maker-published product-link ranking;69- maker-homepage gap filling that can replace rejected product-site candidates without duplicating addresses;70- maker-name splitting and same-domain Contact Finder filtering; and71- deterministic lead qualification without inflating unresolved products.72
73Local release command:74
75```bash76npm test && jq empty .actor/input_schema.json .actor/dataset_schema.json \77 .actor/output_schema.json .actor/key_value_store_schema.json \78 .actor/pay_per_event.json .actor/actor.json && git diff --check79```80
81## Final-build cloud matrix82
83The deployed build must pass at least 20 bounded cases with delayed readback of dataset rows, `OUTPUT`, `RUN_SUMMARY`, run status, build ID, and named charges.84
85| Group | Cases | Assertions |86| --- | ---: | --- |87| Input routing | 5 | Direct post, product launch, daily range, monthly `2026/7`, and weekly `2026/26` route to intended API records. |88| Website resolution | 5 | ProductLink, override, redirect, maker site, and organic search either pass homepage identity checks or remain unresolved without guessing. |89| Crawl graph | 4 | Homepage-only, linked contact, linked about/team/privacy/terms priority, off-origin rejection, and five-page cap report exact coverage. |90| Email graph | 3 | Zero/one/three public contacts, exact verification statuses, fallback-only/mixed contacts, and invalid/risky separation are correct. |91| Lifecycle/economics | 3 | Invalid input, valid empty, and useful partial/complete runs write terminal KV records; primary persistence precedes email charge; event/provider counts reconcile. |9293Release thresholds:94
95- 20 or more final-build cloud cases;96- observed pass rate at least 99%;97- zero unexpected failures for samples below 100 cases;98- at least one verified external website;99- at least one public-page email branch;100- at least one EmailListVerify verification result;101- at least one contact-finder branch when provider credits permit;102- no secret in source, logs, input, dataset, KV output, or evidence files;103- live pricing and public visibility unchanged unless separately authorized.104
105A full historical month is not acceptance evidence by itself. Run it only after bounded final-build canaries prove website resolution, email extraction, verification, persistence, and charge reconciliation.{ "name": "producthunt-scraper-pro", "version": "1.4.5", "type": "module", "description": "Product Hunt lead engine with verified websites, bounded five-page contact discovery, email verification, and lead scoring", "main": "dist/main.js", "scripts": { "start": "node dist/main.js", "build": "tsc", "dev": "tsc && node dist/main.js", "test": "npm run build && node --test dist/enrichment-utils.test.js dist/lead-engine.test.js" }, "dependencies": { "apify": "^3.7.2", "cheerio": "^1.0.0", "crawlee": "^3.18.1", "playwright": "^1.62.1", "tldts": "^7.4.10" }, "devDependencies": { "@types/node": "^20.19.43", "typescript": "^5.3.0" }}{ "compilerOptions": { "target": "ES2022", "module": "NodeNext", "moduleResolution": "NodeNext", "outDir": "dist", "rootDir": "src", "strict": true, "esModuleInterop": true, "skipLibCheck": true, "resolveJsonModule": true }, "include": ["src/**/*"]}{ "actorSpecification": 1, "name": "producthunt-scraper-pro", "title": "Product Hunt Lead Generator — Products, Websites & Verified Emails", "description": "Turn Product Hunt daily, weekly, monthly, yearly, search, topic, or direct-launch inputs into qualified company leads. The Actor resolves official websites with identity checks, assesses up to five linked public pages, returns up to three contacts, verifies deliverability, fills missing business contacts with a managed fallback, and reports website/email provenance plus lead score and tier.", "readme": "../README.md", "version": "1.4", "buildTag": "latest", "environmentVariables": { "PH_API_TOKEN": "@producthuntApiToken", "EMAILLISTVERIFY_API_KEY": "@emailListVerifyApiKey", "DATAFORSEO_LOGIN": "@dataforseo-login", "DATAFORSEO_PASSWORD": "@dataforseo-password", "SERPAPI_KEY": "@SERPAPI_KEY" }, "dockerfile": "./Dockerfile", "output": "./output_schema.json", "storages": { "dataset": "./dataset_schema.json", "keyValueStore": "./key_value_store_schema.json" }, "input": "./input_schema.json", "changelog": "\n\nv1.4 (2026-09-06) — Added stable newest-first pagination for bounded timelines, cost-efficient resumable exclusions, opaque cursor handoff, and periodic safe RUN_CHECKPOINT records for long periods. New runs can continue directly from RUN_SUMMARY.nextProductHuntCursor or an interrupted run's checkpoint; older continuations page through a lightweight Product Hunt index, skip prior launch URLs before enrichment, and switch to full Product Hunt pages at the first unseen launch. Exclusions do not consume maxResults. Rate-limit recovery honors Product Hunt's documented full X-Rate-Limit-Reset window. Improved website resolution with maker-published links, conservative core-brand aliases, identity-confirmed direct domains, directory rejection, and protected-model vendor ownership checks. Product-site assessment remains capped at five homepage-linked pages, merges static and rendered navigation, recovers common encoded and obfuscated public addresses, keeps a six-candidate reserve to survive invalid-address verification, stops verification once three usable addresses are secured, retries transient verification transport/429/5xx failures once, and caps final output at three ranked provenance-bearing contacts.\n\nv1.3 (2026-08-24) — Rebuilt enrichment as a cost-bounded Product Hunt lead engine. Added direct post and dated leaderboard URL routing through the official API; verified website resolution through redirect metadata, maker evidence, DataForSEO, and SerpApi fallback; static-first five-page email discovery; managed EmailListVerify verification and contact discovery; up to three ranked contacts; lead scores/tiers; provenance; atomic persist-and-charge; and OUTPUT/RUN_SUMMARY terminal contracts. No Product Hunt challenge bypass is used.\n\nv1.2 (2026-08-15) — Historical release; superseded by the verified lead-enrichment contract in v1.3.\n\nv1.1 (2026-05-22) — Historical input-validation update.", "chargedEvents": "./pay_per_event.json"}{ "actorSpecification": 1, "fields": { "type": "object", "properties": { "product_name": { "type": "string", "title": "Product Name", "description": "The official product name as listed on ProductHunt.", "example": "Notion AI" }, "tagline": { "type": ["string", "null"], "title": "Tagline", "description": "Short one-line pitch shown beneath the product name on ProductHunt.", "example": "AI-powered writing for everyone" }, "description": { "type": ["string", "null"], "title": "Description", "description": "Full product description written by the makers on ProductHunt.", "example": "Notion AI helps you write faster, think bigger, and augment your knowledge base." }, "upvote_count": { "type": ["integer", "null"], "title": "Upvotes", "description": "Total upvotes received. Higher = more popular on launch day.", "example": 1284 }, "comment_count": { "type": ["integer", "null"], "title": "Comments", "description": "Total number of comments on the ProductHunt listing.", "example": 87 }, "reviews_count": { "type": ["integer", "null"], "title": "Review Count", "description": "Number of user reviews the product has received on ProductHunt.", "example": 42 }, "reviews_rating": { "type": ["number", "null"], "title": "Avg Rating", "description": "Average user review rating on ProductHunt (typically 1–5 scale).", "example": 4.7 }, "daily_rank": { "type": ["integer", "null"], "title": "Daily Rank", "description": "#1 means most upvoted product of the day. Populated for daily leaderboard scrapes.", "example": 1 }, "weekly_rank": { "type": ["integer", "null"], "title": "Weekly Rank", "description": "Product's position in the weekly leaderboard.", "example": 3 }, "monthly_rank": { "type": ["integer", "null"], "title": "Monthly Rank", "description": "Product's position in the monthly leaderboard.", "example": 12 }, "yearly_rank": { "type": ["integer", "null"], "title": "Yearly Rank", "description": "Product's position in the annual leaderboard.", "example": 47 }, "launch_date": { "type": ["string", "null"], "title": "Launch Date", "description": "Date the product was featured on ProductHunt (YYYY-MM-DD).", "example": "2026-03-28" }, "product_hunt_url": { "type": ["string", "null"], "title": "ProductHunt URL", "description": "Full URL to the product's page on ProductHunt.", "example": "https://www.producthunt.com/posts/notion-ai" }, "website_url": { "type": ["string", "null"], "title": "Website", "description": "Verified public external product homepage. Resolution may use an official ProductLink, redirect metadata, maker-site evidence, an identity-confirmed direct-domain hypothesis, or a search result that passed page identity checks.", "example": "https://notion.so" }, "product_hunt_website_url": { "type": ["string", "null"], "title": "Product Hunt Website Link", "description": "Original Product Hunt website field. This can be a Product Hunt tracking redirect and is retained for provenance, not treated as a verified external website.", "example": "https://www.producthunt.com/r/ABC123" }, "website_resolution_source": { "type": "string", "title": "Website Resolution Source", "description": "How website_url was resolved: official ProductLink, input override, Product Hunt redirect metadata, maker website or maker-published link, protected vendor domain, DataForSEO organic result, identity-confirmed direct-domain hypothesis, SerpApi organic result, or none.", "example": "dataforseo_organic" }, "website_resolution_status": { "type": "string", "title": "Website Resolution Status", "description": "verified_external, verified_search_match, unresolved, or not_available.", "example": "verified_external" }, "website_resolution_confidence": { "type": "integer", "title": "Website Confidence", "description": "Identity confidence from 0 to 100. Search and maker-site candidates are accepted only after homepage evidence passes the resolver threshold.", "example": 91 }, "website_resolution_signals": { "type": "array", "title": "Website Resolution Signals", "description": "Public evidence used to accept the website, such as domain, page-title, metadata, or tagline matches.", "example": ["full_product_name_matches_domain", "product_name_matches_page_title"] }, "website_resolution_provider_cost_usd": { "type": "number", "title": "Website Resolver Provider Cost", "description": "Provider-reported DataForSEO cost attributed to this website resolution attempt. Zero when no DataForSEO call was made.", "example": 0.002 }, "company_domain": { "type": ["string", "null"], "title": "Company Domain", "description": "Normalized company domain used for same-domain email ranking and contact discovery.", "example": "notion.so" }, "emails": { "type": "array", "title": "Emails Found", "description": "Up to three ranked contact emails from the public website crawl and, when needed, the managed business-contact finder.", "example": ["hello@notion.so", "support@notion.so"] }, "verified_emails": { "type": "array", "title": "Verified Deliverable Emails", "description": "Subset of emails for which EmailListVerify returned the exact deliverable status ok.", "example": ["hello@notion.so"] }, "risky_emails": { "type": "array", "title": "Other or Risky Emails", "description": "Selected contacts that were not confirmed with status ok, retained with their exact verification status for transparent filtering.", "example": ["support@notion.so"] }, "rejected_emails": { "type": "array", "title": "Rejected Emails", "description": "Public emails rejected from the final lead list because verification returned dead server, invalid MX, disabled, invalid syntax, disposable, or spam-trap status.", "example": ["old@notion.so"] }, "rejected_email_verifications": { "type": "array", "title": "Rejected Email Verification Results", "description": "Exact EmailListVerify records for rejected public emails, retained for audit without counting them as contacts or email-found events.", "example": [{"email": "old@notion.so", "provider": "emaillistverify", "status": "invalid_mx", "source": "public_website", "confidence": null, "checkedAt": "2026-08-24T14:30:00.000Z"}] }, "email_verifications": { "type": "array", "title": "EmailListVerify Results", "description": "One verification record per selected email, including exact provider status, discovery source, optional contact-finder confidence, and check time.", "example": [{"email": "hello@notion.so", "provider": "emaillistverify", "status": "ok", "source": "public_website", "confidence": null, "checkedAt": "2026-08-24T14:30:00.000Z"}] }, "email_verification_requested": { "type": "boolean", "title": "Email Verification Requested", "description": "True when this run requested EmailListVerify verification. A configured owner or caller key is also required before any provider call is made.", "example": true }, "email_source": { "type": "string", "title": "Email Source", "description": "page_scrape, maker_website, emaillistverify_contact_finder, mixed, or none.", "example": "mixed" }, "email_result": { "type": "string", "title": "Email Result", "description": "emails_found, no_public_email_found, or not_assessed.", "example": "emails_found" }, "email_coverage": { "type": "string", "title": "Email Crawl Coverage", "description": "complete when every selected page was assessed, partial when one or more eligible pages could not be assessed, or none when no verified external site was available.", "example": "complete" }, "email_terminal_reason": { "type": "string", "title": "Email Terminal Reason", "description": "Bounded reason for the enrichment result, such as max_public_email_candidates_reached, eligible_pages_exhausted, homepage_timeout, or website_not_available.", "example": "eligible_pages_exhausted" }, "email_pages_visited": { "type": "integer", "title": "Website Pages Visited", "description": "Number of verified external website pages assessed for email extraction, capped at five.", "example": 3 }, "email_page_urls": { "type": "array", "title": "Email Page URLs", "description": "Final same-origin website pages actually assessed for email extraction.", "example": ["https://notion.so/", "https://notion.so/contact"] }, "maker_email_pages_visited": { "type": "integer", "title": "Maker Homepages Visited", "description": "Number of public maker homepages from the Product Hunt API assessed as a final published-email fallback, capped at two.", "example": 1 }, "maker_email_page_urls": { "type": "array", "title": "Maker Email Page URLs", "description": "Product Hunt maker homepages actually assessed for publicly listed founder contact emails.", "example": ["https://founder.example/"] }, "contact_discovery_requested": { "type": "boolean", "title": "Contact Discovery Requested", "description": "True when the managed Contact Finder fallback was eligible after public-page verification.", "example": true }, "contact_discovery_attempts": { "type": "integer", "title": "Contact Discovery Attempts", "description": "Number of bounded EmailListVerify Contact Finder requests made for this product.", "example": 1 }, "contact_discovery_credits": { "type": "integer", "title": "Contact Discovery Credit Ceiling Used", "description": "Documented EmailListVerify credits associated with attempted named or domain contact-finder requests.", "example": 5 }, "lead_status": { "type": "string", "title": "Lead Status", "description": "verified_contact, catch_all_contact, unverified_contact, no_contact, or unresolved_website.", "example": "verified_contact" }, "lead_score": { "type": "integer", "title": "Lead Score", "description": "Transparent 0–100 qualification score based on website confidence, email deliverability, public provenance, and Product Hunt traction.", "example": 84 }, "lead_tier": { "type": "string", "title": "Lead Tier", "description": "A, B, C, or unqualified, derived deterministically from lead_score.", "example": "A" }, "lead_signals": { "type": "array", "title": "Lead Signals", "description": "Plain-language signals explaining the lead score and tier.", "example": ["website_resolved", "deliverable_email_verified", "email_published_on_website"] }, "enrichment_warnings": { "type": "array", "title": "Enrichment Warnings", "description": "Safe per-record warnings, for example an off-origin redirect or a page timeout. Never contains credentials.", "example": [] }, "topics": { "type": "array", "title": "Topics", "description": "ProductHunt category tags (e.g. Artificial Intelligence, Developer Tools, Productivity).", "example": ["Productivity", "Artificial Intelligence", "Writing"] }, "thumbnail_url": { "type": ["string", "null"], "title": "Thumbnail", "description": "URL of the product logo/thumbnail hosted on ProductHunt's CDN.", "example": "https://ph-files.imgix.net/abc123.png?auto=format&fit=crop&h=256&w=256" }, "makers": { "type": "array", "title": "Makers", "description": "People who built the product. Each entry includes maker identity, public Product Hunt/Twitter links, and the maker website when exposed by the official API.", "example": [{"maker_name": "Ivan Zhao", "maker_id": "ivanhzhao", "maker_ph_id": "1234567", "twitter_url": "https://twitter.com/ivanhzhao", "maker_url": "https://www.producthunt.com/@ivanhzhao", "website_url": "https://notion.so"}] }, "featured": { "type": "boolean", "title": "Featured", "description": "True if the product was editorially featured on ProductHunt's homepage.", "example": true }, "scraped_at": { "type": "string", "title": "Scraped At", "description": "ISO 8601 timestamp of when this record was extracted.", "example": "2026-03-28T14:30:00.000Z" } } }, "views": { "overview": { "title": "🚀 Products Overview", "transformation": { "fields": [ "daily_rank", "product_name", "tagline", "upvote_count", "comment_count", "reviews_rating", "launch_date", "topics", "website_url", "website_resolution_source", "website_resolution_confidence", "product_hunt_url", "featured" ] }, "display": { "component": "table", "properties": { "daily_rank": { "label": "Rank" }, "product_name": { "label": "Product" }, "tagline": { "label": "Tagline" }, "upvote_count": { "label": "Upvotes" }, "comment_count": { "label": "Comments" }, "reviews_rating": { "label": "Rating" }, "launch_date": { "label": "Launch Date" }, "topics": { "label": "Topics" }, "website_url": { "label": "Website" }, "website_resolution_source": { "label": "Website Source" }, "website_resolution_confidence": { "label": "Website Confidence" }, "product_hunt_url": { "label": "PH Link" }, "featured": { "label": "Featured?" } } } }, "leads": { "title": "✉️ Email Leads", "transformation": { "fields": [ "product_name", "tagline", "emails", "verified_emails", "risky_emails", "rejected_emails", "email_verifications", "email_source", "lead_status", "lead_score", "lead_tier", "website_url", "company_domain", "topics", "upvote_count", "launch_date", "product_hunt_url" ] }, "display": { "component": "table", "properties": { "product_name": { "label": "Product" }, "tagline": { "label": "Tagline" }, "emails": { "label": "Emails Found" }, "verified_emails": { "label": "Verified Emails" }, "risky_emails": { "label": "Other / Risky Emails" }, "rejected_emails": { "label": "Rejected Emails" }, "email_verifications": { "label": "Verification" }, "email_source": { "label": "Source" }, "lead_status": { "label": "Lead Status" }, "lead_score": { "label": "Lead Score" }, "lead_tier": { "label": "Lead Tier" }, "website_url": { "label": "Website" }, "company_domain": { "label": "Company Domain" }, "topics": { "label": "Topics" }, "upvote_count": { "label": "Upvotes" }, "launch_date": { "label": "Launch Date" }, "product_hunt_url": { "label": "PH Link" } } } }, "rankings": { "title": "🏆 Rankings", "transformation": { "fields": [ "daily_rank", "weekly_rank", "monthly_rank", "yearly_rank", "product_name", "upvote_count", "comment_count", "reviews_count", "reviews_rating", "launch_date", "featured" ] }, "display": { "component": "table", "properties": { "daily_rank": { "label": "Daily #" }, "weekly_rank": { "label": "Weekly #" }, "monthly_rank": { "label": "Monthly #" }, "yearly_rank": { "label": "Yearly #" }, "product_name": { "label": "Product" }, "upvote_count": { "label": "Upvotes" }, "comment_count": { "label": "Comments" }, "reviews_count": { "label": "Reviews" }, "reviews_rating":{ "label": "Avg Rating" }, "launch_date": { "label": "Launch Date" }, "featured": { "label": "Featured?" } } } }, "makers": { "title": "👥 Makers", "transformation": { "fields": [ "product_name", "makers", "upvote_count", "launch_date", "website_url", "product_hunt_url" ] }, "display": { "component": "table", "properties": { "product_name": { "label": "Product" }, "makers": { "label": "Makers" }, "upvote_count": { "label": "Upvotes" }, "launch_date": { "label": "Launch Date" }, "website_url": { "label": "Website" }, "product_hunt_url": { "label": "PH Link" } } } } }}{ "title": "Product Hunt Lead Generator Input", "type": "object", "schemaVersion": 1, "properties": { "mode": { "title": "Scraping Mode", "type": "string", "description": "How to find products. Use 'leaderboard' for top-ranked products by time period. Use 'search' when the user provides a keyword. Use 'topic' for a ProductHunt topic slug. Use 'urls' for specific ProductHunt URLs.", "enum": ["leaderboard", "search", "topic", "urls"], "default": "leaderboard", "editor": "select" }, "leaderboardPeriod": { "title": "Leaderboard Period", "type": "string", "description": "Only used when mode='leaderboard'. Choose 'daily', 'weekly', 'monthly', or 'yearly'.", "enum": ["daily", "weekly", "monthly", "yearly"], "default": "daily", "editor": "select" }, "searchQuery": { "title": "Search Query", "type": "string", "description": "Used when mode='search'. Filters products returned by Product Hunt's current official API window against name, tagline, description, and topics (for example 'AI writing tools').", "editor": "textfield" }, "topic": { "title": "Topic Slug", "type": "string", "description": "Used when mode='topic'. The ProductHunt topic URL slug (e.g. 'artificial-intelligence', 'developer-tools', 'productivity').", "editor": "textfield" }, "startUrls": { "title": "Start URLs", "type": "array", "description": "Used when mode='urls'. Accepts specific Product Hunt post/product-launch URLs and dated leaderboard URLs (for example /leaderboard/monthly/2026/7). Each URL is resolved through the official Product Hunt API, not by scraping Product Hunt pages.", "editor": "requestListSources", "prefill": [{"url": "https://www.producthunt.com/leaderboard/monthly/2026/7"}] }, "maxResults": { "title": "Max Results", "type": "integer", "description": "Maximum number of products to return. Default is 100.", "default": 100, "minimum": 1, "maximum": 20000, "editor": "number" }, "excludeProductHuntUrls": { "title": "Exclude Product Hunt Launch URLs", "type": "array", "description": "Optional resumable-run list. Products whose Product Hunt post or product-launch URL appears here are skipped before website or email enrichment and do not consume maxResults. Use URLs from an earlier dataset when a large week, month, or year reaches its result limit.", "editor": "json", "items": { "type": "string" } }, "productHuntResumeCursor": { "title": "Product Hunt Resume Cursor", "type": "string", "description": "Optional opaque cursor from RUN_SUMMARY.nextProductHuntCursor. Use it with the same mode, filters, date range, and exclusions to continue directly after a capped page without re-indexing all prior Product Hunt pages.", "editor": "textfield" }, "includeAllProducts": { "title": "Include All Published Products", "type": "boolean", "description": "Enabled by default so date ranges and leaderboard URLs include featured and non-featured Product Hunt posts up to maxResults. Disable to request featured posts only.", "default": true, "editor": "checkbox" }, "maxConcurrency": { "title": "Max Concurrency", "type": "integer", "description": "Maximum concurrent product-enrichment jobs. Default is 3 to keep browser and website traffic bounded.", "default": 3, "minimum": 1, "maximum": 5, "editor": "number" }, "enrichEmails": { "title": "Find Email Addresses", "type": "boolean", "description": "When enabled (the default), enriches each verified product website. It visits the homepage and up to four same-site contact, about, team, privacy, or terms pages linked from that homepage, then checks up to two public maker homepages and can use the managed contact finder if deliverable-email slots remain.", "default": true, "editor": "checkbox" }, "resolveWebsites": { "title": "Resolve Product Websites", "type": "boolean", "description": "Recommended and enabled by default. Resolves Product Hunt redirect-only links through redirect metadata, official maker sites and their product links, bounded domain hypotheses, then owner-managed search providers. Every candidate must pass product-identity and ownership checks before it is accepted.", "default": true, "editor": "checkbox" }, "maxWebsitePages": { "title": "Maximum Website Pages per Product", "type": "integer", "description": "Bounded public-site crawl limit. Default and maximum are 5 total pages: homepage plus up to four directly linked allowed pages.", "default": 5, "minimum": 1, "maximum": 5, "editor": "number" }, "maxEmailsPerProduct": { "title": "Maximum Public Emails per Product", "type": "integer", "description": "Returns and bills for up to this many ranked public email addresses associated with the product or its listed makers. Default and maximum are 3.", "default": 3, "minimum": 1, "maximum": 3, "editor": "number" }, "verifyEmails": { "title": "Verify Emails with EmailListVerify", "type": "boolean", "description": "Enabled by default. Verifies every selected public email with the Actor's managed EmailListVerify integration. Each result includes the exact provider status; invalid, disposable, and spam-trap results are never presented as verified emails.", "default": true, "editor": "checkbox" }, "findContacts": { "title": "Fill Missing Contacts", "type": "boolean", "description": "Enabled by default. After product-site and maker-homepage public discovery, use the managed EmailListVerify Contact Finder with a Product Hunt maker name first, then a domain-only fallback only if deliverable-email slots remain. Provider calls and credit ceilings remain bounded per product.", "default": true, "editor": "checkbox" }, "websiteUrlOverrides": { "title": "Approved External Website Overrides", "type": "array", "description": "Optional approved mapping for a Product Hunt product. Each item accepts productHuntUrl, productSlug, productName, or productId plus websiteUrl. A valid override takes priority over managed automatic website resolution.", "editor": "json" }, "startDate": { "title": "Start Date (YYYY-MM-DD)", "type": "string", "description": "For leaderboard daily mode: query a specific date. Leave blank for today. Use with endDate to scrape a date range in one run (e.g. startDate=2026-03-01, endDate=2026-03-07 → full week).", "editor": "textfield" }, "endDate": { "title": "End Date (YYYY-MM-DD)", "type": "string", "description": "Optional end date for multi-day scraping in one run. Only used in daily leaderboard mode. Example: startDate=2026-03-01, endDate=2026-03-07 returns all products launched across that 7-day window.", "editor": "textfield" }, "lookbackDays": { "title": "Lookback Days", "type": "integer", "description": "Alternative to startDate: scrape the last N days from today. E.g. lookbackDays=7 pulls the last week. Ignored when startDate is provided.", "default": 1, "minimum": 1, "maximum": 365, "editor": "number" }, "outputMode": { "title": "Output Mode", "type": "string", "description": "Controls what gets returned. 'full' = all fields (default). 'lean' = minimal fields optimized for LLM/AI agent context windows (name, tagline, website, upvotes, rank, emails, topics). 'leads' = full fields but only products where at least 1 email was found — perfect for outreach lists.", "enum": ["full", "lean", "leads"], "default": "full", "editor": "select" } }}{ "actorKeyValueStoreSchemaVersion": 1, "title": "Product Hunt Lead Generator KV Store", "collections": { "OUTPUT": { "title": "Run summary", "description": "Machine-readable run summary and final output metadata.", "keyPrefix": "OUTPUT" }, "RUN_SUMMARY": { "title": "Run summary alias", "description": "Alias for tools that expect a RUN_SUMMARY record.", "keyPrefix": "RUN_SUMMARY" }, "RUN_CHECKPOINT": { "title": "Resume checkpoint", "description": "Latest safe Product Hunt cursor and progress counts for recovering an interrupted long-period run.", "keyPrefix": "RUN_CHECKPOINT" } }}{ "actorOutputSchemaVersion": 1, "title": "Product Hunt Lead Generator Output", "description": "Product Hunt lead records with launch data, verified company websites, up to five assessed product-site pages, maker-homepage fallback provenance, up to three ranked contacts, exact EmailListVerify statuses, and deterministic lead scores/tiers.", "properties": { "results": { "type": "string", "title": "Scraped Products", "description": "All Product Hunt lead records. Each item contains product metadata, website URL/source/confidence, company domain, emails, verified_emails, risky_emails, exact verification records, crawl coverage, contact-finder usage, lead status/score/tier, makers, topics, and rankings.", "template": "{{links.apiDefaultDatasetUrl}}/items" }, "output": { "type": "string", "title": "Terminal output", "description": "Machine-readable outcome, item counts, named charge counts, and run warnings.", "template": "{{links.apiDefaultKeyValueStoreUrl}}/records/OUTPUT" }, "runSummary": { "type": "string", "title": "Detailed run summary", "description": "Detailed run diagnostics covering website resolution, email verification, maker fallback, contact discovery, lead tiers, and charges.", "template": "{{links.apiDefaultKeyValueStoreUrl}}/records/RUN_SUMMARY" } }}{ "actorSpecification": 1, "events": { "apify-default-dataset-item": { "eventTitle": "result", "eventDescription": "Single result in the default dataset.", "eventPriceUsd": 0.002, "isOneTimeEvent": false, "isPrimaryEvent": true }, "email-found": { "eventTitle": "Email Found", "eventDescription": "One ranked contact email returned for a Product Hunt product", "eventPriceUsd": 0.04, "isOneTimeEvent": false }, "apify-actor-start": { "eventTitle": "Actor Start", "eventDescription": "Charged when the Actor starts running. Number of events charged depends on Actor memory (one event per GB, minimum one event).", "eventPriceUsd": 0.00005, "isOneTimeEvent": true } }}1import assert from 'node:assert/strict';2import test from 'node:test';3import {4 buildWebsitePagePlan,5 canonicalWebsiteUrl,6 decodeCloudflareEmail,7 emailCandidateLimit,8 extractEmailCandidatesFromText,9 hasReachedUsableEmailTarget,10 hasUnexcludedProductHuntLaunch,11 isRetryableEmailListVerifyHttpStatus,12 normalizeEmailListVerifyStatus,13 resumableCheckpointCursor,14 productHuntLaunchKey,15 parseProductHuntSeedUrl,16 productHuntRateLimitDelaySeconds,17 productHuntStreamOrder,18 shouldUseLightweightProductHuntIndex,19 rankEmails,20} from './enrichment-utils.js';21
22test('normalizes Product Hunt launch URLs for resumable exclusions', () => {23 assert.equal(24 productHuntLaunchKey('https://www.producthunt.com/products/context-dev/launches/context-dev-2?utm_source=api#launch'),25 '/products/context-dev/launches/context-dev-2',26 );27 assert.equal(productHuntLaunchKey('https://producthunt.com/posts/notion-ai/'), '/posts/notion-ai');28});29
30test('rejects non-launch URLs as resumable exclusion keys', () => {31 assert.equal(productHuntLaunchKey('https://example.com/posts/notion-ai'), null);32 assert.equal(productHuntLaunchKey('https://www.producthunt.com/products/notion'), null);33 assert.equal(productHuntLaunchKey('not a url'), null);34});35
36test('honors Product Hunt reset seconds instead of truncating long rate-limit windows', () => {37 assert.equal(productHuntRateLimitDelaySeconds('850', null), 852);38 assert.equal(productHuntRateLimitDelaySeconds(null, '30'), 32);39 assert.equal(productHuntRateLimitDelaySeconds('not-a-number', null), 32);40 assert.equal(productHuntRateLimitDelaySeconds('2000', null), 930);41});42
43test('uses stable newest-first pagination for bounded timelines', () => {44 assert.equal(productHuntStreamOrder(true), 'NEWEST');45 assert.equal(productHuntStreamOrder(false), 'VOTES');46});47
48test('uses lightweight Product Hunt pagination only for resumable leaderboard streams', () => {49 assert.equal(shouldUseLightweightProductHuntIndex('leaderboard', 750), true);50 assert.equal(shouldUseLightweightProductHuntIndex('urls', 750), true);51 assert.equal(shouldUseLightweightProductHuntIndex('search', 750), false);52 assert.equal(shouldUseLightweightProductHuntIndex('topic', 750), false);53 assert.equal(shouldUseLightweightProductHuntIndex('leaderboard', 0), false);54 assert.equal(shouldUseLightweightProductHuntIndex('leaderboard', 750, 'opaque-page-cursor'), false);55});56
57test('detects the first page containing an unseen Product Hunt launch', () => {58 const excluded = new Set(['/posts/old-one', '/products/old-two/launches/old-two']);59 assert.equal(hasUnexcludedProductHuntLaunch([60 'https://www.producthunt.com/posts/old-one',61 'https://www.producthunt.com/products/old-two/launches/old-two?ref=api',62 ], excluded), false);63 assert.equal(hasUnexcludedProductHuntLaunch([64 'https://www.producthunt.com/posts/old-one',65 'https://www.producthunt.com/posts/new-one',66 ], excluded), true);67});68
69test('checkpoints the next page only after a page completes without new write failures', () => {70 assert.equal(resumableCheckpointCursor('page-start', true, 'page-end', 0, 0), 'page-end');71 assert.equal(resumableCheckpointCursor('page-start', true, 'page-end', 0, 1), 'page-start');72 assert.equal(resumableCheckpointCursor('', true, 'page-end', 2, 3), null);73 assert.equal(resumableCheckpointCursor('page-start', false, null, 0, 0), null);74});75
76test('parses a direct Product Hunt post URL', () => {77 assert.deepEqual(parseProductHuntSeedUrl('https://www.producthunt.com/posts/notion-ai?ref=feed'), {78 kind: 'post',79 sourceUrl: 'https://www.producthunt.com/posts/notion-ai?ref=feed',80 slug: 'notion-ai',81 });82});83
84test('parses a Product Hunt product-launch URL', () => {85 assert.deepEqual(parseProductHuntSeedUrl('https://www.producthunt.com/products/pazi/launches/pazi-2'), {86 kind: 'post',87 sourceUrl: 'https://www.producthunt.com/products/pazi/launches/pazi-2',88 slug: 'pazi-2',89 });90});91
92test('parses a monthly leaderboard URL as a stable month anchor', () => {93 assert.deepEqual(parseProductHuntSeedUrl('https://www.producthunt.com/leaderboard/monthly/2026/7'), {94 kind: 'leaderboard',95 sourceUrl: 'https://www.producthunt.com/leaderboard/monthly/2026/7',96 period: 'monthly',97 anchorDate: '2026-07-01',98 });99});100
101test('parses Product Hunt\'s current ISO-week leaderboard URL', () => {102 assert.deepEqual(parseProductHuntSeedUrl('https://www.producthunt.com/leaderboard/weekly/2026/26'), {103 kind: 'leaderboard',104 sourceUrl: 'https://www.producthunt.com/leaderboard/weekly/2026/26',105 period: 'weekly',106 anchorDate: '2026-06-22',107 });108});109
110test('supports the older explicit-date weekly leaderboard URL', () => {111 assert.deepEqual(parseProductHuntSeedUrl('https://www.producthunt.com/leaderboard/weekly/2026/7/14'), {112 kind: 'leaderboard',113 sourceUrl: 'https://www.producthunt.com/leaderboard/weekly/2026/7/14',114 period: 'weekly',115 anchorDate: '2026-07-14',116 });117});118
119test('parses a yearly leaderboard URL', () => {120 assert.deepEqual(parseProductHuntSeedUrl('https://www.producthunt.com/leaderboard/yearly/2026'), {121 kind: 'leaderboard',122 sourceUrl: 'https://www.producthunt.com/leaderboard/yearly/2026',123 period: 'yearly',124 anchorDate: '2026-01-01',125 });126});127
128test('rejects unrelated and malformed URLs', () => {129 assert.equal(parseProductHuntSeedUrl('https://example.com/posts/nope').kind, 'invalid');130 assert.equal(parseProductHuntSeedUrl('https://www.producthunt.com/r/ABC123').kind, 'invalid');131 assert.equal(parseProductHuntSeedUrl('https://www.producthunt.com/products/pazi').kind, 'invalid');132 assert.equal(parseProductHuntSeedUrl('not a url').kind, 'invalid');133});134
135test('caps the site plan at homepage plus four directly linked pages', () => {136 const plan = buildWebsitePagePlan('https://acme.com/', [137 { url: 'https://acme.com/contact', kind: 'contact', priority: 0 },138 { url: 'https://acme.com/about', kind: 'about', priority: 1 },139 { url: 'https://acme.com/team', kind: 'team', priority: 2 },140 { url: 'https://acme.com/privacy', kind: 'privacy', priority: 3 },141 { url: 'https://acme.com/terms', kind: 'terms', priority: 4 },142 ]);143 assert.equal(plan.length, 5);144 assert.deepEqual(plan.map((page) => page.kind), ['homepage', 'contact', 'about', 'team', 'privacy']);145});146
147test('allows lower-priority terms only when higher-priority categories are absent', () => {148 const plan = buildWebsitePagePlan('https://acme.com/', [149 { url: 'https://acme.com/contact', kind: 'contact', priority: 0 },150 { url: 'https://acme.com/terms', kind: 'terms', priority: 4 },151 ]);152 assert.deepEqual(plan.map((page) => page.kind), ['homepage', 'contact', 'terms']);153});154
155test('uses one linked page per category so duplicate contact links do not crowd out later fallbacks', () => {156 const plan = buildWebsitePagePlan('https://acme.com/', [157 { url: 'https://acme.com/contact', kind: 'contact', priority: 0 },158 { url: 'https://acme.com/contact-sales', kind: 'contact', priority: 0 },159 { url: 'https://acme.com/about', kind: 'about', priority: 1 },160 { url: 'https://acme.com/team', kind: 'team', priority: 2 },161 { url: 'https://acme.com/privacy', kind: 'privacy', priority: 3 },162 ]);163 assert.deepEqual(plan.map((page) => page.kind), ['homepage', 'contact', 'about', 'team', 'privacy']);164});165
166test('merges browser-discovered fallback categories into a static page plan without exceeding five pages', () => {167 const staticPlan = buildWebsitePagePlan('https://acme.com/', [168 { url: 'https://acme.com/contact', kind: 'contact', priority: 0 },169 { url: 'https://acme.com/privacy', kind: 'privacy', priority: 3 },170 ]);171 const mergedPlan = buildWebsitePagePlan('https://acme.com/', [172 ...staticPlan.slice(1),173 { url: 'https://acme.com/about', kind: 'about', priority: 1 },174 { url: 'https://acme.com/team', kind: 'team', priority: 2 },175 { url: 'https://acme.com/terms', kind: 'terms', priority: 4 },176 ]);177 assert.deepEqual(mergedPlan.map((page) => page.kind), ['homepage', 'contact', 'about', 'team', 'privacy']);178});179
180test('recovers common public email obfuscations without inventing addresses', () => {181 assert.deepEqual(extractEmailCandidatesFromText([182 'hello [at] acme [dot] com',183 'sales(at)acme(dot)io',184 String.raw`team\x40acme\x2edev`,185 String.raw`founder\u0040acme\u002eai`,186 'not an address at all',187 ].join(' ')), [188 'hello@acme.com',189 'sales@acme.io',190 'team@acme.dev',191 'founder@acme.ai',192 ]);193});194
195test('decodes Cloudflare protected public email attributes safely', () => {196 assert.equal(decodeCloudflareEmail('6d05080101022d0c0e0008430e0200'), 'hello@acme.com');197 assert.equal(decodeCloudflareEmail('not-hex'), null);198 assert.equal(decodeCloudflareEmail('00'), null);199});200
201test('prefers the site domain and caps public email output at three', () => {202 assert.deepEqual(rankEmails([203 'random@vendor.com', 'support@acme.com', 'hello@acme.com', 'sales@acme.com', 'noreply@acme.com',204 ], 'acme.com'), ['hello@acme.com', 'sales@acme.com', 'support@acme.com']);205});206
207test('keeps a bounded reserve of email candidates for post-crawl verification', () => {208 assert.equal(emailCandidateLimit(3), 6);209 assert.deepEqual(rankEmails([210 'hello@acme.com', 'sales@acme.com', 'support@acme.com',211 'team@acme.com', 'press@acme.com', 'privacy@acme.com', 'legal@acme.com',212 ], 'acme.com', emailCandidateLimit(3)), [213 'hello@acme.com', 'sales@acme.com', 'team@acme.com',214 'support@acme.com', 'press@acme.com', 'privacy@acme.com',215 ]);216});217
218test('verification reserve stops only after the requested usable-email target', () => {219 assert.equal(hasReachedUsableEmailTarget(['dead_server', 'ok', 'provider_error'], 3), false);220 assert.equal(hasReachedUsableEmailTarget(['ok', 'ok_for_all', 'smtp_protocol'], 3), true);221 assert.equal(hasReachedUsableEmailTarget(['ok', 'email_disabled', 'invalid_mx'], 1), true);222});223
224test('ranks outreach roles ahead of compliance-only inboxes', () => {225 assert.deepEqual(rankEmails([226 'privacy@acme.com', 'founder@acme.com', 'contact@acme.com', 'legal@acme.com',227 ], 'acme.com'), ['founder@acme.com', 'contact@acme.com', 'privacy@acme.com']);228});229
230test('filters placeholders and no-reply values before ranking', () => {231 assert.deepEqual(rankEmails([232 'example@example.com', 'noreply@acme.com', 'contact@acme.com', 'person@other.test',233 ], 'acme.com'), ['contact@acme.com', 'person@other.test']);234});235
236test('normalizes only documented EmailListVerify statuses', () => {237 assert.equal(normalizeEmailListVerifyStatus('ok'), 'ok');238 assert.equal(normalizeEmailListVerifyStatus('ok-for-all'), 'ok_for_all');239 assert.equal(normalizeEmailListVerifyStatus('error_credit'), 'error_credit');240 assert.equal(normalizeEmailListVerifyStatus('unexpected response'), 'provider_error');241});242
243test('retries only transient EmailListVerify transport statuses', () => {244 assert.equal(isRetryableEmailListVerifyHttpStatus(408), true);245 assert.equal(isRetryableEmailListVerifyHttpStatus(429), true);246 assert.equal(isRetryableEmailListVerifyHttpStatus(503), true);247 assert.equal(isRetryableEmailListVerifyHttpStatus(400), false);248 assert.equal(isRetryableEmailListVerifyHttpStatus(402), false);249});250
251test('rejects private and malformed web targets', () => {252 assert.equal(canonicalWebsiteUrl('http://127.0.0.1/internal'), null);253 assert.equal(canonicalWebsiteUrl('https://localhost:3000'), null);254 assert.equal(canonicalWebsiteUrl('https://user:pass@acme.com'), null);255 assert.equal(canonicalWebsiteUrl('https://acme.com:8443'), null);256 assert.equal(canonicalWebsiteUrl('http://100.64.0.1/internal'), null);257 assert.equal(canonicalWebsiteUrl('http://203.0.113.7/internal'), null);258 assert.equal(canonicalWebsiteUrl('http://[::1]/internal'), null);259 assert.equal(canonicalWebsiteUrl('https://acme.com/contact/?ref=ph#footer'), 'https://acme.com/contact?ref=ph');260});1export const DEFAULT_MAX_EMAILS_PER_PRODUCT = 3;2export const MAX_EMAILS_PER_PRODUCT = 3;3export const MAX_EMAIL_CANDIDATES_PER_PRODUCT = MAX_EMAILS_PER_PRODUCT * 2;4export const DEFAULT_MAX_WEBSITE_PAGES = 5;5export const MAX_WEBSITE_PAGES = 5;6
7export type FallbackPageKind = 'homepage' | 'contact' | 'about' | 'team' | 'privacy' | 'terms';8
9export type WebsitePage = {10 url: string;11 kind: FallbackPageKind;12 priority: number;13};14
15export type ProductHuntSeed =16 | { kind: 'post'; sourceUrl: string; slug: string }17 | { kind: 'leaderboard'; sourceUrl: string; period: 'daily' | 'weekly' | 'monthly' | 'yearly'; anchorDate: string }18 | { kind: 'invalid'; sourceUrl: string; reason: string };19
20export const EMAIL_LIST_VERIFY_STATUSES = [21 'ok',22 'unknown',23 'dead_server',24 'invalid_mx',25 'email_disabled',26 'antispam_system',27 'ok_for_all',28 'smtp_protocol',29 'invalid_syntax',30 'disposable',31 'spamtrap',32 'error_credit',33] as const;34
35export type EmailListVerifyStatus = typeof EMAIL_LIST_VERIFY_STATUSES[number] | 'provider_error' | 'not_configured' | 'not_requested';36
37export function normalizeEmailListVerifyStatus(value: string): EmailListVerifyStatus {38 const normalized = value.trim().toLowerCase().replace(/[\s-]+/g, '_');39 return (EMAIL_LIST_VERIFY_STATUSES as readonly string[]).includes(normalized)40 ? normalized as EmailListVerifyStatus41 : 'provider_error';42}43
44export function isRetryableEmailListVerifyHttpStatus(status: number): boolean {45 return status === 408 || status === 429 || status >= 500;46}47
48export function emailCandidateLimit(maxEmails: unknown): number {49 return clampInteger(maxEmails, DEFAULT_MAX_EMAILS_PER_PRODUCT, 1, MAX_EMAILS_PER_PRODUCT) * 2;50}51
52const SETTLED_USABLE_EMAIL_STATUSES = new Set<EmailListVerifyStatus>([53 'ok', 'ok_for_all', 'unknown', 'antispam_system', 'smtp_protocol',54]);55
56export function hasReachedUsableEmailTarget(statuses: EmailListVerifyStatus[], maxEmails: unknown): boolean {57 const target = clampInteger(maxEmails, DEFAULT_MAX_EMAILS_PER_PRODUCT, 1, MAX_EMAILS_PER_PRODUCT);58 return statuses.filter((status) => SETTLED_USABLE_EMAIL_STATUSES.has(status)).length >= target;59}60
61export function decodeCloudflareEmail(encoded: string): string | null {62 if (!/^[0-9a-f]+$/i.test(encoded) || encoded.length < 4 || encoded.length % 2 !== 0) return null;63 const key = Number.parseInt(encoded.slice(0, 2), 16);64 let decoded = '';65 for (let index = 2; index < encoded.length; index += 2) {66 decoded += String.fromCharCode(Number.parseInt(encoded.slice(index, index + 2), 16) ^ key);67 }68 return decoded.includes('@') ? decoded : null;69}70
71export function extractEmailCandidatesFromText(text: string): string[] {72 const normalized = text73 .replace(/\\(?:x40|u0040)/gi, '@')74 .replace(/\\(?:x2e|u002e)/gi, '.')75 .replace(/&#(?:64|x40);|@/gi, '@')76 .replace(/&#(?:46|x2e);|./gi, '.');77 const obfuscatedPattern = /\b[a-zA-Z0-9._%+\-]+\s*(?:\[at\]|\(at\)|\sat\s)\s*[a-zA-Z0-9.\-]+\s*(?:\[dot\]|\(dot\)|\sdot\s)\s*[a-zA-Z]{2,}\b/gi;78 const deobfuscated = normalized.replace(obfuscatedPattern, (match) => match79 .replace(/\s*(?:\[at\]|\(at\)|\sat\s)\s*/i, '@')80 .replace(/\s*(?:\[dot\]|\(dot\)|\sdot\s)\s*/ig, '.'));81 const plainPattern = /\b[a-zA-Z0-9._%+\-]+@[a-zA-Z0-9.\-]+\.[a-zA-Z]{2,}\b/g;82 const found = deobfuscated.match(plainPattern) ?? [];83 return [...new Set(found.map((email) => email.toLowerCase()))];84}85
86export function clampInteger(value: unknown, fallback: number, minimum: number, maximum: number): number {87 const numeric = Number(value);88 if (!Number.isFinite(numeric)) return fallback;89 return Math.min(maximum, Math.max(minimum, Math.floor(numeric)));90}91
92export function isProductHuntHost(hostname: string): boolean {93 const host = hostname.toLowerCase();94 return host === 'producthunt.com' || host.endsWith('.producthunt.com');95}96
97export function productHuntLaunchKey(rawUrl: string): string | null {98 try {99 const url = new URL(rawUrl);100 if (!isProductHuntHost(url.hostname)) return null;101 const parts = url.pathname.split('/').filter(Boolean);102 const launchesIndex = parts.indexOf('launches');103 const isPost = parts[0] === 'posts' && Boolean(parts[1]);104 const isLaunch = parts[0] === 'products' && launchesIndex > 0 && Boolean(parts[launchesIndex + 1]);105 if (!isPost && !isLaunch) return null;106 return `/${parts.join('/')}`.toLowerCase();107 } catch {108 return null;109 }110}111
112export function productHuntRateLimitDelaySeconds(resetHeader: string | null, retryAfterHeader: string | null): number {113 const parsed = Number(resetHeader ?? retryAfterHeader ?? 30);114 const seconds = Number.isFinite(parsed) && parsed >= 0 ? parsed : 30;115 // Product Hunt documents X-Rate-Limit-Reset as seconds remaining in its116 // 15-minute quota window. Add a small boundary cushion and permit the full117 // documented window instead of repeatedly retrying every two minutes.118 return Math.min(930, Math.max(5, Math.ceil(seconds) + 2));119}120
121export function productHuntStreamOrder(hasBoundedDateRange: boolean): 'NEWEST' | 'VOTES' {122 // Vote totals can change while a long historical stream is being paged,123 // moving launches across cursors. Creation time is stable for bounded124 // daily/weekly/monthly/yearly discovery and each row still carries ranks.125 return hasBoundedDateRange ? 'NEWEST' : 'VOTES';126}127
128export function shouldUseLightweightProductHuntIndex(mode: string, exclusionCount: number, resumeCursor = ''): boolean {129 return !resumeCursor && exclusionCount > 0 && (mode === 'leaderboard' || mode === 'urls');130}131
132export function hasUnexcludedProductHuntLaunch(urls: string[], excludedKeys: Set<string>): boolean {133 return urls.some((url) => {134 const key = productHuntLaunchKey(url);135 return Boolean(key && !excludedKeys.has(key));136 });137}138
139export function resumableCheckpointCursor(140 pageStartCursor: string | null,141 hasNextPage: boolean,142 endCursor: string | null | undefined,143 failuresBeforePage: number,144 failuresAfterPage: number,145): string | null {146 if (failuresAfterPage > failuresBeforePage) return pageStartCursor || null;147 return hasNextPage && endCursor ? endCursor : null;148}149
150export function isPublicHttpUrl(rawUrl: string): boolean {151 try {152 const url = new URL(rawUrl);153 if (url.protocol !== 'http:' && url.protocol !== 'https:') return false;154 if (url.username || url.password) return false;155 if (url.port && url.port !== '80' && url.port !== '443') return false;156 const host = url.hostname.toLowerCase().replace(/^\[|\]$/g, '');157 if (!host || host === 'localhost' || host.endsWith('.localhost') || host.endsWith('.local')) return false;158 // Literal IPv6 targets are not needed for product websites and are159 // rejected outright, including loopback and IPv4-mapped addresses.160 if (host.includes(':')) return false;161
162 const ipv4 = host.match(/^(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})$/);163 if (!ipv4) return true;164 const octets = ipv4.slice(1).map(Number);165 if (octets.some((part) => part > 255)) return false;166 const [a, b] = octets;167 return !(168 a === 0 || a === 10 || a === 127 || a >= 224 ||169 (a === 100 && b >= 64 && b <= 127) ||170 (a === 169 && b === 254) ||171 (a === 172 && b >= 16 && b <= 31) ||172 (a === 192 && (b === 0 || b === 168)) ||173 (a === 198 && (b === 18 || b === 19 || b === 51)) ||174 (a === 203 && b === 0)175 );176 } catch {177 return false;178 }179}180
181export function canonicalWebsiteUrl(rawUrl: string): string | null {182 if (!isPublicHttpUrl(rawUrl)) return null;183 try {184 const url = new URL(rawUrl);185 url.hash = '';186 url.pathname = url.pathname.replace(/\/+$/, '') || '/';187 return url.toString();188 } catch {189 return null;190 }191}192
193function validDateParts(year: string | undefined, month: string | undefined, day: string | undefined): string | null {194 if (!year || !month) return null;195 const y = Number(year);196 const m = Number(month);197 const d = day ? Number(day) : 1;198 if (!Number.isInteger(y) || !Number.isInteger(m) || !Number.isInteger(d) || m < 1 || m > 12 || d < 1 || d > 31) return null;199 const date = new Date(Date.UTC(y, m - 1, d));200 if (date.getUTCFullYear() !== y || date.getUTCMonth() !== m - 1 || date.getUTCDate() !== d) return null;201 return date.toISOString().slice(0, 10);202}203
204function isoWeekStart(year: string | undefined, week: string | undefined): string | null {205 if (!year || !week) return null;206 const y = Number(year);207 const w = Number(week);208 if (!Number.isInteger(y) || !Number.isInteger(w) || w < 1 || w > 53) return null;209
210 // ISO week 1 is the week that contains January 4 and begins on Monday.211 const januaryFourth = new Date(Date.UTC(y, 0, 4));212 const januaryFourthDay = januaryFourth.getUTCDay() || 7;213 const firstMonday = new Date(Date.UTC(y, 0, 4 - januaryFourthDay + 1));214 const start = new Date(firstMonday);215 start.setUTCDate(firstMonday.getUTCDate() + (w - 1) * 7);216
217 const nextJanuaryFourth = new Date(Date.UTC(y + 1, 0, 4));218 const nextJanuaryFourthDay = nextJanuaryFourth.getUTCDay() || 7;219 const nextFirstMonday = new Date(Date.UTC(y + 1, 0, 4 - nextJanuaryFourthDay + 1));220 if (start >= nextFirstMonday) return null;221 return start.toISOString().slice(0, 10);222}223
224export function parseProductHuntSeedUrl(rawUrl: string): ProductHuntSeed {225 let url: URL;226 try {227 url = new URL(rawUrl);228 } catch {229 return { kind: 'invalid', sourceUrl: rawUrl, reason: 'URL is malformed.' };230 }231 if (!isProductHuntHost(url.hostname)) {232 return { kind: 'invalid', sourceUrl: rawUrl, reason: 'URL must use producthunt.com.' };233 }234
235 const parts = url.pathname.split('/').filter(Boolean);236 if (parts[0] === 'leaderboard') {237 const period = parts[1];238 if (!['daily', 'weekly', 'monthly', 'yearly'].includes(period)) {239 return { kind: 'invalid', sourceUrl: rawUrl, reason: 'Leaderboard URL must include daily, weekly, monthly, or yearly.' };240 }241 const anchorDate = period === 'weekly'242 ? parts[4]243 // Support the older explicit date form as well as Product Hunt's244 // current ISO-week form: /leaderboard/weekly/2026/26.245 ? validDateParts(parts[2], parts[3], parts[4])246 : isoWeekStart(parts[2], parts[3])247 : period === 'monthly'248 ? validDateParts(parts[2], parts[3], undefined)249 : period === 'yearly'250 ? validDateParts(parts[2], '1', undefined)251 : validDateParts(parts[2], parts[3], parts[4]);252 if (!anchorDate) {253 return { kind: 'invalid', sourceUrl: rawUrl, reason: 'Leaderboard URL must include a valid period-specific date (daily YYYY/M/D, weekly YYYY/week, monthly YYYY/M, yearly YYYY).' };254 }255 return { kind: 'leaderboard', sourceUrl: url.toString(), period: period as 'daily' | 'weekly' | 'monthly' | 'yearly', anchorDate };256 }257
258 if (parts[0] === 'r') {259 return { kind: 'invalid', sourceUrl: rawUrl, reason: 'Product Hunt redirect URLs are not valid inputs. Use the Product Hunt post/product/leaderboard URL or an approved external website override.' };260 }261
262 const launchesIndex = parts.indexOf('launches');263 if (parts[0] === 'products' && launchesIndex < 0) {264 return { kind: 'invalid', sourceUrl: rawUrl, reason: 'Product root URLs do not identify a specific launch in the official API. Use the Product Hunt post URL or a /products/{product}/launches/{launch} URL.' };265 }266 const slug = parts[0] === 'posts'267 ? parts[1]268 : launchesIndex >= 0269 ? parts[launchesIndex + 1]270 : undefined;271 if (!slug) {272 return { kind: 'invalid', sourceUrl: rawUrl, reason: 'URL must be a Product Hunt post, product launch, or dated leaderboard.' };273 }274 return { kind: 'post', sourceUrl: url.toString(), slug };275}276
277const PLACEHOLDER_EMAIL = /(?:example\.|test@|placeholder|noreply|no-reply|@sentry\.|@logrocket\.|@segment\.|your@|email@email|\d{10,}@)/i;278const EMAIL_PATTERN = /^[a-zA-Z0-9._%+\-]+@[a-zA-Z0-9.\-]+\.[a-zA-Z]{2,}$/;279
280export function rankEmails(candidates: string[], baseDomain: string, maxEmails = DEFAULT_MAX_EMAILS_PER_PRODUCT): string[] {281 const boundedMax = clampInteger(maxEmails, DEFAULT_MAX_EMAILS_PER_PRODUCT, 1, MAX_EMAIL_CANDIDATES_PER_PRODUCT);282 const normalizedDomain = baseDomain.replace(/^www\./i, '').toLowerCase();283 const unique: Array<{ email: string; score: number; index: number }> = [];284 const seen = new Set<string>();285 for (const [index, raw] of candidates.entries()) {286 const email = raw.trim().toLowerCase().replace(/^mailto:/, '').split('?')[0];287 if (!EMAIL_PATTERN.test(email) || PLACEHOLDER_EMAIL.test(email) || seen.has(email)) continue;288 seen.add(email);289 const domain = email.split('@')[1] ?? '';290 const sameDomain = domain === normalizedDomain || domain.endsWith(`.${normalizedDomain}`) || normalizedDomain.endsWith(`.${domain}`);291 const localPart = email.split('@')[0] ?? '';292 const highValueRole = /^(hello|contact|info|team|sales|partners|business|founders?)$/i.test(localPart);293 const serviceRole = /^(support|success|press|media)$/i.test(localPart);294 const complianceRole = /^(privacy|legal|abuse|security|dpo)$/i.test(localPart);295 unique.push({296 email,297 score: (sameDomain ? 10 : 0) + (highValueRole ? 4 : serviceRole ? 2 : complianceRole ? -2 : 0),298 index,299 });300 }301 return unique302 .sort((a, b) => b.score - a.score || a.index - b.index)303 .slice(0, boundedMax)304 .map(({ email }) => email);305}306
307export function buildWebsitePagePlan(homepageUrl: string, candidates: WebsitePage[], maxPages = DEFAULT_MAX_WEBSITE_PAGES): WebsitePage[] {308 const boundedPages = clampInteger(maxPages, DEFAULT_MAX_WEBSITE_PAGES, 1, MAX_WEBSITE_PAGES);309 const homepage = canonicalWebsiteUrl(homepageUrl);310 if (!homepage) return [];311 const seen = new Set([homepage]);312 const seenKinds = new Set<FallbackPageKind>();313 const fallback = candidates314 .filter((candidate) => candidate.kind !== 'homepage')315 .sort((a, b) => a.priority - b.priority)316 .map((candidate) => ({ ...candidate, url: canonicalWebsiteUrl(candidate.url) }))317 .filter((candidate): candidate is WebsitePage & { url: string } => Boolean(candidate.url))318 .filter((candidate) => {319 if (seen.has(candidate.url) || seenKinds.has(candidate.kind)) return false;320 seen.add(candidate.url);321 seenKinds.add(candidate.kind);322 return true;323 })324 .slice(0, Math.max(0, boundedPages - 1));325 return [{ url: homepage, kind: 'homepage', priority: -1 }, ...fallback];326}1import assert from 'node:assert/strict';2import test from 'node:test';3import {4 buildLeadProfile,5 buildDomainHypothesisCandidates,6 buildProtectedProductCandidates,7 buildProductIdentityNames,8 buildWebsiteSearchQuery,9 buildWebsiteSearchQueries,10 evaluateWebsiteIdentity,11 hasDirectOwnershipSignal,12 hasProtectedProductVendorMatch,13 isRetryableDataForSeoStatus,14 isPrivateNetworkAddress,15 isProtectedProductHostAllowed,16 isUsableEmailStatus,17 matchesProductSearch,18 normalizeContactFinderResults,19 parseDataForSeoCandidates,20 parseSerpApiCandidates,21 rankMakerWebsiteLinkCandidates,22 scoreSearchCandidate,23 selectMakerFallbackEmails,24 splitMakerName,25} from './lead-engine.js';26
27test('builds a bounded plain-language website query without paid search operators', () => {28 const query = buildWebsiteSearchQuery('Acme AI', 'Automate customer support');29 assert.equal(query, 'Acme AI Automate customer support official website');30 assert.doesNotMatch(query, /\b(?:site|intitle|inurl):/i);31});32
33test('builds at most two cost-bounded website queries using a conservative core brand alias', () => {34 assert.deepEqual(buildProductIdentityNames('MailAdept by mailwarm'), ['MailAdept by mailwarm', 'MailAdept']);35 assert.deepEqual(buildProductIdentityNames('Modelence Mobile Builder'), ['Modelence Mobile Builder', 'Modelence']);36 assert.deepEqual(buildProductIdentityNames('OASIS 1 Ring'), ['OASIS 1 Ring', 'OASIS']);37 assert.deepEqual(buildProductIdentityNames('N71'), ['N71']);38 assert.deepEqual(buildWebsiteSearchQueries('Modelence Mobile Builder', 'Build mobile apps with AI'), [39 'Modelence Mobile Builder Build mobile apps with AI official website',40 'Modelence Build mobile apps with AI official website',41 ]);42});43
44test('retries only transient DataForSEO task failures', () => {45 for (const status of [40101, 40103, 50301, 50303, 50401]) assert.equal(isRetryableDataForSeoStatus(status), true);46 for (const status of [20000, 40100, 40102, 40200]) assert.equal(isRetryableDataForSeoStatus(status), false);47});48
49test('parses DataForSEO organic candidates and provider cost', () => {50 const parsed = parseDataForSeoCandidates({51 tasks: [{ cost: 0.002, result: [{ items: [52 { type: 'organic', url: 'https://acme.ai/', title: 'Acme AI', description: 'Official site', rank_group: 1 },53 { type: 'people_also_ask', url: 'https://example.com/' },54 ] }] }],55 });56 assert.equal(parsed.costUsd, 0.002);57 assert.deepEqual(parsed.candidates, [{ url: 'https://acme.ai/', title: 'Acme AI', snippet: 'Official site', rank: 1 }]);58});59
60test('parses SerpApi organic candidates', () => {61 assert.deepEqual(parseSerpApiCandidates({ organic_results: [62 { link: 'https://acme.ai/', title: 'Acme AI', snippet: 'Official site', position: 2 },63 ] }), [{ url: 'https://acme.ai/', title: 'Acme AI', snippet: 'Official site', rank: 2 }]);64});65
66test('filters Product Hunt search locally using product text and topics', () => {67 const product = {68 name: 'WriterFlow',69 tagline: 'Draft marketing copy with AI',70 description: 'A writing workspace for growth teams',71 topics: ['Artificial Intelligence', 'Marketing'],72 };73 assert.equal(matchesProductSearch('AI writing tool', product), true);74 assert.equal(matchesProductSearch('database monitoring', product), false);75});76
77test('blocks directory results even when their title contains the product name', () => {78 for (const url of [79 'https://www.g2.com/products/acme-ai',80 'https://launly.com/products/acme-ai',81 'https://www.needbase.net/product/acme-ai',82 'https://tools.launchllama.co/acme-ai',83 'https://learn.microsoft.com/en-us/tools/acme-ai',84 ]) {85 const score = scoreSearchCandidate({ url, title: 'Acme AI reviews', snippet: 'Software listing', rank: 1 }, 'Acme AI', 'Automate support');86 assert.equal(score, -1, url);87 }88});89
90test('builds a bounded direct-domain recovery set for distinctive product names', () => {91 assert.deepEqual(buildDomainHypothesisCandidates('Browser Notes').map(({ url }) => url), [92 'https://browsernotes.com/',93 'https://browsernotes.ai/',94 'https://browsernotes.app/',95 'https://browsernotes.io/',96 'https://browsernotes.xyz/',97 ]);98 assert.deepEqual(buildDomainHypothesisCandidates('N71').map(({ url }) => url).slice(0, 2), [99 'https://n71.com/',100 'https://n71.ai/',101 ]);102});103
104test('blocks private, loopback, link-local, documentation, and mapped addresses', () => {105 for (const address of ['127.0.0.1', '10.0.0.5', '169.254.169.254', '192.168.1.1', '203.0.113.9', '::1', 'fd00::1', 'fe80::1', '::ffff:127.0.0.1']) {106 assert.equal(isPrivateNetworkAddress(address), true, address);107 }108 assert.equal(isPrivateNetworkAddress('8.8.8.8'), false);109 assert.equal(isPrivateNetworkAddress('2606:4700:4700::1111'), false);110});111
112test('restricts official foundation-model launches to their vendor domains', () => {113 assert.equal(isProtectedProductHostAllowed('Claude Sonnet 5', 'https://www.anthropic.com/news/claude-sonnet-5'), true);114 assert.equal(isProtectedProductHostAllowed('Claude Sonnet 5', 'https://claudesonnet.io/'), false);115 assert.equal(isProtectedProductHostAllowed('Gemini Omni Flash', 'https://ai.google.dev/gemini-api/docs'), true);116 assert.equal(isProtectedProductHostAllowed('Gemini Omni Flash', 'https://geminiomniflash.ai/'), false);117 assert.equal(isProtectedProductHostAllowed('Claude Desktop Buddy', 'https://indiehardware.example/'), true);118 assert.equal(hasProtectedProductVendorMatch('Claude Sonnet 5', 'https://www.anthropic.com/news/claude-sonnet-5'), true);119 assert.equal(hasProtectedProductVendorMatch('Claude Desktop Buddy', 'https://www.anthropic.com/'), false);120 assert.deepEqual(buildProtectedProductCandidates('Claude Sonnet 5').map(({ url }) => url), [121 'https://www.anthropic.com/news/claude-sonnet-5',122 'https://claude.com/',123 ]);124 assert.deepEqual(buildProtectedProductCandidates('Gemini Omni Flash').map(({ url }) => url), [125 'https://gemini.google/overview/video-generation',126 'https://ai.google.dev/',127 ]);128});129
130test('accepts a search candidate only when its homepage confirms product identity', () => {131 const candidate = { url: 'https://acme.ai/', title: 'Acme AI — Official', snippet: 'Automate support', rank: 1 };132 const identity = evaluateWebsiteIdentity({133 candidate,134 productName: 'Acme AI',135 tagline: 'Automate customer support',136 pageTitle: 'Acme AI — Automate customer support',137 pageDescription: 'Acme AI helps support teams move faster.',138 pageText: 'Acme AI is the customer support automation platform.',139 });140 assert.equal(identity.accepted, true);141 assert.ok(identity.confidence >= 70);142});143
144test('rejects a plausible search result when the homepage is unrelated', () => {145 const candidate = { url: 'https://acmeventures.example/', title: 'Acme AI', snippet: 'Result', rank: 1 };146 const identity = evaluateWebsiteIdentity({147 candidate,148 productName: 'Acme AI',149 tagline: 'Automate customer support',150 pageTitle: 'Acme Ventures',151 pageDescription: 'Real estate investment company',152 pageText: 'We buy and operate commercial property.',153 });154 assert.equal(identity.accepted, false);155});156
157test('rejects a same-name single-token product when its purpose does not match', () => {158 const candidate = { url: 'https://openacti.com/', title: 'Acti — Agentic Keyboard', snippet: 'AI keyboard', rank: 1 };159 const identity = evaluateWebsiteIdentity({160 candidate,161 productName: 'Acti',162 tagline: 'Your life and daily events journal',163 pageTitle: "Acti - The World's First Agentic Keyboard",164 pageDescription: 'A unified command layer for apps and APIs.',165 pageText: 'Acti is an agentic keyboard for iOS and Android.',166 });167 assert.equal(identity.accepted, false);168 assert.ok(identity.signals.includes('single_token_tagline_mismatch'));169});170
171test('accepts a core-brand domain when the homepage confirms the launch purpose', () => {172 const identity = evaluateWebsiteIdentity({173 candidate: { url: 'https://modelence.com/', title: 'Modelence', snippet: 'Build mobile apps with AI', rank: 1 },174 productName: 'Modelence',175 tagline: 'Build production-ready mobile apps with AI',176 pageTitle: 'Modelence — AI Mobile App Builder',177 pageDescription: 'Build and publish production-ready mobile applications with AI.',178 pageText: 'Modelence is the full-stack mobile app builder for teams.',179 });180 assert.equal(identity.accepted, true);181 assert.ok(identity.signals.includes('full_product_name_matches_domain'));182});183
184test('requires a direct domain-ownership signal before trusting an exact product page', () => {185 assert.equal(hasDirectOwnershipSignal(['product_name_matches_page_title', 'tagline_matches_page']), false);186 assert.equal(hasDirectOwnershipSignal(['product_name_matches_domain', 'product_name_matches_page_title']), true);187 assert.equal(hasDirectOwnershipSignal(['full_product_name_matches_domain']), true);188});189
190test('accepts a distinctive alphanumeric name on its exact domain without rendered purpose text', () => {191 const identity = evaluateWebsiteIdentity({192 candidate: { url: 'https://n71.ai/', title: 'N71', snippet: '', rank: 2 },193 productName: 'N71',194 tagline: 'Mission control for coordinated AI agents',195 pageTitle: 'N71',196 pageDescription: '',197 pageText: '',198 });199 assert.equal(identity.accepted, true);200 assert.ok(identity.signals.includes('distinctive_alphanumeric_domain'));201});202
203test('ranks product links found on an authoritative maker homepage', () => {204 const candidates = rankMakerWebsiteLinkCandidates([205 {206 url: 'https://getacti.app/',207 text: 'Acti — Travel tracker & life journal for iOS',208 context: 'My pet project. Your life, remembered automatically.',209 rank: 1,210 },211 {212 url: 'https://www.linkedin.com/in/example/',213 text: 'LinkedIn',214 context: 'Find me on LinkedIn.',215 rank: 2,216 },217 ], 'https://denismilovanov.com/', 'Acti', 'Your life and daily events journal');218 assert.deepEqual(candidates.map(({ url }) => url), ['https://getacti.app/']);219});220
221test('uses maker-homepage emails only to fill gaps after product-site emails', () => {222 assert.deepEqual(selectMakerFallbackEmails(223 ['hello@acme.ai'],224 ['hello@acme.ai', 'founder@example.dev', 'team@example.dev'],225 2,226 ), ['founder@example.dev']);227});228
229test('lets verified maker emails replace rejected product-site candidates without duplicating them', () => {230 assert.deepEqual(selectMakerFallbackEmails(231 ['bad@acme.ai', 'duplicate@acme.ai'],232 ['duplicate@acme.ai', 'founder@example.dev', 'team@example.dev'],233 2,234 0,235 ), ['founder@example.dev', 'team@example.dev']);236});237
238test('splits maker names for lower-cost named contact discovery', () => {239 assert.deepEqual(splitMakerName('Ada Lovelace'), { firstName: 'Ada', lastName: 'Lovelace' });240 assert.deepEqual(splitMakerName('Prince'), { firstName: 'Prince' });241 assert.deepEqual(splitMakerName('🤖'), {});242 assert.deepEqual(splitMakerName('[REDACTED]'), {});243});244
245test('keeps only same-domain deliverable high/medium contact-finder results', () => {246 const results = normalizeContactFinderResults([247 { email: 'ada@acme.ai', confidence: 'high', result: 'ok' },248 { email: 'team@acme.ai', confidence: 'medium', result: 'ok' },249 { email: 'bad@acme.ai', confidence: 'high', result: 'invalid_mx' },250 { email: 'ada@gmail.com', confidence: 'high', result: 'ok' },251 { email: 'ada@acme.ai', confidence: 'high', result: 'ok' },252 ], 'acme.ai', 3);253 assert.deepEqual(results.map(({ email }) => email), ['ada@acme.ai', 'team@acme.ai']);254});255
256test('excludes hard-invalid verification outcomes from usable lead contacts', () => {257 for (const status of ['dead_server', 'invalid_mx', 'email_disabled', 'invalid_syntax', 'disposable', 'spamtrap']) {258 assert.equal(isUsableEmailStatus(status), false, status);259 }260 for (const status of ['ok', 'ok_for_all', 'unknown', 'provider_error', 'error_credit']) {261 assert.equal(isUsableEmailStatus(status), true, status);262 }263});264
265test('scores a deliverable published contact as an A-tier lead', () => {266 const profile = buildLeadProfile({267 websiteResolved: true,268 websiteConfidence: 90,269 emailStatuses: ['ok', 'ok'],270 publicEmailCount: 1,271 upvotes: 500,272 pagesVisited: 5,273 });274 assert.equal(profile.leadStatus, 'verified_contact');275 assert.equal(profile.leadTier, 'A');276 assert.ok(profile.leadScore >= 75);277});278
279test('does not inflate an unresolved product into a lead', () => {280 assert.deepEqual(buildLeadProfile({281 websiteResolved: false,282 websiteConfidence: 0,283 emailStatuses: [],284 publicEmailCount: 0,285 upvotes: 0,286 pagesVisited: 0,287 }), {288 leadStatus: 'unresolved_website',289 leadScore: 0,290 leadTier: 'unqualified',291 signals: [],292 });293});1export type SearchCandidate = {2 url: string;3 title: string;4 snippet: string;5 rank: number;6};7
8export type MakerWebsiteLink = {9 url: string;10 text: string;11 context: string;12 rank: number;13};14
15export type WebsiteIdentity = {16 accepted: boolean;17 confidence: number;18 signals: string[];19};20
21export type ContactFinderResult = {22 email: string;23 confidence: 'high' | 'medium' | 'low' | 'unknown';24 result: string;25};26
27export type LeadProfile = {28 leadStatus: 'verified_contact' | 'catch_all_contact' | 'unverified_contact' | 'no_contact' | 'unresolved_website';29 leadScore: number;30 leadTier: 'A' | 'B' | 'C' | 'unqualified';31 signals: string[];32};33
34const DIRECTORY_HOSTS = new Set([35 'alternativeto.net', 'appsumo.com', 'apps.apple.com', 'capterra.com',36 'crunchbase.com', 'discord.com', 'discord.gg', 'facebook.com', 'g2.com',37 'github.com', 'indiehackers.com', 'instagram.com', 'linkedin.com',38 'launly.com', 'learn.microsoft.com', 'medium.com', 'microlaunch.net',39 'needbase.net', 'play.google.com', 'producthunt.com', 'reddit.com',40 'saasworthy.com', 'sourceforge.net', 'stackshare.io', 'substack.com',41 'theresanaiforthat.com', 'toolify.ai', 'trustpilot.com', 'twitter.com',42 'tools.launchllama.co', 'uneed.best', 'wikipedia.org', 'x.com', 'youtube.com',43]);44
45const STOP_WORDS = new Set([46 'a', 'ai', 'an', 'and', 'app', 'by', 'for', 'from', 'in', 'io', 'of', 'on',47 'platform', 'software', 'the', 'to', 'tool', 'tools', 'with',48]);49
50const EMAIL_PATTERN = /^[a-zA-Z0-9._%+\-]+@[a-zA-Z0-9.\-]+\.[a-zA-Z]{2,}$/;51const HARD_REJECTED_EMAIL_STATUSES = new Set([52 'dead_server', 'invalid_mx', 'email_disabled', 'invalid_syntax', 'disposable', 'spamtrap',53]);54
55function normalizedText(value: unknown): string {56 return String(value ?? '')57 .normalize('NFKD')58 .replace(/[\u0300-\u036f]/g, '')59 .toLowerCase()60 .replace(/[^a-z0-9]+/g, ' ')61 .trim();62}63
64function identityTokens(value: unknown): string[] {65 const all = normalizedText(value).split(/\s+/).filter(Boolean);66 const meaningful = all.filter((token) => token.length >= 2 && !STOP_WORDS.has(token));67 return [...new Set(meaningful.length > 0 ? meaningful : all.filter((token) => token.length >= 2))];68}69
70function coverage(tokens: string[], haystack: string): number {71 if (tokens.length === 0) return 0;72 const matches = tokens.filter((token) => haystack.includes(token)).length;73 return matches / tokens.length;74}75
76function hostnameOf(rawUrl: string): string {77 try {78 return new URL(rawUrl).hostname.toLowerCase().replace(/^www\./, '');79 } catch {80 return '';81 }82}83
84function isDirectoryHost(hostname: string): boolean {85 return [...DIRECTORY_HOSTS].some((blocked) => hostname === blocked || hostname.endsWith(`.${blocked}`));86}87
88export function buildWebsiteSearchQuery(productName: string, tagline: string): string {89 const cleanName = String(productName ?? '').trim().replace(/\s+/g, ' ').slice(0, 100);90 const cleanTagline = String(tagline ?? '').trim().replace(/\s+/g, ' ').slice(0, 100);91 return [cleanName, cleanTagline, 'official website'].filter(Boolean).join(' ').slice(0, 240);92}93
94export function buildProductIdentityNames(productName: string): string[] {95 const original = String(productName ?? '').trim().replace(/\s+/g, ' ').slice(0, 100);96 if (!original) return [];97 const aliases = [original];98 let core = original99 .replace(/\s+by\s+.+$/i, '')100 .replace(/\s+(?:v?\d+(?:\.\d+)*)(?:\s+.*)?$/i, '')101 .replace(/\s+(?:mobile\s+builder|website\s+builder|app\s+builder|apps)$/i, '')102 .trim();103 if (core.length < 2) core = original;104 if (normalizedText(core) !== normalizedText(original)) aliases.push(core);105 return [...new Set(aliases)].slice(0, 2);106}107
108export function buildWebsiteSearchQueries(productName: string, tagline: string): string[] {109 return buildProductIdentityNames(productName)110 .map((identity) => buildWebsiteSearchQuery(identity, tagline))111 .filter((query, index, queries) => query && queries.indexOf(query) === index)112 .slice(0, 2);113}114
115export function buildDomainHypothesisCandidates(productName: string): SearchCandidate[] {116 const identity = buildProductIdentityNames(productName).at(-1) ?? '';117 const compact = normalizedText(identity).replace(/\s+/g, '');118 if (compact.length < 3 || compact.length > 40) return [];119 return ['com', 'ai', 'app', 'io', 'xyz'].map((tld, index) => ({120 url: `https://${compact}.${tld}/`,121 title: identity,122 snippet: '',123 rank: index + 1,124 }));125}126
127export function isRetryableDataForSeoStatus(status: number): boolean {128 return new Set([40101, 40103, 50301, 50303, 50401]).has(status);129}130
131export function hasDirectOwnershipSignal(signals: string[]): boolean {132 return signals.includes('product_name_matches_domain') || signals.includes('full_product_name_matches_domain');133}134
135export function isUsableEmailStatus(status: string): boolean {136 return !HARD_REJECTED_EMAIL_STATUSES.has(status);137}138
139export function matchesProductSearch(140 query: string,141 product: { name?: unknown; tagline?: unknown; description?: unknown; topics?: unknown[] },142): boolean {143 const normalizedQuery = normalizedText(query);144 if (!normalizedQuery) return false;145 const topics = Array.isArray(product.topics) ? product.topics.map(normalizedText).join(' ') : '';146 const haystack = normalizedText(`${String(product.name ?? '')} ${String(product.tagline ?? '')} ${String(product.description ?? '')} ${topics}`);147 if (haystack.includes(normalizedQuery)) return true;148 const tokens = identityTokens(normalizedQuery);149 if (tokens.length === 0) return false;150 const matches = tokens.filter((token) => haystack.includes(token)).length;151 return matches >= Math.max(1, Math.ceil(tokens.length * 0.6));152}153
154export function isPrivateNetworkAddress(address: string): boolean {155 const normalized = address.toLowerCase().replace(/^\[|\]$/g, '');156 const mappedIpv4 = normalized.match(/^(?:::ffff:)?(\d{1,3}(?:\.\d{1,3}){3})$/)?.[1];157 if (mappedIpv4) {158 const octets = mappedIpv4.split('.').map(Number);159 if (octets.length !== 4 || octets.some((part) => !Number.isInteger(part) || part < 0 || part > 255)) return true;160 const [a, b, c] = octets;161 return a === 0162 || a === 10163 || a === 127164 || a >= 224165 || (a === 100 && b >= 64 && b <= 127)166 || (a === 169 && b === 254)167 || (a === 172 && b >= 16 && b <= 31)168 || (a === 192 && (b === 0 || b === 168))169 || (a === 198 && (b === 18 || b === 19 || (b === 51 && c === 100)))170 || (a === 203 && b === 0 && c === 113);171 }172 return normalized === '::'173 || normalized === '::1'174 || normalized.startsWith('fc')175 || normalized.startsWith('fd')176 || /^fe[89ab]/.test(normalized)177 || normalized.startsWith('ff')178 || normalized.startsWith('2001:db8:');179}180
181function protectedProductVendorRoots(productName: string): string[] | null {182 const normalizedName = normalizedText(productName);183 if (/\bclaude\s+(?:sonnet|opus|haiku)\b/.test(normalizedName)) return ['anthropic.com', 'claude.com'];184 if (/\bgemini\s+(?:omni|flash|pro|ultra|\d)/.test(normalizedName)) return ['google.com', 'google.dev', 'gemini.google'];185 if (/\b(?:openai\s+)?gpt\s*\d/.test(normalizedName)) return ['openai.com', 'chatgpt.com'];186 return null;187}188
189export function hasProtectedProductVendorMatch(productName: string, rawUrl: string): boolean {190 const roots = protectedProductVendorRoots(productName);191 if (!roots) return false;192 const hostname = hostnameOf(rawUrl);193 if (!hostname) return false;194 return roots.some((root) => hostname === root || hostname.endsWith(`.${root}`));195}196
197export function isProtectedProductHostAllowed(productName: string, rawUrl: string): boolean {198 return protectedProductVendorRoots(productName) === null || hasProtectedProductVendorMatch(productName, rawUrl);199}200
201export function buildProtectedProductCandidates(productName: string): SearchCandidate[] {202 const normalizedName = normalizedText(productName);203 const slug = normalizedName.replace(/\s+/g, '-');204 const urls = /\bclaude\s+(?:sonnet|opus|haiku)\b/.test(normalizedName)205 ? [`https://www.anthropic.com/news/${slug}`, 'https://claude.com/']206 : /\bgemini\s+(?:omni|flash|pro|ultra|\d)/.test(normalizedName)207 ? ['https://gemini.google/overview/video-generation', 'https://ai.google.dev/']208 : /\b(?:openai\s+)?gpt\s*\d/.test(normalizedName)209 ? [`https://openai.com/index/${slug}/`, 'https://chatgpt.com/']210 : [];211 return urls.map((url, index) => ({ url, title: productName, snippet: '', rank: index + 1 }));212}213
214export function parseDataForSeoCandidates(payload: unknown): { candidates: SearchCandidate[]; costUsd: number } {215 const body = payload && typeof payload === 'object' ? payload as Record<string, any> : {};216 const task = Array.isArray(body.tasks) ? body.tasks[0] : null;217 const result = Array.isArray(task?.result) ? task.result[0] : null;218 const items = Array.isArray(result?.items) ? result.items : [];219 const candidates = items220 .filter((item: any) => item && item.type === 'organic' && typeof item.url === 'string')221 .map((item: any, index: number): SearchCandidate => ({222 url: item.url,223 title: String(item.title ?? '').slice(0, 500),224 snippet: String(item.description ?? item.snippet ?? '').slice(0, 800),225 rank: Number.isInteger(item.rank_group) ? item.rank_group : index + 1,226 }));227 return { candidates, costUsd: Number(task?.cost ?? 0) || 0 };228}229
230export function parseSerpApiCandidates(payload: unknown): SearchCandidate[] {231 const body = payload && typeof payload === 'object' ? payload as Record<string, any> : {};232 const items = Array.isArray(body.organic_results) ? body.organic_results : [];233 return items234 .filter((item: any) => item && typeof item.link === 'string')235 .map((item: any, index: number): SearchCandidate => ({236 url: item.link,237 title: String(item.title ?? '').slice(0, 500),238 snippet: String(item.snippet ?? '').slice(0, 800),239 rank: Number.isInteger(item.position) ? item.position : index + 1,240 }));241}242
243export function scoreSearchCandidate(candidate: SearchCandidate, productName: string, tagline: string): number {244 const hostname = hostnameOf(candidate.url);245 if (!hostname || isDirectoryHost(hostname)) return -1;246
247 const nameTokens = identityTokens(productName);248 if (nameTokens.length === 0) return -1;249 const taglineTokens = identityTokens(tagline).slice(0, 8);250 const hostText = normalizedText(hostname);251 const titleText = normalizedText(candidate.title);252 const snippetText = normalizedText(candidate.snippet);253 const namePhrase = normalizedText(productName);254 const joinedName = nameTokens.join('');255
256 let score = candidate.rank <= 1 ? 18 : candidate.rank <= 3 ? 13 : candidate.rank <= 5 ? 8 : 3;257 const hostCoverage = coverage(nameTokens, hostText);258 const titleCoverage = coverage(nameTokens, titleText);259 score += Math.round(hostCoverage * 32);260 score += Math.round(titleCoverage * 30);261 score += Math.round(coverage(taglineTokens, `${titleText} ${snippetText}`) * 10);262 if (namePhrase.length >= 3 && titleText.includes(namePhrase)) score += 18;263 if (joinedName.length >= 3 && hostText.replace(/\s+/g, '').includes(joinedName)) score += 22;264 if (/\b(official|homepage|home)\b/.test(`${titleText} ${snippetText}`)) score += 4;265 return Math.min(100, score);266}267
268export function rankMakerWebsiteLinkCandidates(269 links: MakerWebsiteLink[],270 makerWebsiteUrl: string,271 productName: string,272 tagline: string,273): SearchCandidate[] {274 const makerHost = hostnameOf(makerWebsiteUrl);275 const identityNames = buildProductIdentityNames(productName);276 const seen = new Set<string>();277 return links278 .map((link): { candidate: SearchCandidate; score: number } | null => {279 let candidateUrl: URL;280 try {281 candidateUrl = new URL(link.url);282 } catch {283 return null;284 }285 if (!['http:', 'https:'].includes(candidateUrl.protocol) || candidateUrl.username || candidateUrl.password) return null;286 const host = hostnameOf(candidateUrl.toString());287 if (!host || host === makerHost || host.endsWith(`.${makerHost}`) || isDirectoryHost(host)) return null;288 candidateUrl.hash = '';289 const key = `${candidateUrl.origin}${candidateUrl.pathname}`.replace(/\/$/, '');290 if (seen.has(key)) return null;291 seen.add(key);292 const candidate: SearchCandidate = {293 url: candidateUrl.toString(),294 title: String(link.text ?? '').slice(0, 500),295 snippet: String(link.context ?? '').slice(0, 800),296 rank: Number.isInteger(link.rank) && link.rank > 0 ? link.rank : links.indexOf(link) + 1,297 };298 const score = Math.max(...identityNames.map((identity) => scoreSearchCandidate(candidate, identity, tagline)));299 return { candidate, score };300 })301 .filter((entry): entry is { candidate: SearchCandidate; score: number } => entry !== null && entry.score >= 34)302 .sort((left, right) => right.score - left.score || left.candidate.rank - right.candidate.rank)303 .slice(0, 5)304 .map(({ candidate }) => candidate);305}306
307export function selectMakerFallbackEmails(308 productWebsiteEmails: string[],309 makerWebsiteEmails: string[],310 maxEmails: number,311 filledEmailSlots = productWebsiteEmails.length,312): string[] {313 const existing = new Set(productWebsiteEmails.map((email) => email.toLowerCase()));314 const selected: string[] = [];315 const remaining = Math.max(0, Math.min(3, Math.floor(maxEmails)) - Math.max(0, filledEmailSlots));316 for (const rawEmail of makerWebsiteEmails) {317 const email = String(rawEmail ?? '').trim().toLowerCase();318 if (!EMAIL_PATTERN.test(email) || existing.has(email) || selected.includes(email)) continue;319 selected.push(email);320 if (selected.length >= remaining) break;321 }322 return selected;323}324
325export function evaluateWebsiteIdentity(input: {326 candidate: SearchCandidate;327 productName: string;328 tagline: string;329 pageTitle: string;330 pageDescription: string;331 pageText: string;332}): WebsiteIdentity {333 const { candidate, productName, tagline, pageTitle, pageDescription, pageText } = input;334 const searchScore = scoreSearchCandidate(candidate, productName, tagline);335 if (searchScore < 0) return { accepted: false, confidence: 0, signals: ['blocked_or_invalid_search_host'] };336
337 const hostname = hostnameOf(candidate.url);338 const hostText = normalizedText(hostname);339 const titleText = normalizedText(pageTitle);340 const descriptionText = normalizedText(pageDescription);341 const bodyText = normalizedText(pageText).slice(0, 80_000);342 const nameTokens = identityTokens(productName);343 const taglineTokens = identityTokens(tagline).slice(0, 8);344 const namePhrase = normalizedText(productName);345 const hostCoverage = coverage(nameTokens, hostText);346 const titleCoverage = coverage(nameTokens, `${titleText} ${descriptionText}`);347 const bodyCoverage = coverage(nameTokens, bodyText);348 const taglineCoverage = coverage(taglineTokens, `${descriptionText} ${bodyText}`);349 const exactTitle = namePhrase.length >= 3 && `${titleText} ${descriptionText}`.includes(namePhrase);350 const compactName = namePhrase.replace(/\s+/g, '');351 const compactDomainMatch = compactName.length >= 3 && hostText.replace(/\s+/g, '').includes(compactName);352
353 let pageScore = Math.round(hostCoverage * 35 + titleCoverage * 35 + bodyCoverage * 15 + taglineCoverage * 10);354 if (exactTitle) pageScore += 15;355 pageScore = Math.min(100, pageScore);356 const confidence = Math.min(100, Math.round(searchScore * 0.52 + pageScore * 0.48));357 const distinctiveAlphanumericDomain = nameTokens.length === 1358 && /[a-z]/.test(nameTokens[0])359 && /\d/.test(nameTokens[0])360 && compactDomainMatch;361 const singleTokenPurposeMatch = nameTokens.length !== 1362 || distinctiveAlphanumericDomain363 || taglineTokens.length === 0364 || taglineCoverage >= 0.25;365 const strongIdentitySignal = exactTitle366 || compactDomainMatch367 || (nameTokens.length >= 2 && hostCoverage >= 0.5)368 || (titleCoverage >= 0.75 && bodyCoverage >= 0.75 && taglineCoverage >= 0.5);369 const accepted = searchScore >= 42 && pageScore >= 38 && confidence >= 48 && strongIdentitySignal && singleTokenPurposeMatch;370 const signals: string[] = [];371 if (hostCoverage >= 0.5) signals.push('product_name_matches_domain');372 if (compactDomainMatch) signals.push('full_product_name_matches_domain');373 if (exactTitle) signals.push('product_name_matches_page_title');374 if (titleCoverage >= 0.75) signals.push('name_tokens_match_metadata');375 if (taglineCoverage >= 0.5) signals.push('tagline_matches_page');376 if (distinctiveAlphanumericDomain) signals.push('distinctive_alphanumeric_domain');377 if (!singleTokenPurposeMatch) signals.push('single_token_tagline_mismatch');378 if (!accepted) signals.push('identity_threshold_not_met');379 return { accepted, confidence, signals };380}381
382export function splitMakerName(fullName: unknown): { firstName?: string; lastName?: string } {383 const clean = String(fullName ?? '').replace(/[^\p{L}\p{M}'’\-\s]/gu, ' ').replace(/\s+/g, ' ').trim();384 if (!clean || /^(?:redacted|anonymous|deleted|unknown|private|n\/a)$/i.test(clean)) return {};385 const parts = clean.split(' ');386 if (parts.length === 1) return { firstName: parts[0] };387 return { firstName: parts[0], lastName: parts.slice(1).join(' ') };388}389
390export function normalizeContactFinderResults(payload: unknown, domain: string, maxEmails: number): ContactFinderResult[] {391 const normalizedDomain = domain.toLowerCase().replace(/^www\./, '');392 const rows = Array.isArray(payload) ? payload : [];393 const seen = new Set<string>();394 const confidenceRank: Record<ContactFinderResult['confidence'], number> = { high: 4, medium: 3, low: 2, unknown: 1 };395 return rows396 .map((row: any): ContactFinderResult => ({397 email: String(row?.email ?? '').trim().toLowerCase(),398 confidence: ['high', 'medium', 'low', 'unknown'].includes(row?.confidence) ? row.confidence : 'unknown',399 result: String(row?.result ?? 'unknown').trim().toLowerCase(),400 }))401 .filter((row) => {402 if (!EMAIL_PATTERN.test(row.email) || row.result !== 'ok' || !['high', 'medium'].includes(row.confidence)) return false;403 const emailDomain = row.email.split('@')[1] ?? '';404 if (!(emailDomain === normalizedDomain || emailDomain.endsWith(`.${normalizedDomain}`))) return false;405 if (seen.has(row.email)) return false;406 seen.add(row.email);407 return true;408 })409 .sort((left, right) => confidenceRank[right.confidence] - confidenceRank[left.confidence])410 .slice(0, Math.max(0, Math.min(3, Math.floor(maxEmails))));411}412
413export function buildLeadProfile(input: {414 websiteResolved: boolean;415 websiteConfidence: number;416 emailStatuses: string[];417 publicEmailCount: number;418 upvotes: number;419 pagesVisited: number;420}): LeadProfile {421 const { websiteResolved, websiteConfidence, emailStatuses, publicEmailCount, upvotes, pagesVisited } = input;422 const verifiedCount = emailStatuses.filter((status) => status === 'ok').length;423 const catchAllCount = emailStatuses.filter((status) => status === 'ok_for_all').length;424 const hasContact = emailStatuses.length > 0;425 const signals: string[] = [];426 let score = 0;427
428 if (websiteResolved) {429 score += 15;430 signals.push('website_resolved');431 }432 if (websiteConfidence >= 75) {433 score += 10;434 signals.push('high_confidence_website');435 } else if (websiteConfidence >= 50) {436 score += 5;437 }438 if (verifiedCount > 0) {439 score += 45 + Math.min(15, (verifiedCount - 1) * 8);440 signals.push('deliverable_email_verified');441 } else if (catchAllCount > 0) {442 score += 28;443 signals.push('catch_all_email');444 } else if (hasContact) {445 score += 15;446 signals.push('unverified_contact');447 }448 if (publicEmailCount > 0) {449 score += 8;450 signals.push('email_published_on_website');451 }452 if (pagesVisited >= 3) signals.push('multi_page_website_assessed');453 score += Math.min(7, Math.floor(Math.log10(Math.max(1, upvotes) + 1) * 3));454 score = Math.min(100, score);455
456 const leadStatus: LeadProfile['leadStatus'] = verifiedCount > 0457 ? 'verified_contact'458 : catchAllCount > 0459 ? 'catch_all_contact'460 : hasContact461 ? 'unverified_contact'462 : websiteResolved463 ? 'no_contact'464 : 'unresolved_website';465 const leadTier: LeadProfile['leadTier'] = score >= 75 ? 'A' : score >= 55 ? 'B' : score >= 35 ? 'C' : 'unqualified';466 return { leadStatus, leadScore: score, leadTier, signals };467}1import { Actor } from 'apify';2import { log } from 'crawlee';3import { load } from 'cheerio';4import { lookup } from 'node:dns/promises';5import { isIP } from 'node:net';6import { chromium, type Browser } from 'playwright';7import { getDomain } from 'tldts';8import {9 buildWebsitePagePlan,10 canonicalWebsiteUrl,11 clampInteger,12 decodeCloudflareEmail,13 DEFAULT_MAX_EMAILS_PER_PRODUCT,14 DEFAULT_MAX_WEBSITE_PAGES,15 emailCandidateLimit,16 extractEmailCandidatesFromText,17 hasUnexcludedProductHuntLaunch,18 hasReachedUsableEmailTarget,19 isRetryableEmailListVerifyHttpStatus,20 isProductHuntHost,21 isPublicHttpUrl,22 MAX_EMAILS_PER_PRODUCT,23 MAX_WEBSITE_PAGES,24 normalizeEmailListVerifyStatus,25 parseProductHuntSeedUrl,26 productHuntLaunchKey,27 productHuntRateLimitDelaySeconds,28 productHuntStreamOrder,29 shouldUseLightweightProductHuntIndex,30 rankEmails,31 resumableCheckpointCursor,32 type EmailListVerifyStatus,33 type WebsitePage,34} from './enrichment-utils.js';35import {36 buildLeadProfile,37 buildDomainHypothesisCandidates,38 buildProtectedProductCandidates,39 buildProductIdentityNames,40 buildWebsiteSearchQueries,41 evaluateWebsiteIdentity,42 hasDirectOwnershipSignal,43 hasProtectedProductVendorMatch,44 isPrivateNetworkAddress,45 isProtectedProductHostAllowed,46 isRetryableDataForSeoStatus,47 isUsableEmailStatus,48 matchesProductSearch,49 normalizeContactFinderResults,50 parseDataForSeoCandidates,51 parseSerpApiCandidates,52 rankMakerWebsiteLinkCandidates,53 scoreSearchCandidate,54 selectMakerFallbackEmails,55 splitMakerName,56 type ContactFinderResult,57 type MakerWebsiteLink,58 type SearchCandidate,59} from './lead-engine.js';60
61await Actor.init();62
63type Mode = 'leaderboard' | 'search' | 'topic' | 'urls';64type LeaderboardPeriod = 'daily' | 'weekly' | 'monthly' | 'yearly';65type RunOutcome = 'COMPLETE' | 'PARTIAL' | 'VALID_EMPTY' | 'INVALID_INPUT' | 'UPSTREAM_FAILED' | 'CONFIG_ERROR';66type EmailSource = 'page_scrape' | 'maker_website' | 'emaillistverify_contact_finder' | 'mixed' | 'none';67type EmailResult = 'emails_found' | 'no_public_email_found' | 'not_assessed';68type Coverage = 'complete' | 'partial' | 'none';69type WebsiteResolutionStatus = 'verified_external' | 'verified_search_match' | 'unresolved' | 'not_available';70type WebsiteResolutionSource = 'api_website_product_link' | 'input_override' | 'product_hunt_redirect' | 'maker_website' | 'maker_website_link' | 'protected_vendor_domain' | 'dataforseo_organic' | 'direct_domain_hypothesis' | 'serpapi_organic' | 'none';71type EmailVerification = {72 email: string;73 provider: 'emaillistverify';74 status: EmailListVerifyStatus;75 checkedAt: string | null;76 source: 'public_website' | 'maker_website' | 'emaillistverify_contact_finder';77 confidence: 'high' | 'medium' | 'low' | 'unknown' | null;78};79
80type DateRange = { postedAfter: string; postedBefore: string };81type WebsiteOverride = { keys: string[]; websiteUrl: string };82type EnrichmentResult = {83 emails: string[];84 emailSource: EmailSource;85 result: EmailResult;86 coverage: Coverage;87 terminalReason: string;88 pagesVisited: number;89 pageUrls: string[];90 warnings: string[];91};92
93type WebsiteResolution = {94 websiteUrl: string | null;95 source: WebsiteResolutionSource;96 status: WebsiteResolutionStatus;97 confidence: number;98 signals: string[];99 providerCostUsd: number;100 providerAttempts: number;101 warnings: string[];102};103
104type ContactDiscovery = {105 contacts: ContactFinderResult[];106 attempts: number;107 credits: number;108 warnings: string[];109};110
111const PH_API = 'https://api.producthunt.com/v2/api/graphql';112const RESULT_EVENT = 'apify-default-dataset-item';113const EMAIL_EVENT = 'email-found';114const RESULT_PRICE_USD = 0.002;115const EMAIL_PRICE_USD = 0.04;116const EMAIL_LIST_VERIFY_API = 'https://api.emaillistverify.com/api/verifyEmail';117const EMAIL_LIST_VERIFY_CONTACT_API = 'https://api.emaillistverify.com/api/findContact';118const DATAFORSEO_SEARCH_API = 'https://api.dataforseo.com/v3/serp/google/organic/live/advanced';119const SERPAPI_SEARCH_API = 'https://serpapi.com/search.json';120const EMAIL_LIST_VERIFY_MIN_INTERVAL_MS = 225; // Contact Finder is limited to 5 requests/second; keep headroom.121const FAILED_OUTCOMES = new Set<RunOutcome>(['UPSTREAM_FAILED', 'CONFIG_ERROR']);122
123const SKIP_DOMAINS = new Set([124 'apps.apple.com', 'play.google.com', 'github.com', 'youtube.com',125 'twitter.com', 'x.com', 'linkedin.com', 'facebook.com', 'instagram.com',126 'reddit.com', 'discord.com', 'discord.gg', 'slack.com', 'medium.com',127 'substack.com', 'producthunt.com', 'amazon.com', 'chrome.google.com',128 'marketplace.visualstudio.com', 'notion.so', 'figma.com', 'trello.com',129 'airtable.com',130]);131
132const PRODUCT_HUNT_USER_AGENT = 'ProductHunt Lead Generator/1.4 (authorized API client)';133const WEBSITE_USER_AGENT = 'Mozilla/5.0 (compatible; ProductHuntLeadGenerator/1.4; +https://apify.com/khadinakbar/producthunt-scraper-pro)';134
135let nextEmailListVerifyRequestAt = 0;136let emailListVerifyCreditsExhausted = false;137const publicHostCache = new Map<string, Promise<boolean>>();138
139function isoDate(date: Date): string {140 return date.toISOString().slice(0, 10);141}142
143function getWeekBounds(date: Date): { start: string; end: string } {144 const start = new Date(date);145 // Product Hunt's weekly archive uses ISO weeks: Monday through Sunday.146 const weekday = date.getUTCDay() || 7;147 start.setUTCDate(date.getUTCDate() - weekday + 1);148 const end = new Date(start);149 end.setUTCDate(start.getUTCDate() + 6);150 return { start: isoDate(start), end: isoDate(end) };151}152
153function getMonthBounds(date: Date): { start: string; end: string } {154 const year = date.getUTCFullYear();155 const month = date.getUTCMonth();156 return {157 start: isoDate(new Date(Date.UTC(year, month, 1))),158 end: isoDate(new Date(Date.UTC(year, month + 1, 0))),159 };160}161
162function getYearBounds(date: Date): { start: string; end: string } {163 const year = date.getUTCFullYear();164 return { start: `${year}-01-01`, end: `${year}-12-31` };165}166
167function validIsoDate(value: string): boolean {168 if (!/^\d{4}-\d{2}-\d{2}$/.test(value)) return false;169 const parsed = new Date(`${value}T12:00:00Z`);170 return Number.isFinite(parsed.getTime()) && isoDate(parsed) === value;171}172
173function dateRangeForPeriod(period: LeaderboardPeriod, anchorDate: string, endDate = ''): DateRange {174 const base = new Date(`${anchorDate}T12:00:00Z`);175 if (period === 'daily') {176 return {177 postedAfter: `${anchorDate}T00:00:00+00:00`,178 postedBefore: `${endDate || anchorDate}T23:59:59+00:00`,179 };180 }181 const bounds = period === 'weekly'182 ? getWeekBounds(base)183 : period === 'monthly'184 ? getMonthBounds(base)185 : getYearBounds(base);186 return {187 postedAfter: `${bounds.start}T00:00:00+00:00`,188 postedBefore: `${bounds.end}T23:59:59+00:00`,189 };190}191
192function normalizeKey(value: unknown): string | null {193 if (typeof value !== 'string' || !value.trim()) return null;194 return value.trim().toLowerCase().replace(/[?#].*$/, '').replace(/\/$/, '');195}196
197function parseWebsiteOverrides(value: unknown): WebsiteOverride[] {198 const overrides: WebsiteOverride[] = [];199 const add = (keys: unknown[], websiteUrl: unknown): void => {200 if (typeof websiteUrl !== 'string' || !isPublicHttpUrl(websiteUrl)) return;201 const canonical = canonicalWebsiteUrl(websiteUrl);202 if (!canonical || isProductHuntHost(new URL(canonical).hostname)) return;203 const usableKeys = keys.map(normalizeKey).filter((key): key is string => Boolean(key));204 if (usableKeys.length > 0) overrides.push({ keys: usableKeys, websiteUrl: canonical });205 };206
207 if (Array.isArray(value)) {208 for (const entry of value) {209 if (!entry || typeof entry !== 'object') continue;210 const row = entry as Record<string, unknown>;211 add([row.productHuntUrl, row.productSlug, row.productName, row.productId], row.websiteUrl);212 }213 } else if (value && typeof value === 'object') {214 for (const [key, websiteUrl] of Object.entries(value as Record<string, unknown>)) add([key], websiteUrl);215 }216 return overrides;217}218
219function lookupWebsiteOverride(node: Record<string, unknown>, overrides: WebsiteOverride[]): string | null {220 const candidates = [node.url, node.slug, node.id, node.name]221 .map(normalizeKey)222 .filter((value): value is string => Boolean(value));223 return overrides.find((override) => override.keys.some((key) => candidates.includes(key)))?.websiteUrl ?? null;224}225
226function rootDomain(hostname: string): string {227 const labels = hostname.replace(/^www\./i, '').toLowerCase().split('.');228 return labels.slice(-2).join('.');229}230
231function isAllowedExternalWebsite(url: string): boolean {232 if (!isPublicHttpUrl(url)) return false;233 try {234 const host = new URL(url).hostname.toLowerCase();235 return !isProductHuntHost(host) && !SKIP_DOMAINS.has(host) && !SKIP_DOMAINS.has(rootDomain(host));236 } catch {237 return false;238 }239}240
241async function isSafePublicNetworkUrl(rawUrl: string): Promise<boolean> {242 if (!isPublicHttpUrl(rawUrl)) return false;243 let hostname: string;244 try {245 hostname = new URL(rawUrl).hostname.toLowerCase().replace(/^\[|\]$/g, '');246 } catch {247 return false;248 }249 if (isIP(hostname)) return !isPrivateNetworkAddress(hostname);250
251 let assessment = publicHostCache.get(hostname);252 if (!assessment) {253 assessment = lookup(hostname, { all: true, verbatim: true })254 .then((records) => records.length > 0 && records.every(({ address }) => !isPrivateNetworkAddress(address)))255 .catch(() => false);256 publicHostCache.set(hostname, assessment);257 }258 return assessment;259}260
261// Executed inside a website page. It intentionally extracts only public page262// content; no forms, login state, or challenge handling is involved.263function browserExtractEmails(): string[] {264 const emailRe = /\b[a-zA-Z0-9._%+\-]+@[a-zA-Z0-9.\-]+\.[a-zA-Z]{2,}\b/g;265 const obfuscatedRe = /\b[a-zA-Z0-9._%+\-]+\s*(?:\[at\]|\(at\)|\sat\s)\s*[a-zA-Z0-9.\-]+\s*(?:\[dot\]|\(dot\)|\sdot\s)\s*[a-zA-Z]{2,}\b/gi;266 const collected: string[] = [];267 const add = (value: unknown): void => {268 if (typeof value !== 'string') return;269 let decoded = value;270 try { decoded = decodeURIComponent(value); } catch { /* malformed percent encoding */ }271 const email = decoded.trim().toLowerCase().replace(/^mailto:/, '').split('?')[0];272 if (email.includes('@')) collected.push(email);273 };274 const walkJson = (value: unknown, depth = 0): void => {275 if (depth > 8 || value === null || value === undefined) return;276 if (Array.isArray(value)) {277 value.forEach((item) => walkJson(item, depth + 1));278 return;279 }280 if (typeof value !== 'object') return;281 for (const [key, child] of Object.entries(value as Record<string, unknown>)) {282 if (key.toLowerCase() === 'email') add(child);283 walkJson(child, depth + 1);284 }285 };286 const decodeCfEmail = (encoded: string): string | null => {287 if (!/^[0-9a-f]+$/i.test(encoded) || encoded.length < 4) return null;288 const key = Number.parseInt(encoded.slice(0, 2), 16);289 let decoded = '';290 for (let index = 2; index < encoded.length; index += 2) {291 decoded += String.fromCharCode(Number.parseInt(encoded.slice(index, index + 2), 16) ^ key);292 }293 return decoded;294 };295
296 document.querySelectorAll('a[href^="mailto:"]').forEach((element) => add((element as HTMLAnchorElement).href));297 document.querySelectorAll('[data-email]').forEach((element) => add(element.getAttribute('data-email')));298 document.querySelectorAll('[data-cfemail]').forEach((element) => {299 const decoded = decodeCfEmail(element.getAttribute('data-cfemail') ?? '');300 if (decoded) add(decoded);301 });302 document.querySelectorAll('script[type="application/ld+json"]').forEach((element) => {303 try { walkJson(JSON.parse(element.textContent ?? '')); } catch { /* malformed JSON-LD */ }304 });305
306 const combined = `${document.body?.innerText ?? ''}\n${document.documentElement.innerHTML}`307 .replace(/\\(?:x40|u0040)/gi, '@')308 .replace(/\\(?:x2e|u002e)/gi, '.')309 .replace(/&#(?:64|x40);|@/gi, '@')310 .replace(/&#(?:46|x2e);|./gi, '.');311 (combined.match(emailRe) ?? []).forEach(add);312 for (const match of combined.match(obfuscatedRe) ?? []) {313 add(match.replace(/\s*(?:\[at\]|\(at\)|\sat\s)\s*/i, '@').replace(/\s*(?:\[dot\]|\(dot\)|\sdot\s)\s*/ig, '.'));314 }315 return collected;316}317
318// Executed only on the verified external website homepage. It never invents a319// route and exposes only allowlisted paths directly linked by that homepage.320function browserExtractLinkedFallbackPages(origin: string): WebsitePage[] {321 const matchers: Array<{ kind: Exclude<WebsitePage['kind'], 'homepage'>; priority: number; pattern: RegExp }> = [322 { kind: 'contact', priority: 0, pattern: /(contact|contact-us|get-in-touch|reach-us)/i },323 { kind: 'about', priority: 1, pattern: /(about|about-us|company|our-story)/i },324 { kind: 'team', priority: 2, pattern: /(team|our-team|meet-the-team|people|founders)/i },325 { kind: 'privacy', priority: 3, pattern: /(privacy|privacy-policy)/i },326 { kind: 'terms', priority: 4, pattern: /(terms|terms-of-service|terms-and-conditions|conditions)/i },327 ];328 const links = new Map<string, WebsitePage>();329 document.querySelectorAll('a[href]').forEach((element) => {330 const anchor = element as HTMLAnchorElement;331 try {332 const url = new URL(anchor.href);333 if (url.origin !== origin || !['http:', 'https:'].includes(url.protocol) || url.username || url.password) return;334 const label = (anchor.innerText || anchor.textContent || '').trim();335 const matched = matchers.find((candidate) => candidate.pattern.test(`${url.pathname} ${label}`));336 if (!matched) return;337 url.hash = '';338 const clean = url.toString();339 if (clean === `${origin}/` || clean === origin || links.has(clean)) return;340 links.set(clean, { url: clean, kind: matched.kind, priority: matched.priority });341 } catch { /* ignore malformed hrefs */ }342 });343 return [...links.values()].sort((left, right) => left.priority - right.priority);344}345
346async function waitForEmailListVerifySlot(): Promise<void> {347 const scheduledAt = Math.max(Date.now(), nextEmailListVerifyRequestAt);348 nextEmailListVerifyRequestAt = scheduledAt + EMAIL_LIST_VERIFY_MIN_INTERVAL_MS;349 const delayMs = scheduledAt - Date.now();350 if (delayMs > 0) await new Promise((resolve) => setTimeout(resolve, delayMs));351}352
353async function verifyEmailsWithEmailListVerify(354 emails: string[],355 requested: boolean,356 apiKey: string,357 source: EmailVerification['source'] = 'public_website',358 maxUsableEmails = emails.length,359): Promise<{ verifications: EmailVerification[]; attempts: number; warnings: string[] }> {360 if (emails.length === 0) return { verifications: [], attempts: 0, warnings: [] };361 if (!requested) {362 return {363 verifications: emails.map((email) => ({364 email, provider: 'emaillistverify', status: 'not_requested', checkedAt: null,365 source, confidence: null,366 })),367 attempts: 0,368 warnings: [],369 };370 }371 if (!apiKey) {372 return {373 verifications: emails.map((email) => ({374 email, provider: 'emaillistverify', status: 'not_configured', checkedAt: null,375 source, confidence: null,376 })),377 attempts: 0,378 warnings: ['EmailListVerify was requested but no owner-managed API key was configured; public emails were kept unverified.'],379 };380 }381 if (emailListVerifyCreditsExhausted) {382 return {383 verifications: emails.map((email) => ({384 email, provider: 'emaillistverify', status: 'error_credit', checkedAt: null,385 source, confidence: null,386 })),387 attempts: 0,388 warnings: ['EmailListVerify credits were exhausted earlier in this run; no additional verification requests were sent.'],389 };390 }391
392 const warnings = new Set<string>();393 const verifications: EmailVerification[] = [];394 let attempts = 0;395 for (const email of emails) {396 if (hasReachedUsableEmailTarget(verifications.map(({ status }) => status), maxUsableEmails)) break;397 for (let requestAttempt = 0; requestAttempt < 2; requestAttempt += 1) {398 await waitForEmailListVerifySlot();399 attempts += 1;400 const checkedAt = new Date().toISOString();401 try {402 const requestUrl = new URL(EMAIL_LIST_VERIFY_API);403 requestUrl.searchParams.set('email', email);404 const response = await fetch(requestUrl, {405 headers: { 'x-api-key': apiKey, Accept: 'text/plain, application/json;q=0.9' },406 signal: AbortSignal.timeout(12_000),407 });408 if (!response.ok) {409 if (requestAttempt === 0 && isRetryableEmailListVerifyHttpStatus(response.status)) {410 await new Promise((resolve) => setTimeout(resolve, 500));411 continue;412 }413 warnings.add(`EmailListVerify returned HTTP ${response.status}; affected public emails were retained with provider_error status.`);414 if (response.status === 402) emailListVerifyCreditsExhausted = true;415 verifications.push({416 email, provider: 'emaillistverify', status: response.status === 402 ? 'error_credit' : 'provider_error', checkedAt,417 source, confidence: null,418 });419 break;420 }421 const status = normalizeEmailListVerifyStatus(await response.text());422 if (status === 'provider_error') warnings.add('EmailListVerify returned an undocumented response; affected public emails were retained with provider_error status.');423 if (status === 'error_credit') {424 emailListVerifyCreditsExhausted = true;425 warnings.add('EmailListVerify reported insufficient verification credits; remaining public emails were retained with error_credit status.');426 }427 verifications.push({ email, provider: 'emaillistverify', status, checkedAt, source, confidence: null });428 break;429 } catch {430 if (requestAttempt === 0) {431 await new Promise((resolve) => setTimeout(resolve, 500));432 continue;433 }434 warnings.add('EmailListVerify request failed twice; affected public emails were retained with provider_error status.');435 verifications.push({ email, provider: 'emaillistverify', status: 'provider_error', checkedAt, source, confidence: null });436 }437 }438 if (emailListVerifyCreditsExhausted) break;439 }440 return { verifications, attempts, warnings: [...warnings] };441}442
443let browser: Browser | null = null;444
445async function ensureBrowser(): Promise<Browser> {446 if (!browser) {447 browser = await chromium.launch({448 headless: true,449 args: [450 '--no-sandbox', '--disable-setuid-sandbox', '--disable-dev-shm-usage',451 '--disable-gpu', '--no-zygote', '--disable-extensions',452 ],453 });454 log.info('Browser launched for verified external website enrichment.');455 }456 return browser;457}458
459async function closeBrowser(): Promise<void> {460 if (!browser) return;461 await browser.close().catch(() => undefined);462 browser = null;463}464
465function extractStaticEmails(html: string): string[] {466 const $ = load(html);467 const collected: string[] = [];468 const add = (value: unknown): void => {469 if (typeof value !== 'string') return;470 let decoded = value;471 try { decoded = decodeURIComponent(value); } catch { /* malformed percent encoding */ }472 const email = decoded.trim().toLowerCase().replace(/^mailto:/, '').split('?')[0];473 if (email.includes('@')) collected.push(email);474 };475 const walkJson = (value: unknown, depth = 0): void => {476 if (depth > 8 || value === null || value === undefined) return;477 if (Array.isArray(value)) return value.forEach((item) => walkJson(item, depth + 1));478 if (typeof value !== 'object') return;479 for (const [key, child] of Object.entries(value as Record<string, unknown>)) {480 if (key.toLowerCase() === 'email') add(child);481 walkJson(child, depth + 1);482 }483 };484 $('a[href^="mailto:"]').each((_, element) => add($(element).attr('href')));485 $('[data-email]').each((_, element) => add($(element).attr('data-email')));486 $('[data-cfemail]').each((_, element) => add(decodeCloudflareEmail($(element).attr('data-cfemail') ?? '')));487 $('script[type="application/ld+json"]').each((_, element) => {488 try { walkJson(JSON.parse($(element).text())); } catch { /* malformed JSON-LD */ }489 });490 const combined = `${$.text()}\n${html}`;491 extractEmailCandidatesFromText(combined).forEach(add);492 return collected;493}494
495function extractStaticFallbackPages(html: string, origin: string): WebsitePage[] {496 const $ = load(html);497 const matchers: Array<{ kind: Exclude<WebsitePage['kind'], 'homepage'>; priority: number; pattern: RegExp }> = [498 { kind: 'contact', priority: 0, pattern: /(contact|contact-us|get-in-touch|reach-us)/i },499 { kind: 'about', priority: 1, pattern: /(about|about-us|company|our-story)/i },500 { kind: 'team', priority: 2, pattern: /(team|our-team|meet-the-team|people|founders)/i },501 { kind: 'privacy', priority: 3, pattern: /(privacy|privacy-policy)/i },502 { kind: 'terms', priority: 4, pattern: /(terms|terms-of-service|terms-and-conditions|conditions)/i },503 ];504 const candidates = new Map<string, WebsitePage>();505 $('a[href]').each((_, element) => {506 try {507 const url = new URL($(element).attr('href') ?? '', origin);508 if (url.origin !== origin || !['http:', 'https:'].includes(url.protocol) || url.username || url.password) return;509 const label = $(element).text().trim();510 const matched = matchers.find((candidate) => candidate.pattern.test(`${url.pathname} ${label}`));511 if (!matched) return;512 url.hash = '';513 const clean = url.toString();514 if (clean === `${origin}/` || clean === origin || candidates.has(clean)) return;515 candidates.set(clean, { url: clean, kind: matched.kind, priority: matched.priority });516 } catch { /* ignore malformed hrefs */ }517 });518 return [...candidates.values()].sort((left, right) => left.priority - right.priority);519}520
521type StaticPage = { ok: true; url: string; html: string } | { ok: false; reason: string };522
523async function fetchPublicHtml(rawUrl: string, timeoutMs: number): Promise<StaticPage> {524 let current = canonicalWebsiteUrl(rawUrl);525 if (!current || !isAllowedExternalWebsite(current)) return { ok: false, reason: 'invalid_or_private_target' };526 try {527 for (let hop = 0; hop <= 3; hop += 1) {528 if (!await isSafePublicNetworkUrl(current)) return { ok: false, reason: 'private_or_unresolvable_target' };529 const response = await fetch(current, {530 method: 'GET',531 redirect: 'manual',532 headers: { Accept: 'text/html,application/xhtml+xml', 'User-Agent': WEBSITE_USER_AGENT },533 signal: AbortSignal.timeout(timeoutMs),534 });535 if (response.status >= 300 && response.status < 400) {536 const location = response.headers.get('location');537 if (!location || hop === 3) return { ok: false, reason: 'redirect_hop_limit' };538 const next = canonicalWebsiteUrl(new URL(location, current).toString());539 if (!next || !isAllowedExternalWebsite(next)) return { ok: false, reason: 'redirect_to_private_or_unsupported_target' };540 if (!await isSafePublicNetworkUrl(next)) return { ok: false, reason: 'redirect_to_private_or_unresolvable_target' };541 current = next;542 continue;543 }544 if (!response.ok) return { ok: false, reason: `http_${response.status}` };545 const contentType = response.headers.get('content-type') ?? '';546 if (!/(?:text\/html|application\/xhtml\+xml)/i.test(contentType)) return { ok: false, reason: 'non_html_response' };547 const contentLength = Number(response.headers.get('content-length') ?? '0');548 if (Number.isFinite(contentLength) && contentLength > 1_500_000) return { ok: false, reason: 'response_too_large' };549 const html = (await response.text()).slice(0, 1_500_000);550 return { ok: true, url: current, html };551 }552 return { ok: false, reason: 'redirect_hop_limit' };553 } catch (error) {554 const message = error instanceof Error ? error.message : '';555 return { ok: false, reason: /timeout|abort/i.test(message) ? 'timeout' : 'network_error' };556 }557}558
559function pageIdentity(html: string): { title: string; description: string; text: string } {560 const $ = load(html);561 $('script,style,noscript,svg').remove();562 return {563 title: $('title').first().text().trim(),564 description: $('meta[name="description"]').attr('content')?.trim()565 || $('meta[property="og:description"]').attr('content')?.trim()566 || '',567 text: $('body').text().replace(/\s+/g, ' ').trim().slice(0, 100_000),568 };569}570
571async function resolveProductHuntRedirect(rawUrl: string): Promise<string | null> {572 let current = canonicalWebsiteUrl(rawUrl);573 if (!current) return null;574 try {575 if (!isProductHuntHost(new URL(current).hostname)) return null;576 for (let hop = 0; hop < 4; hop += 1) {577 if (!await isSafePublicNetworkUrl(current)) return null;578 let response = await fetch(current, {579 method: 'HEAD',580 redirect: 'manual',581 headers: { 'User-Agent': PRODUCT_HUNT_USER_AGENT, Accept: 'text/html,*/*;q=0.5' },582 signal: AbortSignal.timeout(8_000),583 });584 if (response.status === 405) {585 response = await fetch(current, {586 method: 'GET',587 redirect: 'manual',588 headers: { 'User-Agent': PRODUCT_HUNT_USER_AGENT, Accept: 'text/html,*/*;q=0.5' },589 signal: AbortSignal.timeout(8_000),590 });591 }592 if (response.status < 300 || response.status >= 400) return null;593 const location = response.headers.get('location');594 if (!location) return null;595 const next = canonicalWebsiteUrl(new URL(location, current).toString());596 if (!next) return null;597 if (!isProductHuntHost(new URL(next).hostname)) {598 return isAllowedExternalWebsite(next) && await isSafePublicNetworkUrl(next) ? next : null;599 }600 current = next;601 }602 } catch {603 return null;604 }605 return null;606}607
608async function verifyWebsiteCandidates(609 candidates: SearchCandidate[],610 productName: string,611 tagline: string,612 description = '',613): Promise<{ websiteUrl: string; confidence: number; signals: string[] } | null> {614 const identityNames = buildProductIdentityNames(productName);615 const purpose = `${tagline} ${description}`.trim().slice(0, 1_200);616 const ranked = [...candidates]617 .map((candidate) => ({618 candidate,619 score: Math.max(...identityNames.map((identity) => scoreSearchCandidate(candidate, identity, purpose))),620 }))621 .filter(({ score }) => score >= 34)622 .sort((left, right) => right.score - left.score || left.candidate.rank - right.candidate.rank)623 .slice(0, 5);624
625 for (const { candidate } of ranked) {626 let candidateUrl: URL;627 try {628 candidateUrl = new URL(candidate.url);629 } catch {630 continue;631 }632 if (!isProtectedProductHostAllowed(productName, candidateUrl.toString())) continue;633 const homepage = `${candidateUrl.origin}/`;634 const verificationUrls = candidateUrl.pathname !== '/' || candidateUrl.search635 ? [candidateUrl.toString(), homepage]636 : [homepage];637 for (const verificationUrl of [...new Set(verificationUrls)]) {638 const fetched = await fetchPublicHtml(verificationUrl, 6_000);639 if (!fetched.ok) continue;640 const identity = pageIdentity(fetched.html);641 const assessments = identityNames.map((identityName) => ({642 identityName,643 assessment: evaluateWebsiteIdentity({644 candidate: { ...candidate, url: fetched.url },645 productName: identityName,646 tagline: purpose,647 pageTitle: identity.title,648 pageDescription: identity.description,649 pageText: identity.text,650 }),651 })).filter(({ assessment }) => assessment.accepted)652 .sort((left, right) => right.assessment.confidence - left.assessment.confidence);653 if (assessments.length === 0) continue;654 const { assessment, identityName } = assessments[0];655 if (new URL(verificationUrl).pathname !== '/'656 && !hasDirectOwnershipSignal(assessment.signals)657 && !hasProtectedProductVendorMatch(productName, verificationUrl)) {658 continue;659 }660 const finalUrl = new URL(fetched.url);661 const signals = [...assessment.signals];662 if (identityName !== identityNames[0]) signals.push('core_brand_alias_confirmed');663 if (new URL(verificationUrl).pathname !== '/') signals.push('official_product_page_confirmed');664 return {665 websiteUrl: `${finalUrl.origin}/`,666 confidence: assessment.confidence,667 signals: [...new Set(signals)],668 };669 }670 }671 return null;672}673
674async function discoverMakerLinkedProductCandidates(675 makerWebsiteUrls: string[],676 productName: string,677 tagline: string,678): Promise<SearchCandidate[]> {679 const collected = new Map<string, SearchCandidate>();680 for (const makerWebsiteUrl of makerWebsiteUrls.filter((url) => isAllowedExternalWebsite(url)).slice(0, 2)) {681 const fetched = await fetchPublicHtml(makerWebsiteUrl, 6_000);682 if (!fetched.ok) continue;683 const makerOrigin = new URL(fetched.url).origin;684 const $ = load(fetched.html);685 const links: MakerWebsiteLink[] = [];686 $('a[href]').each((index, element) => {687 try {688 const target = new URL($(element).attr('href') ?? '', fetched.url);689 if (!['http:', 'https:'].includes(target.protocol) || target.username || target.password) return;690 if (target.origin === makerOrigin) return;691 const text = $(element).text().replace(/\s+/g, ' ').trim();692 const context = $(element).closest('p,li,article,section,div').first().text().replace(/\s+/g, ' ').trim();693 links.push({ url: target.toString(), text, context, rank: index + 1 });694 } catch { /* ignore malformed maker-page links */ }695 });696 for (const candidate of rankMakerWebsiteLinkCandidates(links, fetched.url, productName, tagline)) {697 let key: string;698 try { key = new URL(candidate.url).origin; } catch { continue; }699 if (!collected.has(key)) collected.set(key, candidate);700 }701 }702 return [...collected.values()]703 .sort((left, right) => scoreSearchCandidate(right, productName, tagline) - scoreSearchCandidate(left, productName, tagline))704 .slice(0, 5);705}706
707class DataForSeoSearchError extends Error {708 constructor(message: string, readonly retryable: boolean, readonly costUsd: number) {709 super(message);710 }711}712
713async function callDataForSeoWebsiteSearch(714 query: string,715 login: string,716 password: string,717): Promise<{ candidates: SearchCandidate[]; costUsd: number }> {718 const response = await fetch(DATAFORSEO_SEARCH_API, {719 method: 'POST',720 headers: {721 Authorization: `Basic ${Buffer.from(`${login}:${password}`).toString('base64')}`,722 'Content-Type': 'application/json',723 Accept: 'application/json',724 },725 body: JSON.stringify([{726 keyword: query,727 location_code: 2840,728 language_code: 'en',729 device: 'desktop',730 os: 'windows',731 depth: 10,732 group_organic_results: true,733 }]),734 signal: AbortSignal.timeout(45_000),735 });736 if (!response.ok) {737 throw new DataForSeoSearchError(738 `DataForSEO website search returned HTTP ${response.status}.`,739 response.status === 429 || response.status >= 500,740 0,741 );742 }743 const body = await response.json() as Record<string, any>;744 const parsed = parseDataForSeoCandidates(body);745 if (body.status_code && body.status_code !== 20000) {746 throw new DataForSeoSearchError(747 `DataForSEO website search failed with provider code ${body.status_code}: ${String(body.status_message ?? 'unknown error')}.`,748 isRetryableDataForSeoStatus(Number(body.status_code)),749 parsed.costUsd,750 );751 }752 const task = Array.isArray(body.tasks) ? body.tasks[0] : null;753 if (task?.status_code && task.status_code !== 20000) {754 throw new DataForSeoSearchError(755 `DataForSEO website search task failed with provider code ${task.status_code}: ${String(task.status_message ?? 'unknown error')}.`,756 isRetryableDataForSeoStatus(Number(task.status_code)),757 parsed.costUsd,758 );759 }760 return parsed;761}762
763async function callSerpApiWebsiteSearch(query: string, apiKey: string): Promise<SearchCandidate[]> {764 const url = new URL(SERPAPI_SEARCH_API);765 url.searchParams.set('engine', 'google');766 url.searchParams.set('q', query);767 url.searchParams.set('num', '10');768 url.searchParams.set('gl', 'us');769 url.searchParams.set('hl', 'en');770 url.searchParams.set('api_key', apiKey);771 const response = await fetch(url, { signal: AbortSignal.timeout(30_000), headers: { Accept: 'application/json' } });772 if (!response.ok) throw new Error(`SerpApi website search returned HTTP ${response.status}.`);773 const body = await response.json() as Record<string, unknown>;774 if (body.error) throw new Error('SerpApi website search returned a provider error.');775 return parseSerpApiCandidates(body);776}777
778async function resolveWebsite(input: {779 productName: string;780 tagline: string;781 description: string;782 productHuntRedirect: string;783 makerWebsiteUrls: string[];784 dataForSeoLogin: string;785 dataForSeoPassword: string;786 serpApiKey: string;787}): Promise<WebsiteResolution> {788 const warnings: string[] = [];789 let providerCostUsd = 0;790 let providerAttempts = 0;791 const unresolved = (): WebsiteResolution => ({792 websiteUrl: null,793 source: 'none',794 status: input.productHuntRedirect ? 'unresolved' : 'not_available',795 confidence: 0,796 signals: [],797 providerCostUsd,798 providerAttempts,799 warnings,800 });801
802 if (input.productHuntRedirect) {803 const destination = await resolveProductHuntRedirect(input.productHuntRedirect);804 if (destination) {805 const verified = await verifyWebsiteCandidates([{806 url: destination, title: input.productName, snippet: input.tagline, rank: 1,807 }], input.productName, input.tagline, input.description);808 if (verified) {809 return {810 ...verified,811 source: 'product_hunt_redirect',812 status: 'verified_external',813 providerCostUsd,814 providerAttempts,815 warnings,816 };817 }818 }819 }820
821 const makerCandidates = input.makerWebsiteUrls822 .filter((url) => isAllowedExternalWebsite(url))823 .slice(0, 2)824 .map((url, index): SearchCandidate => ({ url, title: '', snippet: '', rank: index + 1 }));825 if (makerCandidates.length > 0) {826 const verified = await verifyWebsiteCandidates(makerCandidates, input.productName, input.tagline, input.description);827 if (verified) {828 return {829 ...verified,830 source: 'maker_website',831 status: 'verified_external',832 providerCostUsd,833 providerAttempts,834 warnings,835 };836 }837
838 const linkedCandidates = await discoverMakerLinkedProductCandidates(839 input.makerWebsiteUrls,840 input.productName,841 input.tagline,842 );843 if (linkedCandidates.length > 0) {844 const linked = await verifyWebsiteCandidates(linkedCandidates, input.productName, input.tagline, input.description);845 if (linked) {846 return {847 ...linked,848 signals: [...new Set([...linked.signals, 'product_linked_from_maker_website'])],849 source: 'maker_website_link',850 status: 'verified_external',851 providerCostUsd,852 providerAttempts,853 warnings,854 };855 }856 warnings.push('Maker websites exposed product-like links, but none passed product-identity verification.');857 }858 }859
860 const vendorCandidate = await verifyWebsiteCandidates(861 buildProtectedProductCandidates(input.productName).map((candidate) => ({862 ...candidate,863 snippet: input.tagline,864 })),865 input.productName,866 input.tagline,867 input.description,868 );869 if (vendorCandidate) {870 return {871 ...vendorCandidate,872 signals: [...new Set([...vendorCandidate.signals, 'protected_vendor_domain_confirmed'])],873 source: 'protected_vendor_domain',874 status: 'verified_external',875 providerCostUsd,876 providerAttempts,877 warnings,878 };879 }880
881 const queries = buildWebsiteSearchQueries(input.productName, input.tagline);882 if (input.dataForSeoLogin && input.dataForSeoPassword) {883 let queryIndex = 0;884 while (queryIndex < queries.length && providerAttempts < 2) {885 const query = queries[queryIndex];886 providerAttempts += 1;887 try {888 const search = await callDataForSeoWebsiteSearch(query, input.dataForSeoLogin, input.dataForSeoPassword);889 providerCostUsd += search.costUsd;890 const verified = await verifyWebsiteCandidates(891 search.candidates,892 input.productName,893 input.tagline,894 input.description,895 );896 if (verified) {897 return {898 ...verified,899 source: 'dataforseo_organic',900 status: 'verified_search_match',901 providerCostUsd,902 providerAttempts,903 warnings,904 };905 }906 warnings.push(`DataForSEO query ${queryIndex + 1} returned no website candidate that passed product-identity verification.`);907 queryIndex += 1;908 } catch (error) {909 if (error instanceof DataForSeoSearchError) providerCostUsd += error.costUsd;910 const message = error instanceof Error ? error.message : 'DataForSEO website search failed.';911 warnings.push(message.slice(0, 200));912 if (error instanceof DataForSeoSearchError && error.retryable && providerAttempts < 2) {913 continue;914 }915 queryIndex += 1;916 }917 }918 }919
920 const directDomain = await verifyWebsiteCandidates(921 buildDomainHypothesisCandidates(input.productName),922 input.productName,923 input.tagline,924 input.description,925 );926 if (directDomain) {927 return {928 ...directDomain,929 signals: [...new Set([...directDomain.signals, 'direct_domain_hypothesis_confirmed'])],930 source: 'direct_domain_hypothesis',931 status: 'verified_external',932 providerCostUsd,933 providerAttempts,934 warnings,935 };936 }937
938 if (input.serpApiKey) {939 providerAttempts += 1;940 try {941 const query = queries.at(-1) ?? buildWebsiteSearchQueries(input.productName, '')[0];942 const candidates = await callSerpApiWebsiteSearch(query, input.serpApiKey);943 const verified = await verifyWebsiteCandidates(candidates, input.productName, input.tagline, input.description);944 if (verified) {945 return {946 ...verified,947 source: 'serpapi_organic',948 status: 'verified_search_match',949 providerCostUsd,950 providerAttempts,951 warnings,952 };953 }954 warnings.push('SerpApi returned no website candidate that passed product-identity verification.');955 } catch (error) {956 const message = error instanceof Error ? error.message : 'SerpApi website search failed.';957 warnings.push(message.slice(0, 160));958 }959 }960
961 if (!input.dataForSeoLogin && !input.serpApiKey) {962 warnings.push('No owner-managed website search provider was configured for unresolved Product Hunt links.');963 }964 return unresolved();965}966
967function companyDomainFromWebsite(websiteUrl: string): string {968 try {969 const hostname = new URL(websiteUrl).hostname.toLowerCase().replace(/^www\./, '');970 return getDomain(hostname, { allowPrivateDomains: true }) ?? hostname;971 } catch {972 return '';973 }974}975
976async function requestEmailListVerifyContacts(977 apiKey: string,978 body: { domain: string; firstName?: string; lastName?: string },979): Promise<unknown> {980 await waitForEmailListVerifySlot();981 const response = await fetch(EMAIL_LIST_VERIFY_CONTACT_API, {982 method: 'POST',983 headers: { 'x-api-key': apiKey, 'Content-Type': 'application/json', Accept: 'application/json' },984 body: JSON.stringify(body),985 signal: AbortSignal.timeout(30_000),986 });987 if (!response.ok) {988 if (response.status === 402) emailListVerifyCreditsExhausted = true;989 throw new Error(`EmailListVerify Contact Finder returned HTTP ${response.status}.`);990 }991 return response.json();992}993
994async function discoverContacts(995 websiteUrl: string,996 makerNames: string[],997 maxEmails: number,998 apiKey: string,999): Promise<ContactDiscovery> {1000 const domain = companyDomainFromWebsite(websiteUrl);1001 if (!domain || maxEmails <= 0) return { contacts: [], attempts: 0, credits: 0, warnings: [] };1002 if (!apiKey) {1003 return {1004 contacts: [], attempts: 0, credits: 0,1005 warnings: ['Contact discovery was requested but the owner-managed EmailListVerify key is not configured.'],1006 };1007 }1008 if (emailListVerifyCreditsExhausted) {1009 return {1010 contacts: [], attempts: 0, credits: 0,1011 warnings: ['EmailListVerify credits were exhausted earlier in this run; contact discovery was skipped.'],1012 };1013 }1014
1015 const warnings: string[] = [];1016 let attempts = 0;1017 let credits = 0;1018 let contacts: ContactFinderResult[] = [];1019 const parsedMakers = makerNames.map(splitMakerName).filter(({ firstName, lastName }) => firstName || lastName);1020 const maker = parsedMakers.find(({ firstName, lastName }) => firstName && lastName) ?? parsedMakers[0];1021 if (maker) {1022 attempts += 1;1023 credits += 5;1024 try {1025 const payload = await requestEmailListVerifyContacts(apiKey, { domain, ...maker });1026 contacts = normalizeContactFinderResults(payload, domain, maxEmails);1027 if (contacts.length >= maxEmails) return { contacts, attempts, credits, warnings };1028 } catch (error) {1029 const message = error instanceof Error ? error.message : 'EmailListVerify named contact discovery failed.';1030 warnings.push(message.slice(0, 160));1031 }1032 }1033
1034 if (!emailListVerifyCreditsExhausted) {1035 attempts += 1;1036 credits += 10;1037 try {1038 const payload = await requestEmailListVerifyContacts(apiKey, { domain });1039 contacts = normalizeContactFinderResults([1040 ...contacts,1041 ...(Array.isArray(payload) ? payload : []),1042 ], domain, maxEmails);1043 return { contacts, attempts, credits, warnings };1044 } catch (error) {1045 const message = error instanceof Error ? error.message : 'EmailListVerify domain contact discovery failed.';1046 warnings.push(message.slice(0, 160));1047 }1048 }1049 return { contacts, attempts, credits, warnings };1050}1051
1052function selectBestEmails(1053 publicVerifications: EmailVerification[],1054 contacts: ContactFinderResult[],1055 maxEmails: number,1056): EmailVerification[] {1057 const contactVerifications: EmailVerification[] = contacts.map((contact) => ({1058 email: contact.email,1059 provider: 'emaillistverify',1060 status: contact.result as EmailListVerifyStatus,1061 checkedAt: new Date().toISOString(),1062 source: 'emaillistverify_contact_finder',1063 confidence: contact.confidence,1064 }));1065 const statusScore: Record<string, number> = {1066 ok: 100, ok_for_all: 70, unknown: 55, antispam_system: 50,1067 not_requested: 48, not_configured: 45, provider_error: 40, error_credit: 38,1068 smtp_protocol: 30, dead_server: 10, email_disabled: 5, invalid_mx: 0,1069 invalid_syntax: 0, disposable: 0, spamtrap: 0,1070 };1071 const best = new Map<string, EmailVerification>();1072 for (const verification of [...publicVerifications, ...contactVerifications]) {1073 const current = best.get(verification.email);1074 const sourceBonus = verification.source === 'public_website' ? 3 : verification.source === 'maker_website' ? 2 : 0;1075 const score = (statusScore[verification.status] ?? 20)1076 + sourceBonus1077 + (verification.confidence === 'high' ? 2 : verification.confidence === 'medium' ? 1 : 0);1078 const currentScore = current1079 ? (statusScore[current.status] ?? 20) + (current.source === 'public_website' ? 3 : current.source === 'maker_website' ? 2 : 0)1080 : -1;1081 if (!current || score > currentScore) best.set(verification.email, verification);1082 }1083 return [...best.values()]1084 .filter(({ status }) => isUsableEmailStatus(status))1085 .sort((left, right) => {1086 const leftScore = (statusScore[left.status] ?? 20) + (left.source === 'public_website' ? 3 : left.source === 'maker_website' ? 2 : 0);1087 const rightScore = (statusScore[right.status] ?? 20) + (right.source === 'public_website' ? 3 : right.source === 'maker_website' ? 2 : 0);1088 return rightScore - leftScore;1089 })1090 .slice(0, maxEmails);1091}1092
1093async function findEmails(1094 rawWebsiteUrl: string,1095 maxWebsitePages: number,1096 maxEmailsPerProduct: number,1097): Promise<EnrichmentResult> {1098 const websiteUrl = canonicalWebsiteUrl(rawWebsiteUrl);1099 if (!websiteUrl || !isAllowedExternalWebsite(websiteUrl)) {1100 return {1101 emails: [], emailSource: 'none', result: 'not_assessed', coverage: 'none',1102 terminalReason: 'invalid_or_private_target', pagesVisited: 0, pageUrls: [],1103 warnings: ['Website URL was not a permitted public external HTTP(S) target.'],1104 };1105 }1106
1107 const initialUrl = new URL(websiteUrl);1108 const maxEmailCandidates = emailCandidateLimit(maxEmailsPerProduct);1109 const warnings: string[] = [];1110 const pageUrls: string[] = [];1111 const addPageUrl = (url: string): void => {1112 const canonical = canonicalWebsiteUrl(url);1113 if (canonical && !pageUrls.includes(canonical) && pageUrls.length < maxWebsitePages) pageUrls.push(canonical);1114 };1115 let canonicalOrigin = initialUrl.origin;1116 let baseDomain = initialUrl.hostname.replace(/^www\./i, '');1117 let emails: string[] = [];1118 let coverage: Coverage = 'complete';1119 let pagePlan: WebsitePage[] = [];1120
1121 const staticHomepage = await fetchPublicHtml(`${initialUrl.origin}/`, 4_000);1122 if (staticHomepage.ok) {1123 const finalHomepage = new URL(staticHomepage.url);1124 canonicalOrigin = finalHomepage.origin;1125 baseDomain = finalHomepage.hostname.replace(/^www\./i, '');1126 addPageUrl(staticHomepage.url);1127 emails = rankEmails(extractStaticEmails(staticHomepage.html), baseDomain, maxEmailCandidates);1128 pagePlan = buildWebsitePagePlan(`${canonicalOrigin}/`, extractStaticFallbackPages(staticHomepage.html, canonicalOrigin), maxWebsitePages);1129 for (const target of pagePlan.slice(1)) {1130 if (emails.length >= maxEmailCandidates) break;1131 const staticPage = await fetchPublicHtml(target.url, 3_000);1132 if (!staticPage.ok) {1133 coverage = 'partial';1134 warnings.push(`${target.kind} static assessment failed: ${staticPage.reason}.`);1135 continue;1136 }1137 const finalPage = new URL(staticPage.url);1138 if (finalPage.origin !== canonicalOrigin) {1139 coverage = 'partial';1140 warnings.push(`${target.kind} redirected off the verified homepage origin; it was not extracted.`);1141 continue;1142 }1143 addPageUrl(staticPage.url);1144 emails = rankEmails([...emails, ...extractStaticEmails(staticPage.html)], baseDomain, maxEmailCandidates);1145 }1146 } else {1147 coverage = 'partial';1148 warnings.push(`Homepage static assessment failed: ${staticHomepage.reason}.`);1149 }1150
1151 if (emails.length < maxEmailCandidates) {1152 const context = await (await ensureBrowser()).newContext({1153 userAgent: WEBSITE_USER_AGENT,1154 locale: 'en-US',1155 viewport: { width: 1280, height: 800 },1156 extraHTTPHeaders: { 'Accept-Language': 'en-US,en;q=0.9' },1157 });1158 await context.route('**/*', async (route) => {1159 if (['image', 'media', 'font'].includes(route.request().resourceType())) return route.abort();1160 const requestUrl = route.request().url();1161 if (!/^(?:data|blob|about):/i.test(requestUrl) && !await isSafePublicNetworkUrl(requestUrl)) return route.abort();1162 return route.continue();1163 });1164 try {1165 const page = await context.newPage();1166 const response = await page.goto(`${canonicalOrigin}/`, { waitUntil: 'domcontentloaded', timeout: 6_000 });1167 const finalHomepage = new URL(page.url());1168 if (!response || finalHomepage.origin !== canonicalOrigin) {1169 coverage = 'partial';1170 warnings.push('Homepage DOM assessment redirected off origin or did not return a response.');1171 } else {1172 addPageUrl(finalHomepage.toString());1173 await page.locator('body').waitFor({ state: 'attached', timeout: 750 }).catch(() => undefined);1174 emails = rankEmails([...emails, ...await page.evaluate(browserExtractEmails)], baseDomain, maxEmailCandidates);1175 const browserFallbackPages = await page.evaluate(browserExtractLinkedFallbackPages, canonicalOrigin);1176 pagePlan = buildWebsitePagePlan(1177 `${canonicalOrigin}/`,1178 [...pagePlan.slice(1), ...browserFallbackPages],1179 maxWebsitePages,1180 );1181 for (const target of pagePlan.slice(1)) {1182 if (emails.length >= maxEmailCandidates) break;1183 try {1184 const fallbackResponse = await page.goto(target.url, { waitUntil: 'domcontentloaded', timeout: 4_500 });1185 const finalFallback = new URL(page.url());1186 if (!fallbackResponse || finalFallback.origin !== canonicalOrigin) {1187 coverage = 'partial';1188 warnings.push(`${target.kind} DOM assessment redirected off origin.`);1189 continue;1190 }1191 addPageUrl(finalFallback.toString());1192 await page.locator('body').waitFor({ state: 'attached', timeout: 750 }).catch(() => undefined);1193 emails = rankEmails([...emails, ...await page.evaluate(browserExtractEmails)], baseDomain, maxEmailCandidates);1194 } catch (error) {1195 coverage = 'partial';1196 const message = error instanceof Error ? error.message : 'navigation error';1197 warnings.push(`${target.kind} DOM assessment failed: ${message.slice(0, 90)}.`);1198 }1199 }1200 }1201 } catch (error) {1202 coverage = 'partial';1203 const message = error instanceof Error ? error.message : 'browser error';1204 warnings.push(`Homepage DOM assessment failed: ${message.slice(0, 90)}.`);1205 } finally {1206 await context.close();1207 }1208 }1209
1210 const result: EmailResult = emails.length > 01211 ? 'emails_found'1212 : pageUrls.length > 0 && coverage === 'complete'1213 ? 'no_public_email_found'1214 : 'not_assessed';1215 return {1216 emails,1217 emailSource: emails.length > 0 ? 'page_scrape' : 'none',1218 result,1219 coverage: pageUrls.length === 0 ? 'none' : coverage,1220 terminalReason: emails.length >= maxEmailCandidates1221 ? 'max_public_email_candidates_reached'1222 : result === 'no_public_email_found'1223 ? 'eligible_pages_exhausted'1224 : 'eligible_pages_partially_assessed',1225 pagesVisited: pageUrls.length,1226 pageUrls,1227 warnings,1228 };1229}1230
1231async function findMakerHomepageEmails(1232 makerWebsiteUrls: string[],1233 productWebsiteEmails: string[],1234 maxEmailsPerProduct: number,1235 filledEmailSlots: number,1236): Promise<{ emails: string[]; pageUrls: string[]; warnings: string[] }> {1237 const candidates: string[] = [];1238 const pageUrls: string[] = [];1239 const warnings: string[] = [];1240 const seenOrigins = new Set<string>();1241
1242 for (const rawUrl of makerWebsiteUrls) {1243 if (selectMakerFallbackEmails(productWebsiteEmails, candidates, maxEmailsPerProduct, filledEmailSlots).length1244 >= Math.max(0, maxEmailsPerProduct - filledEmailSlots)) break;1245 const websiteUrl = canonicalWebsiteUrl(rawUrl);1246 if (!websiteUrl || !isAllowedExternalWebsite(websiteUrl)) continue;1247 const origin = new URL(websiteUrl).origin;1248 if (seenOrigins.has(origin) || seenOrigins.size >= 2) continue;1249 seenOrigins.add(origin);1250
1251 const staticPage = await fetchPublicHtml(`${origin}/`, 4_000);1252 if (staticPage.ok) {1253 pageUrls.push(staticPage.url);1254 const emailDomain = new URL(staticPage.url).hostname.replace(/^www\./i, '');1255 candidates.push(...rankEmails(extractStaticEmails(staticPage.html), emailDomain, maxEmailsPerProduct));1256 if (selectMakerFallbackEmails(productWebsiteEmails, candidates, maxEmailsPerProduct, filledEmailSlots).length > 0) continue;1257 } else {1258 warnings.push(`Maker homepage static assessment failed: ${staticPage.reason}.`);1259 }1260
1261 const context = await (await ensureBrowser()).newContext({1262 userAgent: WEBSITE_USER_AGENT,1263 locale: 'en-US',1264 viewport: { width: 1280, height: 800 },1265 extraHTTPHeaders: { 'Accept-Language': 'en-US,en;q=0.9' },1266 });1267 await context.route('**/*', async (route) => {1268 if (['image', 'media', 'font'].includes(route.request().resourceType())) return route.abort();1269 const requestUrl = route.request().url();1270 if (!/^(?:data|blob|about):/i.test(requestUrl) && !await isSafePublicNetworkUrl(requestUrl)) return route.abort();1271 return route.continue();1272 });1273 try {1274 const page = await context.newPage();1275 const response = await page.goto(`${origin}/`, { waitUntil: 'domcontentloaded', timeout: 10_000 });1276 if (!response || response.status() >= 400 || new URL(page.url()).origin !== origin) {1277 warnings.push('Maker homepage browser assessment did not reach the approved origin.');1278 continue;1279 }1280 if (!pageUrls.includes(page.url())) pageUrls.push(page.url());1281 const emailDomain = new URL(page.url()).hostname.replace(/^www\./i, '');1282 candidates.push(...rankEmails(await page.evaluate(browserExtractEmails), emailDomain, maxEmailsPerProduct));1283 } catch {1284 warnings.push('Maker homepage browser assessment failed.');1285 } finally {1286 await context.close();1287 }1288 }1289
1290 return {1291 emails: selectMakerFallbackEmails(productWebsiteEmails, candidates, maxEmailsPerProduct, filledEmailSlots),1292 pageUrls,1293 warnings,1294 };1295}1296
1297const POST_FIELDS = `1298 id name tagline description votesCount commentsCount url website slug1299 featuredAt createdAt dailyRank weeklyRank monthlyRank yearlyRank1300 reviewsCount reviewsRating1301 thumbnail { url }1302 productLinks { type url }1303 topics(first: 10) { edges { node { name } } }1304 makers { id name username twitterUsername url websiteUrl }1305`;1306
1307const POSTS_QUERY = `query GetPosts($first:Int $after:String $order:PostsOrder $featured:Boolean $postedAfter:DateTime $postedBefore:DateTime $topic:String) {1308 posts(first:$first after:$after order:$order featured:$featured postedAfter:$postedAfter postedBefore:$postedBefore topic:$topic) {1309 edges { node { ${POST_FIELDS} } }1310 pageInfo { endCursor hasNextPage }1311 }1312}`;1313
1314const POSTS_INDEX_QUERY = `query GetPostIndex($first:Int $after:String $order:PostsOrder $featured:Boolean $postedAfter:DateTime $postedBefore:DateTime) {1315 posts(first:$first after:$after order:$order featured:$featured postedAfter:$postedAfter postedBefore:$postedBefore) {1316 edges { node { id url slug } }1317 pageInfo { endCursor hasNextPage }1318 }1319}`;1320
1321const POST_BY_SLUG_QUERY = `query GetPost($slug:String!) { post(slug:$slug) { ${POST_FIELDS} } }`;1322
1323function toInt(value: unknown): number | null {1324 if (value === null || value === undefined || value === '') return null;1325 const parsed = Number.parseInt(String(value), 10);1326 return Number.isNaN(parsed) ? null : parsed;1327}1328
1329function toFloat(value: unknown): number | null {1330 if (value === null || value === undefined || value === '') return null;1331 const parsed = Number.parseFloat(String(value));1332 return Number.isNaN(parsed) ? null : parsed;1333}1334
1335function nodeToRecord(node: Record<string, any>, rank: number): Record<string, unknown> {1336 const makers = (node.makers ?? []).map((maker: Record<string, unknown>) => ({1337 maker_name: maker.name ?? null,1338 maker_id: maker.username ?? null,1339 maker_ph_id: maker.id ?? null,1340 twitter_url: maker.twitterUsername ? `https://twitter.com/${String(maker.twitterUsername)}` : null,1341 maker_url: maker.url ?? null,1342 website_url: maker.websiteUrl ?? null,1343 }));1344 const topics = (node.topics?.edges ?? []).map((edge: any) => edge?.node?.name).filter(Boolean);1345 const productLinks = Array.isArray(node.productLinks) ? node.productLinks : [];1346 const directWebsite = productLinks.find((link: Record<string, unknown>) => {1347 const type = String(link.type ?? '').toLowerCase();1348 return type === 'website' && typeof link.url === 'string' && isAllowedExternalWebsite(link.url);1349 })?.url as string | undefined;1350
1351 const websiteUrl = directWebsite ? canonicalWebsiteUrl(directWebsite) : null;1352 return {1353 product_name: node.name ? String(node.name).trim() : null,1354 tagline: node.tagline ?? null,1355 description: node.description ?? null,1356 upvote_count: toInt(node.votesCount),1357 comment_count: toInt(node.commentsCount),1358 reviews_count: toInt(node.reviewsCount),1359 reviews_rating: toFloat(node.reviewsRating),1360 daily_rank: toInt(node.dailyRank) ?? rank,1361 weekly_rank: toInt(node.weeklyRank),1362 monthly_rank: toInt(node.monthlyRank),1363 yearly_rank: toInt(node.yearlyRank),1364 launch_date: node.featuredAt ? isoDate(new Date(node.featuredAt)) : node.createdAt ? isoDate(new Date(node.createdAt)) : null,1365 product_hunt_url: node.url ?? (node.slug ? `https://www.producthunt.com/posts/${node.slug}` : null),1366 product_hunt_website_url: node.website ?? null,1367 website_url: websiteUrl,1368 website_resolution_source: websiteUrl ? 'api_website_product_link' satisfies WebsiteResolutionSource : 'none' satisfies WebsiteResolutionSource,1369 website_resolution_status: websiteUrl1370 ? 'verified_external' satisfies WebsiteResolutionStatus1371 : node.website1372 ? 'unresolved' satisfies WebsiteResolutionStatus1373 : 'not_available' satisfies WebsiteResolutionStatus,1374 website_resolution_confidence: websiteUrl ? 100 : 0,1375 website_resolution_signals: websiteUrl ? ['official_api_product_link'] : [],1376 website_resolution_provider_cost_usd: 0,1377 company_domain: websiteUrl ? companyDomainFromWebsite(websiteUrl) : null,1378 topics,1379 thumbnail_url: node.thumbnail?.url ?? null,1380 makers,1381 featured: Boolean(node.featuredAt),1382 emails: [] as string[],1383 verified_emails: [] as string[],1384 risky_emails: [] as string[],1385 rejected_emails: [] as string[],1386 rejected_email_verifications: [] as EmailVerification[],1387 email_verifications: [] as EmailVerification[],1388 email_verification_requested: false,1389 email_source: 'none' as EmailSource,1390 email_result: 'not_assessed' as EmailResult,1391 email_coverage: 'none' as Coverage,1392 email_terminal_reason: 'website_not_available',1393 email_pages_visited: 0,1394 email_page_urls: [] as string[],1395 maker_email_pages_visited: 0,1396 maker_email_page_urls: [] as string[],1397 contact_discovery_requested: false,1398 contact_discovery_attempts: 0,1399 contact_discovery_credits: 0,1400 lead_status: 'unresolved_website',1401 lead_score: 0,1402 lead_tier: 'unqualified',1403 lead_signals: [] as string[],1404 enrichment_warnings: [] as string[],1405 scraped_at: new Date().toISOString(),1406 };1407}1408
1409const LEAN_FIELDS = new Set([1410 'product_name', 'tagline', 'website_url', 'company_domain', 'product_hunt_website_url', 'website_resolution_status',1411 'website_resolution_source', 'website_resolution_confidence', 'upvote_count', 'daily_rank', 'emails', 'verified_emails',1412 'risky_emails', 'rejected_emails', 'email_verifications', 'rejected_email_verifications', 'email_verification_requested', 'email_source', 'email_result', 'email_coverage',1413 'email_pages_visited', 'contact_discovery_attempts', 'lead_status', 'lead_score', 'lead_tier', 'lead_signals',1414 'maker_email_pages_visited',1415 'topics', 'launch_date', 'product_hunt_url', 'featured', 'scraped_at',1416]);1417
1418function applyOutputMode(record: Record<string, unknown>, outputMode: 'full' | 'lean' | 'leads'): Record<string, unknown> | null {1419 if (outputMode === 'leads' && (!Array.isArray(record.emails) || record.emails.length === 0)) return null;1420 if (outputMode !== 'lean') return record;1421 const output: Record<string, unknown> = {};1422 for (const field of LEAN_FIELDS) output[field] = record[field] ?? null;1423 return output;1424}1425
1426function validateOutputRecord(record: Record<string, unknown>): string | null {1427 if (typeof record.product_name !== 'string' || !record.product_name.trim()) return 'product_name is missing.';1428 if (typeof record.product_hunt_url !== 'string' || !record.product_hunt_url.startsWith('https://')) return 'product_hunt_url is missing or invalid.';1429 if (!Array.isArray(record.emails) || !record.emails.every((email) => typeof email === 'string')) return 'emails must be a string array.';1430 if (typeof record.scraped_at !== 'string' || Number.isNaN(Date.parse(record.scraped_at))) return 'scraped_at is invalid.';1431 return null;1432}1433
1434async function run(): Promise<void> {1435 const rawInput = (await Actor.getInput() ?? {}) as Record<string, unknown>;1436 const mode = (rawInput.mode ?? rawInput.scrapeMode ?? 'leaderboard') as Mode;1437 const leaderboardPeriod = (rawInput.leaderboardPeriod ?? rawInput.period ?? 'daily') as LeaderboardPeriod;1438 const startDate = String(rawInput.startDate ?? rawInput.date ?? '');1439 const endDate = String(rawInput.endDate ?? '');1440 const lookbackDays = clampInteger(rawInput.lookbackDays, 1, 1, 365);1441 const outputMode = (rawInput.outputMode ?? 'full') as 'full' | 'lean' | 'leads';1442 const searchQuery = typeof (rawInput.searchQuery ?? rawInput.query ?? rawInput.q) === 'string'1443 ? String(rawInput.searchQuery ?? rawInput.query ?? rawInput.q).trim()1444 : '';1445 const topic = typeof (rawInput.topic ?? rawInput.topicSlug) === 'string'1446 ? String(rawInput.topic ?? rawInput.topicSlug).trim()1447 : '';1448 const startUrls = Array.isArray(rawInput.startUrls) ? rawInput.startUrls as Array<{ url?: unknown } | string> : [];1449 const maxResults = clampInteger(rawInput.maxResults ?? rawInput.maxItems ?? rawInput.limit, 100, 1, 20_000);1450 const maxConcurrency = clampInteger(rawInput.maxConcurrency, 3, 1, 5);1451 const maxWebsitePages = clampInteger(rawInput.maxWebsitePages, DEFAULT_MAX_WEBSITE_PAGES, 1, MAX_WEBSITE_PAGES);1452 const maxEmailsPerProduct = clampInteger(rawInput.maxEmailsPerProduct, DEFAULT_MAX_EMAILS_PER_PRODUCT, 1, MAX_EMAILS_PER_PRODUCT);1453 const includeAllProducts = Boolean(rawInput.includeAllProducts ?? rawInput.allProducts ?? rawInput.includeAll ?? true);1454 const enrichEmails = Boolean(rawInput.enrichEmails ?? rawInput.findEmails ?? true);1455 const resolveWebsites = Boolean(rawInput.resolveWebsites ?? true);1456 const verifyEmails = Boolean(rawInput.verifyEmails ?? rawInput.verifyWithEmailListVerify ?? true);1457 const findContacts = Boolean(rawInput.findContacts ?? true);1458 const emailListVerifyApiKey = String(process.env.EMAILLISTVERIFY_API_KEY ?? '').trim();1459 const phApiToken = String(process.env.PH_API_TOKEN ?? '');1460 const dataForSeoLogin = String(process.env.DATAFORSEO_LOGIN ?? '').trim();1461 const dataForSeoPassword = String(process.env.DATAFORSEO_PASSWORD ?? '').trim();1462 const serpApiKey = String(process.env.SERPAPI_KEY ?? '').trim();1463 const websiteOverrides = parseWebsiteOverrides(rawInput.websiteUrlOverrides);1464 const excludedProductHuntUrls = Array.isArray(rawInput.excludeProductHuntUrls)1465 ? rawInput.excludeProductHuntUrls1466 .slice(0, 20_000)1467 .map((value) => productHuntLaunchKey(String(value ?? '')))1468 .filter((value): value is string => Boolean(value))1469 : [];1470 const excludedProductHuntKeys = new Set(excludedProductHuntUrls);1471 const productHuntResumeCursor = typeof rawInput.productHuntResumeCursor === 'string'1472 ? rawInput.productHuntResumeCursor.trim().slice(0, 2_000)1473 : '';1474 const startedAt = new Date().toISOString();1475 emailListVerifyCreditsExhausted = false;1476
1477 const diagnostics = {1478 itemsPushed: 0,1479 itemsProcessed: 0,1480 itemsFiltered: 0,1481 itemsExcluded: 0,1482 itemsFailed: 0,1483 invalidRecords: 0,1484 apiFailures: 0,1485 productHuntIndexPages: 0,1486 productHuntDetailLookups: 0,1487 productHuntFullPageLookups: 0,1488 nextProductHuntCursor: null as string | null,1489 websiteResolved: 0,1490 websiteUnresolved: 0,1491 websiteResolutionProviderAttempts: 0,1492 websiteResolutionProviderCostUsd: 0,1493 emailsFound: 0,1494 verifiedEmailsFound: 0,1495 emailVerificationAttempts: 0,1496 emailVerificationProviderErrors: 0,1497 makerEmailPagesVisited: 0,1498 makerEmailsFound: 0,1499 contactDiscoveryAttempts: 0,1500 contactDiscoveryCredits: 0,1501 contactDiscoveryEmailsFound: 0,1502 leadTiers: { A: 0, B: 0, C: 0, unqualified: 0 },1503 resultEvents: 0,1504 emailEvents: 0,1505 chargeLimitReached: false,1506 resultLimitReached: false,1507 warnings: [] as string[],1508 };1509 let finalised = false;1510
1511 const paidItemLimit = clampInteger(process.env.ACTOR_MAX_PAID_DATASET_ITEMS, maxResults, 1, maxResults);1512 const effectiveMaxResults = Math.min(maxResults, paidItemLimit);1513
1514 const addWarning = (message: string): void => {1515 if (diagnostics.warnings.length < 100 && !diagnostics.warnings.includes(message)) diagnostics.warnings.push(message);1516 };1517
1518 const finalizeRun = async (outcome: RunOutcome, message: string): Promise<never> => {1519 if (finalised) throw new Error(message);1520 finalised = true;1521 const finishedAt = new Date().toISOString();1522 const summary = {1523 outcome,1524 message,1525 startedAt,1526 finishedAt,1527 buildNumber: process.env.ACTOR_BUILD_NUMBER ?? null,1528 runId: process.env.ACTOR_RUN_ID ?? null,1529 datasetId: process.env.ACTOR_DEFAULT_DATASET_ID ?? null,1530 keyValueStoreId: process.env.ACTOR_DEFAULT_KEY_VALUE_STORE_ID ?? null,1531 input: {1532 mode, leaderboardPeriod, startDate: startDate || null, endDate: endDate || null,1533 lookbackDays, maxResults, effectiveMaxResults, includeAllProducts, enrichEmails, resolveWebsites, verifyEmails, findContacts,1534 excludedProductHuntUrls: excludedProductHuntKeys.size,1535 productHuntResumeCursorUsed: Boolean(productHuntResumeCursor),1536 emailListVerifyConfigured: Boolean(emailListVerifyApiKey), maxWebsitePages,1537 websiteSearchConfigured: Boolean((dataForSeoLogin && dataForSeoPassword) || serpApiKey),1538 maxEmailsPerProduct, outputMode,1539 },1540 ...diagnostics,1541 chargedEventCounts: {1542 [RESULT_EVENT]: diagnostics.resultEvents,1543 [EMAIL_EVENT]: diagnostics.emailEvents,1544 },1545 };1546 const output = {1547 outcome,1548 message,1549 itemsPushed: diagnostics.itemsPushed,1550 itemsFailed: diagnostics.itemsFailed,1551 invalidRecords: diagnostics.invalidRecords,1552 chargedEventCounts: summary.chargedEventCounts,1553 warnings: diagnostics.warnings,1554 };1555 await Actor.setValue('OUTPUT', output);1556 await Actor.setValue('RUN_SUMMARY', summary);1557 const failed = FAILED_OUTCOMES.has(outcome);1558 await Actor.setStatusMessage(message, {1559 isStatusMessageTerminal: true,1560 level: failed ? 'ERROR' : outcome === 'COMPLETE' ? 'INFO' : 'WARNING',1561 });1562 if (failed) {1563 // release-validator: allow-actor-fail only CONFIG_ERROR and UPSTREAM_FAILED are failed after OUTPUT and RUN_SUMMARY are persisted.1564 await Actor.fail(message);1565 }1566 await Actor.exit();1567 throw new Error('Actor terminal call returned unexpectedly.');1568 };1569
1570 try {1571 const modes: Mode[] = ['leaderboard', 'search', 'topic', 'urls'];1572 const periods: LeaderboardPeriod[] = ['daily', 'weekly', 'monthly', 'yearly'];1573 const outputModes = ['full', 'lean', 'leads'];1574 if (!modes.includes(mode)) await finalizeRun('INVALID_INPUT', 'mode must be leaderboard, search, topic, or urls.');1575 if (!periods.includes(leaderboardPeriod)) await finalizeRun('INVALID_INPUT', 'leaderboardPeriod must be daily, weekly, monthly, or yearly.');1576 if (!outputModes.includes(outputMode)) await finalizeRun('INVALID_INPUT', 'outputMode must be full, lean, or leads.');1577 if (!phApiToken) await finalizeRun('CONFIG_ERROR', 'Product Hunt API token is not configured. Set the owner-managed PH_API_TOKEN secret and re-run.');1578 if (mode === 'search' && !searchQuery) await finalizeRun('INVALID_INPUT', 'searchQuery is required for search mode.');1579 if (mode === 'topic' && !topic) await finalizeRun('INVALID_INPUT', 'topic is required for topic mode.');1580 if (mode === 'urls' && startUrls.length === 0) await finalizeRun('INVALID_INPUT', 'startUrls is required for urls mode.');1581 if (startDate && !validIsoDate(startDate)) await finalizeRun('INVALID_INPUT', 'startDate must be a valid YYYY-MM-DD date.');1582 if (endDate && !validIsoDate(endDate)) await finalizeRun('INVALID_INPUT', 'endDate must be a valid YYYY-MM-DD date.');1583 if (startDate && endDate && startDate > endDate) await finalizeRun('INVALID_INPUT', 'endDate must be on or after startDate.');1584 if (leaderboardPeriod !== 'daily' && endDate) await finalizeRun('INVALID_INPUT', 'endDate is supported only with daily leaderboard ranges.');1585
1586 const maxPotentialEventCost = effectiveMaxResults * (RESULT_PRICE_USD + (enrichEmails ? maxEmailsPerProduct * EMAIL_PRICE_USD : 0));1587 const maxPotentialVerificationCredits = enrichEmails && emailListVerifyApiKey1588 ? effectiveMaxResults * ((verifyEmails ? maxEmailsPerProduct * 3 * 2 : 0) + (findContacts ? 15 : 0))1589 : 0;1590 log.info('ProductHunt Scraper starting', {1591 mode, leaderboardPeriod, startDate: startDate || '(default)', endDate: endDate || null,1592 lookbackDays, effectiveMaxResults, includeAllProducts, enrichEmails, resolveWebsites, verifyEmails, findContacts,1593 emailListVerifyConfigured: Boolean(emailListVerifyApiKey), maxWebsitePages, maxEmailsPerProduct,1594 websiteSearchProviders: {1595 dataForSeo: Boolean(dataForSeoLogin && dataForSeoPassword),1596 serpApiFallback: Boolean(serpApiKey),1597 },1598 eventCostCeilingUsd: Number(maxPotentialEventCost.toFixed(3)),1599 excludedProductHuntUrls: excludedProductHuntKeys.size,1600 emailListVerifyCreditCeiling: maxPotentialVerificationCredits,1601 websiteSearchQueryCeiling: resolveWebsites1602 ? effectiveMaxResults * ((dataForSeoLogin && dataForSeoPassword ? 2 : 0) + (serpApiKey ? 1 : 0))1603 : 0,1604 platformUsage: 'Additional compute and proxy usage is billed by Apify separately where applicable.',1605 });1606 await Actor.setStatusMessage(`Starting: up to ${effectiveMaxResults} products, ${maxWebsitePages} public website pages/product, and ${maxEmailsPerProduct} public emails/product.`);1607
1608 const phQuery = async (query: string, variables: Record<string, unknown>, attempt = 0): Promise<any> => {1609 const response = await fetch(PH_API, {1610 method: 'POST',1611 headers: {1612 'Content-Type': 'application/json',1613 Accept: 'application/json',1614 Authorization: `Bearer ${phApiToken}`,1615 'User-Agent': PRODUCT_HUNT_USER_AGENT,1616 },1617 body: JSON.stringify({ query, variables }),1618 signal: AbortSignal.timeout(30_000),1619 });1620 if (response.status === 429 && attempt < 3) {1621 const waitSeconds = productHuntRateLimitDelaySeconds(1622 response.headers.get('x-rate-limit-reset'),1623 response.headers.get('retry-after'),1624 );1625 log.warning(`Product Hunt API rate limited; waiting ${waitSeconds}s before retry ${attempt + 1}/3.`);1626 await new Promise((resolve) => setTimeout(resolve, waitSeconds * 1000));1627 return phQuery(query, variables, attempt + 1);1628 }1629 if (!response.ok) throw new Error(`Product Hunt API returned HTTP ${response.status}.`);1630 const payload = await response.json() as { data?: unknown; errors?: Array<{ message?: string }> };1631 if (payload.errors?.length) throw new Error(`Product Hunt GraphQL: ${payload.errors[0]?.message ?? 'unknown error'}`);1632 return payload.data as any;1633 };1634
1635 const seenIds = new Set<string>();1636 let collectedCandidates = 0;1637
1638 const pushNode = async (node: Record<string, any>, rank: number): Promise<void> => {1639 const record = nodeToRecord(node, rank);1640 record.email_verification_requested = verifyEmails;1641 const override = lookupWebsiteOverride(node, websiteOverrides);1642 if (override) {1643 record.website_url = override;1644 record.website_resolution_source = 'input_override' satisfies WebsiteResolutionSource;1645 record.website_resolution_status = 'verified_external' satisfies WebsiteResolutionStatus;1646 record.website_resolution_confidence = 100;1647 record.website_resolution_signals = ['approved_input_override'];1648 }1649
1650 let websiteUrl = typeof record.website_url === 'string' ? record.website_url : '';1651 if (!websiteUrl && resolveWebsites) {1652 const resolution = await resolveWebsite({1653 productName: String(record.product_name ?? ''),1654 tagline: String(record.tagline ?? ''),1655 description: String(record.description ?? ''),1656 productHuntRedirect: typeof record.product_hunt_website_url === 'string' ? record.product_hunt_website_url : '',1657 makerWebsiteUrls: (node.makers ?? [])1658 .map((maker: Record<string, unknown>) => typeof maker.websiteUrl === 'string' ? maker.websiteUrl : '')1659 .filter(Boolean),1660 dataForSeoLogin,1661 dataForSeoPassword,1662 serpApiKey,1663 });1664 diagnostics.websiteResolutionProviderAttempts += resolution.providerAttempts;1665 diagnostics.websiteResolutionProviderCostUsd += resolution.providerCostUsd;1666 resolution.warnings.forEach(addWarning);1667 record.website_resolution_provider_cost_usd = Number(resolution.providerCostUsd.toFixed(6));1668 if (resolution.websiteUrl) {1669 websiteUrl = resolution.websiteUrl;1670 record.website_url = resolution.websiteUrl;1671 record.website_resolution_source = resolution.source;1672 record.website_resolution_status = resolution.status;1673 record.website_resolution_confidence = resolution.confidence;1674 record.website_resolution_signals = resolution.signals;1675 }1676 }1677
1678 if (websiteUrl) {1679 diagnostics.websiteResolved += 1;1680 record.company_domain = companyDomainFromWebsite(websiteUrl);1681 } else {1682 diagnostics.websiteUnresolved += 1;1683 record.website_resolution_status = record.product_hunt_website_url ? 'unresolved' : 'not_available';1684 }1685
1686 let publicEmailCount = 0;1687 let publicVerifications: EmailVerification[] = [];1688 if (enrichEmails && websiteUrl) {1689 const enriched = await findEmails(websiteUrl, maxWebsitePages, maxEmailsPerProduct);1690 record.emails = enriched.emails;1691 record.email_source = enriched.emailSource;1692 record.email_result = enriched.result;1693 record.email_coverage = enriched.coverage;1694 record.email_terminal_reason = enriched.terminalReason;1695 record.email_pages_visited = enriched.pagesVisited;1696 record.email_page_urls = enriched.pageUrls;1697 record.enrichment_warnings = enriched.warnings;1698 publicEmailCount = enriched.emails.length;1699 enriched.warnings.forEach(addWarning);1700 log.info('Website enrichment complete', {1701 product: record.product_name,1702 emailsFound: enriched.emails.length,1703 source: enriched.emailSource,1704 pagesVisited: enriched.pagesVisited,1705 coverage: enriched.coverage,1706 terminalReason: enriched.terminalReason,1707 });1708 }1709
1710 const selectedEmails = Array.isArray(record.emails) ? record.emails.filter((email): email is string => typeof email === 'string') : [];1711 if (selectedEmails.length > 0) {1712 const verification = await verifyEmailsWithEmailListVerify(1713 selectedEmails,1714 verifyEmails,1715 emailListVerifyApiKey,1716 'public_website',1717 maxEmailsPerProduct,1718 );1719 publicVerifications = verification.verifications;1720 diagnostics.emailVerificationAttempts += verification.attempts;1721 diagnostics.emailVerificationProviderErrors += verification.verifications.filter(({ status }) => status === 'provider_error' || status === 'error_credit').length;1722 verification.warnings.forEach(addWarning);1723 log.info('Public email verification complete', {1724 product: record.product_name,1725 emailsAssessed: selectedEmails.length,1726 requested: verifyEmails,1727 attempts: verification.attempts,1728 statuses: verification.verifications.map(({ status }) => status),1729 });1730 }1731
1732 const makerWebsiteUrls = (node.makers ?? [])1733 .map((maker: Record<string, unknown>) => typeof maker.websiteUrl === 'string' ? maker.websiteUrl : '')1734 .filter(Boolean);1735 const productDeliverableCount = verifyEmails1736 ? publicVerifications.filter(({ status }) => status === 'ok').length1737 : publicVerifications.filter(({ status }) => isUsableEmailStatus(status)).length;1738 if (enrichEmails && productDeliverableCount < maxEmailsPerProduct && makerWebsiteUrls.length > 0) {1739 const makerEnrichment = await findMakerHomepageEmails(1740 makerWebsiteUrls,1741 selectedEmails,1742 maxEmailsPerProduct,1743 productDeliverableCount,1744 );1745 record.maker_email_pages_visited = makerEnrichment.pageUrls.length;1746 record.maker_email_page_urls = makerEnrichment.pageUrls;1747 diagnostics.makerEmailPagesVisited += makerEnrichment.pageUrls.length;1748 makerEnrichment.warnings.forEach(addWarning);1749 record.enrichment_warnings = [1750 ...new Set([1751 ...(Array.isArray(record.enrichment_warnings) ? record.enrichment_warnings as string[] : []),1752 ...makerEnrichment.warnings,1753 ]),1754 ];1755 if (makerEnrichment.emails.length > 0) {1756 const makerVerification = await verifyEmailsWithEmailListVerify(1757 makerEnrichment.emails,1758 verifyEmails,1759 emailListVerifyApiKey,1760 'maker_website',1761 );1762 publicVerifications.push(...makerVerification.verifications);1763 publicEmailCount += makerEnrichment.emails.length;1764 diagnostics.makerEmailsFound += makerEnrichment.emails.length;1765 diagnostics.emailVerificationAttempts += makerVerification.attempts;1766 diagnostics.emailVerificationProviderErrors += makerVerification.verifications1767 .filter(({ status }) => status === 'provider_error' || status === 'error_credit').length;1768 makerVerification.warnings.forEach(addWarning);1769 log.info('Maker homepage email fallback complete', {1770 product: record.product_name,1771 emailsFound: makerEnrichment.emails.length,1772 pagesVisited: makerEnrichment.pageUrls.length,1773 statuses: makerVerification.verifications.map(({ status }) => status),1774 });1775 }1776 }1777
1778 let discoveredContacts: ContactFinderResult[] = [];1779 const deliverablePublicCount = verifyEmails1780 ? publicVerifications.filter(({ status }) => status === 'ok').length1781 : publicVerifications.filter(({ status }) => isUsableEmailStatus(status)).length;1782 if (enrichEmails && findContacts && websiteUrl && deliverablePublicCount < maxEmailsPerProduct) {1783 record.contact_discovery_requested = true;1784 const discovery = await discoverContacts(1785 websiteUrl,1786 (node.makers ?? []).map((maker: Record<string, unknown>) => String(maker.name ?? '')).filter(Boolean),1787 Math.max(1, maxEmailsPerProduct - deliverablePublicCount),1788 emailListVerifyApiKey,1789 );1790 discoveredContacts = discovery.contacts;1791 record.contact_discovery_attempts = discovery.attempts;1792 record.contact_discovery_credits = discovery.credits;1793 diagnostics.contactDiscoveryAttempts += discovery.attempts;1794 diagnostics.contactDiscoveryCredits += discovery.credits;1795 diagnostics.contactDiscoveryEmailsFound += discovery.contacts.length;1796 discovery.warnings.forEach(addWarning);1797 }1798
1799 const finalVerifications = selectBestEmails(publicVerifications, discoveredContacts, maxEmailsPerProduct);1800 const rejectedVerifications = publicVerifications.filter(({ status }) => !isUsableEmailStatus(status));1801 const finalEmails = finalVerifications.map(({ email }) => email);1802 const publicSelected = finalVerifications.filter(({ source }) => source === 'public_website').length;1803 const makerSelected = finalVerifications.filter(({ source }) => source === 'maker_website').length;1804 const finderSelected = finalVerifications.filter(({ source }) => source === 'emaillistverify_contact_finder').length;1805 record.emails = finalEmails;1806 record.verified_emails = finalVerifications.filter(({ status }) => status === 'ok').map(({ email }) => email);1807 record.risky_emails = finalVerifications.filter(({ status }) => status !== 'ok').map(({ email }) => email);1808 record.rejected_emails = rejectedVerifications.map(({ email }) => email);1809 record.email_verifications = finalVerifications;1810 record.rejected_email_verifications = rejectedVerifications;1811 const selectedSourceCount = [publicSelected, makerSelected, finderSelected].filter((count) => count > 0).length;1812 record.email_source = selectedSourceCount > 11813 ? 'mixed'1814 : publicSelected > 01815 ? 'page_scrape'1816 : makerSelected > 01817 ? 'maker_website'1818 : finderSelected > 01819 ? 'emaillistverify_contact_finder'1820 : 'none';1821 record.email_result = finalEmails.length > 0 ? 'emails_found' : record.email_result;1822 if (finderSelected > 0) record.email_terminal_reason = 'contact_finder_enriched';1823 else if (makerSelected > 0) record.email_terminal_reason = 'maker_homepage_enriched';1824 diagnostics.emailsFound += finalEmails.length;1825 diagnostics.verifiedEmailsFound += (record.verified_emails as string[]).length;1826
1827 const lead = buildLeadProfile({1828 websiteResolved: Boolean(websiteUrl),1829 websiteConfidence: Number(record.website_resolution_confidence ?? 0),1830 emailStatuses: finalVerifications.map(({ status }) => status),1831 publicEmailCount,1832 upvotes: Number(record.upvote_count ?? 0),1833 pagesVisited: Number(record.email_pages_visited ?? 0),1834 });1835 record.lead_status = lead.leadStatus;1836 record.lead_score = lead.leadScore;1837 record.lead_tier = lead.leadTier;1838 record.lead_signals = lead.signals;1839 diagnostics.leadTiers[lead.leadTier] += 1;1840
1841 log.info('Lead enrichment complete', {1842 product: record.product_name,1843 websiteResolutionSource: record.website_resolution_source,1844 websiteResolutionConfidence: record.website_resolution_confidence,1845 publicEmailsFound: publicEmailCount,1846 contactFinderEmailsFound: discoveredContacts.length,1847 finalEmails: finalEmails.length,1848 verifiedEmails: (record.verified_emails as string[]).length,1849 leadTier: lead.leadTier,1850 });1851
1852 const outputRecord = applyOutputMode(record, outputMode);1853 if (!outputRecord) {1854 diagnostics.itemsFiltered += 1;1855 return;1856 }1857 const validationError = validateOutputRecord(outputRecord);1858 if (validationError) {1859 diagnostics.invalidRecords += 1;1860 diagnostics.itemsFailed += 1;1861 addWarning(`A Product Hunt record was not persisted: ${validationError}`);1862 return;1863 }1864
1865 try {1866 // release-validator: allow-direct-push complete record validated above; the SDK automatically1867 // couples default-dataset persistence to the synthetic primary event.1868 // release-validator: allow-direct-push validated full row; default dataset persistence and the primary event are SDK-coupled.1869 const primaryResult = await Actor.pushData(outputRecord) as unknown as { chargedCount?: number; eventChargeLimitReached?: boolean };1870 if (primaryResult.eventChargeLimitReached && (primaryResult.chargedCount ?? 0) === 0) {1871 diagnostics.chargeLimitReached = true;1872 addWarning('The event-charge limit was reached before this validated row could be persisted.');1873 return;1874 }1875 diagnostics.itemsPushed += 1;1876 diagnostics.resultEvents += primaryResult.chargedCount ?? 1;1877 diagnostics.chargeLimitReached ||= Boolean(primaryResult.eventChargeLimitReached);1878
1879 const emails = outputRecord.emails as string[];1880 if (emails.length > 0 && !diagnostics.chargeLimitReached) {1881 const emailResult = await Actor.charge({ eventName: EMAIL_EVENT, count: emails.length }) as unknown as { chargedCount?: number; eventChargeLimitReached?: boolean };1882 diagnostics.emailEvents += emailResult.chargedCount ?? emails.length;1883 diagnostics.chargeLimitReached ||= Boolean(emailResult.eventChargeLimitReached);1884 if (emailResult.eventChargeLimitReached) addWarning('The event-charge limit was reached after a persisted record; remaining work stopped at the configured cap.');1885 }1886 } catch (error) {1887 diagnostics.itemsFailed += 1;1888 const message = error instanceof Error ? error.message : 'Unknown dataset-write failure.';1889 addWarning(`Validated dataset write failed: ${message.slice(0, 120)}`);1890 log.warning(`Validated dataset write failed for ${String(record.product_name)}: ${message.slice(0, 120)}`);1891 }1892 };1893
1894 const processNodes = async (nodes: Array<Record<string, any>>): Promise<void> => {1895 const uniqueNodes: Array<Record<string, any>> = [];1896 for (const node of nodes) {1897 if (collectedCandidates >= effectiveMaxResults || diagnostics.chargeLimitReached) break;1898 const id = String(node.id ?? '');1899 if (!id || seenIds.has(id)) continue;1900 seenIds.add(id);1901 const launchKey = productHuntLaunchKey(String(node.url ?? ''));1902 if (launchKey && excludedProductHuntKeys.has(launchKey)) {1903 diagnostics.itemsExcluded += 1;1904 continue;1905 }1906 uniqueNodes.push(node);1907 collectedCandidates += 1;1908 }1909 for (let index = 0; index < uniqueNodes.length; index += maxConcurrency) {1910 if (diagnostics.chargeLimitReached) break;1911 const batch = uniqueNodes.slice(index, index + maxConcurrency);1912 await Promise.all(batch.map((node, batchIndex) => pushNode(node, diagnostics.itemsProcessed + batchIndex + 1)));1913 diagnostics.itemsProcessed += batch.length;1914 }1915 };1916
1917 const streamPosts = async (range: DateRange | null, label: string): Promise<void> => {1918 let cursor: string | null = productHuntResumeCursor || null;1919 let pageNumber = 1;1920 let useLightweightIndex = shouldUseLightweightProductHuntIndex(1921 mode,1922 excludedProductHuntKeys.size,1923 productHuntResumeCursor,1924 );1925 while (collectedCandidates < effectiveMaxResults && !diagnostics.chargeLimitReached) {1926 const pageStartCursor = cursor;1927 const failuresBeforePage = diagnostics.itemsFailed;1928 const remaining = effectiveMaxResults - collectedCandidates;1929 const variables: Record<string, unknown> = {1930 first: Math.min(20, remaining),1931 after: cursor,1932 order: productHuntStreamOrder(Boolean(range)),1933 };1934 if (!includeAllProducts) variables.featured = true;1935 if (range) {1936 variables.postedAfter = range.postedAfter;1937 variables.postedBefore = range.postedBefore;1938 }1939 if (mode === 'topic') variables.topic = topic;1940 try {1941 let data: any = await phQuery(useLightweightIndex ? POSTS_INDEX_QUERY : POSTS_QUERY, variables);1942 let edges = data?.posts?.edges ?? [];1943 let pageInfo: { hasNextPage?: boolean; endCursor?: string | null } = data?.posts?.pageInfo ?? {};1944 if (!edges.length) break;1945 log.info(`Product Hunt ${label} page ${pageNumber}`, { received: edges.length, processed: diagnostics.itemsProcessed });1946 let nodes = edges.map((edge: { node?: Record<string, any> }) => edge.node).filter(Boolean);1947 if (useLightweightIndex) {1948 diagnostics.productHuntIndexPages += 1;1949 const hasUnseenLaunch = hasUnexcludedProductHuntLaunch(1950 nodes.map((node: Record<string, any>) => String(node.url ?? '')),1951 excludedProductHuntKeys,1952 );1953 if (hasUnseenLaunch) {1954 data = await phQuery(POSTS_QUERY, variables);1955 diagnostics.productHuntFullPageLookups += 1;1956 edges = data?.posts?.edges ?? [];1957 pageInfo = data?.posts?.pageInfo ?? pageInfo;1958 nodes = edges.map((edge: { node?: Record<string, any> }) => edge.node).filter(Boolean);1959 useLightweightIndex = false;1960 } else {1961 for (const node of nodes) {1962 const id = String(node.id ?? '');1963 if (!id || seenIds.has(id)) continue;1964 const launchKey = productHuntLaunchKey(String(node.url ?? ''));1965 if (launchKey && excludedProductHuntKeys.has(launchKey)) {1966 seenIds.add(id);1967 diagnostics.itemsExcluded += 1;1968 }1969 }1970 nodes = [];1971 }1972 }1973 const selectedNodes = mode === 'search'1974 ? nodes.filter((node: Record<string, any>) => matchesProductSearch(searchQuery, {1975 name: node.name,1976 tagline: node.tagline,1977 description: node.description,1978 topics: (node.topics?.edges ?? []).map((edge: any) => edge?.node?.name).filter(Boolean),1979 }))1980 : nodes;1981 await processNodes(selectedNodes);1982 diagnostics.nextProductHuntCursor = pageInfo.hasNextPage && pageInfo.endCursor1983 ? pageInfo.endCursor1984 : null;1985 const checkpointCursor = resumableCheckpointCursor(1986 pageStartCursor,1987 Boolean(pageInfo.hasNextPage),1988 pageInfo.endCursor,1989 failuresBeforePage,1990 diagnostics.itemsFailed,1991 );1992 if (!useLightweightIndex || pageNumber === 1 || pageNumber % 10 === 0 || !pageInfo.hasNextPage) {1993 try {1994 await Actor.setValue('RUN_CHECKPOINT', {1995 status: 'RUNNING',1996 updatedAt: new Date().toISOString(),1997 runId: process.env.ACTOR_RUN_ID ?? null,1998 buildNumber: process.env.ACTOR_BUILD_NUMBER ?? null,1999 mode,2000 label,2001 range,2002 productHuntResumeCursor: checkpointCursor,2003 pageNumber,2004 itemsProcessed: diagnostics.itemsProcessed,2005 itemsPushed: diagnostics.itemsPushed,2006 itemsExcluded: diagnostics.itemsExcluded,2007 itemsFailed: diagnostics.itemsFailed,2008 });2009 } catch (error) {2010 const message = error instanceof Error ? error.message : 'unknown checkpoint failure';2011 addWarning(`Run checkpoint could not be persisted: ${message.slice(0, 120)}`);2012 }2013 }2014 if (collectedCandidates >= effectiveMaxResults && pageInfo.hasNextPage) {2015 diagnostics.resultLimitReached = true;2016 addWarning('The result limit was reached while Product Hunt still had another page. Continue with excludeProductHuntUrls from this dataset to avoid duplicate enrichment charges.');2017 }2018 if (!pageInfo.hasNextPage || !pageInfo.endCursor) break;2019 cursor = pageInfo.endCursor;2020 pageNumber += 1;2021 } catch (error) {2022 diagnostics.apiFailures += 1;2023 const message = error instanceof Error ? error.message : 'Unknown Product Hunt API error.';2024 addWarning(`Product Hunt ${label} query failed: ${message}`);2025 break;2026 }2027 }2028 };2029
2030 if (mode === 'urls') {2031 const rawSeeds = startUrls2032 .map((entry) => typeof entry === 'string' ? entry : typeof entry?.url === 'string' ? entry.url : '')2033 .filter(Boolean);2034 const seeds = rawSeeds.map(parseProductHuntSeedUrl);2035 const validSeeds = seeds.filter((seed) => seed.kind !== 'invalid');2036 seeds.filter((seed) => seed.kind === 'invalid').forEach((seed) => addWarning(`Ignored start URL: ${seed.reason}`));2037 if (validSeeds.length === 0) await finalizeRun('INVALID_INPUT', 'No valid Product Hunt post, product launch, or dated leaderboard URLs were supplied.');2038
2039 for (const seed of validSeeds) {2040 if (collectedCandidates >= effectiveMaxResults || diagnostics.chargeLimitReached) break;2041 if (seed.kind === 'post') {2042 try {2043 const data = await phQuery(POST_BY_SLUG_QUERY, { slug: seed.slug });2044 if (!data?.post) {2045 addWarning(`No Product Hunt post matched the supplied URL slug: ${seed.slug}.`);2046 continue;2047 }2048 await processNodes([data.post]);2049 } catch (error) {2050 diagnostics.apiFailures += 1;2051 const message = error instanceof Error ? error.message : 'Unknown Product Hunt API error.';2052 addWarning(`Product Hunt URL lookup failed for ${seed.slug}: ${message}`);2053 }2054 } else if (seed.kind === 'leaderboard') {2055 await streamPosts(dateRangeForPeriod(seed.period, seed.anchorDate), `${seed.period} leaderboard URL`);2056 }2057 }2058 } else {2059 let range: DateRange | null = null;2060 if (mode === 'leaderboard') {2061 if (startDate) {2062 range = dateRangeForPeriod(leaderboardPeriod, startDate, endDate);2063 } else if (lookbackDays > 1) {2064 const end = new Date();2065 const begin = new Date(end);2066 begin.setUTCDate(end.getUTCDate() - lookbackDays + 1);2067 range = { postedAfter: `${isoDate(begin)}T00:00:00+00:00`, postedBefore: `${isoDate(end)}T23:59:59+00:00` };2068 } else {2069 range = dateRangeForPeriod(leaderboardPeriod, isoDate(new Date()));2070 }2071 }2072 await streamPosts(range, mode);2073 }2074
2075 await closeBrowser();2076 const hasUsefulRows = diagnostics.itemsPushed > 0;2077 const outcome: RunOutcome = !hasUsefulRows2078 ? diagnostics.apiFailures > 0 || diagnostics.itemsFailed > 0 || diagnostics.invalidRecords > 02079 ? 'UPSTREAM_FAILED'2080 : 'VALID_EMPTY'2081 : diagnostics.apiFailures > 0 || diagnostics.itemsFailed > 0 || (resolveWebsites && diagnostics.websiteUnresolved > 0) || diagnostics.chargeLimitReached2082 || diagnostics.resultLimitReached2083 ? 'PARTIAL'2084 : 'COMPLETE';2085 const eventCost = diagnostics.resultEvents * RESULT_PRICE_USD + diagnostics.emailEvents * EMAIL_PRICE_USD;2086 await finalizeRun(outcome, `${outcome}: processed ${diagnostics.itemsProcessed} Product Hunt products, persisted ${diagnostics.itemsPushed} rows, and found ${diagnostics.emailsFound} public email(s). Event charges: $${eventCost.toFixed(3)} plus applicable platform usage.`);2087 } catch (error) {2088 await closeBrowser();2089 if (finalised) return;2090 const message = error instanceof Error ? error.message : 'Unknown actor failure.';2091 diagnostics.itemsFailed += 1;2092 addWarning(`Unhandled actor error: ${message.slice(0, 160)}`);2093 await finalizeRun('UPSTREAM_FAILED', `The run stopped before completion: ${message.slice(0, 160)}`);2094 }2095}2096
2097await run();