Product Hunt Scraper (with Email Enrichment)
Pricing
from $2.00 / 1,000 results
Extract product names, upvotes, makers, taglines, and website URLs from ProductHunt MCP-ready. $2.00/1K.
Pricing
from $2.00 / 1,000 results
Rating
0.0
(0)
Developer
Khadin Akbar
Maintained by CommunityActor stats
2
Bookmarked
63
Total users
11
Monthly active users
3 days ago
Last modified
Categories
Share
node_modules/dist/storage/.env.env.**.log.DS_Storenode_modules/dist/storage/.env.env.**.log.DS_StoreFROM apify/actor-node-playwright-chrome:20
# Detect and save the image's pre-installed playwright version BEFORE any COPY.# The image ships playwright + matching Chromium. We must install the exact same# playwright version so browser executable paths always match.RUN node -e "try{process.stdout.write(require('./node_modules/playwright/package.json').version)}catch(e){process.stdout.write('unknown')}" > /tmp/IMAGE_PW_VERSION && \ echo "Image playwright: $(cat /tmp/IMAGE_PW_VERSION)"
# Copy ALL source (including package.json, package-lock.json, src/, .actor/)COPY . .
# Patch package.json + regenerate lock file to use the image's playwright versionRUN IMAGE_PW=$(cat /tmp/IMAGE_PW_VERSION) && \ npm pkg set dependencies.playwright="$IMAGE_PW" && \ npm install --package-lock-only --ignore-scripts 2>/dev/null && \ echo "Playwright pinned to $IMAGE_PW (matches image browsers)"
# Install ALL deps (devDependencies included so tsc is available)RUN NODE_ENV=development npm install --audit=false
# Compile TypeScriptRUN npm run build
# Remove devDependencies from the final imageRUN npm prune --omit=dev
CMD ["node", "dist/main.js"]1# Product Hunt lead-engine reliability graph — v1.32
3This graph is both the implementation model and the release gate.4
5```text6Product Hunt date / period / URL / search / topic input7 -> validate and normalize8 -> official Product Hunt GraphQL API9 -> for each unique product10 -> official external ProductLink or approved override?11 -> yes: accept with provenance12 -> no: Product Hunt redirect metadata13 -> no verified match: maker website evidence (up to 2)14 -> no verified match: DataForSEO organic query (1)15 -> no verified match/failure: SerpApi fallback query (1)16 -> every candidate homepage must pass identity scoring17 -> verified external homepage?18 -> no: persist unresolved provenance19 -> yes: static homepage assessment20 -> homepage-linked same-origin page plan21 contact -> about -> team -> privacy -> terms22 -> cap at 5 pages total23 -> browser DOM fallback only below email target24 -> rank/dedupe/cap public emails at 325 -> verify public emails with EmailListVerify26 -> below deliverable target: maker-name finder (5 credits)27 -> still empty: domain finder (10 credits)28 -> keep only same-domain, deliverable, high/medium matches29 -> merge/rank/cap final contacts at 330 -> calculate status/score/tier/signals31 -> validate complete row32 -> persist row with primary event33 -> charge one email-found event per persisted selected contact34 -> OUTPUT + RUN_SUMMARY35```36
37## Local graph loop38
39`npm test` covers:40
41- post, product-launch, daily/monthly/weekly/yearly URL normalization;42- current ISO-week archive routing;43- invalid Product Hunt hosts, redirect seeds, malformed dates, and private targets;44- homepage plus four linked-page cap and contact/about/team/privacy/terms priority;45- public email dedupe, placeholder filtering, same-domain ranking, and three-email cap;46- documented EmailListVerify status normalization;47- DataForSEO and SerpApi result normalization;48- search-directory rejection, positive homepage identity confirmation, and false-positive rejection;49- maker-name splitting and same-domain Contact Finder filtering; and50- deterministic lead qualification without inflating unresolved products.51
52Local release command:53
54```bash55npm test && jq empty .actor/input_schema.json .actor/dataset_schema.json \56 .actor/output_schema.json .actor/key_value_store_schema.json \57 .actor/pay_per_event.json .actor/actor.json && git diff --check58```59
60## Final-build cloud matrix61
62The deployed build must pass at least 20 bounded cases with delayed readback of dataset rows, `OUTPUT`, `RUN_SUMMARY`, run status, build ID, and named charges.63
64| Group | Cases | Assertions |65| --- | ---: | --- |66| Input routing | 5 | Direct post, product launch, daily range, monthly `2026/7`, and weekly `2026/26` route to intended API records. |67| Website resolution | 5 | ProductLink, override, redirect, maker site, and organic search either pass homepage identity checks or remain unresolved without guessing. |68| Crawl graph | 4 | Homepage-only, linked contact, linked about/team/privacy/terms priority, off-origin rejection, and five-page cap report exact coverage. |69| Email graph | 3 | Zero/one/three public contacts, exact verification statuses, fallback-only/mixed contacts, and invalid/risky separation are correct. |70| Lifecycle/economics | 3 | Invalid input, valid empty, and useful partial/complete runs write terminal KV records; primary persistence precedes email charge; event/provider counts reconcile. |7172Release thresholds:73
74- 20 or more final-build cloud cases;75- observed pass rate at least 99%;76- zero unexpected failures for samples below 100 cases;77- at least one verified external website;78- at least one public-page email branch;79- at least one EmailListVerify verification result;80- at least one contact-finder branch when provider credits permit;81- no secret in source, logs, input, dataset, KV output, or evidence files;82- live pricing and public visibility unchanged unless separately authorized.83
84A full historical month is not acceptance evidence by itself. Run it only after bounded final-build canaries prove website resolution, email extraction, verification, persistence, and charge reconciliation.{ "name": "producthunt-scraper-pro", "version": "1.3.0", "type": "module", "description": "Product Hunt lead engine with verified websites, bounded five-page contact discovery, email verification, and lead scoring", "main": "dist/main.js", "scripts": { "start": "node dist/main.js", "build": "tsc", "dev": "tsc && node dist/main.js", "test": "npm run build && node --test dist/enrichment-utils.test.js dist/lead-engine.test.js" }, "dependencies": { "apify": "^3.7.2", "cheerio": "^1.0.0", "crawlee": "^3.18.1", "playwright": "^1.62.1", "tldts": "^7.4.10" }, "devDependencies": { "@types/node": "^20.19.43", "typescript": "^5.3.0" }}{ "compilerOptions": { "target": "ES2022", "module": "NodeNext", "moduleResolution": "NodeNext", "outDir": "dist", "rootDir": "src", "strict": true, "esModuleInterop": true, "skipLibCheck": true, "resolveJsonModule": true }, "include": ["src/**/*"]}{ "actorSpecification": 1, "name": "producthunt-scraper-pro", "title": "Product Hunt Lead Generator — Products, Websites & Verified Emails", "description": "Turn Product Hunt daily, weekly, monthly, yearly, search, topic, or direct-launch inputs into qualified company leads. The Actor resolves official websites with identity checks, assesses up to five linked public pages, returns up to three contacts, verifies deliverability, fills missing business contacts with a managed fallback, and reports website/email provenance plus lead score and tier.", "version": "1.3", "buildTag": "latest", "environmentVariables": { "PH_API_TOKEN": "@producthuntApiToken", "EMAILLISTVERIFY_API_KEY": "@emailListVerifyApiKey", "DATAFORSEO_LOGIN": "@dataforseo-login", "DATAFORSEO_PASSWORD": "@dataforseo-password", "SERPAPI_KEY": "@SERPAPI_KEY" }, "dockerfile": "./Dockerfile", "output": "./output_schema.json", "storages": { "dataset": "./dataset_schema.json", "keyValueStore": "./key_value_store_schema.json" }, "pricingInfo": { "pricingModel": "PAY_PER_EVENT", "pricingPerEvent": { "actorChargeEvents": [ { "eventName": "product-scraped", "eventDescription": "One ProductHunt product record scraped and pushed to the dataset.", "eventPriceUsd": 0.001 } ] } }, "input": "./input_schema.json", "changelog": "\n\nv1.3 (2026-08-24) — Rebuilt enrichment as a cost-bounded Product Hunt lead engine. Added direct post and dated leaderboard URL routing through the official API; verified website resolution through redirect metadata, maker evidence, DataForSEO, and SerpApi fallback; static-first five-page email discovery; managed EmailListVerify verification and contact discovery; up to three ranked contacts; lead scores/tiers; provenance; atomic persist-and-charge; and OUTPUT/RUN_SUMMARY terminal contracts. No Product Hunt challenge bypass is used.\n\nv1.2 (2026-08-15) — Historical release; superseded by the verified lead-enrichment contract in v1.3.\n\nv1.1 (2026-05-22) — Historical input-validation update.", "chargedEvents": "./pay_per_event.json"}{ "actorSpecification": 1, "fields": { "type": "object", "properties": { "product_name": { "type": "string", "title": "Product Name", "description": "The official product name as listed on ProductHunt.", "example": "Notion AI" }, "tagline": { "type": ["string", "null"], "title": "Tagline", "description": "Short one-line pitch shown beneath the product name on ProductHunt.", "example": "AI-powered writing for everyone" }, "description": { "type": ["string", "null"], "title": "Description", "description": "Full product description written by the makers on ProductHunt.", "example": "Notion AI helps you write faster, think bigger, and augment your knowledge base." }, "upvote_count": { "type": ["integer", "null"], "title": "Upvotes", "description": "Total upvotes received. Higher = more popular on launch day.", "example": 1284 }, "comment_count": { "type": ["integer", "null"], "title": "Comments", "description": "Total number of comments on the ProductHunt listing.", "example": 87 }, "reviews_count": { "type": ["integer", "null"], "title": "Review Count", "description": "Number of user reviews the product has received on ProductHunt.", "example": 42 }, "reviews_rating": { "type": ["number", "null"], "title": "Avg Rating", "description": "Average user review rating on ProductHunt (typically 1–5 scale).", "example": 4.7 }, "daily_rank": { "type": ["integer", "null"], "title": "Daily Rank", "description": "#1 means most upvoted product of the day. Populated for daily leaderboard scrapes.", "example": 1 }, "weekly_rank": { "type": ["integer", "null"], "title": "Weekly Rank", "description": "Product's position in the weekly leaderboard.", "example": 3 }, "monthly_rank": { "type": ["integer", "null"], "title": "Monthly Rank", "description": "Product's position in the monthly leaderboard.", "example": 12 }, "yearly_rank": { "type": ["integer", "null"], "title": "Yearly Rank", "description": "Product's position in the annual leaderboard.", "example": 47 }, "launch_date": { "type": ["string", "null"], "title": "Launch Date", "description": "Date the product was featured on ProductHunt (YYYY-MM-DD).", "example": "2026-03-28" }, "product_hunt_url": { "type": ["string", "null"], "title": "ProductHunt URL", "description": "Full URL to the product's page on ProductHunt.", "example": "https://www.producthunt.com/posts/notion-ai" }, "website_url": { "type": ["string", "null"], "title": "Website", "description": "Verified public external product homepage. Resolution may use an official ProductLink, redirect metadata, maker-site evidence, or a search result that passed homepage identity checks.", "example": "https://notion.so" }, "product_hunt_website_url": { "type": ["string", "null"], "title": "Product Hunt Website Link", "description": "Original Product Hunt website field. This can be a Product Hunt tracking redirect and is retained for provenance, not treated as a verified external website.", "example": "https://www.producthunt.com/r/ABC123" }, "website_resolution_source": { "type": "string", "title": "Website Resolution Source", "description": "How website_url was resolved: official ProductLink, input override, Product Hunt redirect metadata, maker website, DataForSEO organic result, SerpApi organic result, or none.", "example": "dataforseo_organic" }, "website_resolution_status": { "type": "string", "title": "Website Resolution Status", "description": "verified_external, verified_search_match, unresolved, or not_available.", "example": "verified_external" }, "website_resolution_confidence": { "type": "integer", "title": "Website Confidence", "description": "Identity confidence from 0 to 100. Search and maker-site candidates are accepted only after homepage evidence passes the resolver threshold.", "example": 91 }, "website_resolution_signals": { "type": "array", "title": "Website Resolution Signals", "description": "Public evidence used to accept the website, such as domain, page-title, metadata, or tagline matches.", "example": ["full_product_name_matches_domain", "product_name_matches_page_title"] }, "website_resolution_provider_cost_usd": { "type": "number", "title": "Website Resolver Provider Cost", "description": "Provider-reported DataForSEO cost attributed to this website resolution attempt. Zero when no DataForSEO call was made.", "example": 0.002 }, "company_domain": { "type": ["string", "null"], "title": "Company Domain", "description": "Normalized company domain used for same-domain email ranking and contact discovery.", "example": "notion.so" }, "emails": { "type": "array", "title": "Emails Found", "description": "Up to three ranked contact emails from the public website crawl and, when needed, the managed business-contact finder.", "example": ["hello@notion.so", "support@notion.so"] }, "verified_emails": { "type": "array", "title": "Verified Deliverable Emails", "description": "Subset of emails for which EmailListVerify returned the exact deliverable status ok.", "example": ["hello@notion.so"] }, "risky_emails": { "type": "array", "title": "Other or Risky Emails", "description": "Selected contacts that were not confirmed with status ok, retained with their exact verification status for transparent filtering.", "example": ["support@notion.so"] }, "rejected_emails": { "type": "array", "title": "Rejected Emails", "description": "Public emails rejected from the final lead list because verification returned dead server, invalid MX, disabled, invalid syntax, disposable, or spam-trap status.", "example": ["old@notion.so"] }, "rejected_email_verifications": { "type": "array", "title": "Rejected Email Verification Results", "description": "Exact EmailListVerify records for rejected public emails, retained for audit without counting them as contacts or email-found events.", "example": [{"email": "old@notion.so", "provider": "emaillistverify", "status": "invalid_mx", "source": "public_website", "confidence": null, "checkedAt": "2026-08-24T14:30:00.000Z"}] }, "email_verifications": { "type": "array", "title": "EmailListVerify Results", "description": "One verification record per selected email, including exact provider status, discovery source, optional contact-finder confidence, and check time.", "example": [{"email": "hello@notion.so", "provider": "emaillistverify", "status": "ok", "source": "public_website", "confidence": null, "checkedAt": "2026-08-24T14:30:00.000Z"}] }, "email_verification_requested": { "type": "boolean", "title": "Email Verification Requested", "description": "True when this run requested EmailListVerify verification. A configured owner or caller key is also required before any provider call is made.", "example": true }, "email_source": { "type": "string", "title": "Email Source", "description": "page_scrape, emaillistverify_contact_finder, mixed, or none.", "example": "mixed" }, "email_result": { "type": "string", "title": "Email Result", "description": "emails_found, no_public_email_found, or not_assessed.", "example": "emails_found" }, "email_coverage": { "type": "string", "title": "Email Crawl Coverage", "description": "complete when every selected page was assessed, partial when one or more eligible pages could not be assessed, or none when no verified external site was available.", "example": "complete" }, "email_terminal_reason": { "type": "string", "title": "Email Terminal Reason", "description": "Bounded reason for the enrichment result, such as max_public_emails_reached, eligible_pages_exhausted, homepage_timeout, or website_not_available.", "example": "eligible_pages_exhausted" }, "email_pages_visited": { "type": "integer", "title": "Website Pages Visited", "description": "Number of verified external website pages assessed for email extraction, capped at five.", "example": 3 }, "email_page_urls": { "type": "array", "title": "Email Page URLs", "description": "Final same-origin website pages actually assessed for email extraction.", "example": ["https://notion.so/", "https://notion.so/contact"] }, "contact_discovery_requested": { "type": "boolean", "title": "Contact Discovery Requested", "description": "True when the managed Contact Finder fallback was eligible after public-page verification.", "example": true }, "contact_discovery_attempts": { "type": "integer", "title": "Contact Discovery Attempts", "description": "Number of bounded EmailListVerify Contact Finder requests made for this product.", "example": 1 }, "contact_discovery_credits": { "type": "integer", "title": "Contact Discovery Credit Ceiling Used", "description": "Documented EmailListVerify credits associated with attempted named or domain contact-finder requests.", "example": 5 }, "lead_status": { "type": "string", "title": "Lead Status", "description": "verified_contact, catch_all_contact, unverified_contact, no_contact, or unresolved_website.", "example": "verified_contact" }, "lead_score": { "type": "integer", "title": "Lead Score", "description": "Transparent 0–100 qualification score based on website confidence, email deliverability, public provenance, and Product Hunt traction.", "example": 84 }, "lead_tier": { "type": "string", "title": "Lead Tier", "description": "A, B, C, or unqualified, derived deterministically from lead_score.", "example": "A" }, "lead_signals": { "type": "array", "title": "Lead Signals", "description": "Plain-language signals explaining the lead score and tier.", "example": ["website_resolved", "deliverable_email_verified", "email_published_on_website"] }, "enrichment_warnings": { "type": "array", "title": "Enrichment Warnings", "description": "Safe per-record warnings, for example an off-origin redirect or a page timeout. Never contains credentials.", "example": [] }, "topics": { "type": "array", "title": "Topics", "description": "ProductHunt category tags (e.g. Artificial Intelligence, Developer Tools, Productivity).", "example": ["Productivity", "Artificial Intelligence", "Writing"] }, "thumbnail_url": { "type": ["string", "null"], "title": "Thumbnail", "description": "URL of the product logo/thumbnail hosted on ProductHunt's CDN.", "example": "https://ph-files.imgix.net/abc123.png?auto=format&fit=crop&h=256&w=256" }, "makers": { "type": "array", "title": "Makers", "description": "People who built the product. Each entry includes maker identity, public Product Hunt/Twitter links, and the maker website when exposed by the official API.", "example": [{"maker_name": "Ivan Zhao", "maker_id": "ivanhzhao", "maker_ph_id": "1234567", "twitter_url": "https://twitter.com/ivanhzhao", "maker_url": "https://www.producthunt.com/@ivanhzhao", "website_url": "https://notion.so"}] }, "featured": { "type": "boolean", "title": "Featured", "description": "True if the product was editorially featured on ProductHunt's homepage.", "example": true }, "scraped_at": { "type": "string", "title": "Scraped At", "description": "ISO 8601 timestamp of when this record was extracted.", "example": "2026-03-28T14:30:00.000Z" } } }, "views": { "overview": { "title": "🚀 Products Overview", "transformation": { "fields": [ "daily_rank", "product_name", "tagline", "upvote_count", "comment_count", "reviews_rating", "launch_date", "topics", "website_url", "website_resolution_source", "website_resolution_confidence", "product_hunt_url", "featured" ] }, "display": { "component": "table", "properties": { "daily_rank": { "label": "Rank" }, "product_name": { "label": "Product" }, "tagline": { "label": "Tagline" }, "upvote_count": { "label": "Upvotes" }, "comment_count": { "label": "Comments" }, "reviews_rating": { "label": "Rating" }, "launch_date": { "label": "Launch Date" }, "topics": { "label": "Topics" }, "website_url": { "label": "Website" }, "website_resolution_source": { "label": "Website Source" }, "website_resolution_confidence": { "label": "Website Confidence" }, "product_hunt_url": { "label": "PH Link" }, "featured": { "label": "Featured?" } } } }, "leads": { "title": "✉️ Email Leads", "transformation": { "fields": [ "product_name", "tagline", "emails", "verified_emails", "risky_emails", "rejected_emails", "email_verifications", "email_source", "lead_status", "lead_score", "lead_tier", "website_url", "company_domain", "topics", "upvote_count", "launch_date", "product_hunt_url" ] }, "display": { "component": "table", "properties": { "product_name": { "label": "Product" }, "tagline": { "label": "Tagline" }, "emails": { "label": "Emails Found" }, "verified_emails": { "label": "Verified Emails" }, "risky_emails": { "label": "Other / Risky Emails" }, "rejected_emails": { "label": "Rejected Emails" }, "email_verifications": { "label": "Verification" }, "email_source": { "label": "Source" }, "lead_status": { "label": "Lead Status" }, "lead_score": { "label": "Lead Score" }, "lead_tier": { "label": "Lead Tier" }, "website_url": { "label": "Website" }, "company_domain": { "label": "Company Domain" }, "topics": { "label": "Topics" }, "upvote_count": { "label": "Upvotes" }, "launch_date": { "label": "Launch Date" }, "product_hunt_url": { "label": "PH Link" } } } }, "rankings": { "title": "🏆 Rankings", "transformation": { "fields": [ "daily_rank", "weekly_rank", "monthly_rank", "yearly_rank", "product_name", "upvote_count", "comment_count", "reviews_count", "reviews_rating", "launch_date", "featured" ] }, "display": { "component": "table", "properties": { "daily_rank": { "label": "Daily #" }, "weekly_rank": { "label": "Weekly #" }, "monthly_rank": { "label": "Monthly #" }, "yearly_rank": { "label": "Yearly #" }, "product_name": { "label": "Product" }, "upvote_count": { "label": "Upvotes" }, "comment_count": { "label": "Comments" }, "reviews_count": { "label": "Reviews" }, "reviews_rating":{ "label": "Avg Rating" }, "launch_date": { "label": "Launch Date" }, "featured": { "label": "Featured?" } } } }, "makers": { "title": "👥 Makers", "transformation": { "fields": [ "product_name", "makers", "upvote_count", "launch_date", "website_url", "product_hunt_url" ] }, "display": { "component": "table", "properties": { "product_name": { "label": "Product" }, "makers": { "label": "Makers" }, "upvote_count": { "label": "Upvotes" }, "launch_date": { "label": "Launch Date" }, "website_url": { "label": "Website" }, "product_hunt_url": { "label": "PH Link" } } } } }}{ "title": "ProductHunt Scraper Pro Input", "type": "object", "schemaVersion": 1, "properties": { "mode": { "title": "Scraping Mode", "type": "string", "description": "How to find products. Use 'leaderboard' for top-ranked products by time period. Use 'search' when the user provides a keyword. Use 'topic' for a ProductHunt topic slug. Use 'urls' for specific ProductHunt URLs.", "enum": ["leaderboard", "search", "topic", "urls"], "default": "leaderboard", "editor": "select" }, "leaderboardPeriod": { "title": "Leaderboard Period", "type": "string", "description": "Only used when mode='leaderboard'. Choose 'daily', 'weekly', 'monthly', or 'yearly'.", "enum": ["daily", "weekly", "monthly", "yearly"], "default": "daily", "editor": "select" }, "searchQuery": { "title": "Search Query", "type": "string", "description": "Used when mode='search'. Filters products returned by Product Hunt's current official API window against name, tagline, description, and topics (for example 'AI writing tools').", "editor": "textfield" }, "topic": { "title": "Topic Slug", "type": "string", "description": "Used when mode='topic'. The ProductHunt topic URL slug (e.g. 'artificial-intelligence', 'developer-tools', 'productivity').", "editor": "textfield" }, "startUrls": { "title": "Start URLs", "type": "array", "description": "Used when mode='urls'. Accepts specific Product Hunt post/product-launch URLs and dated leaderboard URLs (for example /leaderboard/monthly/2026/7). Each URL is resolved through the official Product Hunt API, not by scraping Product Hunt pages.", "editor": "requestListSources", "prefill": [{"url": "https://www.producthunt.com/leaderboard/monthly/2026/7"}] }, "maxResults": { "title": "Max Results", "type": "integer", "description": "Maximum number of products to return. Default is 100.", "default": 100, "minimum": 1, "maximum": 20000, "editor": "number" }, "includeAllProducts": { "title": "Include All Published Products", "type": "boolean", "description": "Enabled by default so date ranges and leaderboard URLs include featured and non-featured Product Hunt posts up to maxResults. Disable to request featured posts only.", "default": true, "editor": "checkbox" }, "maxConcurrency": { "title": "Max Concurrency", "type": "integer", "description": "Maximum concurrent product-enrichment jobs. Default is 3 to keep browser and website traffic bounded.", "default": 3, "minimum": 1, "maximum": 5, "editor": "number" }, "enrichEmails": { "title": "Find Email Addresses", "type": "boolean", "description": "When enabled (the default), enriches each verified product website. It visits the homepage and up to four same-site contact, about, team, privacy, or terms pages linked from that homepage, then can use the managed contact finder if the public crawl has fewer than three deliverable emails.", "default": true, "editor": "checkbox" }, "resolveWebsites": { "title": "Resolve Product Websites", "type": "boolean", "description": "Recommended and enabled by default. Resolves Product Hunt redirect-only links through redirect metadata, maker-site evidence, and owner-managed search providers. Every search candidate must pass an identity check against its homepage before it is accepted.", "default": true, "editor": "checkbox" }, "maxWebsitePages": { "title": "Maximum Website Pages per Product", "type": "integer", "description": "Bounded public-site crawl limit. Default and maximum are 5 total pages: homepage plus up to four directly linked allowed pages.", "default": 5, "minimum": 1, "maximum": 5, "editor": "number" }, "maxEmailsPerProduct": { "title": "Maximum Public Emails per Product", "type": "integer", "description": "Returns and bills for up to this many ranked, public email addresses from a verified external website. Default and maximum are 3.", "default": 3, "minimum": 1, "maximum": 3, "editor": "number" }, "verifyEmails": { "title": "Verify Emails with EmailListVerify", "type": "boolean", "description": "Enabled by default. Verifies every selected public email with the Actor's managed EmailListVerify integration. Each result includes the exact provider status; invalid, disposable, and spam-trap results are never presented as verified emails.", "default": true, "editor": "checkbox" }, "findContacts": { "title": "Fill Missing Contacts", "type": "boolean", "description": "Enabled by default. When the five-page public crawl yields fewer than the requested number of deliverable emails, use the managed EmailListVerify Contact Finder with a Product Hunt maker name first, then a domain-only fallback only if needed. Provider calls and credit ceilings remain bounded per product.", "default": true, "editor": "checkbox" }, "websiteUrlOverrides": { "title": "Approved External Website Overrides", "type": "array", "description": "Optional approved mapping for a Product Hunt product. Each item accepts productHuntUrl, productSlug, productName, or productId plus websiteUrl. A valid override takes priority over managed automatic website resolution.", "editor": "json" }, "startDate": { "title": "Start Date (YYYY-MM-DD)", "type": "string", "description": "For leaderboard daily mode: query a specific date. Leave blank for today. Use with endDate to scrape a date range in one run (e.g. startDate=2026-03-01, endDate=2026-03-07 → full week).", "editor": "textfield" }, "endDate": { "title": "End Date (YYYY-MM-DD)", "type": "string", "description": "Optional end date for multi-day scraping in one run. Only used in daily leaderboard mode. Example: startDate=2026-03-01, endDate=2026-03-07 returns all products launched across that 7-day window.", "editor": "textfield" }, "lookbackDays": { "title": "Lookback Days", "type": "integer", "description": "Alternative to startDate: scrape the last N days from today. E.g. lookbackDays=7 pulls the last week. Ignored when startDate is provided.", "default": 1, "minimum": 1, "maximum": 365, "editor": "number" }, "outputMode": { "title": "Output Mode", "type": "string", "description": "Controls what gets returned. 'full' = all fields (default). 'lean' = minimal fields optimized for LLM/AI agent context windows (name, tagline, website, upvotes, rank, emails, topics). 'leads' = full fields but only products where at least 1 email was found — perfect for outreach lists.", "enum": ["full", "lean", "leads"], "default": "full", "editor": "select" } }}{ "actorKeyValueStoreSchemaVersion": 1, "title": "Product Hunt Scraper (with Email Enrichment) KV Store", "collections": { "OUTPUT": { "title": "Run summary", "description": "Machine-readable run summary and final output metadata.", "keyPrefix": "OUTPUT" }, "RUN_SUMMARY": { "title": "Run summary alias", "description": "Alias for tools that expect a RUN_SUMMARY record.", "keyPrefix": "RUN_SUMMARY" } }}{ "actorOutputSchemaVersion": 1, "title": "ProductHunt Scraper Pro Output", "description": "Product Hunt lead records with launch data, verified company websites, up to five assessed website pages, up to three ranked contacts, exact EmailListVerify statuses, contact-discovery provenance, and deterministic lead scores/tiers. Results are stored in the default dataset.", "properties": { "results": { "type": "string", "title": "Scraped Products", "description": "All Product Hunt lead records. Each item contains product metadata, website URL/source/confidence, company domain, emails, verified_emails, risky_emails, exact verification records, crawl coverage, contact-finder usage, lead status/score/tier, makers, topics, and rankings.", "template": "{{links.apiDefaultDatasetUrl}}/items" } }}{ "actorSpecification": 1, "events": { "apify-default-dataset-item": { "eventTitle": "result", "eventDescription": "Single result in the default dataset.", "eventPriceUsd": 0.002, "isOneTimeEvent": false, "isPrimaryEvent": true }, "email-found": { "eventTitle": "Email Found", "eventDescription": "One email address extracted from a product's website", "eventPriceUsd": 0.04, "isOneTimeEvent": false }, "apify-actor-start": { "eventTitle": "Actor Start", "eventDescription": "Charged when the Actor starts running. Number of events charged depends on Actor memory (one event per GB, minimum one event).", "eventPriceUsd": 0.00005, "isOneTimeEvent": true } }}1import assert from 'node:assert/strict';2import test from 'node:test';3import {4 buildWebsitePagePlan,5 canonicalWebsiteUrl,6 normalizeEmailListVerifyStatus,7 parseProductHuntSeedUrl,8 rankEmails,9} from './enrichment-utils.js';10
11test('parses a direct Product Hunt post URL', () => {12 assert.deepEqual(parseProductHuntSeedUrl('https://www.producthunt.com/posts/notion-ai?ref=feed'), {13 kind: 'post',14 sourceUrl: 'https://www.producthunt.com/posts/notion-ai?ref=feed',15 slug: 'notion-ai',16 });17});18
19test('parses a Product Hunt product-launch URL', () => {20 assert.deepEqual(parseProductHuntSeedUrl('https://www.producthunt.com/products/pazi/launches/pazi-2'), {21 kind: 'post',22 sourceUrl: 'https://www.producthunt.com/products/pazi/launches/pazi-2',23 slug: 'pazi-2',24 });25});26
27test('parses a monthly leaderboard URL as a stable month anchor', () => {28 assert.deepEqual(parseProductHuntSeedUrl('https://www.producthunt.com/leaderboard/monthly/2026/7'), {29 kind: 'leaderboard',30 sourceUrl: 'https://www.producthunt.com/leaderboard/monthly/2026/7',31 period: 'monthly',32 anchorDate: '2026-07-01',33 });34});35
36test('parses Product Hunt\'s current ISO-week leaderboard URL', () => {37 assert.deepEqual(parseProductHuntSeedUrl('https://www.producthunt.com/leaderboard/weekly/2026/26'), {38 kind: 'leaderboard',39 sourceUrl: 'https://www.producthunt.com/leaderboard/weekly/2026/26',40 period: 'weekly',41 anchorDate: '2026-06-22',42 });43});44
45test('supports the older explicit-date weekly leaderboard URL', () => {46 assert.deepEqual(parseProductHuntSeedUrl('https://www.producthunt.com/leaderboard/weekly/2026/7/14'), {47 kind: 'leaderboard',48 sourceUrl: 'https://www.producthunt.com/leaderboard/weekly/2026/7/14',49 period: 'weekly',50 anchorDate: '2026-07-14',51 });52});53
54test('parses a yearly leaderboard URL', () => {55 assert.deepEqual(parseProductHuntSeedUrl('https://www.producthunt.com/leaderboard/yearly/2026'), {56 kind: 'leaderboard',57 sourceUrl: 'https://www.producthunt.com/leaderboard/yearly/2026',58 period: 'yearly',59 anchorDate: '2026-01-01',60 });61});62
63test('rejects unrelated and malformed URLs', () => {64 assert.equal(parseProductHuntSeedUrl('https://example.com/posts/nope').kind, 'invalid');65 assert.equal(parseProductHuntSeedUrl('https://www.producthunt.com/r/ABC123').kind, 'invalid');66 assert.equal(parseProductHuntSeedUrl('https://www.producthunt.com/products/pazi').kind, 'invalid');67 assert.equal(parseProductHuntSeedUrl('not a url').kind, 'invalid');68});69
70test('caps the site plan at homepage plus four directly linked pages', () => {71 const plan = buildWebsitePagePlan('https://acme.com/', [72 { url: 'https://acme.com/contact', kind: 'contact', priority: 0 },73 { url: 'https://acme.com/about', kind: 'about', priority: 1 },74 { url: 'https://acme.com/team', kind: 'team', priority: 2 },75 { url: 'https://acme.com/privacy', kind: 'privacy', priority: 3 },76 { url: 'https://acme.com/terms', kind: 'terms', priority: 4 },77 ]);78 assert.equal(plan.length, 5);79 assert.deepEqual(plan.map((page) => page.kind), ['homepage', 'contact', 'about', 'team', 'privacy']);80});81
82test('allows lower-priority terms only when higher-priority categories are absent', () => {83 const plan = buildWebsitePagePlan('https://acme.com/', [84 { url: 'https://acme.com/contact', kind: 'contact', priority: 0 },85 { url: 'https://acme.com/terms', kind: 'terms', priority: 4 },86 ]);87 assert.deepEqual(plan.map((page) => page.kind), ['homepage', 'contact', 'terms']);88});89
90test('uses one linked page per category so duplicate contact links do not crowd out later fallbacks', () => {91 const plan = buildWebsitePagePlan('https://acme.com/', [92 { url: 'https://acme.com/contact', kind: 'contact', priority: 0 },93 { url: 'https://acme.com/contact-sales', kind: 'contact', priority: 0 },94 { url: 'https://acme.com/about', kind: 'about', priority: 1 },95 { url: 'https://acme.com/team', kind: 'team', priority: 2 },96 { url: 'https://acme.com/privacy', kind: 'privacy', priority: 3 },97 ]);98 assert.deepEqual(plan.map((page) => page.kind), ['homepage', 'contact', 'about', 'team', 'privacy']);99});100
101test('prefers the site domain and caps public email output at three', () => {102 assert.deepEqual(rankEmails([103 'random@vendor.com', 'support@acme.com', 'hello@acme.com', 'sales@acme.com', 'noreply@acme.com',104 ], 'acme.com'), ['hello@acme.com', 'sales@acme.com', 'support@acme.com']);105});106
107test('ranks outreach roles ahead of compliance-only inboxes', () => {108 assert.deepEqual(rankEmails([109 'privacy@acme.com', 'founder@acme.com', 'contact@acme.com', 'legal@acme.com',110 ], 'acme.com'), ['founder@acme.com', 'contact@acme.com', 'privacy@acme.com']);111});112
113test('filters placeholders and no-reply values before ranking', () => {114 assert.deepEqual(rankEmails([115 'example@example.com', 'noreply@acme.com', 'contact@acme.com', 'person@other.test',116 ], 'acme.com'), ['contact@acme.com', 'person@other.test']);117});118
119test('normalizes only documented EmailListVerify statuses', () => {120 assert.equal(normalizeEmailListVerifyStatus('ok'), 'ok');121 assert.equal(normalizeEmailListVerifyStatus('ok-for-all'), 'ok_for_all');122 assert.equal(normalizeEmailListVerifyStatus('error_credit'), 'error_credit');123 assert.equal(normalizeEmailListVerifyStatus('unexpected response'), 'provider_error');124});125
126test('rejects private and malformed web targets', () => {127 assert.equal(canonicalWebsiteUrl('http://127.0.0.1/internal'), null);128 assert.equal(canonicalWebsiteUrl('https://localhost:3000'), null);129 assert.equal(canonicalWebsiteUrl('https://user:pass@acme.com'), null);130 assert.equal(canonicalWebsiteUrl('https://acme.com:8443'), null);131 assert.equal(canonicalWebsiteUrl('http://100.64.0.1/internal'), null);132 assert.equal(canonicalWebsiteUrl('http://203.0.113.7/internal'), null);133 assert.equal(canonicalWebsiteUrl('http://[::1]/internal'), null);134 assert.equal(canonicalWebsiteUrl('https://acme.com/contact/?ref=ph#footer'), 'https://acme.com/contact?ref=ph');135});1export const DEFAULT_MAX_EMAILS_PER_PRODUCT = 3;2export const MAX_EMAILS_PER_PRODUCT = 3;3export const DEFAULT_MAX_WEBSITE_PAGES = 5;4export const MAX_WEBSITE_PAGES = 5;5
6export type FallbackPageKind = 'homepage' | 'contact' | 'about' | 'team' | 'privacy' | 'terms';7
8export type WebsitePage = {9 url: string;10 kind: FallbackPageKind;11 priority: number;12};13
14export type ProductHuntSeed =15 | { kind: 'post'; sourceUrl: string; slug: string }16 | { kind: 'leaderboard'; sourceUrl: string; period: 'daily' | 'weekly' | 'monthly' | 'yearly'; anchorDate: string }17 | { kind: 'invalid'; sourceUrl: string; reason: string };18
19export const EMAIL_LIST_VERIFY_STATUSES = [20 'ok',21 'unknown',22 'dead_server',23 'invalid_mx',24 'email_disabled',25 'antispam_system',26 'ok_for_all',27 'smtp_protocol',28 'invalid_syntax',29 'disposable',30 'spamtrap',31 'error_credit',32] as const;33
34export type EmailListVerifyStatus = typeof EMAIL_LIST_VERIFY_STATUSES[number] | 'provider_error' | 'not_configured' | 'not_requested';35
36export function normalizeEmailListVerifyStatus(value: string): EmailListVerifyStatus {37 const normalized = value.trim().toLowerCase().replace(/[\s-]+/g, '_');38 return (EMAIL_LIST_VERIFY_STATUSES as readonly string[]).includes(normalized)39 ? normalized as EmailListVerifyStatus40 : 'provider_error';41}42
43export function clampInteger(value: unknown, fallback: number, minimum: number, maximum: number): number {44 const numeric = Number(value);45 if (!Number.isFinite(numeric)) return fallback;46 return Math.min(maximum, Math.max(minimum, Math.floor(numeric)));47}48
49export function isProductHuntHost(hostname: string): boolean {50 const host = hostname.toLowerCase();51 return host === 'producthunt.com' || host.endsWith('.producthunt.com');52}53
54export function isPublicHttpUrl(rawUrl: string): boolean {55 try {56 const url = new URL(rawUrl);57 if (url.protocol !== 'http:' && url.protocol !== 'https:') return false;58 if (url.username || url.password) return false;59 if (url.port && url.port !== '80' && url.port !== '443') return false;60 const host = url.hostname.toLowerCase().replace(/^\[|\]$/g, '');61 if (!host || host === 'localhost' || host.endsWith('.localhost') || host.endsWith('.local')) return false;62 // Literal IPv6 targets are not needed for product websites and are63 // rejected outright, including loopback and IPv4-mapped addresses.64 if (host.includes(':')) return false;65
66 const ipv4 = host.match(/^(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})$/);67 if (!ipv4) return true;68 const octets = ipv4.slice(1).map(Number);69 if (octets.some((part) => part > 255)) return false;70 const [a, b] = octets;71 return !(72 a === 0 || a === 10 || a === 127 || a >= 224 ||73 (a === 100 && b >= 64 && b <= 127) ||74 (a === 169 && b === 254) ||75 (a === 172 && b >= 16 && b <= 31) ||76 (a === 192 && (b === 0 || b === 168)) ||77 (a === 198 && (b === 18 || b === 19 || b === 51)) ||78 (a === 203 && b === 0)79 );80 } catch {81 return false;82 }83}84
85export function canonicalWebsiteUrl(rawUrl: string): string | null {86 if (!isPublicHttpUrl(rawUrl)) return null;87 try {88 const url = new URL(rawUrl);89 url.hash = '';90 url.pathname = url.pathname.replace(/\/+$/, '') || '/';91 return url.toString();92 } catch {93 return null;94 }95}96
97function validDateParts(year: string | undefined, month: string | undefined, day: string | undefined): string | null {98 if (!year || !month) return null;99 const y = Number(year);100 const m = Number(month);101 const d = day ? Number(day) : 1;102 if (!Number.isInteger(y) || !Number.isInteger(m) || !Number.isInteger(d) || m < 1 || m > 12 || d < 1 || d > 31) return null;103 const date = new Date(Date.UTC(y, m - 1, d));104 if (date.getUTCFullYear() !== y || date.getUTCMonth() !== m - 1 || date.getUTCDate() !== d) return null;105 return date.toISOString().slice(0, 10);106}107
108function isoWeekStart(year: string | undefined, week: string | undefined): string | null {109 if (!year || !week) return null;110 const y = Number(year);111 const w = Number(week);112 if (!Number.isInteger(y) || !Number.isInteger(w) || w < 1 || w > 53) return null;113
114 // ISO week 1 is the week that contains January 4 and begins on Monday.115 const januaryFourth = new Date(Date.UTC(y, 0, 4));116 const januaryFourthDay = januaryFourth.getUTCDay() || 7;117 const firstMonday = new Date(Date.UTC(y, 0, 4 - januaryFourthDay + 1));118 const start = new Date(firstMonday);119 start.setUTCDate(firstMonday.getUTCDate() + (w - 1) * 7);120
121 const nextJanuaryFourth = new Date(Date.UTC(y + 1, 0, 4));122 const nextJanuaryFourthDay = nextJanuaryFourth.getUTCDay() || 7;123 const nextFirstMonday = new Date(Date.UTC(y + 1, 0, 4 - nextJanuaryFourthDay + 1));124 if (start >= nextFirstMonday) return null;125 return start.toISOString().slice(0, 10);126}127
128export function parseProductHuntSeedUrl(rawUrl: string): ProductHuntSeed {129 let url: URL;130 try {131 url = new URL(rawUrl);132 } catch {133 return { kind: 'invalid', sourceUrl: rawUrl, reason: 'URL is malformed.' };134 }135 if (!isProductHuntHost(url.hostname)) {136 return { kind: 'invalid', sourceUrl: rawUrl, reason: 'URL must use producthunt.com.' };137 }138
139 const parts = url.pathname.split('/').filter(Boolean);140 if (parts[0] === 'leaderboard') {141 const period = parts[1];142 if (!['daily', 'weekly', 'monthly', 'yearly'].includes(period)) {143 return { kind: 'invalid', sourceUrl: rawUrl, reason: 'Leaderboard URL must include daily, weekly, monthly, or yearly.' };144 }145 const anchorDate = period === 'weekly'146 ? parts[4]147 // Support the older explicit date form as well as Product Hunt's148 // current ISO-week form: /leaderboard/weekly/2026/26.149 ? validDateParts(parts[2], parts[3], parts[4])150 : isoWeekStart(parts[2], parts[3])151 : period === 'monthly'152 ? validDateParts(parts[2], parts[3], undefined)153 : period === 'yearly'154 ? validDateParts(parts[2], '1', undefined)155 : validDateParts(parts[2], parts[3], parts[4]);156 if (!anchorDate) {157 return { kind: 'invalid', sourceUrl: rawUrl, reason: 'Leaderboard URL must include a valid period-specific date (daily YYYY/M/D, weekly YYYY/week, monthly YYYY/M, yearly YYYY).' };158 }159 return { kind: 'leaderboard', sourceUrl: url.toString(), period: period as 'daily' | 'weekly' | 'monthly' | 'yearly', anchorDate };160 }161
162 if (parts[0] === 'r') {163 return { kind: 'invalid', sourceUrl: rawUrl, reason: 'Product Hunt redirect URLs are not valid inputs. Use the Product Hunt post/product/leaderboard URL or an approved external website override.' };164 }165
166 const launchesIndex = parts.indexOf('launches');167 if (parts[0] === 'products' && launchesIndex < 0) {168 return { kind: 'invalid', sourceUrl: rawUrl, reason: 'Product root URLs do not identify a specific launch in the official API. Use the Product Hunt post URL or a /products/{product}/launches/{launch} URL.' };169 }170 const slug = parts[0] === 'posts'171 ? parts[1]172 : launchesIndex >= 0173 ? parts[launchesIndex + 1]174 : undefined;175 if (!slug) {176 return { kind: 'invalid', sourceUrl: rawUrl, reason: 'URL must be a Product Hunt post, product launch, or dated leaderboard.' };177 }178 return { kind: 'post', sourceUrl: url.toString(), slug };179}180
181const PLACEHOLDER_EMAIL = /(?:example\.|test@|placeholder|noreply|no-reply|@sentry\.|@logrocket\.|@segment\.|your@|email@email|\d{10,}@)/i;182const EMAIL_PATTERN = /^[a-zA-Z0-9._%+\-]+@[a-zA-Z0-9.\-]+\.[a-zA-Z]{2,}$/;183
184export function rankEmails(candidates: string[], baseDomain: string, maxEmails = DEFAULT_MAX_EMAILS_PER_PRODUCT): string[] {185 const boundedMax = clampInteger(maxEmails, DEFAULT_MAX_EMAILS_PER_PRODUCT, 1, MAX_EMAILS_PER_PRODUCT);186 const normalizedDomain = baseDomain.replace(/^www\./i, '').toLowerCase();187 const unique: Array<{ email: string; score: number; index: number }> = [];188 const seen = new Set<string>();189 for (const [index, raw] of candidates.entries()) {190 const email = raw.trim().toLowerCase().replace(/^mailto:/, '').split('?')[0];191 if (!EMAIL_PATTERN.test(email) || PLACEHOLDER_EMAIL.test(email) || seen.has(email)) continue;192 seen.add(email);193 const domain = email.split('@')[1] ?? '';194 const sameDomain = domain === normalizedDomain || domain.endsWith(`.${normalizedDomain}`) || normalizedDomain.endsWith(`.${domain}`);195 const localPart = email.split('@')[0] ?? '';196 const highValueRole = /^(hello|contact|info|team|sales|partners|business|founders?)$/i.test(localPart);197 const serviceRole = /^(support|success|press|media)$/i.test(localPart);198 const complianceRole = /^(privacy|legal|abuse|security|dpo)$/i.test(localPart);199 unique.push({200 email,201 score: (sameDomain ? 10 : 0) + (highValueRole ? 4 : serviceRole ? 2 : complianceRole ? -2 : 0),202 index,203 });204 }205 return unique206 .sort((a, b) => b.score - a.score || a.index - b.index)207 .slice(0, boundedMax)208 .map(({ email }) => email);209}210
211export function buildWebsitePagePlan(homepageUrl: string, candidates: WebsitePage[], maxPages = DEFAULT_MAX_WEBSITE_PAGES): WebsitePage[] {212 const boundedPages = clampInteger(maxPages, DEFAULT_MAX_WEBSITE_PAGES, 1, MAX_WEBSITE_PAGES);213 const homepage = canonicalWebsiteUrl(homepageUrl);214 if (!homepage) return [];215 const seen = new Set([homepage]);216 const seenKinds = new Set<FallbackPageKind>();217 const fallback = candidates218 .filter((candidate) => candidate.kind !== 'homepage')219 .sort((a, b) => a.priority - b.priority)220 .map((candidate) => ({ ...candidate, url: canonicalWebsiteUrl(candidate.url) }))221 .filter((candidate): candidate is WebsitePage & { url: string } => Boolean(candidate.url))222 .filter((candidate) => {223 if (seen.has(candidate.url) || seenKinds.has(candidate.kind)) return false;224 seen.add(candidate.url);225 seenKinds.add(candidate.kind);226 return true;227 })228 .slice(0, Math.max(0, boundedPages - 1));229 return [{ url: homepage, kind: 'homepage', priority: -1 }, ...fallback];230}1import assert from 'node:assert/strict';2import test from 'node:test';3import {4 buildLeadProfile,5 buildWebsiteSearchQuery,6 evaluateWebsiteIdentity,7 isPrivateNetworkAddress,8 isUsableEmailStatus,9 matchesProductSearch,10 normalizeContactFinderResults,11 parseDataForSeoCandidates,12 parseSerpApiCandidates,13 scoreSearchCandidate,14 splitMakerName,15} from './lead-engine.js';16
17test('builds a bounded plain-language website query without paid search operators', () => {18 const query = buildWebsiteSearchQuery('Acme AI', 'Automate customer support');19 assert.equal(query, 'Acme AI Automate customer support official website');20 assert.doesNotMatch(query, /\b(?:site|intitle|inurl):/i);21});22
23test('parses DataForSEO organic candidates and provider cost', () => {24 const parsed = parseDataForSeoCandidates({25 tasks: [{ cost: 0.002, result: [{ items: [26 { type: 'organic', url: 'https://acme.ai/', title: 'Acme AI', description: 'Official site', rank_group: 1 },27 { type: 'people_also_ask', url: 'https://example.com/' },28 ] }] }],29 });30 assert.equal(parsed.costUsd, 0.002);31 assert.deepEqual(parsed.candidates, [{ url: 'https://acme.ai/', title: 'Acme AI', snippet: 'Official site', rank: 1 }]);32});33
34test('parses SerpApi organic candidates', () => {35 assert.deepEqual(parseSerpApiCandidates({ organic_results: [36 { link: 'https://acme.ai/', title: 'Acme AI', snippet: 'Official site', position: 2 },37 ] }), [{ url: 'https://acme.ai/', title: 'Acme AI', snippet: 'Official site', rank: 2 }]);38});39
40test('filters Product Hunt search locally using product text and topics', () => {41 const product = {42 name: 'WriterFlow',43 tagline: 'Draft marketing copy with AI',44 description: 'A writing workspace for growth teams',45 topics: ['Artificial Intelligence', 'Marketing'],46 };47 assert.equal(matchesProductSearch('AI writing tool', product), true);48 assert.equal(matchesProductSearch('database monitoring', product), false);49});50
51test('blocks directory results even when their title contains the product name', () => {52 const score = scoreSearchCandidate({53 url: 'https://www.g2.com/products/acme-ai', title: 'Acme AI reviews', snippet: 'Software listing', rank: 1,54 }, 'Acme AI', 'Automate support');55 assert.equal(score, -1);56});57
58test('blocks private, loopback, link-local, documentation, and mapped addresses', () => {59 for (const address of ['127.0.0.1', '10.0.0.5', '169.254.169.254', '192.168.1.1', '203.0.113.9', '::1', 'fd00::1', 'fe80::1', '::ffff:127.0.0.1']) {60 assert.equal(isPrivateNetworkAddress(address), true, address);61 }62 assert.equal(isPrivateNetworkAddress('8.8.8.8'), false);63 assert.equal(isPrivateNetworkAddress('2606:4700:4700::1111'), false);64});65
66test('accepts a search candidate only when its homepage confirms product identity', () => {67 const candidate = { url: 'https://acme.ai/', title: 'Acme AI — Official', snippet: 'Automate support', rank: 1 };68 const identity = evaluateWebsiteIdentity({69 candidate,70 productName: 'Acme AI',71 tagline: 'Automate customer support',72 pageTitle: 'Acme AI — Automate customer support',73 pageDescription: 'Acme AI helps support teams move faster.',74 pageText: 'Acme AI is the customer support automation platform.',75 });76 assert.equal(identity.accepted, true);77 assert.ok(identity.confidence >= 70);78});79
80test('rejects a plausible search result when the homepage is unrelated', () => {81 const candidate = { url: 'https://acmeventures.example/', title: 'Acme AI', snippet: 'Result', rank: 1 };82 const identity = evaluateWebsiteIdentity({83 candidate,84 productName: 'Acme AI',85 tagline: 'Automate customer support',86 pageTitle: 'Acme Ventures',87 pageDescription: 'Real estate investment company',88 pageText: 'We buy and operate commercial property.',89 });90 assert.equal(identity.accepted, false);91});92
93test('splits maker names for lower-cost named contact discovery', () => {94 assert.deepEqual(splitMakerName('Ada Lovelace'), { firstName: 'Ada', lastName: 'Lovelace' });95 assert.deepEqual(splitMakerName('Prince'), { firstName: 'Prince' });96 assert.deepEqual(splitMakerName('🤖'), {});97 assert.deepEqual(splitMakerName('[REDACTED]'), {});98});99
100test('keeps only same-domain deliverable high/medium contact-finder results', () => {101 const results = normalizeContactFinderResults([102 { email: 'ada@acme.ai', confidence: 'high', result: 'ok' },103 { email: 'team@acme.ai', confidence: 'medium', result: 'ok' },104 { email: 'bad@acme.ai', confidence: 'high', result: 'invalid_mx' },105 { email: 'ada@gmail.com', confidence: 'high', result: 'ok' },106 { email: 'ada@acme.ai', confidence: 'high', result: 'ok' },107 ], 'acme.ai', 3);108 assert.deepEqual(results.map(({ email }) => email), ['ada@acme.ai', 'team@acme.ai']);109});110
111test('excludes hard-invalid verification outcomes from usable lead contacts', () => {112 for (const status of ['dead_server', 'invalid_mx', 'email_disabled', 'invalid_syntax', 'disposable', 'spamtrap']) {113 assert.equal(isUsableEmailStatus(status), false, status);114 }115 for (const status of ['ok', 'ok_for_all', 'unknown', 'provider_error', 'error_credit']) {116 assert.equal(isUsableEmailStatus(status), true, status);117 }118});119
120test('scores a deliverable published contact as an A-tier lead', () => {121 const profile = buildLeadProfile({122 websiteResolved: true,123 websiteConfidence: 90,124 emailStatuses: ['ok', 'ok'],125 publicEmailCount: 1,126 upvotes: 500,127 pagesVisited: 5,128 });129 assert.equal(profile.leadStatus, 'verified_contact');130 assert.equal(profile.leadTier, 'A');131 assert.ok(profile.leadScore >= 75);132});133
134test('does not inflate an unresolved product into a lead', () => {135 assert.deepEqual(buildLeadProfile({136 websiteResolved: false,137 websiteConfidence: 0,138 emailStatuses: [],139 publicEmailCount: 0,140 upvotes: 0,141 pagesVisited: 0,142 }), {143 leadStatus: 'unresolved_website',144 leadScore: 0,145 leadTier: 'unqualified',146 signals: [],147 });148});1export type SearchCandidate = {2 url: string;3 title: string;4 snippet: string;5 rank: number;6};7
8export type WebsiteIdentity = {9 accepted: boolean;10 confidence: number;11 signals: string[];12};13
14export type ContactFinderResult = {15 email: string;16 confidence: 'high' | 'medium' | 'low' | 'unknown';17 result: string;18};19
20export type LeadProfile = {21 leadStatus: 'verified_contact' | 'catch_all_contact' | 'unverified_contact' | 'no_contact' | 'unresolved_website';22 leadScore: number;23 leadTier: 'A' | 'B' | 'C' | 'unqualified';24 signals: string[];25};26
27const DIRECTORY_HOSTS = new Set([28 'alternativeto.net', 'appsumo.com', 'capterra.com', 'crunchbase.com', 'g2.com',29 'github.com', 'indiehackers.com', 'producthunt.com', 'saasworthy.com',30 'sourceforge.net', 'stackshare.io', 'theresanaiforthat.com', 'toolify.ai',31 'trustpilot.com', 'wikipedia.org', 'youtube.com',32]);33
34const STOP_WORDS = new Set([35 'a', 'ai', 'an', 'and', 'app', 'by', 'for', 'from', 'in', 'io', 'of', 'on',36 'platform', 'software', 'the', 'to', 'tool', 'tools', 'with',37]);38
39const EMAIL_PATTERN = /^[a-zA-Z0-9._%+\-]+@[a-zA-Z0-9.\-]+\.[a-zA-Z]{2,}$/;40const HARD_REJECTED_EMAIL_STATUSES = new Set([41 'dead_server', 'invalid_mx', 'email_disabled', 'invalid_syntax', 'disposable', 'spamtrap',42]);43
44function normalizedText(value: unknown): string {45 return String(value ?? '')46 .normalize('NFKD')47 .replace(/[\u0300-\u036f]/g, '')48 .toLowerCase()49 .replace(/[^a-z0-9]+/g, ' ')50 .trim();51}52
53function identityTokens(value: unknown): string[] {54 const all = normalizedText(value).split(/\s+/).filter(Boolean);55 const meaningful = all.filter((token) => token.length >= 2 && !STOP_WORDS.has(token));56 return [...new Set(meaningful.length > 0 ? meaningful : all.filter((token) => token.length >= 2))];57}58
59function coverage(tokens: string[], haystack: string): number {60 if (tokens.length === 0) return 0;61 const matches = tokens.filter((token) => haystack.includes(token)).length;62 return matches / tokens.length;63}64
65function hostnameOf(rawUrl: string): string {66 try {67 return new URL(rawUrl).hostname.toLowerCase().replace(/^www\./, '');68 } catch {69 return '';70 }71}72
73function isDirectoryHost(hostname: string): boolean {74 return [...DIRECTORY_HOSTS].some((blocked) => hostname === blocked || hostname.endsWith(`.${blocked}`));75}76
77export function buildWebsiteSearchQuery(productName: string, tagline: string): string {78 const cleanName = String(productName ?? '').trim().replace(/\s+/g, ' ').slice(0, 100);79 const cleanTagline = String(tagline ?? '').trim().replace(/\s+/g, ' ').slice(0, 100);80 return [cleanName, cleanTagline, 'official website'].filter(Boolean).join(' ').slice(0, 240);81}82
83export function isUsableEmailStatus(status: string): boolean {84 return !HARD_REJECTED_EMAIL_STATUSES.has(status);85}86
87export function matchesProductSearch(88 query: string,89 product: { name?: unknown; tagline?: unknown; description?: unknown; topics?: unknown[] },90): boolean {91 const normalizedQuery = normalizedText(query);92 if (!normalizedQuery) return false;93 const topics = Array.isArray(product.topics) ? product.topics.map(normalizedText).join(' ') : '';94 const haystack = normalizedText(`${String(product.name ?? '')} ${String(product.tagline ?? '')} ${String(product.description ?? '')} ${topics}`);95 if (haystack.includes(normalizedQuery)) return true;96 const tokens = identityTokens(normalizedQuery);97 if (tokens.length === 0) return false;98 const matches = tokens.filter((token) => haystack.includes(token)).length;99 return matches >= Math.max(1, Math.ceil(tokens.length * 0.6));100}101
102export function isPrivateNetworkAddress(address: string): boolean {103 const normalized = address.toLowerCase().replace(/^\[|\]$/g, '');104 const mappedIpv4 = normalized.match(/^(?:::ffff:)?(\d{1,3}(?:\.\d{1,3}){3})$/)?.[1];105 if (mappedIpv4) {106 const octets = mappedIpv4.split('.').map(Number);107 if (octets.length !== 4 || octets.some((part) => !Number.isInteger(part) || part < 0 || part > 255)) return true;108 const [a, b, c] = octets;109 return a === 0110 || a === 10111 || a === 127112 || a >= 224113 || (a === 100 && b >= 64 && b <= 127)114 || (a === 169 && b === 254)115 || (a === 172 && b >= 16 && b <= 31)116 || (a === 192 && (b === 0 || b === 168))117 || (a === 198 && (b === 18 || b === 19 || (b === 51 && c === 100)))118 || (a === 203 && b === 0 && c === 113);119 }120 return normalized === '::'121 || normalized === '::1'122 || normalized.startsWith('fc')123 || normalized.startsWith('fd')124 || /^fe[89ab]/.test(normalized)125 || normalized.startsWith('ff')126 || normalized.startsWith('2001:db8:');127}128
129export function parseDataForSeoCandidates(payload: unknown): { candidates: SearchCandidate[]; costUsd: number } {130 const body = payload && typeof payload === 'object' ? payload as Record<string, any> : {};131 const task = Array.isArray(body.tasks) ? body.tasks[0] : null;132 const result = Array.isArray(task?.result) ? task.result[0] : null;133 const items = Array.isArray(result?.items) ? result.items : [];134 const candidates = items135 .filter((item: any) => item && item.type === 'organic' && typeof item.url === 'string')136 .map((item: any, index: number): SearchCandidate => ({137 url: item.url,138 title: String(item.title ?? '').slice(0, 500),139 snippet: String(item.description ?? item.snippet ?? '').slice(0, 800),140 rank: Number.isInteger(item.rank_group) ? item.rank_group : index + 1,141 }));142 return { candidates, costUsd: Number(task?.cost ?? 0) || 0 };143}144
145export function parseSerpApiCandidates(payload: unknown): SearchCandidate[] {146 const body = payload && typeof payload === 'object' ? payload as Record<string, any> : {};147 const items = Array.isArray(body.organic_results) ? body.organic_results : [];148 return items149 .filter((item: any) => item && typeof item.link === 'string')150 .map((item: any, index: number): SearchCandidate => ({151 url: item.link,152 title: String(item.title ?? '').slice(0, 500),153 snippet: String(item.snippet ?? '').slice(0, 800),154 rank: Number.isInteger(item.position) ? item.position : index + 1,155 }));156}157
158export function scoreSearchCandidate(candidate: SearchCandidate, productName: string, tagline: string): number {159 const hostname = hostnameOf(candidate.url);160 if (!hostname || isDirectoryHost(hostname)) return -1;161
162 const nameTokens = identityTokens(productName);163 if (nameTokens.length === 0) return -1;164 const taglineTokens = identityTokens(tagline).slice(0, 8);165 const hostText = normalizedText(hostname);166 const titleText = normalizedText(candidate.title);167 const snippetText = normalizedText(candidate.snippet);168 const namePhrase = normalizedText(productName);169 const joinedName = nameTokens.join('');170
171 let score = candidate.rank <= 1 ? 18 : candidate.rank <= 3 ? 13 : candidate.rank <= 5 ? 8 : 3;172 const hostCoverage = coverage(nameTokens, hostText);173 const titleCoverage = coverage(nameTokens, titleText);174 score += Math.round(hostCoverage * 32);175 score += Math.round(titleCoverage * 30);176 score += Math.round(coverage(taglineTokens, `${titleText} ${snippetText}`) * 10);177 if (namePhrase.length >= 3 && titleText.includes(namePhrase)) score += 18;178 if (joinedName.length >= 3 && hostText.replace(/\s+/g, '').includes(joinedName)) score += 22;179 if (/\b(official|homepage|home)\b/.test(`${titleText} ${snippetText}`)) score += 4;180 return Math.min(100, score);181}182
183export function evaluateWebsiteIdentity(input: {184 candidate: SearchCandidate;185 productName: string;186 tagline: string;187 pageTitle: string;188 pageDescription: string;189 pageText: string;190}): WebsiteIdentity {191 const { candidate, productName, tagline, pageTitle, pageDescription, pageText } = input;192 const searchScore = scoreSearchCandidate(candidate, productName, tagline);193 if (searchScore < 0) return { accepted: false, confidence: 0, signals: ['blocked_or_invalid_search_host'] };194
195 const hostname = hostnameOf(candidate.url);196 const hostText = normalizedText(hostname);197 const titleText = normalizedText(pageTitle);198 const descriptionText = normalizedText(pageDescription);199 const bodyText = normalizedText(pageText).slice(0, 80_000);200 const nameTokens = identityTokens(productName);201 const taglineTokens = identityTokens(tagline).slice(0, 8);202 const namePhrase = normalizedText(productName);203 const hostCoverage = coverage(nameTokens, hostText);204 const titleCoverage = coverage(nameTokens, `${titleText} ${descriptionText}`);205 const bodyCoverage = coverage(nameTokens, bodyText);206 const taglineCoverage = coverage(taglineTokens, `${descriptionText} ${bodyText}`);207 const exactTitle = namePhrase.length >= 3 && `${titleText} ${descriptionText}`.includes(namePhrase);208 const compactName = namePhrase.replace(/\s+/g, '');209 const compactDomainMatch = compactName.length >= 3 && hostText.replace(/\s+/g, '').includes(compactName);210
211 let pageScore = Math.round(hostCoverage * 35 + titleCoverage * 35 + bodyCoverage * 15 + taglineCoverage * 10);212 if (exactTitle) pageScore += 15;213 pageScore = Math.min(100, pageScore);214 const confidence = Math.min(100, Math.round(searchScore * 0.52 + pageScore * 0.48));215 const strongIdentitySignal = exactTitle216 || compactDomainMatch217 || (nameTokens.length >= 2 && hostCoverage >= 0.5)218 || (titleCoverage >= 0.75 && bodyCoverage >= 0.75 && taglineCoverage >= 0.5);219 const accepted = searchScore >= 42 && pageScore >= 38 && confidence >= 48 && strongIdentitySignal;220 const signals: string[] = [];221 if (hostCoverage >= 0.5) signals.push('product_name_matches_domain');222 if (compactDomainMatch) signals.push('full_product_name_matches_domain');223 if (exactTitle) signals.push('product_name_matches_page_title');224 if (titleCoverage >= 0.75) signals.push('name_tokens_match_metadata');225 if (taglineCoverage >= 0.5) signals.push('tagline_matches_page');226 if (!accepted) signals.push('identity_threshold_not_met');227 return { accepted, confidence, signals };228}229
230export function splitMakerName(fullName: unknown): { firstName?: string; lastName?: string } {231 const clean = String(fullName ?? '').replace(/[^\p{L}\p{M}'’\-\s]/gu, ' ').replace(/\s+/g, ' ').trim();232 if (!clean || /^(?:redacted|anonymous|deleted|unknown|private|n\/a)$/i.test(clean)) return {};233 const parts = clean.split(' ');234 if (parts.length === 1) return { firstName: parts[0] };235 return { firstName: parts[0], lastName: parts.slice(1).join(' ') };236}237
238export function normalizeContactFinderResults(payload: unknown, domain: string, maxEmails: number): ContactFinderResult[] {239 const normalizedDomain = domain.toLowerCase().replace(/^www\./, '');240 const rows = Array.isArray(payload) ? payload : [];241 const seen = new Set<string>();242 const confidenceRank: Record<ContactFinderResult['confidence'], number> = { high: 4, medium: 3, low: 2, unknown: 1 };243 return rows244 .map((row: any): ContactFinderResult => ({245 email: String(row?.email ?? '').trim().toLowerCase(),246 confidence: ['high', 'medium', 'low', 'unknown'].includes(row?.confidence) ? row.confidence : 'unknown',247 result: String(row?.result ?? 'unknown').trim().toLowerCase(),248 }))249 .filter((row) => {250 if (!EMAIL_PATTERN.test(row.email) || row.result !== 'ok' || !['high', 'medium'].includes(row.confidence)) return false;251 const emailDomain = row.email.split('@')[1] ?? '';252 if (!(emailDomain === normalizedDomain || emailDomain.endsWith(`.${normalizedDomain}`))) return false;253 if (seen.has(row.email)) return false;254 seen.add(row.email);255 return true;256 })257 .sort((left, right) => confidenceRank[right.confidence] - confidenceRank[left.confidence])258 .slice(0, Math.max(0, Math.min(3, Math.floor(maxEmails))));259}260
261export function buildLeadProfile(input: {262 websiteResolved: boolean;263 websiteConfidence: number;264 emailStatuses: string[];265 publicEmailCount: number;266 upvotes: number;267 pagesVisited: number;268}): LeadProfile {269 const { websiteResolved, websiteConfidence, emailStatuses, publicEmailCount, upvotes, pagesVisited } = input;270 const verifiedCount = emailStatuses.filter((status) => status === 'ok').length;271 const catchAllCount = emailStatuses.filter((status) => status === 'ok_for_all').length;272 const hasContact = emailStatuses.length > 0;273 const signals: string[] = [];274 let score = 0;275
276 if (websiteResolved) {277 score += 15;278 signals.push('website_resolved');279 }280 if (websiteConfidence >= 75) {281 score += 10;282 signals.push('high_confidence_website');283 } else if (websiteConfidence >= 50) {284 score += 5;285 }286 if (verifiedCount > 0) {287 score += 45 + Math.min(15, (verifiedCount - 1) * 8);288 signals.push('deliverable_email_verified');289 } else if (catchAllCount > 0) {290 score += 28;291 signals.push('catch_all_email');292 } else if (hasContact) {293 score += 15;294 signals.push('unverified_contact');295 }296 if (publicEmailCount > 0) {297 score += 8;298 signals.push('email_published_on_website');299 }300 if (pagesVisited >= 3) signals.push('multi_page_website_assessed');301 score += Math.min(7, Math.floor(Math.log10(Math.max(1, upvotes) + 1) * 3));302 score = Math.min(100, score);303
304 const leadStatus: LeadProfile['leadStatus'] = verifiedCount > 0305 ? 'verified_contact'306 : catchAllCount > 0307 ? 'catch_all_contact'308 : hasContact309 ? 'unverified_contact'310 : websiteResolved311 ? 'no_contact'312 : 'unresolved_website';313 const leadTier: LeadProfile['leadTier'] = score >= 75 ? 'A' : score >= 55 ? 'B' : score >= 35 ? 'C' : 'unqualified';314 return { leadStatus, leadScore: score, leadTier, signals };315}1import { Actor } from 'apify';2import { log } from 'crawlee';3import { load } from 'cheerio';4import { lookup } from 'node:dns/promises';5import { isIP } from 'node:net';6import { chromium, type Browser } from 'playwright';7import { getDomain } from 'tldts';8import {9 buildWebsitePagePlan,10 canonicalWebsiteUrl,11 clampInteger,12 DEFAULT_MAX_EMAILS_PER_PRODUCT,13 DEFAULT_MAX_WEBSITE_PAGES,14 isProductHuntHost,15 isPublicHttpUrl,16 MAX_EMAILS_PER_PRODUCT,17 MAX_WEBSITE_PAGES,18 normalizeEmailListVerifyStatus,19 parseProductHuntSeedUrl,20 rankEmails,21 type EmailListVerifyStatus,22 type WebsitePage,23} from './enrichment-utils.js';24import {25 buildLeadProfile,26 buildWebsiteSearchQuery,27 evaluateWebsiteIdentity,28 isPrivateNetworkAddress,29 isUsableEmailStatus,30 matchesProductSearch,31 normalizeContactFinderResults,32 parseDataForSeoCandidates,33 parseSerpApiCandidates,34 scoreSearchCandidate,35 splitMakerName,36 type ContactFinderResult,37 type SearchCandidate,38} from './lead-engine.js';39
40await Actor.init();41
42type Mode = 'leaderboard' | 'search' | 'topic' | 'urls';43type LeaderboardPeriod = 'daily' | 'weekly' | 'monthly' | 'yearly';44type RunOutcome = 'COMPLETE' | 'PARTIAL' | 'VALID_EMPTY' | 'INVALID_INPUT' | 'UPSTREAM_FAILED' | 'CONFIG_ERROR';45type EmailSource = 'page_scrape' | 'emaillistverify_contact_finder' | 'mixed' | 'none';46type EmailResult = 'emails_found' | 'no_public_email_found' | 'not_assessed';47type Coverage = 'complete' | 'partial' | 'none';48type WebsiteResolutionStatus = 'verified_external' | 'verified_search_match' | 'unresolved' | 'not_available';49type WebsiteResolutionSource = 'api_website_product_link' | 'input_override' | 'product_hunt_redirect' | 'maker_website' | 'dataforseo_organic' | 'serpapi_organic' | 'none';50type EmailVerification = {51 email: string;52 provider: 'emaillistverify';53 status: EmailListVerifyStatus;54 checkedAt: string | null;55 source: 'public_website' | 'emaillistverify_contact_finder';56 confidence: 'high' | 'medium' | 'low' | 'unknown' | null;57};58
59type DateRange = { postedAfter: string; postedBefore: string };60type WebsiteOverride = { keys: string[]; websiteUrl: string };61type EnrichmentResult = {62 emails: string[];63 emailSource: EmailSource;64 result: EmailResult;65 coverage: Coverage;66 terminalReason: string;67 pagesVisited: number;68 pageUrls: string[];69 warnings: string[];70};71
72type WebsiteResolution = {73 websiteUrl: string | null;74 source: WebsiteResolutionSource;75 status: WebsiteResolutionStatus;76 confidence: number;77 signals: string[];78 providerCostUsd: number;79 providerAttempts: number;80 warnings: string[];81};82
83type ContactDiscovery = {84 contacts: ContactFinderResult[];85 attempts: number;86 credits: number;87 warnings: string[];88};89
90const PH_API = 'https://api.producthunt.com/v2/api/graphql';91const RESULT_EVENT = 'apify-default-dataset-item';92const EMAIL_EVENT = 'email-found';93const RESULT_PRICE_USD = 0.002;94const EMAIL_PRICE_USD = 0.04;95const EMAIL_LIST_VERIFY_API = 'https://api.emaillistverify.com/api/verifyEmail';96const EMAIL_LIST_VERIFY_CONTACT_API = 'https://api.emaillistverify.com/api/findContact';97const DATAFORSEO_SEARCH_API = 'https://api.dataforseo.com/v3/serp/google/organic/live/advanced';98const SERPAPI_SEARCH_API = 'https://serpapi.com/search.json';99const EMAIL_LIST_VERIFY_MIN_INTERVAL_MS = 225; // Contact Finder is limited to 5 requests/second; keep headroom.100const FAILED_OUTCOMES = new Set<RunOutcome>(['UPSTREAM_FAILED', 'CONFIG_ERROR']);101
102const SKIP_DOMAINS = new Set([103 'apps.apple.com', 'play.google.com', 'github.com', 'youtube.com',104 'twitter.com', 'x.com', 'linkedin.com', 'facebook.com', 'instagram.com',105 'reddit.com', 'discord.com', 'discord.gg', 'slack.com', 'medium.com',106 'substack.com', 'producthunt.com', 'amazon.com', 'chrome.google.com',107 'marketplace.visualstudio.com', 'notion.so', 'figma.com', 'trello.com',108 'airtable.com',109]);110
111const PRODUCT_HUNT_USER_AGENT = 'ProductHunt Scraper Pro/1.3 (authorized API client)';112const WEBSITE_USER_AGENT = 'Mozilla/5.0 (compatible; ProductHuntScraperPro/1.3; +https://apify.com/khadinakbar/producthunt-scraper-pro)';113
114let nextEmailListVerifyRequestAt = 0;115let emailListVerifyCreditsExhausted = false;116const publicHostCache = new Map<string, Promise<boolean>>();117
118function isoDate(date: Date): string {119 return date.toISOString().slice(0, 10);120}121
122function getWeekBounds(date: Date): { start: string; end: string } {123 const start = new Date(date);124 // Product Hunt's weekly archive uses ISO weeks: Monday through Sunday.125 const weekday = date.getUTCDay() || 7;126 start.setUTCDate(date.getUTCDate() - weekday + 1);127 const end = new Date(start);128 end.setUTCDate(start.getUTCDate() + 6);129 return { start: isoDate(start), end: isoDate(end) };130}131
132function getMonthBounds(date: Date): { start: string; end: string } {133 const year = date.getUTCFullYear();134 const month = date.getUTCMonth();135 return {136 start: isoDate(new Date(Date.UTC(year, month, 1))),137 end: isoDate(new Date(Date.UTC(year, month + 1, 0))),138 };139}140
141function getYearBounds(date: Date): { start: string; end: string } {142 const year = date.getUTCFullYear();143 return { start: `${year}-01-01`, end: `${year}-12-31` };144}145
146function validIsoDate(value: string): boolean {147 if (!/^\d{4}-\d{2}-\d{2}$/.test(value)) return false;148 const parsed = new Date(`${value}T12:00:00Z`);149 return Number.isFinite(parsed.getTime()) && isoDate(parsed) === value;150}151
152function dateRangeForPeriod(period: LeaderboardPeriod, anchorDate: string, endDate = ''): DateRange {153 const base = new Date(`${anchorDate}T12:00:00Z`);154 if (period === 'daily') {155 return {156 postedAfter: `${anchorDate}T00:00:00+00:00`,157 postedBefore: `${endDate || anchorDate}T23:59:59+00:00`,158 };159 }160 const bounds = period === 'weekly'161 ? getWeekBounds(base)162 : period === 'monthly'163 ? getMonthBounds(base)164 : getYearBounds(base);165 return {166 postedAfter: `${bounds.start}T00:00:00+00:00`,167 postedBefore: `${bounds.end}T23:59:59+00:00`,168 };169}170
171function normalizeKey(value: unknown): string | null {172 if (typeof value !== 'string' || !value.trim()) return null;173 return value.trim().toLowerCase().replace(/[?#].*$/, '').replace(/\/$/, '');174}175
176function parseWebsiteOverrides(value: unknown): WebsiteOverride[] {177 const overrides: WebsiteOverride[] = [];178 const add = (keys: unknown[], websiteUrl: unknown): void => {179 if (typeof websiteUrl !== 'string' || !isPublicHttpUrl(websiteUrl)) return;180 const canonical = canonicalWebsiteUrl(websiteUrl);181 if (!canonical || isProductHuntHost(new URL(canonical).hostname)) return;182 const usableKeys = keys.map(normalizeKey).filter((key): key is string => Boolean(key));183 if (usableKeys.length > 0) overrides.push({ keys: usableKeys, websiteUrl: canonical });184 };185
186 if (Array.isArray(value)) {187 for (const entry of value) {188 if (!entry || typeof entry !== 'object') continue;189 const row = entry as Record<string, unknown>;190 add([row.productHuntUrl, row.productSlug, row.productName, row.productId], row.websiteUrl);191 }192 } else if (value && typeof value === 'object') {193 for (const [key, websiteUrl] of Object.entries(value as Record<string, unknown>)) add([key], websiteUrl);194 }195 return overrides;196}197
198function lookupWebsiteOverride(node: Record<string, unknown>, overrides: WebsiteOverride[]): string | null {199 const candidates = [node.url, node.slug, node.id, node.name]200 .map(normalizeKey)201 .filter((value): value is string => Boolean(value));202 return overrides.find((override) => override.keys.some((key) => candidates.includes(key)))?.websiteUrl ?? null;203}204
205function rootDomain(hostname: string): string {206 const labels = hostname.replace(/^www\./i, '').toLowerCase().split('.');207 return labels.slice(-2).join('.');208}209
210function isAllowedExternalWebsite(url: string): boolean {211 if (!isPublicHttpUrl(url)) return false;212 try {213 const host = new URL(url).hostname.toLowerCase();214 return !isProductHuntHost(host) && !SKIP_DOMAINS.has(host) && !SKIP_DOMAINS.has(rootDomain(host));215 } catch {216 return false;217 }218}219
220async function isSafePublicNetworkUrl(rawUrl: string): Promise<boolean> {221 if (!isPublicHttpUrl(rawUrl)) return false;222 let hostname: string;223 try {224 hostname = new URL(rawUrl).hostname.toLowerCase().replace(/^\[|\]$/g, '');225 } catch {226 return false;227 }228 if (isIP(hostname)) return !isPrivateNetworkAddress(hostname);229
230 let assessment = publicHostCache.get(hostname);231 if (!assessment) {232 assessment = lookup(hostname, { all: true, verbatim: true })233 .then((records) => records.length > 0 && records.every(({ address }) => !isPrivateNetworkAddress(address)))234 .catch(() => false);235 publicHostCache.set(hostname, assessment);236 }237 return assessment;238}239
240// Executed inside a website page. It intentionally extracts only public page241// content; no forms, login state, or challenge handling is involved.242function browserExtractEmails(): string[] {243 const emailRe = /\b[a-zA-Z0-9._%+\-]+@[a-zA-Z0-9.\-]+\.[a-zA-Z]{2,}\b/g;244 const obfuscatedRe = /\b[a-zA-Z0-9._%+\-]+\s*(?:\[at\]|\(at\)|\sat\s)\s*[a-zA-Z0-9.\-]+\s*(?:\[dot\]|\(dot\)|\sdot\s)\s*[a-zA-Z]{2,}\b/gi;245 const collected: string[] = [];246 const add = (value: unknown): void => {247 if (typeof value !== 'string') return;248 const email = value.trim().toLowerCase().replace(/^mailto:/, '').split('?')[0];249 if (email.includes('@')) collected.push(email);250 };251 const walkJson = (value: unknown, depth = 0): void => {252 if (depth > 8 || value === null || value === undefined) return;253 if (Array.isArray(value)) {254 value.forEach((item) => walkJson(item, depth + 1));255 return;256 }257 if (typeof value !== 'object') return;258 for (const [key, child] of Object.entries(value as Record<string, unknown>)) {259 if (key.toLowerCase() === 'email') add(child);260 walkJson(child, depth + 1);261 }262 };263 const decodeCfEmail = (encoded: string): string | null => {264 if (!/^[0-9a-f]+$/i.test(encoded) || encoded.length < 4) return null;265 const key = Number.parseInt(encoded.slice(0, 2), 16);266 let decoded = '';267 for (let index = 2; index < encoded.length; index += 2) {268 decoded += String.fromCharCode(Number.parseInt(encoded.slice(index, index + 2), 16) ^ key);269 }270 return decoded;271 };272
273 document.querySelectorAll('a[href^="mailto:"]').forEach((element) => add((element as HTMLAnchorElement).href));274 document.querySelectorAll('[data-email]').forEach((element) => add(element.getAttribute('data-email')));275 document.querySelectorAll('[data-cfemail]').forEach((element) => {276 const decoded = decodeCfEmail(element.getAttribute('data-cfemail') ?? '');277 if (decoded) add(decoded);278 });279 document.querySelectorAll('script[type="application/ld+json"]').forEach((element) => {280 try { walkJson(JSON.parse(element.textContent ?? '')); } catch { /* malformed JSON-LD */ }281 });282
283 const combined = `${document.body?.innerText ?? ''}\n${document.documentElement.innerHTML}`;284 (combined.match(emailRe) ?? []).forEach(add);285 for (const match of combined.match(obfuscatedRe) ?? []) {286 add(match.replace(/\s*(?:\[at\]|\(at\)|\sat\s)\s*/i, '@').replace(/\s*(?:\[dot\]|\(dot\)|\sdot\s)\s*/ig, '.'));287 }288 return collected;289}290
291// Executed only on the verified external website homepage. It never invents a292// route and exposes only allowlisted paths directly linked by that homepage.293function browserExtractLinkedFallbackPages(origin: string): WebsitePage[] {294 const matchers: Array<{ kind: Exclude<WebsitePage['kind'], 'homepage'>; priority: number; pattern: RegExp }> = [295 { kind: 'contact', priority: 0, pattern: /(contact|contact-us|get-in-touch|reach-us)/i },296 { kind: 'about', priority: 1, pattern: /(about|about-us|company|our-story)/i },297 { kind: 'team', priority: 2, pattern: /(team|our-team|meet-the-team|people|founders)/i },298 { kind: 'privacy', priority: 3, pattern: /(privacy|privacy-policy)/i },299 { kind: 'terms', priority: 4, pattern: /(terms|terms-of-service|terms-and-conditions|conditions)/i },300 ];301 const links = new Map<string, WebsitePage>();302 document.querySelectorAll('a[href]').forEach((element) => {303 const anchor = element as HTMLAnchorElement;304 try {305 const url = new URL(anchor.href);306 if (url.origin !== origin || !['http:', 'https:'].includes(url.protocol) || url.username || url.password) return;307 const label = (anchor.innerText || anchor.textContent || '').trim();308 const matched = matchers.find((candidate) => candidate.pattern.test(`${url.pathname} ${label}`));309 if (!matched) return;310 url.hash = '';311 const clean = url.toString();312 if (clean === `${origin}/` || clean === origin || links.has(clean)) return;313 links.set(clean, { url: clean, kind: matched.kind, priority: matched.priority });314 } catch { /* ignore malformed hrefs */ }315 });316 return [...links.values()].sort((left, right) => left.priority - right.priority);317}318
319async function waitForEmailListVerifySlot(): Promise<void> {320 const scheduledAt = Math.max(Date.now(), nextEmailListVerifyRequestAt);321 nextEmailListVerifyRequestAt = scheduledAt + EMAIL_LIST_VERIFY_MIN_INTERVAL_MS;322 const delayMs = scheduledAt - Date.now();323 if (delayMs > 0) await new Promise((resolve) => setTimeout(resolve, delayMs));324}325
326async function verifyEmailsWithEmailListVerify(327 emails: string[],328 requested: boolean,329 apiKey: string,330): Promise<{ verifications: EmailVerification[]; attempts: number; warnings: string[] }> {331 if (emails.length === 0) return { verifications: [], attempts: 0, warnings: [] };332 if (!requested) {333 return {334 verifications: emails.map((email) => ({335 email, provider: 'emaillistverify', status: 'not_requested', checkedAt: null,336 source: 'public_website', confidence: null,337 })),338 attempts: 0,339 warnings: [],340 };341 }342 if (!apiKey) {343 return {344 verifications: emails.map((email) => ({345 email, provider: 'emaillistverify', status: 'not_configured', checkedAt: null,346 source: 'public_website', confidence: null,347 })),348 attempts: 0,349 warnings: ['EmailListVerify was requested but no owner-managed API key was configured; public emails were kept unverified.'],350 };351 }352 if (emailListVerifyCreditsExhausted) {353 return {354 verifications: emails.map((email) => ({355 email, provider: 'emaillistverify', status: 'error_credit', checkedAt: null,356 source: 'public_website', confidence: null,357 })),358 attempts: 0,359 warnings: ['EmailListVerify credits were exhausted earlier in this run; no additional verification requests were sent.'],360 };361 }362
363 const warnings = new Set<string>();364 const verifications = await Promise.all(emails.map(async (email): Promise<EmailVerification> => {365 await waitForEmailListVerifySlot();366 const checkedAt = new Date().toISOString();367 try {368 const requestUrl = new URL(EMAIL_LIST_VERIFY_API);369 requestUrl.searchParams.set('email', email);370 const response = await fetch(requestUrl, {371 headers: { 'x-api-key': apiKey, Accept: 'text/plain, application/json;q=0.9' },372 signal: AbortSignal.timeout(12_000),373 });374 if (!response.ok) {375 warnings.add(`EmailListVerify returned HTTP ${response.status}; affected public emails were retained with provider_error status.`);376 if (response.status === 402) emailListVerifyCreditsExhausted = true;377 return {378 email, provider: 'emaillistverify', status: response.status === 402 ? 'error_credit' : 'provider_error', checkedAt,379 source: 'public_website', confidence: null,380 };381 }382 const status = normalizeEmailListVerifyStatus(await response.text());383 if (status === 'provider_error') warnings.add('EmailListVerify returned an undocumented response; affected public emails were retained with provider_error status.');384 if (status === 'error_credit') {385 emailListVerifyCreditsExhausted = true;386 warnings.add('EmailListVerify reported insufficient verification credits; remaining public emails were retained with error_credit status.');387 }388 return { email, provider: 'emaillistverify', status, checkedAt, source: 'public_website', confidence: null };389 } catch {390 warnings.add('EmailListVerify request failed; affected public emails were retained with provider_error status.');391 return { email, provider: 'emaillistverify', status: 'provider_error', checkedAt, source: 'public_website', confidence: null };392 }393 }));394 return { verifications, attempts: emails.length, warnings: [...warnings] };395}396
397let browser: Browser | null = null;398
399async function ensureBrowser(): Promise<Browser> {400 if (!browser) {401 browser = await chromium.launch({402 headless: true,403 args: [404 '--no-sandbox', '--disable-setuid-sandbox', '--disable-dev-shm-usage',405 '--disable-gpu', '--no-zygote', '--disable-extensions',406 ],407 });408 log.info('Browser launched for verified external website enrichment.');409 }410 return browser;411}412
413async function closeBrowser(): Promise<void> {414 if (!browser) return;415 await browser.close().catch(() => undefined);416 browser = null;417}418
419function extractStaticEmails(html: string): string[] {420 const $ = load(html);421 const collected: string[] = [];422 const emailRe = /\b[a-zA-Z0-9._%+\-]+@[a-zA-Z0-9.\-]+\.[a-zA-Z]{2,}\b/g;423 const add = (value: unknown): void => {424 if (typeof value !== 'string') return;425 const email = value.trim().toLowerCase().replace(/^mailto:/, '').split('?')[0];426 if (email.includes('@')) collected.push(email);427 };428 const walkJson = (value: unknown, depth = 0): void => {429 if (depth > 8 || value === null || value === undefined) return;430 if (Array.isArray(value)) return value.forEach((item) => walkJson(item, depth + 1));431 if (typeof value !== 'object') return;432 for (const [key, child] of Object.entries(value as Record<string, unknown>)) {433 if (key.toLowerCase() === 'email') add(child);434 walkJson(child, depth + 1);435 }436 };437 $('a[href^="mailto:"]').each((_, element) => add($(element).attr('href')));438 $('[data-email]').each((_, element) => add($(element).attr('data-email')));439 $('script[type="application/ld+json"]').each((_, element) => {440 try { walkJson(JSON.parse($(element).text())); } catch { /* malformed JSON-LD */ }441 });442 const combined = `${$.text()}\n${html}`;443 (combined.match(emailRe) ?? []).forEach(add);444 return collected;445}446
447function extractStaticFallbackPages(html: string, origin: string): WebsitePage[] {448 const $ = load(html);449 const matchers: Array<{ kind: Exclude<WebsitePage['kind'], 'homepage'>; priority: number; pattern: RegExp }> = [450 { kind: 'contact', priority: 0, pattern: /(contact|contact-us|get-in-touch|reach-us)/i },451 { kind: 'about', priority: 1, pattern: /(about|about-us|company|our-story)/i },452 { kind: 'team', priority: 2, pattern: /(team|our-team|meet-the-team|people|founders)/i },453 { kind: 'privacy', priority: 3, pattern: /(privacy|privacy-policy)/i },454 { kind: 'terms', priority: 4, pattern: /(terms|terms-of-service|terms-and-conditions|conditions)/i },455 ];456 const candidates = new Map<string, WebsitePage>();457 $('a[href]').each((_, element) => {458 try {459 const url = new URL($(element).attr('href') ?? '', origin);460 if (url.origin !== origin || !['http:', 'https:'].includes(url.protocol) || url.username || url.password) return;461 const label = $(element).text().trim();462 const matched = matchers.find((candidate) => candidate.pattern.test(`${url.pathname} ${label}`));463 if (!matched) return;464 url.hash = '';465 const clean = url.toString();466 if (clean === `${origin}/` || clean === origin || candidates.has(clean)) return;467 candidates.set(clean, { url: clean, kind: matched.kind, priority: matched.priority });468 } catch { /* ignore malformed hrefs */ }469 });470 return [...candidates.values()].sort((left, right) => left.priority - right.priority);471}472
473type StaticPage = { ok: true; url: string; html: string } | { ok: false; reason: string };474
475async function fetchPublicHtml(rawUrl: string, timeoutMs: number): Promise<StaticPage> {476 let current = canonicalWebsiteUrl(rawUrl);477 if (!current || !isAllowedExternalWebsite(current)) return { ok: false, reason: 'invalid_or_private_target' };478 try {479 for (let hop = 0; hop <= 3; hop += 1) {480 if (!await isSafePublicNetworkUrl(current)) return { ok: false, reason: 'private_or_unresolvable_target' };481 const response = await fetch(current, {482 method: 'GET',483 redirect: 'manual',484 headers: { Accept: 'text/html,application/xhtml+xml', 'User-Agent': WEBSITE_USER_AGENT },485 signal: AbortSignal.timeout(timeoutMs),486 });487 if (response.status >= 300 && response.status < 400) {488 const location = response.headers.get('location');489 if (!location || hop === 3) return { ok: false, reason: 'redirect_hop_limit' };490 const next = canonicalWebsiteUrl(new URL(location, current).toString());491 if (!next || !isAllowedExternalWebsite(next)) return { ok: false, reason: 'redirect_to_private_or_unsupported_target' };492 if (!await isSafePublicNetworkUrl(next)) return { ok: false, reason: 'redirect_to_private_or_unresolvable_target' };493 current = next;494 continue;495 }496 if (!response.ok) return { ok: false, reason: `http_${response.status}` };497 const contentType = response.headers.get('content-type') ?? '';498 if (!/(?:text\/html|application\/xhtml\+xml)/i.test(contentType)) return { ok: false, reason: 'non_html_response' };499 const contentLength = Number(response.headers.get('content-length') ?? '0');500 if (Number.isFinite(contentLength) && contentLength > 1_500_000) return { ok: false, reason: 'response_too_large' };501 const html = (await response.text()).slice(0, 1_500_000);502 return { ok: true, url: current, html };503 }504 return { ok: false, reason: 'redirect_hop_limit' };505 } catch (error) {506 const message = error instanceof Error ? error.message : '';507 return { ok: false, reason: /timeout|abort/i.test(message) ? 'timeout' : 'network_error' };508 }509}510
511function pageIdentity(html: string): { title: string; description: string; text: string } {512 const $ = load(html);513 $('script,style,noscript,svg').remove();514 return {515 title: $('title').first().text().trim(),516 description: $('meta[name="description"]').attr('content')?.trim()517 || $('meta[property="og:description"]').attr('content')?.trim()518 || '',519 text: $('body').text().replace(/\s+/g, ' ').trim().slice(0, 100_000),520 };521}522
523async function resolveProductHuntRedirect(rawUrl: string): Promise<string | null> {524 let current = canonicalWebsiteUrl(rawUrl);525 if (!current) return null;526 try {527 if (!isProductHuntHost(new URL(current).hostname)) return null;528 for (let hop = 0; hop < 4; hop += 1) {529 if (!await isSafePublicNetworkUrl(current)) return null;530 let response = await fetch(current, {531 method: 'HEAD',532 redirect: 'manual',533 headers: { 'User-Agent': PRODUCT_HUNT_USER_AGENT, Accept: 'text/html,*/*;q=0.5' },534 signal: AbortSignal.timeout(8_000),535 });536 if (response.status === 405) {537 response = await fetch(current, {538 method: 'GET',539 redirect: 'manual',540 headers: { 'User-Agent': PRODUCT_HUNT_USER_AGENT, Accept: 'text/html,*/*;q=0.5' },541 signal: AbortSignal.timeout(8_000),542 });543 }544 if (response.status < 300 || response.status >= 400) return null;545 const location = response.headers.get('location');546 if (!location) return null;547 const next = canonicalWebsiteUrl(new URL(location, current).toString());548 if (!next) return null;549 if (!isProductHuntHost(new URL(next).hostname)) {550 return isAllowedExternalWebsite(next) && await isSafePublicNetworkUrl(next) ? next : null;551 }552 current = next;553 }554 } catch {555 return null;556 }557 return null;558}559
560async function verifyWebsiteCandidates(561 candidates: SearchCandidate[],562 productName: string,563 tagline: string,564): Promise<{ websiteUrl: string; confidence: number; signals: string[] } | null> {565 const ranked = [...candidates]566 .map((candidate) => ({ candidate, score: scoreSearchCandidate(candidate, productName, tagline) }))567 .filter(({ score }) => score >= 34)568 .sort((left, right) => right.score - left.score || left.candidate.rank - right.candidate.rank)569 .slice(0, 5);570
571 for (const { candidate } of ranked) {572 let homepage: string;573 try {574 const candidateUrl = new URL(candidate.url);575 homepage = `${candidateUrl.origin}/`;576 } catch {577 continue;578 }579 const fetched = await fetchPublicHtml(homepage, 6_000);580 if (!fetched.ok) continue;581 const identity = pageIdentity(fetched.html);582 const assessment = evaluateWebsiteIdentity({583 candidate: { ...candidate, url: fetched.url },584 productName,585 tagline,586 pageTitle: identity.title,587 pageDescription: identity.description,588 pageText: identity.text,589 });590 if (!assessment.accepted) continue;591 const finalUrl = new URL(fetched.url);592 return { websiteUrl: `${finalUrl.origin}/`, confidence: assessment.confidence, signals: assessment.signals };593 }594 return null;595}596
597async function callDataForSeoWebsiteSearch(598 query: string,599 login: string,600 password: string,601): Promise<{ candidates: SearchCandidate[]; costUsd: number }> {602 const response = await fetch(DATAFORSEO_SEARCH_API, {603 method: 'POST',604 headers: {605 Authorization: `Basic ${Buffer.from(`${login}:${password}`).toString('base64')}`,606 'Content-Type': 'application/json',607 Accept: 'application/json',608 },609 body: JSON.stringify([{610 keyword: query,611 location_code: 2840,612 language_code: 'en',613 device: 'desktop',614 os: 'windows',615 depth: 10,616 group_organic_results: true,617 }]),618 signal: AbortSignal.timeout(45_000),619 });620 if (!response.ok) throw new Error(`DataForSEO website search returned HTTP ${response.status}.`);621 const body = await response.json() as Record<string, any>;622 if (body.status_code && body.status_code !== 20000) {623 throw new Error(`DataForSEO website search failed with provider code ${body.status_code}.`);624 }625 const task = Array.isArray(body.tasks) ? body.tasks[0] : null;626 if (task?.status_code && task.status_code !== 20000) {627 throw new Error(`DataForSEO website search task failed with provider code ${task.status_code}.`);628 }629 return parseDataForSeoCandidates(body);630}631
632async function callSerpApiWebsiteSearch(query: string, apiKey: string): Promise<SearchCandidate[]> {633 const url = new URL(SERPAPI_SEARCH_API);634 url.searchParams.set('engine', 'google');635 url.searchParams.set('q', query);636 url.searchParams.set('num', '10');637 url.searchParams.set('gl', 'us');638 url.searchParams.set('hl', 'en');639 url.searchParams.set('api_key', apiKey);640 const response = await fetch(url, { signal: AbortSignal.timeout(30_000), headers: { Accept: 'application/json' } });641 if (!response.ok) throw new Error(`SerpApi website search returned HTTP ${response.status}.`);642 const body = await response.json() as Record<string, unknown>;643 if (body.error) throw new Error('SerpApi website search returned a provider error.');644 return parseSerpApiCandidates(body);645}646
647async function resolveWebsite(input: {648 productName: string;649 tagline: string;650 productHuntRedirect: string;651 makerWebsiteUrls: string[];652 dataForSeoLogin: string;653 dataForSeoPassword: string;654 serpApiKey: string;655}): Promise<WebsiteResolution> {656 const warnings: string[] = [];657 let providerCostUsd = 0;658 let providerAttempts = 0;659 const unresolved = (): WebsiteResolution => ({660 websiteUrl: null,661 source: 'none',662 status: input.productHuntRedirect ? 'unresolved' : 'not_available',663 confidence: 0,664 signals: [],665 providerCostUsd,666 providerAttempts,667 warnings,668 });669
670 if (input.productHuntRedirect) {671 const destination = await resolveProductHuntRedirect(input.productHuntRedirect);672 if (destination) {673 const verified = await verifyWebsiteCandidates([{674 url: destination, title: input.productName, snippet: input.tagline, rank: 1,675 }], input.productName, input.tagline);676 if (verified) {677 return {678 ...verified,679 source: 'product_hunt_redirect',680 status: 'verified_external',681 providerCostUsd,682 providerAttempts,683 warnings,684 };685 }686 }687 }688
689 const makerCandidates = input.makerWebsiteUrls690 .filter((url) => isAllowedExternalWebsite(url))691 .slice(0, 2)692 .map((url, index): SearchCandidate => ({ url, title: '', snippet: '', rank: index + 1 }));693 if (makerCandidates.length > 0) {694 const verified = await verifyWebsiteCandidates(makerCandidates, input.productName, input.tagline);695 if (verified) {696 return {697 ...verified,698 source: 'maker_website',699 status: 'verified_external',700 providerCostUsd,701 providerAttempts,702 warnings,703 };704 }705 }706
707 const query = buildWebsiteSearchQuery(input.productName, input.tagline);708 if (input.dataForSeoLogin && input.dataForSeoPassword) {709 providerAttempts += 1;710 try {711 const search = await callDataForSeoWebsiteSearch(query, input.dataForSeoLogin, input.dataForSeoPassword);712 providerCostUsd += search.costUsd;713 const verified = await verifyWebsiteCandidates(search.candidates, input.productName, input.tagline);714 if (verified) {715 return {716 ...verified,717 source: 'dataforseo_organic',718 status: 'verified_search_match',719 providerCostUsd,720 providerAttempts,721 warnings,722 };723 }724 warnings.push('DataForSEO returned no website candidate that passed product-identity verification.');725 } catch (error) {726 const message = error instanceof Error ? error.message : 'DataForSEO website search failed.';727 warnings.push(message.slice(0, 160));728 }729 }730
731 if (input.serpApiKey) {732 providerAttempts += 1;733 try {734 const candidates = await callSerpApiWebsiteSearch(query, input.serpApiKey);735 const verified = await verifyWebsiteCandidates(candidates, input.productName, input.tagline);736 if (verified) {737 return {738 ...verified,739 source: 'serpapi_organic',740 status: 'verified_search_match',741 providerCostUsd,742 providerAttempts,743 warnings,744 };745 }746 warnings.push('SerpApi returned no website candidate that passed product-identity verification.');747 } catch (error) {748 const message = error instanceof Error ? error.message : 'SerpApi website search failed.';749 warnings.push(message.slice(0, 160));750 }751 }752
753 if (!input.dataForSeoLogin && !input.serpApiKey) {754 warnings.push('No owner-managed website search provider was configured for unresolved Product Hunt links.');755 }756 return unresolved();757}758
759function companyDomainFromWebsite(websiteUrl: string): string {760 try {761 const hostname = new URL(websiteUrl).hostname.toLowerCase().replace(/^www\./, '');762 return getDomain(hostname, { allowPrivateDomains: true }) ?? hostname;763 } catch {764 return '';765 }766}767
768async function requestEmailListVerifyContacts(769 apiKey: string,770 body: { domain: string; firstName?: string; lastName?: string },771): Promise<unknown> {772 await waitForEmailListVerifySlot();773 const response = await fetch(EMAIL_LIST_VERIFY_CONTACT_API, {774 method: 'POST',775 headers: { 'x-api-key': apiKey, 'Content-Type': 'application/json', Accept: 'application/json' },776 body: JSON.stringify(body),777 signal: AbortSignal.timeout(30_000),778 });779 if (!response.ok) {780 if (response.status === 402) emailListVerifyCreditsExhausted = true;781 throw new Error(`EmailListVerify Contact Finder returned HTTP ${response.status}.`);782 }783 return response.json();784}785
786async function discoverContacts(787 websiteUrl: string,788 makerNames: string[],789 maxEmails: number,790 apiKey: string,791): Promise<ContactDiscovery> {792 const domain = companyDomainFromWebsite(websiteUrl);793 if (!domain || maxEmails <= 0) return { contacts: [], attempts: 0, credits: 0, warnings: [] };794 if (!apiKey) {795 return {796 contacts: [], attempts: 0, credits: 0,797 warnings: ['Contact discovery was requested but the owner-managed EmailListVerify key is not configured.'],798 };799 }800 if (emailListVerifyCreditsExhausted) {801 return {802 contacts: [], attempts: 0, credits: 0,803 warnings: ['EmailListVerify credits were exhausted earlier in this run; contact discovery was skipped.'],804 };805 }806
807 const warnings: string[] = [];808 let attempts = 0;809 let credits = 0;810 let contacts: ContactFinderResult[] = [];811 const parsedMakers = makerNames.map(splitMakerName).filter(({ firstName, lastName }) => firstName || lastName);812 const maker = parsedMakers.find(({ firstName, lastName }) => firstName && lastName) ?? parsedMakers[0];813 if (maker) {814 attempts += 1;815 credits += 5;816 try {817 const payload = await requestEmailListVerifyContacts(apiKey, { domain, ...maker });818 contacts = normalizeContactFinderResults(payload, domain, maxEmails);819 if (contacts.length >= maxEmails) return { contacts, attempts, credits, warnings };820 } catch (error) {821 const message = error instanceof Error ? error.message : 'EmailListVerify named contact discovery failed.';822 warnings.push(message.slice(0, 160));823 }824 }825
826 if (!emailListVerifyCreditsExhausted) {827 attempts += 1;828 credits += 10;829 try {830 const payload = await requestEmailListVerifyContacts(apiKey, { domain });831 contacts = normalizeContactFinderResults([832 ...contacts,833 ...(Array.isArray(payload) ? payload : []),834 ], domain, maxEmails);835 return { contacts, attempts, credits, warnings };836 } catch (error) {837 const message = error instanceof Error ? error.message : 'EmailListVerify domain contact discovery failed.';838 warnings.push(message.slice(0, 160));839 }840 }841 return { contacts, attempts, credits, warnings };842}843
844function selectBestEmails(845 publicVerifications: EmailVerification[],846 contacts: ContactFinderResult[],847 maxEmails: number,848): EmailVerification[] {849 const contactVerifications: EmailVerification[] = contacts.map((contact) => ({850 email: contact.email,851 provider: 'emaillistverify',852 status: contact.result as EmailListVerifyStatus,853 checkedAt: new Date().toISOString(),854 source: 'emaillistverify_contact_finder',855 confidence: contact.confidence,856 }));857 const statusScore: Record<string, number> = {858 ok: 100, ok_for_all: 70, unknown: 55, antispam_system: 50,859 not_requested: 48, not_configured: 45, provider_error: 40, error_credit: 38,860 smtp_protocol: 30, dead_server: 10, email_disabled: 5, invalid_mx: 0,861 invalid_syntax: 0, disposable: 0, spamtrap: 0,862 };863 const best = new Map<string, EmailVerification>();864 for (const verification of [...publicVerifications, ...contactVerifications]) {865 const current = best.get(verification.email);866 const score = (statusScore[verification.status] ?? 20)867 + (verification.source === 'public_website' ? 3 : 0)868 + (verification.confidence === 'high' ? 2 : verification.confidence === 'medium' ? 1 : 0);869 const currentScore = current870 ? (statusScore[current.status] ?? 20) + (current.source === 'public_website' ? 3 : 0)871 : -1;872 if (!current || score > currentScore) best.set(verification.email, verification);873 }874 return [...best.values()]875 .filter(({ status }) => isUsableEmailStatus(status))876 .sort((left, right) => {877 const leftScore = (statusScore[left.status] ?? 20) + (left.source === 'public_website' ? 3 : 0);878 const rightScore = (statusScore[right.status] ?? 20) + (right.source === 'public_website' ? 3 : 0);879 return rightScore - leftScore;880 })881 .slice(0, maxEmails);882}883
884async function findEmails(885 rawWebsiteUrl: string,886 maxWebsitePages: number,887 maxEmailsPerProduct: number,888): Promise<EnrichmentResult> {889 const websiteUrl = canonicalWebsiteUrl(rawWebsiteUrl);890 if (!websiteUrl || !isAllowedExternalWebsite(websiteUrl)) {891 return {892 emails: [], emailSource: 'none', result: 'not_assessed', coverage: 'none',893 terminalReason: 'invalid_or_private_target', pagesVisited: 0, pageUrls: [],894 warnings: ['Website URL was not a permitted public external HTTP(S) target.'],895 };896 }897
898 const initialUrl = new URL(websiteUrl);899 const warnings: string[] = [];900 const pageUrls: string[] = [];901 const addPageUrl = (url: string): void => {902 const canonical = canonicalWebsiteUrl(url);903 if (canonical && !pageUrls.includes(canonical) && pageUrls.length < maxWebsitePages) pageUrls.push(canonical);904 };905 let canonicalOrigin = initialUrl.origin;906 let baseDomain = initialUrl.hostname.replace(/^www\./i, '');907 let emails: string[] = [];908 let coverage: Coverage = 'complete';909 let pagePlan: WebsitePage[] = [];910
911 const staticHomepage = await fetchPublicHtml(`${initialUrl.origin}/`, 4_000);912 if (staticHomepage.ok) {913 const finalHomepage = new URL(staticHomepage.url);914 canonicalOrigin = finalHomepage.origin;915 baseDomain = finalHomepage.hostname.replace(/^www\./i, '');916 addPageUrl(staticHomepage.url);917 emails = rankEmails(extractStaticEmails(staticHomepage.html), baseDomain, maxEmailsPerProduct);918 pagePlan = buildWebsitePagePlan(`${canonicalOrigin}/`, extractStaticFallbackPages(staticHomepage.html, canonicalOrigin), maxWebsitePages);919 for (const target of pagePlan.slice(1)) {920 if (emails.length >= maxEmailsPerProduct) break;921 const staticPage = await fetchPublicHtml(target.url, 3_000);922 if (!staticPage.ok) {923 coverage = 'partial';924 warnings.push(`${target.kind} static assessment failed: ${staticPage.reason}.`);925 continue;926 }927 const finalPage = new URL(staticPage.url);928 if (finalPage.origin !== canonicalOrigin) {929 coverage = 'partial';930 warnings.push(`${target.kind} redirected off the verified homepage origin; it was not extracted.`);931 continue;932 }933 addPageUrl(staticPage.url);934 emails = rankEmails([...emails, ...extractStaticEmails(staticPage.html)], baseDomain, maxEmailsPerProduct);935 }936 } else {937 coverage = 'partial';938 warnings.push(`Homepage static assessment failed: ${staticHomepage.reason}.`);939 }940
941 if (emails.length < maxEmailsPerProduct) {942 const context = await (await ensureBrowser()).newContext({943 userAgent: WEBSITE_USER_AGENT,944 locale: 'en-US',945 viewport: { width: 1280, height: 800 },946 extraHTTPHeaders: { 'Accept-Language': 'en-US,en;q=0.9' },947 });948 await context.route('**/*', async (route) => {949 if (['image', 'media', 'font'].includes(route.request().resourceType())) return route.abort();950 const requestUrl = route.request().url();951 if (!/^(?:data|blob|about):/i.test(requestUrl) && !await isSafePublicNetworkUrl(requestUrl)) return route.abort();952 return route.continue();953 });954 try {955 const page = await context.newPage();956 const response = await page.goto(`${canonicalOrigin}/`, { waitUntil: 'domcontentloaded', timeout: 6_000 });957 const finalHomepage = new URL(page.url());958 if (!response || finalHomepage.origin !== canonicalOrigin) {959 coverage = 'partial';960 warnings.push('Homepage DOM assessment redirected off origin or did not return a response.');961 } else {962 addPageUrl(finalHomepage.toString());963 await page.locator('body').waitFor({ state: 'attached', timeout: 750 }).catch(() => undefined);964 emails = rankEmails([...emails, ...await page.evaluate(browserExtractEmails)], baseDomain, maxEmailsPerProduct);965 if (pagePlan.length <= 1) {966 pagePlan = buildWebsitePagePlan(`${canonicalOrigin}/`, await page.evaluate(browserExtractLinkedFallbackPages, canonicalOrigin), maxWebsitePages);967 }968 for (const target of pagePlan.slice(1)) {969 if (emails.length >= maxEmailsPerProduct) break;970 try {971 const fallbackResponse = await page.goto(target.url, { waitUntil: 'domcontentloaded', timeout: 4_500 });972 const finalFallback = new URL(page.url());973 if (!fallbackResponse || finalFallback.origin !== canonicalOrigin) {974 coverage = 'partial';975 warnings.push(`${target.kind} DOM assessment redirected off origin.`);976 continue;977 }978 addPageUrl(finalFallback.toString());979 await page.locator('body').waitFor({ state: 'attached', timeout: 750 }).catch(() => undefined);980 emails = rankEmails([...emails, ...await page.evaluate(browserExtractEmails)], baseDomain, maxEmailsPerProduct);981 } catch (error) {982 coverage = 'partial';983 const message = error instanceof Error ? error.message : 'navigation error';984 warnings.push(`${target.kind} DOM assessment failed: ${message.slice(0, 90)}.`);985 }986 }987 }988 } catch (error) {989 coverage = 'partial';990 const message = error instanceof Error ? error.message : 'browser error';991 warnings.push(`Homepage DOM assessment failed: ${message.slice(0, 90)}.`);992 } finally {993 await context.close();994 }995 }996
997 const result: EmailResult = emails.length > 0998 ? 'emails_found'999 : pageUrls.length > 0 && coverage === 'complete'1000 ? 'no_public_email_found'1001 : 'not_assessed';1002 return {1003 emails,1004 emailSource: emails.length > 0 ? 'page_scrape' : 'none',1005 result,1006 coverage: pageUrls.length === 0 ? 'none' : coverage,1007 terminalReason: emails.length >= maxEmailsPerProduct1008 ? 'max_public_emails_reached'1009 : result === 'no_public_email_found'1010 ? 'eligible_pages_exhausted'1011 : 'eligible_pages_partially_assessed',1012 pagesVisited: pageUrls.length,1013 pageUrls,1014 warnings,1015 };1016}1017
1018const POST_FIELDS = `1019 id name tagline description votesCount commentsCount url website slug1020 featuredAt createdAt dailyRank weeklyRank monthlyRank yearlyRank1021 reviewsCount reviewsRating1022 thumbnail { url }1023 productLinks { type url }1024 topics(first: 10) { edges { node { name } } }1025 makers { id name username twitterUsername url websiteUrl }1026`;1027
1028const POSTS_QUERY = `query GetPosts($first:Int $after:String $order:PostsOrder $featured:Boolean $postedAfter:DateTime $postedBefore:DateTime $topic:String) {1029 posts(first:$first after:$after order:$order featured:$featured postedAfter:$postedAfter postedBefore:$postedBefore topic:$topic) {1030 edges { node { ${POST_FIELDS} } }1031 pageInfo { endCursor hasNextPage }1032 }1033}`;1034
1035const POST_BY_SLUG_QUERY = `query GetPost($slug:String!) { post(slug:$slug) { ${POST_FIELDS} } }`;1036
1037function toInt(value: unknown): number | null {1038 if (value === null || value === undefined || value === '') return null;1039 const parsed = Number.parseInt(String(value), 10);1040 return Number.isNaN(parsed) ? null : parsed;1041}1042
1043function toFloat(value: unknown): number | null {1044 if (value === null || value === undefined || value === '') return null;1045 const parsed = Number.parseFloat(String(value));1046 return Number.isNaN(parsed) ? null : parsed;1047}1048
1049function nodeToRecord(node: Record<string, any>, rank: number): Record<string, unknown> {1050 const makers = (node.makers ?? []).map((maker: Record<string, unknown>) => ({1051 maker_name: maker.name ?? null,1052 maker_id: maker.username ?? null,1053 maker_ph_id: maker.id ?? null,1054 twitter_url: maker.twitterUsername ? `https://twitter.com/${String(maker.twitterUsername)}` : null,1055 maker_url: maker.url ?? null,1056 website_url: maker.websiteUrl ?? null,1057 }));1058 const topics = (node.topics?.edges ?? []).map((edge: any) => edge?.node?.name).filter(Boolean);1059 const productLinks = Array.isArray(node.productLinks) ? node.productLinks : [];1060 const directWebsite = productLinks.find((link: Record<string, unknown>) => {1061 const type = String(link.type ?? '').toLowerCase();1062 return type === 'website' && typeof link.url === 'string' && isAllowedExternalWebsite(link.url);1063 })?.url as string | undefined;1064
1065 const websiteUrl = directWebsite ? canonicalWebsiteUrl(directWebsite) : null;1066 return {1067 product_name: node.name ? String(node.name).trim() : null,1068 tagline: node.tagline ?? null,1069 description: node.description ?? null,1070 upvote_count: toInt(node.votesCount),1071 comment_count: toInt(node.commentsCount),1072 reviews_count: toInt(node.reviewsCount),1073 reviews_rating: toFloat(node.reviewsRating),1074 daily_rank: toInt(node.dailyRank) ?? rank,1075 weekly_rank: toInt(node.weeklyRank),1076 monthly_rank: toInt(node.monthlyRank),1077 yearly_rank: toInt(node.yearlyRank),1078 launch_date: node.featuredAt ? isoDate(new Date(node.featuredAt)) : node.createdAt ? isoDate(new Date(node.createdAt)) : null,1079 product_hunt_url: node.url ?? (node.slug ? `https://www.producthunt.com/posts/${node.slug}` : null),1080 product_hunt_website_url: node.website ?? null,1081 website_url: websiteUrl,1082 website_resolution_source: websiteUrl ? 'api_website_product_link' satisfies WebsiteResolutionSource : 'none' satisfies WebsiteResolutionSource,1083 website_resolution_status: websiteUrl1084 ? 'verified_external' satisfies WebsiteResolutionStatus1085 : node.website1086 ? 'unresolved' satisfies WebsiteResolutionStatus1087 : 'not_available' satisfies WebsiteResolutionStatus,1088 website_resolution_confidence: websiteUrl ? 100 : 0,1089 website_resolution_signals: websiteUrl ? ['official_api_product_link'] : [],1090 website_resolution_provider_cost_usd: 0,1091 company_domain: websiteUrl ? companyDomainFromWebsite(websiteUrl) : null,1092 topics,1093 thumbnail_url: node.thumbnail?.url ?? null,1094 makers,1095 featured: Boolean(node.featuredAt),1096 emails: [] as string[],1097 verified_emails: [] as string[],1098 risky_emails: [] as string[],1099 rejected_emails: [] as string[],1100 rejected_email_verifications: [] as EmailVerification[],1101 email_verifications: [] as EmailVerification[],1102 email_verification_requested: false,1103 email_source: 'none' as EmailSource,1104 email_result: 'not_assessed' as EmailResult,1105 email_coverage: 'none' as Coverage,1106 email_terminal_reason: 'website_not_available',1107 email_pages_visited: 0,1108 email_page_urls: [] as string[],1109 contact_discovery_requested: false,1110 contact_discovery_attempts: 0,1111 contact_discovery_credits: 0,1112 lead_status: 'unresolved_website',1113 lead_score: 0,1114 lead_tier: 'unqualified',1115 lead_signals: [] as string[],1116 enrichment_warnings: [] as string[],1117 scraped_at: new Date().toISOString(),1118 };1119}1120
1121const LEAN_FIELDS = new Set([1122 'product_name', 'tagline', 'website_url', 'company_domain', 'product_hunt_website_url', 'website_resolution_status',1123 'website_resolution_source', 'website_resolution_confidence', 'upvote_count', 'daily_rank', 'emails', 'verified_emails',1124 'risky_emails', 'rejected_emails', 'email_verifications', 'rejected_email_verifications', 'email_verification_requested', 'email_source', 'email_result', 'email_coverage',1125 'email_pages_visited', 'contact_discovery_attempts', 'lead_status', 'lead_score', 'lead_tier', 'lead_signals',1126 'topics', 'launch_date', 'product_hunt_url', 'featured', 'scraped_at',1127]);1128
1129function applyOutputMode(record: Record<string, unknown>, outputMode: 'full' | 'lean' | 'leads'): Record<string, unknown> | null {1130 if (outputMode === 'leads' && (!Array.isArray(record.emails) || record.emails.length === 0)) return null;1131 if (outputMode !== 'lean') return record;1132 const output: Record<string, unknown> = {};1133 for (const field of LEAN_FIELDS) output[field] = record[field] ?? null;1134 return output;1135}1136
1137function validateOutputRecord(record: Record<string, unknown>): string | null {1138 if (typeof record.product_name !== 'string' || !record.product_name.trim()) return 'product_name is missing.';1139 if (typeof record.product_hunt_url !== 'string' || !record.product_hunt_url.startsWith('https://')) return 'product_hunt_url is missing or invalid.';1140 if (!Array.isArray(record.emails) || !record.emails.every((email) => typeof email === 'string')) return 'emails must be a string array.';1141 if (typeof record.scraped_at !== 'string' || Number.isNaN(Date.parse(record.scraped_at))) return 'scraped_at is invalid.';1142 return null;1143}1144
1145async function run(): Promise<void> {1146 const rawInput = (await Actor.getInput() ?? {}) as Record<string, unknown>;1147 const mode = (rawInput.mode ?? rawInput.scrapeMode ?? 'leaderboard') as Mode;1148 const leaderboardPeriod = (rawInput.leaderboardPeriod ?? rawInput.period ?? 'daily') as LeaderboardPeriod;1149 const startDate = String(rawInput.startDate ?? rawInput.date ?? '');1150 const endDate = String(rawInput.endDate ?? '');1151 const lookbackDays = clampInteger(rawInput.lookbackDays, 1, 1, 365);1152 const outputMode = (rawInput.outputMode ?? 'full') as 'full' | 'lean' | 'leads';1153 const searchQuery = typeof (rawInput.searchQuery ?? rawInput.query ?? rawInput.q) === 'string'1154 ? String(rawInput.searchQuery ?? rawInput.query ?? rawInput.q).trim()1155 : '';1156 const topic = typeof (rawInput.topic ?? rawInput.topicSlug) === 'string'1157 ? String(rawInput.topic ?? rawInput.topicSlug).trim()1158 : '';1159 const startUrls = Array.isArray(rawInput.startUrls) ? rawInput.startUrls as Array<{ url?: unknown } | string> : [];1160 const maxResults = clampInteger(rawInput.maxResults ?? rawInput.maxItems ?? rawInput.limit, 100, 1, 20_000);1161 const maxConcurrency = clampInteger(rawInput.maxConcurrency, 3, 1, 5);1162 const maxWebsitePages = clampInteger(rawInput.maxWebsitePages, DEFAULT_MAX_WEBSITE_PAGES, 1, MAX_WEBSITE_PAGES);1163 const maxEmailsPerProduct = clampInteger(rawInput.maxEmailsPerProduct, DEFAULT_MAX_EMAILS_PER_PRODUCT, 1, MAX_EMAILS_PER_PRODUCT);1164 const includeAllProducts = Boolean(rawInput.includeAllProducts ?? rawInput.allProducts ?? rawInput.includeAll ?? true);1165 const enrichEmails = Boolean(rawInput.enrichEmails ?? rawInput.findEmails ?? true);1166 const resolveWebsites = Boolean(rawInput.resolveWebsites ?? true);1167 const verifyEmails = Boolean(rawInput.verifyEmails ?? rawInput.verifyWithEmailListVerify ?? true);1168 const findContacts = Boolean(rawInput.findContacts ?? true);1169 const emailListVerifyApiKey = String(process.env.EMAILLISTVERIFY_API_KEY ?? '').trim();1170 const phApiToken = String(process.env.PH_API_TOKEN ?? '');1171 const dataForSeoLogin = String(process.env.DATAFORSEO_LOGIN ?? '').trim();1172 const dataForSeoPassword = String(process.env.DATAFORSEO_PASSWORD ?? '').trim();1173 const serpApiKey = String(process.env.SERPAPI_KEY ?? '').trim();1174 const websiteOverrides = parseWebsiteOverrides(rawInput.websiteUrlOverrides);1175 const startedAt = new Date().toISOString();1176 emailListVerifyCreditsExhausted = false;1177
1178 const diagnostics = {1179 itemsPushed: 0,1180 itemsProcessed: 0,1181 itemsFiltered: 0,1182 itemsFailed: 0,1183 invalidRecords: 0,1184 apiFailures: 0,1185 websiteResolved: 0,1186 websiteUnresolved: 0,1187 websiteResolutionProviderAttempts: 0,1188 websiteResolutionProviderCostUsd: 0,1189 emailsFound: 0,1190 verifiedEmailsFound: 0,1191 emailVerificationAttempts: 0,1192 emailVerificationProviderErrors: 0,1193 contactDiscoveryAttempts: 0,1194 contactDiscoveryCredits: 0,1195 contactDiscoveryEmailsFound: 0,1196 leadTiers: { A: 0, B: 0, C: 0, unqualified: 0 },1197 resultEvents: 0,1198 emailEvents: 0,1199 chargeLimitReached: false,1200 warnings: [] as string[],1201 };1202 let finalised = false;1203
1204 const paidItemLimit = clampInteger(process.env.ACTOR_MAX_PAID_DATASET_ITEMS, maxResults, 1, maxResults);1205 const effectiveMaxResults = Math.min(maxResults, paidItemLimit);1206
1207 const addWarning = (message: string): void => {1208 if (diagnostics.warnings.length < 100 && !diagnostics.warnings.includes(message)) diagnostics.warnings.push(message);1209 };1210
1211 const finalizeRun = async (outcome: RunOutcome, message: string): Promise<never> => {1212 if (finalised) throw new Error(message);1213 finalised = true;1214 const finishedAt = new Date().toISOString();1215 const summary = {1216 outcome,1217 message,1218 startedAt,1219 finishedAt,1220 buildNumber: process.env.ACTOR_BUILD_NUMBER ?? null,1221 runId: process.env.ACTOR_RUN_ID ?? null,1222 datasetId: process.env.ACTOR_DEFAULT_DATASET_ID ?? null,1223 keyValueStoreId: process.env.ACTOR_DEFAULT_KEY_VALUE_STORE_ID ?? null,1224 input: {1225 mode, leaderboardPeriod, startDate: startDate || null, endDate: endDate || null,1226 lookbackDays, maxResults, effectiveMaxResults, includeAllProducts, enrichEmails, resolveWebsites, verifyEmails, findContacts,1227 emailListVerifyConfigured: Boolean(emailListVerifyApiKey), maxWebsitePages,1228 websiteSearchConfigured: Boolean((dataForSeoLogin && dataForSeoPassword) || serpApiKey),1229 maxEmailsPerProduct, outputMode,1230 },1231 ...diagnostics,1232 chargedEventCounts: {1233 [RESULT_EVENT]: diagnostics.resultEvents,1234 [EMAIL_EVENT]: diagnostics.emailEvents,1235 },1236 };1237 const output = {1238 outcome,1239 message,1240 itemsPushed: diagnostics.itemsPushed,1241 itemsFailed: diagnostics.itemsFailed,1242 invalidRecords: diagnostics.invalidRecords,1243 chargedEventCounts: summary.chargedEventCounts,1244 warnings: diagnostics.warnings,1245 };1246 await Actor.setValue('OUTPUT', output);1247 await Actor.setValue('RUN_SUMMARY', summary);1248 const failed = FAILED_OUTCOMES.has(outcome);1249 await Actor.setStatusMessage(message, {1250 isStatusMessageTerminal: true,1251 level: failed ? 'ERROR' : outcome === 'COMPLETE' ? 'INFO' : 'WARNING',1252 });1253 if (failed) {1254 await Actor.fail(message);1255 }1256 await Actor.exit();1257 throw new Error('Actor terminal call returned unexpectedly.');1258 };1259
1260 try {1261 const modes: Mode[] = ['leaderboard', 'search', 'topic', 'urls'];1262 const periods: LeaderboardPeriod[] = ['daily', 'weekly', 'monthly', 'yearly'];1263 const outputModes = ['full', 'lean', 'leads'];1264 if (!modes.includes(mode)) await finalizeRun('INVALID_INPUT', 'mode must be leaderboard, search, topic, or urls.');1265 if (!periods.includes(leaderboardPeriod)) await finalizeRun('INVALID_INPUT', 'leaderboardPeriod must be daily, weekly, monthly, or yearly.');1266 if (!outputModes.includes(outputMode)) await finalizeRun('INVALID_INPUT', 'outputMode must be full, lean, or leads.');1267 if (!phApiToken) await finalizeRun('CONFIG_ERROR', 'Product Hunt API token is not configured. Set the owner-managed PH_API_TOKEN secret and re-run.');1268 if (mode === 'search' && !searchQuery) await finalizeRun('INVALID_INPUT', 'searchQuery is required for search mode.');1269 if (mode === 'topic' && !topic) await finalizeRun('INVALID_INPUT', 'topic is required for topic mode.');1270 if (mode === 'urls' && startUrls.length === 0) await finalizeRun('INVALID_INPUT', 'startUrls is required for urls mode.');1271 if (startDate && !validIsoDate(startDate)) await finalizeRun('INVALID_INPUT', 'startDate must be a valid YYYY-MM-DD date.');1272 if (endDate && !validIsoDate(endDate)) await finalizeRun('INVALID_INPUT', 'endDate must be a valid YYYY-MM-DD date.');1273 if (startDate && endDate && startDate > endDate) await finalizeRun('INVALID_INPUT', 'endDate must be on or after startDate.');1274 if (leaderboardPeriod !== 'daily' && endDate) await finalizeRun('INVALID_INPUT', 'endDate is supported only with daily leaderboard ranges.');1275
1276 const maxPotentialEventCost = effectiveMaxResults * (RESULT_PRICE_USD + (enrichEmails ? maxEmailsPerProduct * EMAIL_PRICE_USD : 0));1277 const maxPotentialVerificationCredits = enrichEmails && emailListVerifyApiKey1278 ? effectiveMaxResults * ((verifyEmails ? maxEmailsPerProduct : 0) + (findContacts ? 15 : 0))1279 : 0;1280 log.info('ProductHunt Scraper starting', {1281 mode, leaderboardPeriod, startDate: startDate || '(default)', endDate: endDate || null,1282 lookbackDays, effectiveMaxResults, includeAllProducts, enrichEmails, resolveWebsites, verifyEmails, findContacts,1283 emailListVerifyConfigured: Boolean(emailListVerifyApiKey), maxWebsitePages, maxEmailsPerProduct,1284 websiteSearchProviders: {1285 dataForSeo: Boolean(dataForSeoLogin && dataForSeoPassword),1286 serpApiFallback: Boolean(serpApiKey),1287 },1288 eventCostCeilingUsd: Number(maxPotentialEventCost.toFixed(3)),1289 emailListVerifyCreditCeiling: maxPotentialVerificationCredits,1290 websiteSearchQueryCeiling: resolveWebsites ? effectiveMaxResults * (serpApiKey ? 2 : 1) : 0,1291 platformUsage: 'Additional compute and proxy usage is billed by Apify separately where applicable.',1292 });1293 await Actor.setStatusMessage(`Starting: up to ${effectiveMaxResults} products, ${maxWebsitePages} public website pages/product, and ${maxEmailsPerProduct} public emails/product.`);1294
1295 const phQuery = async (query: string, variables: Record<string, unknown>, attempt = 0): Promise<any> => {1296 const response = await fetch(PH_API, {1297 method: 'POST',1298 headers: {1299 'Content-Type': 'application/json',1300 Accept: 'application/json',1301 Authorization: `Bearer ${phApiToken}`,1302 'User-Agent': PRODUCT_HUNT_USER_AGENT,1303 },1304 body: JSON.stringify({ query, variables }),1305 signal: AbortSignal.timeout(30_000),1306 });1307 if (response.status === 429 && attempt < 3) {1308 const retryAfter = Number(response.headers.get('x-rate-limit-reset') ?? response.headers.get('retry-after') ?? 30);1309 const waitSeconds = Math.min(Math.max(5, Number.isFinite(retryAfter) ? retryAfter : 30), 120);1310 log.warning(`Product Hunt API rate limited; waiting ${waitSeconds}s before retry ${attempt + 1}/3.`);1311 await new Promise((resolve) => setTimeout(resolve, waitSeconds * 1000));1312 return phQuery(query, variables, attempt + 1);1313 }1314 if (!response.ok) throw new Error(`Product Hunt API returned HTTP ${response.status}.`);1315 const payload = await response.json() as { data?: unknown; errors?: Array<{ message?: string }> };1316 if (payload.errors?.length) throw new Error(`Product Hunt GraphQL: ${payload.errors[0]?.message ?? 'unknown error'}`);1317 return payload.data as any;1318 };1319
1320 const seenIds = new Set<string>();1321 let collectedCandidates = 0;1322
1323 const pushNode = async (node: Record<string, any>, rank: number): Promise<void> => {1324 const record = nodeToRecord(node, rank);1325 record.email_verification_requested = verifyEmails;1326 const override = lookupWebsiteOverride(node, websiteOverrides);1327 if (override) {1328 record.website_url = override;1329 record.website_resolution_source = 'input_override' satisfies WebsiteResolutionSource;1330 record.website_resolution_status = 'verified_external' satisfies WebsiteResolutionStatus;1331 record.website_resolution_confidence = 100;1332 record.website_resolution_signals = ['approved_input_override'];1333 }1334
1335 let websiteUrl = typeof record.website_url === 'string' ? record.website_url : '';1336 if (!websiteUrl && resolveWebsites) {1337 const resolution = await resolveWebsite({1338 productName: String(record.product_name ?? ''),1339 tagline: String(record.tagline ?? ''),1340 productHuntRedirect: typeof record.product_hunt_website_url === 'string' ? record.product_hunt_website_url : '',1341 makerWebsiteUrls: (node.makers ?? [])1342 .map((maker: Record<string, unknown>) => typeof maker.websiteUrl === 'string' ? maker.websiteUrl : '')1343 .filter(Boolean),1344 dataForSeoLogin,1345 dataForSeoPassword,1346 serpApiKey,1347 });1348 diagnostics.websiteResolutionProviderAttempts += resolution.providerAttempts;1349 diagnostics.websiteResolutionProviderCostUsd += resolution.providerCostUsd;1350 resolution.warnings.forEach(addWarning);1351 record.website_resolution_provider_cost_usd = Number(resolution.providerCostUsd.toFixed(6));1352 if (resolution.websiteUrl) {1353 websiteUrl = resolution.websiteUrl;1354 record.website_url = resolution.websiteUrl;1355 record.website_resolution_source = resolution.source;1356 record.website_resolution_status = resolution.status;1357 record.website_resolution_confidence = resolution.confidence;1358 record.website_resolution_signals = resolution.signals;1359 }1360 }1361
1362 if (websiteUrl) {1363 diagnostics.websiteResolved += 1;1364 record.company_domain = companyDomainFromWebsite(websiteUrl);1365 } else {1366 diagnostics.websiteUnresolved += 1;1367 record.website_resolution_status = record.product_hunt_website_url ? 'unresolved' : 'not_available';1368 }1369
1370 let publicEmailCount = 0;1371 let publicVerifications: EmailVerification[] = [];1372 if (enrichEmails && websiteUrl) {1373 const enriched = await findEmails(websiteUrl, maxWebsitePages, maxEmailsPerProduct);1374 record.emails = enriched.emails;1375 record.email_source = enriched.emailSource;1376 record.email_result = enriched.result;1377 record.email_coverage = enriched.coverage;1378 record.email_terminal_reason = enriched.terminalReason;1379 record.email_pages_visited = enriched.pagesVisited;1380 record.email_page_urls = enriched.pageUrls;1381 record.enrichment_warnings = enriched.warnings;1382 publicEmailCount = enriched.emails.length;1383 enriched.warnings.forEach(addWarning);1384 log.info('Website enrichment complete', {1385 product: record.product_name,1386 emailsFound: enriched.emails.length,1387 source: enriched.emailSource,1388 pagesVisited: enriched.pagesVisited,1389 coverage: enriched.coverage,1390 terminalReason: enriched.terminalReason,1391 });1392 }1393
1394 const selectedEmails = Array.isArray(record.emails) ? record.emails.filter((email): email is string => typeof email === 'string') : [];1395 if (selectedEmails.length > 0) {1396 const verification = await verifyEmailsWithEmailListVerify(selectedEmails, verifyEmails, emailListVerifyApiKey);1397 publicVerifications = verification.verifications;1398 diagnostics.emailVerificationAttempts += verification.attempts;1399 diagnostics.emailVerificationProviderErrors += verification.verifications.filter(({ status }) => status === 'provider_error' || status === 'error_credit').length;1400 verification.warnings.forEach(addWarning);1401 log.info('Public email verification complete', {1402 product: record.product_name,1403 emailsAssessed: selectedEmails.length,1404 requested: verifyEmails,1405 attempts: verification.attempts,1406 statuses: verification.verifications.map(({ status }) => status),1407 });1408 }1409
1410 let discoveredContacts: ContactFinderResult[] = [];1411 const deliverablePublicCount = publicVerifications.filter(({ status }) => status === 'ok').length;1412 if (enrichEmails && findContacts && websiteUrl && deliverablePublicCount < maxEmailsPerProduct) {1413 record.contact_discovery_requested = true;1414 const discovery = await discoverContacts(1415 websiteUrl,1416 (node.makers ?? []).map((maker: Record<string, unknown>) => String(maker.name ?? '')).filter(Boolean),1417 Math.max(1, maxEmailsPerProduct - deliverablePublicCount),1418 emailListVerifyApiKey,1419 );1420 discoveredContacts = discovery.contacts;1421 record.contact_discovery_attempts = discovery.attempts;1422 record.contact_discovery_credits = discovery.credits;1423 diagnostics.contactDiscoveryAttempts += discovery.attempts;1424 diagnostics.contactDiscoveryCredits += discovery.credits;1425 diagnostics.contactDiscoveryEmailsFound += discovery.contacts.length;1426 discovery.warnings.forEach(addWarning);1427 }1428
1429 const finalVerifications = selectBestEmails(publicVerifications, discoveredContacts, maxEmailsPerProduct);1430 const rejectedVerifications = publicVerifications.filter(({ status }) => !isUsableEmailStatus(status));1431 const finalEmails = finalVerifications.map(({ email }) => email);1432 const publicSelected = finalVerifications.filter(({ source }) => source === 'public_website').length;1433 const finderSelected = finalVerifications.filter(({ source }) => source === 'emaillistverify_contact_finder').length;1434 record.emails = finalEmails;1435 record.verified_emails = finalVerifications.filter(({ status }) => status === 'ok').map(({ email }) => email);1436 record.risky_emails = finalVerifications.filter(({ status }) => status !== 'ok').map(({ email }) => email);1437 record.rejected_emails = rejectedVerifications.map(({ email }) => email);1438 record.email_verifications = finalVerifications;1439 record.rejected_email_verifications = rejectedVerifications;1440 record.email_source = publicSelected > 0 && finderSelected > 01441 ? 'mixed'1442 : finderSelected > 01443 ? 'emaillistverify_contact_finder'1444 : publicSelected > 01445 ? 'page_scrape'1446 : 'none';1447 record.email_result = finalEmails.length > 0 ? 'emails_found' : record.email_result;1448 if (finderSelected > 0) record.email_terminal_reason = 'contact_finder_enriched';1449 diagnostics.emailsFound += finalEmails.length;1450 diagnostics.verifiedEmailsFound += (record.verified_emails as string[]).length;1451
1452 const lead = buildLeadProfile({1453 websiteResolved: Boolean(websiteUrl),1454 websiteConfidence: Number(record.website_resolution_confidence ?? 0),1455 emailStatuses: finalVerifications.map(({ status }) => status),1456 publicEmailCount,1457 upvotes: Number(record.upvote_count ?? 0),1458 pagesVisited: Number(record.email_pages_visited ?? 0),1459 });1460 record.lead_status = lead.leadStatus;1461 record.lead_score = lead.leadScore;1462 record.lead_tier = lead.leadTier;1463 record.lead_signals = lead.signals;1464 diagnostics.leadTiers[lead.leadTier] += 1;1465
1466 log.info('Lead enrichment complete', {1467 product: record.product_name,1468 websiteResolutionSource: record.website_resolution_source,1469 websiteResolutionConfidence: record.website_resolution_confidence,1470 publicEmailsFound: publicEmailCount,1471 contactFinderEmailsFound: discoveredContacts.length,1472 finalEmails: finalEmails.length,1473 verifiedEmails: (record.verified_emails as string[]).length,1474 leadTier: lead.leadTier,1475 });1476
1477 const outputRecord = applyOutputMode(record, outputMode);1478 if (!outputRecord) {1479 diagnostics.itemsFiltered += 1;1480 return;1481 }1482 const validationError = validateOutputRecord(outputRecord);1483 if (validationError) {1484 diagnostics.invalidRecords += 1;1485 diagnostics.itemsFailed += 1;1486 addWarning(`A Product Hunt record was not persisted: ${validationError}`);1487 return;1488 }1489
1490 try {1491 // release-validator: allow-direct-push complete record validated above; the SDK automatically1492 // couples default-dataset persistence to the synthetic primary event.1493 const primaryResult = await Actor.pushData(outputRecord) as unknown as { chargedCount?: number; eventChargeLimitReached?: boolean };1494 if (primaryResult.eventChargeLimitReached && (primaryResult.chargedCount ?? 0) === 0) {1495 diagnostics.chargeLimitReached = true;1496 addWarning('The event-charge limit was reached before this validated row could be persisted.');1497 return;1498 }1499 diagnostics.itemsPushed += 1;1500 diagnostics.resultEvents += primaryResult.chargedCount ?? 1;1501 diagnostics.chargeLimitReached ||= Boolean(primaryResult.eventChargeLimitReached);1502
1503 const emails = outputRecord.emails as string[];1504 if (emails.length > 0 && !diagnostics.chargeLimitReached) {1505 const emailResult = await Actor.charge({ eventName: EMAIL_EVENT, count: emails.length }) as unknown as { chargedCount?: number; eventChargeLimitReached?: boolean };1506 diagnostics.emailEvents += emailResult.chargedCount ?? emails.length;1507 diagnostics.chargeLimitReached ||= Boolean(emailResult.eventChargeLimitReached);1508 if (emailResult.eventChargeLimitReached) addWarning('The event-charge limit was reached after a persisted record; remaining work stopped at the configured cap.');1509 }1510 } catch (error) {1511 diagnostics.itemsFailed += 1;1512 const message = error instanceof Error ? error.message : 'Unknown dataset-write failure.';1513 addWarning(`Validated dataset write failed: ${message.slice(0, 120)}`);1514 log.warning(`Validated dataset write failed for ${String(record.product_name)}: ${message.slice(0, 120)}`);1515 }1516 };1517
1518 const processNodes = async (nodes: Array<Record<string, any>>): Promise<void> => {1519 const uniqueNodes: Array<Record<string, any>> = [];1520 for (const node of nodes) {1521 if (collectedCandidates >= effectiveMaxResults || diagnostics.chargeLimitReached) break;1522 const id = String(node.id ?? '');1523 if (!id || seenIds.has(id)) continue;1524 seenIds.add(id);1525 uniqueNodes.push(node);1526 collectedCandidates += 1;1527 }1528 for (let index = 0; index < uniqueNodes.length; index += maxConcurrency) {1529 if (diagnostics.chargeLimitReached) break;1530 const batch = uniqueNodes.slice(index, index + maxConcurrency);1531 await Promise.all(batch.map((node, batchIndex) => pushNode(node, diagnostics.itemsProcessed + batchIndex + 1)));1532 diagnostics.itemsProcessed += batch.length;1533 }1534 };1535
1536 const streamPosts = async (range: DateRange | null, label: string): Promise<void> => {1537 let cursor: string | null = null;1538 let pageNumber = 1;1539 while (collectedCandidates < effectiveMaxResults && !diagnostics.chargeLimitReached) {1540 const remaining = effectiveMaxResults - collectedCandidates;1541 const variables: Record<string, unknown> = { first: Math.min(20, remaining), after: cursor, order: 'VOTES' };1542 if (!includeAllProducts) variables.featured = true;1543 if (range) {1544 variables.postedAfter = range.postedAfter;1545 variables.postedBefore = range.postedBefore;1546 }1547 if (mode === 'topic') variables.topic = topic;1548 try {1549 const data: any = await phQuery(POSTS_QUERY, variables);1550 const edges = data?.posts?.edges ?? [];1551 const pageInfo: { hasNextPage?: boolean; endCursor?: string | null } = data?.posts?.pageInfo ?? {};1552 if (!edges.length) break;1553 log.info(`Product Hunt ${label} page ${pageNumber}`, { received: edges.length, processed: diagnostics.itemsProcessed });1554 const nodes = edges.map((edge: { node?: Record<string, any> }) => edge.node).filter(Boolean);1555 const selectedNodes = mode === 'search'1556 ? nodes.filter((node: Record<string, any>) => matchesProductSearch(searchQuery, {1557 name: node.name,1558 tagline: node.tagline,1559 description: node.description,1560 topics: (node.topics?.edges ?? []).map((edge: any) => edge?.node?.name).filter(Boolean),1561 }))1562 : nodes;1563 await processNodes(selectedNodes);1564 if (!pageInfo.hasNextPage || !pageInfo.endCursor) break;1565 cursor = pageInfo.endCursor;1566 pageNumber += 1;1567 } catch (error) {1568 diagnostics.apiFailures += 1;1569 const message = error instanceof Error ? error.message : 'Unknown Product Hunt API error.';1570 addWarning(`Product Hunt ${label} query failed: ${message}`);1571 break;1572 }1573 }1574 };1575
1576 if (mode === 'urls') {1577 const rawSeeds = startUrls1578 .map((entry) => typeof entry === 'string' ? entry : typeof entry?.url === 'string' ? entry.url : '')1579 .filter(Boolean);1580 const seeds = rawSeeds.map(parseProductHuntSeedUrl);1581 const validSeeds = seeds.filter((seed) => seed.kind !== 'invalid');1582 seeds.filter((seed) => seed.kind === 'invalid').forEach((seed) => addWarning(`Ignored start URL: ${seed.reason}`));1583 if (validSeeds.length === 0) await finalizeRun('INVALID_INPUT', 'No valid Product Hunt post, product launch, or dated leaderboard URLs were supplied.');1584
1585 for (const seed of validSeeds) {1586 if (collectedCandidates >= effectiveMaxResults || diagnostics.chargeLimitReached) break;1587 if (seed.kind === 'post') {1588 try {1589 const data = await phQuery(POST_BY_SLUG_QUERY, { slug: seed.slug });1590 if (!data?.post) {1591 addWarning(`No Product Hunt post matched the supplied URL slug: ${seed.slug}.`);1592 continue;1593 }1594 await processNodes([data.post]);1595 } catch (error) {1596 diagnostics.apiFailures += 1;1597 const message = error instanceof Error ? error.message : 'Unknown Product Hunt API error.';1598 addWarning(`Product Hunt URL lookup failed for ${seed.slug}: ${message}`);1599 }1600 } else if (seed.kind === 'leaderboard') {1601 await streamPosts(dateRangeForPeriod(seed.period, seed.anchorDate), `${seed.period} leaderboard URL`);1602 }1603 }1604 } else {1605 let range: DateRange | null = null;1606 if (mode === 'leaderboard') {1607 if (startDate) {1608 range = dateRangeForPeriod(leaderboardPeriod, startDate, endDate);1609 } else if (lookbackDays > 1) {1610 const end = new Date();1611 const begin = new Date(end);1612 begin.setUTCDate(end.getUTCDate() - lookbackDays + 1);1613 range = { postedAfter: `${isoDate(begin)}T00:00:00+00:00`, postedBefore: `${isoDate(end)}T23:59:59+00:00` };1614 } else {1615 range = dateRangeForPeriod(leaderboardPeriod, isoDate(new Date()));1616 }1617 }1618 await streamPosts(range, mode);1619 }1620
1621 await closeBrowser();1622 const hasUsefulRows = diagnostics.itemsPushed > 0;1623 const outcome: RunOutcome = !hasUsefulRows1624 ? diagnostics.apiFailures > 0 || diagnostics.itemsFailed > 0 || diagnostics.invalidRecords > 01625 ? 'UPSTREAM_FAILED'1626 : 'VALID_EMPTY'1627 : diagnostics.apiFailures > 0 || diagnostics.itemsFailed > 0 || (resolveWebsites && diagnostics.websiteUnresolved > 0) || diagnostics.chargeLimitReached1628 ? 'PARTIAL'1629 : 'COMPLETE';1630 const eventCost = diagnostics.resultEvents * RESULT_PRICE_USD + diagnostics.emailEvents * EMAIL_PRICE_USD;1631 await finalizeRun(outcome, `${outcome}: processed ${diagnostics.itemsProcessed} Product Hunt products, persisted ${diagnostics.itemsPushed} rows, and found ${diagnostics.emailsFound} public email(s). Event charges: $${eventCost.toFixed(3)} plus applicable platform usage.`);1632 } catch (error) {1633 await closeBrowser();1634 if (finalised) return;1635 const message = error instanceof Error ? error.message : 'Unknown actor failure.';1636 diagnostics.itemsFailed += 1;1637 addWarning(`Unhandled actor error: ${message.slice(0, 160)}`);1638 await finalizeRun('UPSTREAM_FAILED', `The run stopped before completion: ${message.slice(0, 160)}`);1639 }1640}1641
1642await run();