Startup Deals & Codes Tracker
Pricing
Pay per usage
Aggregates publicly-announced startup credits, free-tier programs, and SaaS discounts from official 'for startups' pages. Outputs a clean, validated dataset of deal records — not a coupon scraper.
Pricing
Pay per usage
Rating
0.0
(0)
Developer
Bob X
Actor stats
0
Bookmarked
2
Total users
1
Monthly active users
5 days ago
Last modified
Categories
Share
# IMPORTANT: leading slash anchors patterns to the build-context root.# Without it, `storage` matches our Python package `src/storage/` too,# causing `ModuleNotFoundError: No module named 'src.storage'` at runtime.__pycache__*.pyc.venvvenv.env/storage/apify_storage/.apify.git.gitignore.idea.vscode*.md.bak# Template for required local secrets. Copy to `.env` and fill in real values.# `.env` is gitignored; `.env.example` is committed so collaborators know what's needed.
# Anthropic Claude API key — required by the actor's LLM extractor.# Get one at: https://console.anthropic.com/settings/keysANTHROPIC_API_KEY=
# Optional: pin a specific Haiku model build.# ANTHROPIC_MODEL=claude-haiku-4-5-20251001# IMPORTANT: leading slash anchors patterns to the repo root.# Without it, `storage/` matches our Python package `src/storage/` too,# which `apify push` then strips from the tarball, causing# `ModuleNotFoundError: No module named 'src.storage'` in the deployed image.__pycache__/*.pyc*.pyo.venv/venv/.env/storage//apify_storage//.apify/*.log.DS_Store.idea/.vscode/1# Development2
3This file documents how to build, push, and operate the actor.4For the user-facing description, see `README.md`.5
6## Project layout7
8```9.actor/10 actor.json Apify manifest (image, memory, categories)11 input_schema.json Run input schema (form fields)12 pay_per_event.json PPE pricing reference (canonical pricing lives in Apify Console)13src/14 __init__.py15 __main__.py Entrypoint: `python -m src`16 main.py Orchestrator (fetch → cache → extract → filter → verify → push)17 models.py Pydantic v2 Deal schema (strict, extra="forbid")18 fetcher.py httpx (fast HTTP) + Crawl4AI Playwright (browser sources) + BM2519 verifier.py Re-fetch + dead/sold-out/waitlist regex; updates verification_status only20 extractors/21 llm_extractor.py Anthropic tool-use, model = claude-haiku-4-5-2025100122 sources/23 registry.py Declarative list of all 18 sources (slug, url, requires_browser, ...)24 storage/25 cache.py Apify KV-store cache, 7-day TTL, hash-based skip-LLM26Dockerfile FROM apify/actor-python-playwright:3.1227requirements.txt Pinned to Apify SDK v3 + Crawlee v1.6 (see note below)28.env.example Template for required local secrets29```30
31## Prerequisites32
33- Node 18+ and npm34- Python 3.10+ (the Apify cloud image is 3.12; local is only used for syntax checks)35- Apify CLI: `npm install -g apify-cli`36- Anthropic API key with access to Claude Haiku37
38## Local secrets39
40The actor needs `ANTHROPIC_API_KEY` at runtime.41
42- For Apify cloud runs: set it as a Console **Secret** and bind it as an env var on the actor (Source tab → Environment variables → Add → Secret).43- For local runs: copy `.env.example` to `.env` and fill in the value. `.env` is gitignored. `apify run` does not auto-source `.env`; export it first or use `env $(cat .env | xargs) apify run`.44
45## Build & deploy46
47```bash48# from the repo root49apify login # one-time browser auth50apify push --force # builds the Docker image on Apify and updates the actor51```52
53`apify push` produces a new build version (e.g. `0.1.7`). The Apify Console will show it under the Builds tab.54
55## Smoke test (recommended after every push)56
57Use these inputs the first time, then expand:58
59```json60{61 "max_deals": 30,62 "verify_freshness": false,63 "force_refresh": false,64 "min_confidence": 0.7,65 "categories": [],66 "eligibility_filter": []67}68```69
70Memory: **4 GB** (Playwright sources will OOM on lower).71Build: pick the latest version from the dropdown.72
73Healthy log signals, in order:741. `Run config: max_deals=30 ...` — env vars and input were read successfully.752. `Processing source <slug>` lines, one per source.763. `LLM extracted N deals from <slug>` (or `Cache hit ... skipping LLM`).774. `After filters: X / Y deals`.785. `Run complete. Pushed=X total_extracted=Y`.79
80Source-level failures are expected and non-fatal — the orchestrator wraps each source in try/except.81
82## Pay-per-event pricing83
84Two events are defined in `.actor/pay_per_event.json`:85
86- `deal-record` — $0.005 per validated deal pushed to the dataset.87- `verification-check` — $0.001 per re-verified source URL when `verify_freshness=true`.88
89The file is a reference; the canonical pricing must be configured in the Apify Console under the actor's Publication tab.90
91## Dependency notes92
93The dependency stack is intentionally pinned to **Apify SDK v3 + Crawlee v1.6**. Earlier versions (Apify SDK v2 / Crawlee 0.x) carry compatibility shims for `browserforge` and Pydantic 2.12 that have rotted against current PyPI releases — runs fail at import time with either:94
95- `TypeError: cannot specify both default and default_factory` (Pydantic 2.12 vs Crawlee 0.x), or96- `AttributeError: module 'browserforge.download' has no attribute 'DATA_FILES'` (browserforge API rename vs Crawlee 0.x workaround).97
98If a future build fails with similar import errors, check upstream changelogs before downgrading.99
100## .dockerignore caveat101
102Patterns in `.dockerignore` without a leading slash match by basename anywhere in the tree. The line `storage` would silently exclude `src/storage/`, breaking `from src.storage.cache import ...`. Always anchor top-level paths with `/`:103
104```105/storage # apify local-emulation only106/apify_storage # apify local-emulation only107/.apify # CLI cache108```109
110## Validation before push111
112```bash113# Python syntax114find src -name '*.py' -exec python3 -m py_compile {} \;115
116# JSON manifests117for f in .actor/actor.json .actor/input_schema.json .actor/pay_per_event.json; do118 python3 -m json.tool "$f" > /dev/null && echo "$f OK"119done120```# Apify base image with Python 3.12 + Playwright + Chromium pre-installed.FROM apify/actor-python-playwright:3.12
# Install Python deps first so Docker can cache the layer.COPY requirements.txt ./RUN echo "Python version:" \ && python --version \ && echo "Pip version:" \ && pip --version \ && echo "Installing dependencies:" \ && pip install --no-cache-dir -r requirements.txt \ && echo "Installed Python packages:" \ && pip freeze
# Copy the rest of the actor source.COPY . ./
CMD ["python", "-m", "src"]1# Apify SDK v3 + Crawlee v1.6 — required because the older 0.x Crawlee line2# carries fragile compatibility shims for `browserforge` and Pydantic 2.12 that3# break against current PyPI releases. v3/v1.6 has the upstream fixes baked in.4apify>=3.3,<4.05crawlee[playwright]>=1.6,<2.06crawl4ai>=0.4,<1.07pydantic>=2.11,<3.08anthropic>=0.40,<1.09httpx>=0.27,<1.010beautifulsoup4>=4.12,<5.011python-dateutil>=2.9,<3.0{ "actorSpecification": 1, "name": "startup-deals-tracker", "title": "Startup Deals Tracker", "description": "Aggregates publicly-announced startup credits, free-tier programs, and SaaS discounts from official 'for startups' pages. Outputs a clean, validated dataset of deal records — not a coupon scraper.", "version": "0.1", "buildTag": "latest", "minMemoryMbytes": 1024, "maxMemoryMbytes": 4096, "input": "./input_schema.json", "dockerfile": "../Dockerfile", "readme": "../README.md", "categories": [ "DEVELOPER_TOOLS", "AI", "OTHER" ]}{ "title": "Startup Deals Tracker input", "type": "object", "schemaVersion": 1, "description": "Filters and run options. Leave filters empty to return everything above min_confidence.", "properties": { "categories": { "title": "Categories filter", "type": "array", "description": "Optional category whitelist. Empty = no category filter.", "editor": "select", "items": { "type": "string", "enum": [ "ai_tools", "hosting", "database", "devops", "design", "productivity", "communication", "analytics", "cloud", "payments", "email", "security", "auth", "video" ] }, "default": [] }, "eligibility_filter": { "title": "Eligibility filter", "type": "array", "description": "Only return deals matching at least one of these eligibility tags. Empty = no eligibility filter.", "editor": "select", "items": { "type": "string", "enum": [ "open", "startup_pre_seed", "startup_seed", "startup_series_a", "funding_cap", "accelerator_required", "vc_partner_required", "student", "oss_maintainer", "new_customer_only", "nonprofit", "edu", "paid_newsletter_sub" ] }, "default": [] }, "verify_freshness": { "title": "Verify freshness (re-check live status)", "type": "boolean", "description": "Re-fetch each source URL to verify the deal is still live. Charges one verification-check event per deal verified.", "default": false }, "min_confidence": { "title": "Minimum confidence score", "type": "number", "description": "Discard deals below this LLM-reported confidence (0-1).", "minimum": 0, "maximum": 1, "default": 0.7 }, "force_refresh": { "title": "Force refresh (bypass 7-day cache)", "type": "boolean", "description": "Bypass the 7-day source content cache and re-extract everything via LLM.", "default": false }, "max_deals": { "title": "Max deals returned", "type": "integer", "description": "Hard cap on the total number of deals returned (and charged for).", "minimum": 1, "default": 500 } }}{ "actorSpecification": 1, "events": { "deal-record": { "eventTitle": "Validated deal record", "eventDescription": "One validated startup deal pushed to the output dataset (passed Pydantic validation and filters).", "eventPriceUsd": 0.005 }, "verification-check": { "eventTitle": "Source URL verification check", "eventDescription": "Re-fetched a source URL and updated the verification_status of an existing deal without re-extraction.", "eventPriceUsd": 0.001 } }}1"""Startup Deals Tracker — Apify Actor."""1"""Entrypoint: `python -m src` runs the Apify Actor."""2import asyncio3
4from src.main import main5
6
7if __name__ == "__main__":8 asyncio.run(main())1"""Source fetcher: routes between fast HTTP (httpx) and headless browser2(Crawl4AI + Playwright) based on the source's `requires_browser` flag.3
4Both paths run their output through Crawl4AI's BM25 content filter so we5strip nav/footer noise and keep deal-related text only — this keeps the6LLM input small and improves extraction quality.7"""8from __future__ import annotations9
10import hashlib11import logging12from dataclasses import dataclass13from typing import Optional14
15import httpx16
17logger = logging.getLogger(__name__)18
19# BM25 query words tuned for "for startups" pages and aggregator listings.20BM25_QUERY = (21 "startup credits free trial discount perk program for startups eligibility "22 "apply application promo code annual months value USD funding cap accelerator "23 "partner waitlist sold out applications closed students nonprofit OSS "24 "free tier credits hosting cloud database AI tools"25)26
27DEFAULT_USER_AGENT = (28 "Mozilla/5.0 (compatible; StartupDealsTrackerBot/1.0; +https://apify.com)"29)30
31
32@dataclass33class FetchResult:34 url: str35 markdown: str36 content_hash: str37 fetched_via: str # "http" | "browser"38 ok: bool39 error: Optional[str] = None40
41
42def _hash(s: str) -> str:43 return hashlib.sha256(s.encode("utf-8")).hexdigest()44
45
46def _bm25_filter_html(html: str) -> str:47 """Run Crawl4AI's BM25 + DefaultMarkdownGenerator on raw HTML and return markdown.48
49 Falls back to BeautifulSoup-extracted text on any error so we never crash50 a single source's pipeline because of a Crawl4AI version drift.51 """52 try:53 from crawl4ai.content_filter_strategy import BM25ContentFilter54 from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator55
56 bm25 = BM25ContentFilter(user_query=BM25_QUERY, bm25_threshold=1.0)57 md_gen = DefaultMarkdownGenerator(content_filter=bm25)58
59 # Crawl4AI's API has shifted across versions; try both signatures.60 md_obj = None61 try:62 md_obj = md_gen.generate_markdown(cleaned_html=html, citations=False) # type: ignore[arg-type]63 except TypeError:64 try:65 md_obj = md_gen.generate_markdown(html=html) # type: ignore[arg-type]66 except Exception:67 md_obj = None68
69 if md_obj is not None:70 md = (71 getattr(md_obj, "fit_markdown", None)72 or getattr(md_obj, "raw_markdown", None)73 or str(md_obj)74 )75 if md:76 return md77 except Exception:78 logger.debug("BM25 filter unavailable, falling back to plaintext", exc_info=True)79
80 # Fallback: BeautifulSoup plaintext.81 try:82 from bs4 import BeautifulSoup83
84 soup = BeautifulSoup(html, "html.parser")85 for tag in soup(["script", "style", "noscript"]):86 tag.decompose()87 return soup.get_text("\n", strip=True)88 except Exception:89 return html # last resort90
91
92async def _fetch_http(url: str) -> FetchResult:93 """Fast path: httpx GET + BM25 markdown extraction."""94 try:95 async with httpx.AsyncClient(96 follow_redirects=True,97 timeout=30.0,98 headers={"User-Agent": DEFAULT_USER_AGENT, "Accept": "text/html,*/*"},99 ) as client:100 r = await client.get(url)101 r.raise_for_status()102 html = r.text103 except Exception as e:104 logger.warning("HTTP fetch failed for %s: %s", url, e)105 return FetchResult(url=url, markdown="", content_hash="", fetched_via="http",106 ok=False, error=str(e))107
108 md = _bm25_filter_html(html)109 return FetchResult(url=url, markdown=md, content_hash=_hash(md), fetched_via="http", ok=True)110
111
112async def _fetch_browser(url: str) -> FetchResult:113 """Browser path: Crawl4AI's AsyncWebCrawler with Playwright + BM25."""114 try:115 from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig116 from crawl4ai.content_filter_strategy import BM25ContentFilter117 from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator118
119 bm25 = BM25ContentFilter(user_query=BM25_QUERY, bm25_threshold=1.0)120 md_gen = DefaultMarkdownGenerator(content_filter=bm25)121
122 bcfg = BrowserConfig(headless=True, verbose=False)123 rcfg = CrawlerRunConfig(124 markdown_generator=md_gen,125 page_timeout=60_000,126 wait_until="domcontentloaded",127 )128
129 async with AsyncWebCrawler(config=bcfg) as crawler:130 result = await crawler.arun(url=url, config=rcfg)131
132 if not getattr(result, "success", False):133 err = str(getattr(result, "error_message", "unknown error"))134 return FetchResult(url=url, markdown="", content_hash="", fetched_via="browser",135 ok=False, error=err)136
137 md_obj = result.markdown138 md = (139 getattr(md_obj, "fit_markdown", None)140 or getattr(md_obj, "raw_markdown", None)141 or str(md_obj)142 )143 return FetchResult(url=url, markdown=md, content_hash=_hash(md),144 fetched_via="browser", ok=True)145 except Exception as e:146 logger.exception("Browser fetch failed for %s", url)147 return FetchResult(url=url, markdown="", content_hash="", fetched_via="browser",148 ok=False, error=str(e))149
150
151async def fetch_source(url: str, requires_browser: bool) -> FetchResult:152 """Top-level fetcher. Routes based on `requires_browser`."""153 if requires_browser:154 return await _fetch_browser(url)155 return await _fetch_http(url)1"""Orchestrator for the Startup Deals Tracker actor.2
3Pipeline per run:4 1. read input5 2. for each source: fetch -> check 7-day cache -> LLM extract if needed -> save cache6 3. apply (categories, eligibility, min_confidence) filters7 4. optionally verify freshness (charges per check)8 5. cap at max_deals, push to dataset, charge per deal9"""10from __future__ import annotations11
12import logging13import os14from datetime import datetime, timezone15from typing import Any, Dict, List16
17from apify import Actor18
19from src.extractors.llm_extractor import LLMExtractor20from src.fetcher import fetch_source21from src.models import Deal22from src.sources.registry import SOURCES, SourceSpec, get_source_by_url23from src.storage.cache import CachedSource, load_cached, save_cached24from src.verifier import verify_deal25
26logger = logging.getLogger(__name__)27
28CHARGE_DEAL = "deal-record"29CHARGE_VERIFY = "verification-check"30
31
32async def _safe_charge(event_name: str, count: int = 1) -> None:33 """Charge an event but never crash the run on charge failure."""34 try:35 await Actor.charge(event_name=event_name, count=count)36 except Exception as e: # noqa: BLE001 — by design37 logger.warning("Actor.charge(%s, %d) failed: %s", event_name, count, e)38
39
40def _matches_filters(41 d: Deal,42 *,43 categories_filter: List[str],44 eligibility_filter: List[str],45 min_confidence: float,46) -> bool:47 if d.confidence_score < min_confidence:48 return False49 if categories_filter:50 deal_cats = set(d.category or [])51 if not deal_cats.intersection(categories_filter):52 return False53 if eligibility_filter:54 deal_elig = set(d.eligibility_tags or [])55 if not deal_elig.intersection(eligibility_filter):56 return False57 return True58
59
60async def _process_source(61 source: SourceSpec,62 *,63 extractor: LLMExtractor,64 kv_store: Any,65 force_refresh: bool,66) -> List[Deal]:67 Actor.log.info("Processing source %s (%s)", source.slug, source.url)68
69 cached = None if force_refresh else await load_cached(kv_store, source.slug)70
71 fetch_res = await fetch_source(source.url, requires_browser=source.requires_browser)72 if not fetch_res.ok:73 Actor.log.warning("Fetch failed for %s: %s", source.url, fetch_res.error)74 if cached and cached.deals:75 Actor.log.info("Using stale cache for %s after fetch failure", source.slug)76 return _revalidate_cached(cached.deals)77 return []78
79 if (80 cached81 and cached.content_hash == fetch_res.content_hash82 and cached.is_fresh()83 ):84 Actor.log.info("Cache hit for %s — skipping LLM", source.slug)85 revalidated = _revalidate_cached(cached.deals)86 if revalidated:87 return revalidated88 Actor.log.info("Cached deals failed re-validation; re-extracting %s", source.slug)89
90 deals = extractor.extract(91 source_slug=source.slug,92 source_url=source.url,93 source_type=source.source_type,94 markdown=fetch_res.markdown,95 )96 Actor.log.info("LLM extracted %d deals from %s", len(deals), source.slug)97
98 payload = CachedSource(99 content_hash=fetch_res.content_hash,100 extracted_at=datetime.now(timezone.utc).isoformat(),101 deals=[d.model_dump(mode="json") for d in deals],102 fetched_via=fetch_res.fetched_via,103 )104 await save_cached(kv_store, source.slug, payload)105 return deals106
107
108def _revalidate_cached(deals_raw: List[Dict[str, Any]]) -> List[Deal]:109 out: List[Deal] = []110 for raw in deals_raw:111 try:112 out.append(Deal.model_validate(raw))113 except Exception:114 logger.debug("Cached deal failed re-validation; dropping", exc_info=True)115 return out116
117
118async def main() -> None:119 async with Actor:120 actor_input: Dict[str, Any] = await Actor.get_input() or {}121
122 categories_filter: List[str] = list(actor_input.get("categories") or [])123 eligibility_filter: List[str] = list(actor_input.get("eligibility_filter") or [])124 verify_freshness: bool = bool(actor_input.get("verify_freshness", False))125 min_confidence: float = float(actor_input.get("min_confidence", 0.7))126 force_refresh: bool = bool(actor_input.get("force_refresh", False))127 max_deals: int = int(actor_input.get("max_deals", 500))128
129 Actor.log.info(130 "Run config: max_deals=%d min_confidence=%.2f verify=%s force_refresh=%s "131 "categories=%s eligibility=%s",132 max_deals, min_confidence, verify_freshness, force_refresh,133 categories_filter, eligibility_filter,134 )135
136 anthropic_api_key = os.environ.get("ANTHROPIC_API_KEY")137 if not anthropic_api_key:138 await Actor.fail(139 status_message="ANTHROPIC_API_KEY env var is required (set as a secret in Apify Console).",140 )141 return142
143 extractor = LLMExtractor(api_key=anthropic_api_key)144 kv_store = await Actor.open_key_value_store()145
146 # --- Phase 1: fetch + extract per source (resilient: one failure ≠ fatal) ---147 all_deals: List[Deal] = []148 for source in SOURCES:149 try:150 deals = await _process_source(151 source,152 extractor=extractor,153 kv_store=kv_store,154 force_refresh=force_refresh,155 )156 Actor.log.info("Source %s contributed %d deals", source.slug, len(deals))157 all_deals.extend(deals)158 except Exception: # noqa: BLE001159 Actor.log.exception("Source %s crashed; continuing with the rest", source.slug)160
161 # --- Phase 2: filter ---162 filtered = [163 d for d in all_deals164 if _matches_filters(165 d,166 categories_filter=categories_filter,167 eligibility_filter=eligibility_filter,168 min_confidence=min_confidence,169 )170 ]171 Actor.log.info("After filters: %d / %d deals", len(filtered), len(all_deals))172
173 # --- Phase 3: optional freshness verification ---174 if verify_freshness:175 verified: List[Deal] = []176 for d in filtered:177 src_match = get_source_by_url(str(d.source_urls[0])) if d.source_urls else None178 requires_browser = bool(src_match.requires_browser) if src_match else False179 try:180 d2, did_check = await verify_deal(d, requires_browser=requires_browser)181 verified.append(d2)182 if did_check:183 await _safe_charge(CHARGE_VERIFY)184 except Exception: # noqa: BLE001185 Actor.log.exception("Verification failed for deal %s", d.id)186 verified.append(d)187 filtered = verified188
189 # --- Phase 4: cap + push + charge per deal ---190 capped = filtered[:max_deals]191 Actor.log.info("Pushing %d deals to dataset", len(capped))192 for d in capped:193 try:194 await Actor.push_data(d.model_dump(mode="json"))195 except Exception: # noqa: BLE001196 Actor.log.exception("Failed to push deal %s; skipping charge", d.id)197 continue198 await _safe_charge(CHARGE_DEAL)199
200 Actor.log.info("Run complete. Pushed=%d total_extracted=%d", len(capped), len(all_deals))201
202
203if __name__ == "__main__":204 import asyncio205
206 asyncio.run(main())1"""Pydantic v2 Deal schema with strict validation.2
3Provenance fields (`id`, `source_urls`, `source_type`, `first_seen_at`,4`last_verified_at`, `verification_status`) are injected by the orchestrator5and never produced by the LLM.6"""7from __future__ import annotations8
9import re10from datetime import date, datetime11from enum import Enum12from typing import List, Literal, Optional13
14from pydantic import BaseModel, ConfigDict, Field, HttpUrl, field_validator15
16
17SLUG_RE = re.compile(r"^[a-z0-9][a-z0-9\-]*[a-z0-9]$")18
19
20class DealType(str, Enum):21 credits = "credits"22 free_period = "free_period"23 percent_off = "percent_off"24 discounted_plan = "discounted_plan"25 lifetime_deal = "lifetime_deal"26 free_tier_upgrade = "free_tier_upgrade"27 waived_fees = "waived_fees"28 bundle = "bundle"29
30
31class EligibilityTag(str, Enum):32 open = "open"33 startup_pre_seed = "startup_pre_seed"34 startup_seed = "startup_seed"35 startup_series_a = "startup_series_a"36 funding_cap = "funding_cap"37 accelerator_required = "accelerator_required"38 vc_partner_required = "vc_partner_required"39 student = "student"40 oss_maintainer = "oss_maintainer"41 new_customer_only = "new_customer_only"42 nonprofit = "nonprofit"43 edu = "edu"44 paid_newsletter_sub = "paid_newsletter_sub"45
46
47class SourceType(str, Enum):48 official = "official"49 aggregator = "aggregator"50 newsletter = "newsletter"51 community = "community"52
53
54class VerificationStatus(str, Enum):55 live = "live"56 likely_live = "likely_live"57 sold_out = "sold_out"58 expired = "expired"59 unverified = "unverified"60 dead_link = "dead_link"61
62
63class Category(str, Enum):64 ai_tools = "ai_tools"65 hosting = "hosting"66 database = "database"67 devops = "devops"68 design = "design"69 productivity = "productivity"70 communication = "communication"71 analytics = "analytics"72 cloud = "cloud"73 payments = "payments"74 email = "email"75 security = "security"76 auth = "auth"77 video = "video"78
79
80VerificationMethod = Literal["page_present", "code_redemption_check", "human", "none"]81
82
83class Deal(BaseModel):84 """Strict, validated representation of a single startup deal.85
86 `model_config.extra = "forbid"` ensures the LLM cannot smuggle unknown87 fields into the dataset, and `use_enum_values=True` keeps JSON output88 flat-string-friendly for downstream consumers.89 """90
91 model_config = ConfigDict(use_enum_values=True, extra="forbid")92
93 # --- Identity ---94 id: str = Field(..., min_length=3, max_length=80, description="Stable slug primary key")95 provider: str = Field(..., min_length=1)96 provider_url: HttpUrl97 program_name: str = Field(..., min_length=1)98
99 # --- Substance ---100 deal_type: DealType101 headline: str = Field(..., min_length=1, max_length=200)102 description: Optional[str] = Field(default=None, max_length=2000)103
104 # --- Quantification (all optional, all >= 0) ---105 credit_value_usd: Optional[float] = Field(default=None, ge=0)106 discount_percent: Optional[float] = Field(default=None, ge=0, le=100)107 free_period_months: Optional[int] = Field(default=None, ge=0)108 duration_months: Optional[int] = Field(default=None, ge=0)109 estimated_total_value_usd: Optional[float] = Field(default=None, ge=0)110
111 # --- Eligibility ---112 eligibility_tags: List[EligibilityTag] = Field(default_factory=list)113 funding_cap_usd: Optional[float] = Field(default=None, ge=0)114 requires_partner: Optional[str] = Field(default=None, max_length=200)115 geographic_restrictions: List[str] = Field(default_factory=list)116
117 # --- Claim ---118 application_url: Optional[HttpUrl] = None119 promo_code: Optional[str] = Field(120 default=None,121 max_length=80,122 description="PUBLIC promo code only — never personalized codes",123 )124 requires_application: bool = True125 expected_response_time_days: Optional[int] = Field(default=None, ge=0)126
127 # --- Time ---128 starts_at: Optional[date] = None129 expires_at: Optional[date] = None130 is_recurring: bool = False131
132 # --- Provenance (injected, not from LLM) ---133 source_urls: List[HttpUrl] = Field(..., min_length=1)134 source_type: SourceType135 first_seen_at: datetime136 last_verified_at: datetime137 verification_status: VerificationStatus138 verification_method: VerificationMethod139
140 # --- Categorization ---141 confidence_score: float = Field(..., ge=0, le=1)142 extraction_notes: Optional[str] = Field(default=None, max_length=500)143 category: List[Category] = Field(default_factory=list)144 tags: List[str] = Field(default_factory=list)145
146 # --- Validators ---147
148 @field_validator("id")149 @classmethod150 def _validate_id_slug(cls, v: str) -> str:151 if not SLUG_RE.match(v):152 raise ValueError(f"id must match {SLUG_RE.pattern}")153 return v154
155 @field_validator("tags")156 @classmethod157 def _normalize_tags(cls, v: List[str]) -> List[str]:158 seen: set[str] = set()159 out: List[str] = []160 for t in v or []:161 t_norm = t.strip().lower()162 if t_norm and t_norm not in seen:163 seen.add(t_norm)164 out.append(t_norm)165 return out166
167 @field_validator("first_seen_at", "last_verified_at")168 @classmethod169 def _require_tz(cls, v: datetime) -> datetime:170 if v.tzinfo is None or v.tzinfo.utcoffset(v) is None:171 raise ValueError("datetime must be timezone-aware (UTC required)")172 return v1"""Deal verifier — re-fetches a deal's source URL and updates its2`verification_status` and `last_verified_at` WITHOUT re-extracting.3
4Charges one `verification-check` event per deal verified.5"""6from __future__ import annotations7
8import logging9import re10from datetime import datetime, timezone11from typing import Tuple12
13from src.fetcher import fetch_source14from src.models import Deal, VerificationStatus15
16logger = logging.getLogger(__name__)17
18
19_DEAD_PATTERNS = [20 r"applications?\s+(are\s+)?closed",21 r"sold\s+out",22 r"wait\s*list",23 r"no\s+longer\s+accepting",24 r"program\s+(is\s+)?(closed|ended|paused)",25 r"this\s+page\s+(is\s+)?no\s+longer\s+available",26 r"this\s+offer\s+has\s+expired",27]28_DEAD_RE = re.compile("|".join(_DEAD_PATTERNS), re.IGNORECASE)29
30
31def _program_likely_present(program_name: str, markdown: str) -> bool:32 md_lower = markdown.lower()33 pn = program_name.lower()34 if pn and pn in md_lower:35 return True36 # Also accept presence of any 5+ char meaningful token from the program name.37 tokens = [t for t in re.split(r"\W+", pn) if len(t) > 4]38 return any(tok in md_lower for tok in tokens)39
40
41async def verify_deal(deal: Deal, requires_browser: bool) -> Tuple[Deal, bool]:42 """Returns (updated_deal, was_actually_checked).43
44 `was_actually_checked=True` means the verification network call was45 attempted and is eligible to be charged for.46 """47 if not deal.source_urls:48 return deal, False49
50 src_url = str(deal.source_urls[0])51 res = await fetch_source(src_url, requires_browser=requires_browser)52 now = datetime.now(timezone.utc)53
54 if not res.ok:55 return (56 deal.model_copy(update={57 "verification_status": VerificationStatus.dead_link.value,58 "last_verified_at": now,59 }),60 True,61 )62
63 if _DEAD_RE.search(res.markdown or ""):64 new_status = VerificationStatus.sold_out.value65 elif _program_likely_present(deal.program_name, res.markdown or ""):66 new_status = VerificationStatus.live.value67 else:68 new_status = VerificationStatus.unverified.value69
70 return (71 deal.model_copy(update={72 "verification_status": new_status,73 "last_verified_at": now,74 }),75 True,76 )1# Package marker.1"""Anthropic-backed LLM extractor.2
3Forces structured output via Claude tool use (`tool_choice` set to the4`extract_deals` tool). Validates each returned item against the strict5Pydantic `Deal` schema and discards any that fail validation.6
7Provenance fields (`id`, `source_urls`, `source_type`, `first_seen_at`,8`last_verified_at`, `verification_status`) are injected by us — the model9never sees them and never produces them.10"""11from __future__ import annotations12
13import logging14import re15from datetime import datetime, timezone16from typing import Any, Dict, List17
18from anthropic import Anthropic19from pydantic import ValidationError20
21from src.models import Deal, VerificationStatus22
23logger = logging.getLogger(__name__)24
25
26# Per spec; falls back to `claude-haiku-4-5` if the dated suffix is rejected.27DEFAULT_MODEL = "claude-haiku-4-5-20251001"28FALLBACK_MODEL = "claude-haiku-4-5"29
30
31# JSON Schema mirrors the LLM-visible subset of `Deal`. Provenance fields32# are intentionally excluded — we inject them after the LLM responds.33EXTRACT_DEALS_TOOL: Dict[str, Any] = {34 "name": "extract_deals",35 "description": (36 "Return validated startup deals extracted from the provided source markdown. "37 "Return an empty array if the page does not describe a public, currently-live deal."38 ),39 "input_schema": {40 "type": "object",41 "properties": {42 "deals": {43 "type": "array",44 "description": "Array of distinct, public deals on the page.",45 "items": {46 "type": "object",47 "additionalProperties": False,48 "properties": {49 "provider": {"type": "string"},50 "provider_url": {51 "type": "string",52 "description": "Canonical homepage URL of the provider, e.g. https://vercel.com",53 },54 "program_name": {"type": "string"},55 "deal_type": {56 "type": "string",57 "enum": [58 "credits", "free_period", "percent_off",59 "discounted_plan", "lifetime_deal",60 "free_tier_upgrade", "waived_fees", "bundle",61 ],62 },63 "headline": {"type": "string", "maxLength": 200},64 "description": {"type": "string", "maxLength": 2000},65 "credit_value_usd": {"type": "number", "minimum": 0},66 "discount_percent": {"type": "number", "minimum": 0, "maximum": 100},67 "free_period_months": {"type": "integer", "minimum": 0},68 "duration_months": {"type": "integer", "minimum": 0},69 "estimated_total_value_usd": {"type": "number", "minimum": 0},70 "eligibility_tags": {71 "type": "array",72 "items": {73 "type": "string",74 "enum": [75 "open", "startup_pre_seed", "startup_seed",76 "startup_series_a", "funding_cap",77 "accelerator_required", "vc_partner_required",78 "student", "oss_maintainer", "new_customer_only",79 "nonprofit", "edu", "paid_newsletter_sub",80 ],81 },82 },83 "funding_cap_usd": {"type": "number", "minimum": 0},84 "requires_partner": {"type": "string", "maxLength": 200},85 "geographic_restrictions": {86 "type": "array", "items": {"type": "string"},87 },88 "application_url": {"type": "string"},89 "promo_code": {90 "type": "string", "maxLength": 80,91 "description": "PUBLIC promo code only — never personalized codes",92 },93 "requires_application": {"type": "boolean"},94 "expected_response_time_days": {"type": "integer", "minimum": 0},95 "starts_at": {"type": "string", "description": "ISO date YYYY-MM-DD if known"},96 "expires_at": {"type": "string", "description": "ISO date YYYY-MM-DD if known"},97 "is_recurring": {"type": "boolean"},98 "verification_method": {99 "type": "string",100 "enum": ["page_present", "code_redemption_check", "human", "none"],101 },102 "confidence_score": {"type": "number", "minimum": 0, "maximum": 1},103 "extraction_notes": {"type": "string", "maxLength": 500},104 "category": {105 "type": "array",106 "items": {107 "type": "string",108 "enum": [109 "ai_tools", "hosting", "database", "devops", "design",110 "productivity", "communication", "analytics", "cloud",111 "payments", "email", "security", "auth", "video",112 ],113 },114 },115 "tags": {"type": "array", "items": {"type": "string"}},116 },117 "required": [118 "provider", "provider_url", "program_name", "deal_type",119 "headline", "verification_method", "confidence_score",120 ],121 },122 }123 },124 "required": ["deals"],125 },126}127
128
129SYSTEM_PROMPT = (130 "You extract structured startup deal records from public 'for startups' marketing pages "131 "and aggregator listings. Output ONLY publicly-announced deals with public terms — never "132 "personalized promo codes, never deals that require a personal invite or referral.\n\n"133 "If a page describes one big program with one offer, return one deal. If it lists many "134 "sub-deals (e.g. an aggregator), return one deal per concrete provider+program.\n\n"135 "Confidence calibration:\n"136 " - 0.9+ : page explicitly states the dollar value, eligibility, and how to apply\n"137 " - 0.6-0.8 : the program is described but lacks concrete numbers\n"138 " - <0.5 : vague mention only — the host will discard these\n\n"139 "Always set verification_method='page_present' unless you actually verified a redemption "140 "code. NEVER fabricate dollar values or expiry dates. If unsure, omit the field."141)142
143
144_SLUG_NONALNUM = re.compile(r"[^a-z0-9]+")145
146
147def slugify(s: str, max_len: int = 30) -> str:148 s2 = _SLUG_NONALNUM.sub("-", (s or "").lower()).strip("-")149 return (s2 or "x")[:max_len].strip("-") or "x"150
151
152def make_deal_id(source_slug: str, provider: str, program_name: str) -> str:153 raw = f"{slugify(source_slug, 24)}--{slugify(provider, 24)}--{slugify(program_name, 24)}"154 raw = raw[:80].strip("-")155 return raw or "deal"156
157
158class LLMExtractor:159 def __init__(self, api_key: str, model: str = DEFAULT_MODEL):160 self.client = Anthropic(api_key=api_key)161 self.model = model162 self._used_fallback = False163
164 def _call(self, system: str, user_msg: str) -> Any:165 """Call Anthropic with model fallback on model-not-found errors."""166 try:167 return self.client.messages.create(168 model=self.model,169 max_tokens=4096,170 system=system,171 tools=[EXTRACT_DEALS_TOOL],172 tool_choice={"type": "tool", "name": "extract_deals"},173 messages=[{"role": "user", "content": user_msg}],174 )175 except Exception as e:176 msg = str(e).lower()177 if (178 not self._used_fallback179 and ("not found" in msg or "model" in msg and "invalid" in msg)180 ):181 logger.warning(182 "Model %s rejected; falling back to %s",183 self.model, FALLBACK_MODEL,184 )185 self.model = FALLBACK_MODEL186 self._used_fallback = True187 return self.client.messages.create(188 model=self.model,189 max_tokens=4096,190 system=system,191 tools=[EXTRACT_DEALS_TOOL],192 tool_choice={"type": "tool", "name": "extract_deals"},193 messages=[{"role": "user", "content": user_msg}],194 )195 raise196
197 def extract(198 self,199 *,200 source_slug: str,201 source_url: str,202 source_type: str,203 markdown: str,204 ) -> List[Deal]:205 if not markdown or not markdown.strip():206 return []207
208 # Cap content to keep cost predictable. ~60k chars ≈ 15-20k tokens.209 capped = markdown[:60_000]210 user_msg = (211 f"Source URL: {source_url}\n"212 f"Source type: {source_type}\n\n"213 f"=== PAGE MARKDOWN START ===\n{capped}\n=== PAGE MARKDOWN END ===\n\n"214 "Call the extract_deals tool with all distinct, public, currently-live "215 "deals you can identify. Return an empty array if none."216 )217
218 try:219 resp = self._call(SYSTEM_PROMPT, user_msg)220 except Exception:221 logger.exception("Anthropic call failed for %s", source_url)222 return []223
224 deals_raw: List[Dict[str, Any]] = []225 for block in getattr(resp, "content", []):226 if getattr(block, "type", None) == "tool_use" and getattr(block, "name", "") == "extract_deals":227 payload = getattr(block, "input", {}) or {}228 deals_raw = list(payload.get("deals", []))229 break230
231 now = datetime.now(timezone.utc)232 out: List[Deal] = []233 for raw in deals_raw:234 try:235 provider = (raw.get("provider") or "").strip()236 program_name = (raw.get("program_name") or "").strip()237 if not provider or not program_name:238 logger.warning(239 "Discarding deal with missing provider/program from %s",240 source_url,241 )242 continue243
244 # Inject provenance fields the LLM doesn't see.245 raw["id"] = make_deal_id(source_slug, provider, program_name)246 raw["source_urls"] = [source_url]247 raw["source_type"] = source_type248 raw["first_seen_at"] = now249 raw["last_verified_at"] = now250 raw["verification_status"] = VerificationStatus.likely_live.value251
252 deal = Deal.model_validate(raw)253 out.append(deal)254 except ValidationError as ve:255 logger.warning(256 "Discarding invalid deal from %s: %s",257 source_url,258 [259 {"loc": e.get("loc"), "msg": e.get("msg")}260 for e in ve.errors()[:3]261 ],262 )263 except Exception as e:264 logger.warning("Discarding deal from %s due to error: %s", source_url, e)265
266 return out1# Package marker.1"""Declarative list of all 18 sources we crawl.2
3`requires_browser=True` => render with headless Chromium via Crawl4AI.4`requires_browser=False` => fast HTTP fetch via httpx + BM25 markdown extraction.5"""6from __future__ import annotations7
8from dataclasses import dataclass9from typing import List10
11
12@dataclass(frozen=True)13class SourceSpec:14 slug: str15 url: str16 provider: str17 program_name: str18 source_type: str # "official" | "aggregator" | "newsletter" | "community"19 extractor_strategy: str # "llm" for now; future: "json_ld", "regex", ...20 requires_browser: bool21
22
23SOURCES: List[SourceSpec] = [24 # --- Tier 1: official program pages ---25 SourceSpec("vercel-startups", "https://vercel.com/startups/credits",26 "Vercel", "Vercel for Startups", "official", "llm", False),27 SourceSpec("vercel-oss", "https://vercel.com/open-source-program",28 "Vercel", "Vercel OSS Program", "official", "llm", False),29 SourceSpec("render-startups", "https://render.com/startups",30 "Render", "Render Startup Program", "official", "llm", False),31 SourceSpec("notion-startups", "https://www.notion.com/startups",32 "Notion", "Notion for Startups", "official", "llm", False),33 SourceSpec("linear-startups", "https://linear.app/customers/startups",34 "Linear", "Linear for Startups", "official", "llm", False),35 SourceSpec("aws-activate", "https://aws.amazon.com/startups",36 "AWS", "AWS Activate", "official", "llm", True),37 SourceSpec("microsoft-startups", "https://startups.microsoft.com",38 "Microsoft", "Microsoft for Startups Founders Hub", "official", "llm", True),39 SourceSpec("google-startups", "https://cloud.google.com/startup",40 "Google Cloud", "Google for Startups Cloud", "official", "llm", True),41 SourceSpec("nvidia-inception", "https://www.nvidia.com/en-us/startups",42 "NVIDIA", "NVIDIA Inception", "official", "llm", True),43 SourceSpec("anthropic-startups", "https://www.anthropic.com/startups",44 "Anthropic", "Anthropic for Startups", "official", "llm", False),45 SourceSpec("mongodb-startups", "https://www.mongodb.com/startups",46 "MongoDB", "MongoDB for Startups", "official", "llm", False),47 SourceSpec("github-startups", "https://github.com/enterprise/startups",48 "GitHub", "GitHub for Startups", "official", "llm", False),49
50 # --- Tier 2: aggregators ---51 SourceSpec("creditsgull", "https://creditsgull.vercel.app/",52 "CreditsGull", "CreditsGull aggregator", "aggregator", "llm", False),53 SourceSpec("startup-perks", "https://startup-perks.com/",54 "StartupPerks", "StartupPerks aggregator", "aggregator", "llm", False),55 SourceSpec("ai-perks", "https://www.getaiperks.com/",56 "AI Perks", "AI Perks aggregator", "aggregator", "llm", False),57 SourceSpec("appsumo-saas", "https://appsumo.com/browse/?category=software",58 "AppSumo", "AppSumo SaaS deals", "aggregator", "llm", True),59
60 # --- Tier 3: newsletter / community ---61 SourceSpec("lennys-product-pass", "https://www.lennysproductpass.com/",62 "Lenny's Product Pass", "Lenny's Product Pass status", "newsletter", "llm", False),63 SourceSpec("product-hunt-stories", "https://www.producthunt.com/stories",64 "Product Hunt", "Product Hunt Stories", "newsletter", "llm", True),65]66
67
68def get_source_by_slug(slug: str) -> SourceSpec | None:69 for s in SOURCES:70 if s.slug == slug:71 return s72 return None73
74
75def get_source_by_url(url: str) -> SourceSpec | None:76 norm = url.rstrip("/")77 for s in SOURCES:78 if s.url.rstrip("/") == norm:79 return s80 return None1# Package marker.1"""Per-source content-hash cache backed by the Apify Key-Value Store.2
3When the source markdown hash matches the cached hash AND the cache is4< 7 days old, we skip the (expensive) LLM call and reuse the cached deals.5This is the main cost optimization for the actor.6"""7from __future__ import annotations8
9import logging10from dataclasses import asdict, dataclass, field11from datetime import datetime, timedelta, timezone12from typing import Any, Dict, List, Optional13
14logger = logging.getLogger(__name__)15
16
17CACHE_TTL_DAYS = 718CACHE_KEY_PREFIX = "src-"19
20
21@dataclass22class CachedSource:23 content_hash: str24 extracted_at: str # ISO 8601 UTC25 deals: List[Dict[str, Any]] = field(default_factory=list)26 fetched_via: str = ""27
28 def is_fresh(self, ttl_days: int = CACHE_TTL_DAYS) -> bool:29 if not self.extracted_at:30 return False31 try:32 ts = datetime.fromisoformat(self.extracted_at)33 except ValueError:34 return False35 if ts.tzinfo is None:36 ts = ts.replace(tzinfo=timezone.utc)37 return (datetime.now(timezone.utc) - ts) < timedelta(days=ttl_days)38
39
40def _cache_key(source_slug: str) -> str:41 # KV keys must be alnum + dash + underscore.42 safe = "".join(c if c.isalnum() or c in "-_" else "-" for c in source_slug)43 return f"{CACHE_KEY_PREFIX}{safe}"44
45
46async def load_cached(kv_store: Any, source_slug: str) -> Optional[CachedSource]:47 try:48 raw = await kv_store.get_value(_cache_key(source_slug))49 except Exception:50 logger.exception("Cache read failed for %s", source_slug)51 return None52
53 if not raw or not isinstance(raw, dict):54 return None55
56 return CachedSource(57 content_hash=raw.get("content_hash", ""),58 extracted_at=raw.get("extracted_at", ""),59 deals=list(raw.get("deals", [])),60 fetched_via=raw.get("fetched_via", ""),61 )62
63
64async def save_cached(kv_store: Any, source_slug: str, cached: CachedSource) -> None:65 try:66 await kv_store.set_value(_cache_key(source_slug), asdict(cached))67 except Exception:68 logger.exception("Cache write failed for %s", source_slug)