Leo Nguyen

I build data extraction that survives contact with production — Go and Python, no browsers where an HTTP client will do.

What I optimise for is not getting the data once. It is still getting it next week:

• Rate limits are the real adversary. I measure them before writing the scraper. For my Google Trends Actor that meant finding two separate limits — a short burst ceiling and a per-IP budget that accumulates over hours — and designing around the second one.
• Cache what does not change. Most sources serve data at a coarser resolution than people poll it. Serving a cached answer means a request never spent against the limit, which makes runs both cheaper and far less likely to fail.
• Rotate, do not wait. On a billed platform, sleeping through a 429 is paid idle time. Switching IP is faster and cheaper.
• State the limits. Every Actor I publish documents what it cannot do, and returns a failed item as data instead of killing the whole run. A 100-input job should give you 99 rows, not an error.

Open source: lotusmarket  — Vietnamese stock market toolkit for Go and Python (MIT, on PyPI).

Actor source is public on GitHub, so you can read exactly what it does before you run it.