Fixes from the first independent review. These four changed what you pay for or how long a run takes.
- The example sources now run only when every source field is empty. Before, leaving out the Medium or Substack publications field added
netflix-techblog or astralcodexten to your run, and you paid for those rows.
- After a platform restart with "Remove duplicates" off, rows saved before the restart were saved and charged again. They're now skipped and not charged, and the rest of the source is still read.
- The limit of 5,000 requests per run now counts every request. A Substack author with few posts in a big publication stops after 300 posts in a row by other people.
- 3 failed sources in a row now skip only the rest of that service (medium.com, Substack, or one custom domain), not the whole run.
Smaller fixes.
- With "Full text of free posts" off, posts from the sitemap also come without the full text. Search terms keep their commas ("Paris, France" is one search).
- Names that resolve to private addresses are refused. Medium custom domains are read over https only and have to serve Medium's feed.
- HTML, RSS and sitemap parsing take linear time, also on broken markup. A maximum cost below one article ends the run cleanly with 0 rows.
First release.
- Medium authors, publications (also on their own domain) and tags, read from Medium's public RSS feeds. That's the 10 newest stories per feed.
- Substack publications (name, subdomain or custom domain) and authors. The newest posts come from the archive API, older ones through the sitemap.
- Substack site-wide search with cursor paging, up to 40 pages per term.
- Full text of free posts. Paid and member-only posts get a preview and
isPaywalled: true, and you can skip them. Substack rows carry likes, comments and restacks.
- Date filters, duplicate removal across sources, a run report in
OUTPUT, and a stop at the run's maximum cost. Requests to a site slow down after a 429, and on the platform the run switches once to Apify Proxy if a site blocks it.
Pricing is pay per event at $0.002 per saved article.
