RSS feed reader and parser: turn RSS, Atom, RDF and JSON Feed into clean JSON with title, link, author, date, summary, categories and full content when the feed has it. Paste a website and it finds the feed. Keyword and date filters, only-new-items mode. $0.80 per 1,000 items.
Unreleased: new default-mode sample feeds and invalid-input crash fixes (1104)
Default mode sample feeds: the NASA feed (https://www.nasa.gov/feed/) answered the Apify platform with HTTP 403, so {} and the prefill returned no items. The schema default and prefill of feedUrls, the exampleRunInput in .actor/actor.json and DEFAULT_FEEDS are now ["https://www.federalreserve.gov/feeds/press_all.xml", "https://www.w3.org/news/feed/"] (Federal Reserve Board press releases, public domain; W3C News, W3C document license). Both robots.txt files allow the feed paths and neither feed has author names.
Default mode still returns at most 10 items (at most $0.008): each sample feed gets an equal share (5 items each; a smaller maxItemsPerFeed wins) and the run also has a total cap of 10 (a smaller maxTotalItems wins). If one sample feed cannot be read, the other still gives up to 5 items. RUN_SUMMARY keeps defaultFeed (now the first sample feed) and adds defaultFeeds (both). The Default mode: ... log line lists both feeds. A feed list that is exactly the old NASA sample is no longer default mode; it is read as a normal feed.
Crash fixes:postedWithinDays: 1000000 crashed the run (OverflowError), and a non-text item in feedUrls, keywords or excludeKeywords (e.g. [5]) crashed it (AttributeError). They now end SUCCEEDED with the free invalid-input status row (Invalid input, nothing was fetched and no results were charged: <reason>) and no RUN_SUMMARY. Reasons: postedWithinDays must be between 1 and 3650 days, not 1000000. and feedUrls must be a list of text values; item 1 is 5. (same for keywords / excludeKeywords). A negative, NaN or infinite postedWithinDays, or true/false, is now invalid input too (a negative value used to filter out every item); 0 or empty still means no date filter. OverflowError and AttributeError are also caught as a fallback ("a field has the wrong type").
Input schema: postedWithinDays now has "maximum": 3650 and says "(1 to 3650)".
Files: CHANGELOG.md, README.md, .actor/actor.json, .actor/input_schema.json, my_actor/main.py, tests/test_default_mode.py, tests/test_1103_tester.py, tests/test_input_validation_1104.py. Version unchanged. Older entries below still name the NASA feed; they describe what was true then.
Unreleased: invalid-input status row documented and reworded (1104)
Docs: README.md (one FAQ line) and .actor/dataset_schema.json (new top-level description) now say that invalid input (e.g. maxItemsPerFeed: "abc") also ends SUCCEEDED with one free status row (status: "error", message starting Invalid input, nothing was fetched and no results were charged: ), no results charged and no RUN_SUMMARY.
Message wording: the invalid-input status row message and the terminal status message now start Invalid input, nothing was fetched and no results were charged: (was Invalid input, nothing was fetched or charged: ); the reason after the prefix and the error field are unchanged. Said "no results were charged" because Apify's own platform charges (e.g. a start event, if the pricing has one) are outside the Actor's control. Code comments in my_actor/main.py and my_actor/runguard.py, tests/test_1103_tester.py (now checks the full new prefix), the README line and the dataset schema description use the same wording.
tests/run_ppe.sh: local PPE simulation now uses the live item price $0.0008 (was $0.0005); the 5-item cap is now ACTOR_MAX_TOTAL_CHARGE_USD=0.004000 (was 0.002500).
Files: CHANGELOG.md, README.md, .actor/dataset_schema.json, my_actor/runguard.py, my_actor/main.py, tests/test_1103_tester.py, tests/run_ppe.sh. Version unchanged.
Unreleased: private-address guard (GAP17)
On the Apify platform (APIFY_IS_AT_HOME set), every request now checks the host first: if any address it resolves to is private, loopback, link-local, reserved, multicast, unspecified or otherwise not global (also IPv4-mapped IPv6), the request is not sent. This covers feed URLs, every redirect hop (an httpx request hook on the client), robots.txt, feed links found on a page and the common feed paths. Local runs are unchanged.
A blocked feed shows status: error, error: "not fetched: private, loopback and link-local addresses are not allowed" in RUN_SUMMARY and costs nothing. A private feed link found on a page is skipped and the next candidate is tried. When no feed could be read, the existing free status row is written. A host whose DNS lookup fails reads DNS lookup failed for <host>.
No change to prices, events, input, output fields or .actor/ files. Files: CHANGELOG.md, README.md (one FAQ entry), my_actor/common.py, my_actor/main.py, tests/test_ssrf_guard_1102t.py.
Unreleased: changelog file lists and clearer time-limit log (GAP13)
Log text after the hard time limit now says exactly what happens. Items from feeds that finished are kept; a feed still being read when the limit cut it off is dropped, and none of its items are pushed or charged. The old line (kept what was read so far) suggested that part of a cut feed was kept. Behaviour is unchanged: a feed is parsed in one step, so it never had partial items to keep. The warning names the cut feeds (up to 5), and their RUN_SUMMARY error is now stopped: time budget reached while the feed was being read; its items were dropped.
Tests added for the cut-off log text, the RUN_SUMMARY error, and a run where one source finishes and one is cut mid-read: only the finished source's rows are pushed and charged.
GAP13 changed these files: CHANGELOG.md, my_actor/main.py, tests/test_gap13_cutoff.py. No change to .actor/input_schema.json or .actor/actor.json.
GAP12 changed these files: .actor/actor.json, CHANGELOG.md, README.md, my_actor/common.py, my_actor/main.py, my_actor/runguard.py, tests/_e2e.py, tests/test_gap12_body_cap.py, tests/test_gap12_example_input.py, tests/test_gap12_fail_soft_e2e.py, tests/test_gap12_runguard.py, tests/test_redirect_robots_e2e.py. In .actor/actor.json GAP12 only added exampleRunInput (shown in the GAP12 entry below); it did not change the description, the version or any other field. GAP12 made no change to .actor/input_schema.json (defaults, prefill and field texts are as before).
Unreleased: fail soft and a time budget for the Store test (GAP12)
Fail soft: a feed that is down, slow or blocked no longer gives an empty dataset. When no item was read (in the demo, or when no feed could be read), the run pushes one free status item (type, status: warning, message, source, scrapedAt; no charged event) with the reason, logs a warning, and ends SUCCEEDED. RUN_SUMMARY is unchanged.
The rss-feed-reader-state key-value store is opened only when onlyNewItems is on, and a store that cannot be opened, read or written is a warning, not a crash.
A feed body is read as a stream and dropped above 32 MB (5 MB in the demo), so a huge or endless response cannot use up a 256 MB run.
Time budget: the demo run ({}, the prefill or the exampleRunInput) starts no new request after about 40 s, uses a 10 s timeout per request with at most 2 attempts and waits at most 2 s between them (a long Retry-After is capped). Every run, demo or not, also stops starting requests 20 s before the platform timeout (ACTOR_TIMEOUT_AT) and gets cut off at most 10 s after that, so a slow or hung source can no longer make the run time out. When the platform aborts the run, no new request is started.
robots.txt is now fetched with one lock per host instead of one lock for all hosts, so one slow host no longer holds up the others.
New exampleRunInput in actor.json: {"feedUrls": ["https://www.nasa.gov/feed/"]}. It matches the prefill and runs the same small demo.
Tests added for a source that is down, slow or hung, a partial failure, the time budget, the free status item and an unreadable state store.
Unreleased: default mode for an empty input (GAP8)
An empty input ({}) no longer fails with HTTP 400 or a missing-field error. feedUrls is no longer required and now has a schema default (and prefill) of ["https://www.nasa.gov/feed/"]. When feedUrls is empty or is exactly that sample, the run is in default mode: it reads the NASA feed, returns at most 10 items (a smaller maxItemsPerFeed wins), logs Default mode: ..., adds defaultMode and defaultFeed to RUN_SUMMARY and prefixes the status message with Default mode (sample feed): .
Input schema: the prefill changed from Hacker News and The Verge to the NASA sample feed, so the Store test runs default mode. Runs that list any other feeds behave exactly as before.
Lint only: removed unused imports in common.py (pyflakes clean). No behaviour change.
Unreleased: robots.txt on every redirect, unreadable robots.txt skips the host
Redirects are now followed one hop at a time, and robots.txt is checked for every hop. A feed or page that redirects to a disallowed path, or to another host whose robots.txt disallows it, is reported as skipped in RUN_SUMMARY and not fetched. Up to 5 redirects are followed.
If a host's robots.txt can't be read (a server error after one retry, a network failure, or a 200 that is an HTML page or not text/plain), the whole host is skipped for the run with a warning, and the feed is skipped with the reason. Before, such a host was read as if it had no rules. A missing robots.txt (any 4xx) still means no rules.
Unreleased: undated items
With postedWithinDays set, items without a publish date are counted in RUN_SUMMARY (undatedKept) and logged. Set keepUndatedItems to false to drop them.
Unreleased: JSON Feed
JSON Feed 1.0 and 1.1 feeds are read, and /feed.json is one of the paths tried when a page has no feed link.