Changelog
[Latest] - 2025-11-15
Added
Streaming to Dataset : Members are now pushed to the dataset immediately as they are fetched, rather than collecting all in memory firstRandom Payload Selection : Each request randomly selects a payload/header pair from the loaded credentialsPayload Indicator : Console now shows which payload index is being used for each request
Changed
Memory Efficiency : Changed from collecting all members in an array to streaming them directly to the datasetReturn Value : scrapeMembers() now returns the total count instead of the members arrayConsole Output : Added streaming confirmation messages showing real-time data push to dataset
Technical Details
Before:
const members = [ ] ;
members . push ( member ) ;
await Actor . pushData ( members ) ;
After:
let totalMembers = 0 ;
const newMembers = [ ] ;
await Actor . pushData ( newMembers ) ;
totalMembers += newMembers . length ;
Benefits
Lower Memory Usage : No need to store all members in memoryReal-time Data : Data appears in dataset as it's scrapedBetter for Large Groups : Can handle groups with millions of membersRandom Payload Rotation : Each request uses a different random payload for better distribution
Testing
Tested with local JSON file (20251113_134806.json)
Verified data extraction matches Python implementation
Confirmed streaming works correctly with Apify dataset
