Multi-Platform Blog Post Aggregator
Pricing
Pay per event
Multi-Platform Blog Post Aggregator
Pricing
Pay per event
Rating
0.0
(0)
Developer
Biosick Johnny
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
9 days ago
Last modified
Categories
Security
Limited permissionsShare
Multi-Platform Blog Post Aggregator — dev.to, Hashnode, Medium & Substack
Collect public blog posts from dev.to, Hashnode, Medium and Substack in one Apify dataset. Supply an author handle or publication URL for any combination of platforms, then receive consistent metadata, canonical URL deduplication, newest-first sorting, an optional publication-date filter, a per-platform limit and a fairly shared total result cap. No source login or API key is required for public API/RSS collection.
Use this blog scraper for author portfolios, content discovery, newsletter research, editorial dashboards, content migration inventories and scheduled exports to databases or spreadsheets. includeContent adds the HTML or markdown made publicly available by the source.
This project is private and has not been deployed or published. The Dockerfile and Actor metadata are prepared for later review; no cloud pricing, account or visibility settings have been changed.
Quick start
Use just one platform or combine up to four. The handles above are independent public accounts used for testing; the Actor does not assume they belong to the same person. At least one nonempty platform field is required. For incremental collection, add "dateFrom": "2026-10-01" and keep previously exported results in your destination system.
Input
| Field | Default | Accepted values |
|---|---|---|
devtoUsername | omitted | ben, @ben, https://dev.to/ben |
hashnode | omitted | A publication host such as townhall.hashnode.com, user.hashnode.dev, a custom blog domain, a username, or https://hashnode.com/@user |
mediumUsername | omitted | user, @user, https://medium.com/@user, or https://user.medium.com |
substack | omitted | on, on.substack.com, or https://on.substack.com |
maxItems | 100 | Integer from 0 to 10,000; total output cap shared fairly across platforms after filtering and deduplication. Zero skips source requests. |
maxItemsPerPlatform | 25 | Integer of at least 1; maximum newest matching unique posts contributed by each platform |
dateFrom | omitted | Inclusive ISO date in UTC, or timestamp with a timezone, such as 2026-10-01T12:00:00Z |
includeContent | false | true returns available HTML/markdown in content; false returns a plain-text summary of at most 400 characters |
Whitespace, @ prefixes, common profile URL forms, trailing slashes and URL query/hash suffixes are normalized. Unsupported hosts and article URLs produce a source-specific error. A malformed platform value does not prevent other requested platforms from running. Invalid top-level options, unknown fields and an input without any platform fail validation before charging.
Sources and coverage
| Platform | Collection method | Availability and limitations |
|---|---|---|
| dev.to | Public /api/articles?username=…&per_page=…&page=… pagination | Gets reactions, comments, tags and reading time. Detail /api/articles/{id} is requested only with includeContent: true. |
| Hashnode | publication(host:) GraphQL cursor pagination, then user(username:) page pagination if the publication is not found | The specified gql.hashnode.com endpoint currently redirects to a retirement announcement. The Actor falls back to the publication's public /rss.xml. Feed history and engagement metadata are limited. |
| Medium | https://medium.com/feed/@user | Usually the latest 10 RSS entries. Claps, comment counts and reading times are generally absent and returned as null. |
| Substack | https://publication.substack.com/feed | Usually recent entries only. Content may be a teaser; likes and comment counts generally are not present. |
Hashnode announced paid access for every GraphQL request on May 13, 2026. The legacy GraphQL adapter remains implemented and fixture-tested, but the live no-login path uses public RSS. RUN_SUMMARY.platforms.hashnode.method and warnings distinguish these paths. No paid API access or authentication is attempted. If a username uses a custom publication domain, supply that actual domain so the RSS fallback can find it.
Each platform contributes up to maxItemsPerPlatform of its newest matching unique posts. Pagination stops on exhaustion, sufficient candidates, or a page entirely older than dateFrom in the sources' newest-first listings. Medium, Substack and Hashnode RSS fallback expose only their available feed window; increasing either limit cannot recover an older archive or guarantee a minimum number of results. A date filter excludes undated posts and cannot extend a feed's history.
After combining and cross-platform deduplication, the Actor allocates the maxItems budget round-robin: each platform with remaining posts gets its next-newest post per round. Posts within each round are considered newest-first, so a partial final round favors newer posts. Empty, failed or exhausted platforms leave their unused share for the others, up to each platform's own cap. The selected rows are then sorted newest-first for output. With four nonempty platforms and maxItems: 8, each gets two posts if available, even when one platform's posts are much newer. Every platform with distinct eligible posts gets at least one slot when the total cap is large enough; if the cap is smaller than the number of platforms, the newest platform heads get the available slots.
At the defaults, each platform supplies at most 25 posts and the total is at most 100. A run with only one platform still returns at most 25; set maxItemsPerPlatform higher to collect more from that source. Deduplicated cross-posts count once under the retained original platform. The billing budget can stop delivery before all selected rows are written.
Unified output
Each dataset row contains the following fields. Missing scalar metadata is null; missing tags are []. Content mode returns content instead of summary.
| Field | Meaning |
|---|---|
platform | devto, hashnode, medium or substack |
title, url | Article title and source permalink |
canonicalUrl | Published canonical URL when available; RSS uses its entry link when no separate canonical link is provided |
publishedAt | ISO 8601 UTC publication timestamp or null |
tags | Lowercase, unique source tags |
reactions | Published reactions/claps, when available; metrics have different meanings across platforms |
comments | Numeric comment count, when available; a comments-page URL is not a count |
readingTime | Published reading time in minutes, when available; not estimated from text |
author | Published display name or source username, when available |
summary or content | Short text, or the available HTML/markdown body |
Illustrative output; the complete real collected dataset is in ./SAMPLE_OUTPUT.json:
The output schema exposes results as {{links.apiDefaultDatasetUrl}}/items. After a future deployment, Apify's dataset exports can provide JSON, CSV, Excel and other supported formats. The dataset schema includes overview and content table views.
Cross-platform deduplication
URL identity ignores HTTP versus HTTPS, a leading www, trailing slashes, query strings and fragments. The original URL values remain in the output. Source permalinks are also indexed as aliases. If a shared canonical link is absent, identical normalized titles provide a fallback: lowercase Unicode text with punctuation and whitespace collapsed. Aliases merge transitively, including duplicates found on later pages.
When a canonical link identifies an original platform record among the collected copies, that record supplies the platform and permalink. Otherwise, the record with the most populated metadata wins, with body length breaking ties. Missing metadata, longer text and tags can be filled from the other copies. Consequently, a merged row can contain information from more than one source.
Title matching can merge unrelated articles with generic identical titles. Ignoring URL query strings can also merge URLs that distinguish articles only by query parameters. Deduplication applies within each run; repeated posts in separate scheduled runs are new delivered results. Store cross-run state in your destination if needed.
Pricing
Configured PPE prices are $0.005 per actor-start and $0.002 per delivered post ($2 per 1,000 posts plus the start event). A 100-post run is $0.205; a 1,000-post run is $2.005. There is no extra full-content event. These are the intended event prices; this private local project has no cloud billing configuration yet. See ./PRICING.md for the event definitions, budget behavior and competitor snapshot.
The start event is attempted once after input validation and the fresh-dataset check, including an empty result, zero maxItems, or later source failure, if the budget permits. Filtered, duplicate and rejected rows create no post event. Each accepted dataset write charges post once through the SDK's Actor.charge integration. Startup and per-post budget preflights prevent the SDK's over-limit fallback; delivery also checks eventChargeLimitReached and chargedCount. The last affordable row is retained and no further rows are pushed.
The event prices must be registered under these exact names if the Actor is later configured on Apify. Do not also price apify-default-dataset-item, which would add another result fee. Local tests make no real charges.
Failures and run summaries
Requests use an identifiable User-Agent, a 30-second timeout and at least one second between starts to the same host. Network errors, HTTP 408, 429 and 5xx receive up to three retries with exponential backoff. Retry-After is honored; waits over 120 seconds stop that source. Other HTTP errors and malformed payloads fail that source immediately. Repeated pages/cursors are detected.
Each source is prepared independently. A failed source logs a warning, discards that source's incomplete preparation and records the error. Other sources continue. Hashnode's successful RSS fallback records a warning and still counts as a successful source. A valid empty source also counts as success. If every requested source fails, the process exits with code 1. Invalid input and dataset/storage failures also fail the run.
RUN_SUMMARY in the default key-value store records status (succeeded, partial or failed), stop reason (sourceExhausted, maxItems, budgetReached or error), request/retry counts, fetched/selected/delivered counts, duplicates removed, and each requested platform's method, warnings, error and counts. Skipped platforms are marked skipped when a cap or budget prevents requests. An exhausted available feed window is not a complete historical archive.
Use fresh storage for every local run. A nonempty default dataset is rejected to avoid repeated output and billing after a restart. Preparation happens before delivery. A failure during storage or charging can leave partial dataset output; inspect it and the summary before starting a new run.
Run and test locally
Requires Node.js 22. From this folder:
In this workspace, use the supplied Node and npm without changing other folders:
npm test uses only JSON/XML fixtures and offline SDK simulations. local-run.js launches the real Actor in a fresh storage-local-* directory inside this folder, with an environment allowlist that excludes Apify credentials and cloud flags. The live run asserts that all four platforms contribute results, saves ./SAMPLE_OUTPUT.json, and writes research/local-live.json. Partial mode deliberately supplies a nonexistent Medium username, checks its HTTP failure and verifies the other three sources. Budget mode uses the SDK's synthetic $1/event simulation with a three-dollar local limit: one start and two posts, no real money. Exact $0.005/$0.002 boundaries are verified separately with the real ChargingManager and a stubbed cloud client in unit tests.
content, zero-budget and all-failed verify public content, no-work stopping and the fatal all-sources-failed path. Reports are under research/. npm run validate checks metadata against the official Apify schemas copied from the neighboring reference Actor, verifies file links and validates all real sample rows. The lockfile is included and the Docker image installs production dependencies only.
For custom input, create a fresh local storage/key_value_stores/default/INPUT.json and run npm start, leaving APIFY_IS_AT_HOME unset. The pinned SDK treats any nonempty value, including false, as cloud mode. Test results and local-run evidence are in ./STATUS.md.
FAQ
Can I collect all posts ever published by an author? dev.to and the legacy Hashnode GraphQL adapter paginate. RSS feeds expose a recent window only. A large cap or old dateFrom cannot restore missing RSS history.
Does full content include paywalled articles? Only content returned publicly by the API/feed is included. A teaser remains a teaser. The Actor does not log in or bypass subscriptions. HTML is source content; sanitize it before rendering in your own application.
Why are claps, likes or authors null? Public feeds often omit them. The Actor preserves missing values and does not invent metrics, authors or reading times.
Why do fewer platforms appear in the dataset? Fair allocation includes every platform with distinct eligible posts when maxItems has enough slots. A smaller total cap favors newer posts in the first round. A source can also be empty, filtered out, merged into original-platform rows, or fail, and a billing budget can stop delivery early. Check its fetched and delivered counts in RUN_SUMMARY.
How does this compare with a generic RSS reader or a single-platform scraper? This Actor combines four platform inputs with unified metadata, API engagement fields where published, original-platform deduplication, one date filter, a per-platform limit and a fairly shared total cap. The supplied ./COMPETITORS.md includes cheaper generic RSS and dev.to tools, Medium tools around $0.002/post, and Substack tools around $0.005/post. The tradeoff is convenience and consistent output; individual public APIs/feeds can also be read directly.
Does this write to my blogs? Collection uses public read endpoints only. Hashnode GraphQL requests contain queries, not mutations. This project has not been pushed, deployed or published.
Source references
- Forem/dev.to API documentation
- Hashnode GraphQL API paid-access announcement
- Public Hashnode RSS example
- Public Medium RSS example
- Public Substack RSS example
- Apify SDK ChargingManager
Public availability does not transfer copyright. Preserve source links and obtain any rights needed for redistributing article content. This is an independent integration with no platform endorsement.
