# Multi-Platform Blog Post Aggregator (`resourcefully_dialect/blog-aggregator`) Actor

- **URL**: https://apify.com/resourcefully_dialect/blog-aggregator.md
- **Developed by:** [Biosick Johnny](https://apify.com/resourcefully_dialect) (community)
- **Categories:** Social media, News, Developer tools
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Multi-Platform Blog Post Aggregator — dev.to, Hashnode, Medium & Substack

Collect public blog posts from **dev.to, Hashnode, Medium and Substack** in one Apify dataset. Supply an author handle or publication URL for any combination of platforms, then receive consistent metadata, canonical URL deduplication, newest-first sorting, an optional publication-date filter, a per-platform limit and a fairly shared total result cap. No source login or API key is required for public API/RSS collection.

Use this blog scraper for author portfolios, content discovery, newsletter research, editorial dashboards, content migration inventories and scheduled exports to databases or spreadsheets. `includeContent` adds the HTML or markdown made publicly available by the source.

This project is private and has not been deployed or published. The Dockerfile and Actor metadata are prepared for later review; no cloud pricing, account or visibility settings have been changed.

### Quick start

```json
{
  "devtoUsername": "ben",
  "hashnode": "townhall.hashnode.com",
  "mediumUsername": "@netflixtechblog",
  "substack": "https://on.substack.com",
  "maxItems": 100,
  "maxItemsPerPlatform": 25,
  "includeContent": false
}
```

Use just one platform or combine up to four. The handles above are independent public accounts used for testing; the Actor does not assume they belong to the same person. At least one nonempty platform field is required. For incremental collection, add `"dateFrom": "2026-10-01"` and keep previously exported results in your destination system.

### Input

| Field | Default | Accepted values |
| --- | --- | --- |
| `devtoUsername` | omitted | `ben`, `@ben`, `https://dev.to/ben` |
| `hashnode` | omitted | A publication host such as `townhall.hashnode.com`, `user.hashnode.dev`, a custom blog domain, a username, or `https://hashnode.com/@user` |
| `mediumUsername` | omitted | `user`, `@user`, `https://medium.com/@user`, or `https://user.medium.com` |
| `substack` | omitted | `on`, `on.substack.com`, or `https://on.substack.com` |
| `maxItems` | `100` | Integer from 0 to 10,000; total output cap shared fairly across platforms after filtering and deduplication. Zero skips source requests. |
| `maxItemsPerPlatform` | `25` | Integer of at least 1; maximum newest matching unique posts contributed by each platform |
| `dateFrom` | omitted | Inclusive ISO date in UTC, or timestamp with a timezone, such as `2026-10-01T12:00:00Z` |
| `includeContent` | `false` | `true` returns available HTML/markdown in `content`; `false` returns a plain-text `summary` of at most 400 characters |

Whitespace, `@` prefixes, common profile URL forms, trailing slashes and URL query/hash suffixes are normalized. Unsupported hosts and article URLs produce a source-specific error. A malformed platform value does not prevent other requested platforms from running. Invalid top-level options, unknown fields and an input without any platform fail validation before charging.

### Sources and coverage

| Platform | Collection method | Availability and limitations |
| --- | --- | --- |
| dev.to | Public `/api/articles?username=…&per_page=…&page=…` pagination | Gets reactions, comments, tags and reading time. Detail `/api/articles/{id}` is requested only with `includeContent: true`. |
| Hashnode | `publication(host:)` GraphQL cursor pagination, then `user(username:)` page pagination if the publication is not found | The specified `gql.hashnode.com` endpoint currently redirects to a retirement announcement. The Actor falls back to the publication's public `/rss.xml`. Feed history and engagement metadata are limited. |
| Medium | `https://medium.com/feed/@user` | Usually the latest 10 RSS entries. Claps, comment counts and reading times are generally absent and returned as `null`. |
| Substack | `https://publication.substack.com/feed` | Usually recent entries only. Content may be a teaser; likes and comment counts generally are not present. |

Hashnode [announced paid access for every GraphQL request on May 13, 2026](https://hashnode.com/changelog/2026-05-13-graphql-api-paid-access). The legacy GraphQL adapter remains implemented and fixture-tested, but the live no-login path uses public RSS. `RUN_SUMMARY.platforms.hashnode.method` and `warnings` distinguish these paths. No paid API access or authentication is attempted. If a username uses a custom publication domain, supply that actual domain so the RSS fallback can find it.

Each platform contributes up to `maxItemsPerPlatform` of its newest matching unique posts. Pagination stops on exhaustion, sufficient candidates, or a page entirely older than `dateFrom` in the sources' newest-first listings. Medium, Substack and Hashnode RSS fallback expose only their available feed window; increasing either limit cannot recover an older archive or guarantee a minimum number of results. A date filter excludes undated posts and cannot extend a feed's history.

After combining and cross-platform deduplication, the Actor allocates the `maxItems` budget round-robin: each platform with remaining posts gets its next-newest post per round. Posts within each round are considered newest-first, so a partial final round favors newer posts. Empty, failed or exhausted platforms leave their unused share for the others, up to each platform's own cap. The selected rows are then sorted newest-first for output. With four nonempty platforms and `maxItems: 8`, each gets two posts if available, even when one platform's posts are much newer. Every platform with distinct eligible posts gets at least one slot when the total cap is large enough; if the cap is smaller than the number of platforms, the newest platform heads get the available slots.

At the defaults, each platform supplies at most 25 posts and the total is at most 100. A run with only one platform still returns at most 25; set `maxItemsPerPlatform` higher to collect more from that source. Deduplicated cross-posts count once under the retained original platform. The billing budget can stop delivery before all selected rows are written.

### Unified output

Each dataset row contains the following fields. Missing scalar metadata is `null`; missing tags are `[]`. Content mode returns `content` instead of `summary`.

| Field | Meaning |
| --- | --- |
| `platform` | `devto`, `hashnode`, `medium` or `substack` |
| `title`, `url` | Article title and source permalink |
| `canonicalUrl` | Published canonical URL when available; RSS uses its entry link when no separate canonical link is provided |
| `publishedAt` | ISO 8601 UTC publication timestamp or `null` |
| `tags` | Lowercase, unique source tags |
| `reactions` | Published reactions/claps, when available; metrics have different meanings across platforms |
| `comments` | Numeric comment count, when available; a comments-page URL is not a count |
| `readingTime` | Published reading time in minutes, when available; not estimated from text |
| `author` | Published display name or source username, when available |
| `summary` or `content` | Short text, or the available HTML/markdown body |

Illustrative output; the complete real collected dataset is in [SAMPLE_OUTPUT.json](./SAMPLE_OUTPUT.json):

```json
{
  "platform": "devto",
  "title": "A portable blog",
  "url": "https://dev.to/alice/portable",
  "canonicalUrl": "https://blog.example.com/portable",
  "publishedAt": "2026-10-20T12:00:00.000Z",
  "tags": ["javascript", "webdev"],
  "reactions": 0,
  "comments": 3,
  "readingTime": 4,
  "author": "Alice",
  "summary": "A portable blog & more."
}
```

The output schema exposes `results` as `{{links.apiDefaultDatasetUrl}}/items`. After a future deployment, Apify's dataset exports can provide JSON, CSV, Excel and other supported formats. The dataset schema includes overview and content table views.

### Cross-platform deduplication

URL identity ignores HTTP versus HTTPS, a leading `www`, trailing slashes, query strings and fragments. The original URL values remain in the output. Source permalinks are also indexed as aliases. If a shared canonical link is absent, identical normalized titles provide a fallback: lowercase Unicode text with punctuation and whitespace collapsed. Aliases merge transitively, including duplicates found on later pages.

When a canonical link identifies an original platform record among the collected copies, that record supplies the platform and permalink. Otherwise, the record with the most populated metadata wins, with body length breaking ties. Missing metadata, longer text and tags can be filled from the other copies. Consequently, a merged row can contain information from more than one source.

Title matching can merge unrelated articles with generic identical titles. Ignoring URL query strings can also merge URLs that distinguish articles only by query parameters. Deduplication applies within each run; repeated posts in separate scheduled runs are new delivered results. Store cross-run state in your destination if needed.

### Pricing

Configured PPE prices are **$0.005 per `actor-start`** and **$0.002 per delivered `post`** ($2 per 1,000 posts plus the start event). A 100-post run is $0.205; a 1,000-post run is $2.005. There is no extra full-content event. These are the intended event prices; this private local project has no cloud billing configuration yet. See [PRICING.md](./PRICING.md) for the event definitions, budget behavior and competitor snapshot.

The start event is attempted once after input validation and the fresh-dataset check, including an empty result, zero `maxItems`, or later source failure, if the budget permits. Filtered, duplicate and rejected rows create no `post` event. Each accepted dataset write charges `post` once through the SDK's `Actor.charge` integration. Startup and per-post budget preflights prevent the SDK's over-limit fallback; delivery also checks `eventChargeLimitReached` and `chargedCount`. The last affordable row is retained and no further rows are pushed.

The event prices must be registered under these exact names if the Actor is later configured on Apify. Do not also price `apify-default-dataset-item`, which would add another result fee. Local tests make no real charges.

### Failures and run summaries

Requests use an identifiable User-Agent, a 30-second timeout and at least one second between starts to the same host. Network errors, HTTP 408, 429 and 5xx receive up to three retries with exponential backoff. `Retry-After` is honored; waits over 120 seconds stop that source. Other HTTP errors and malformed payloads fail that source immediately. Repeated pages/cursors are detected.

Each source is prepared independently. A failed source logs a warning, discards that source's incomplete preparation and records the error. Other sources continue. Hashnode's successful RSS fallback records a warning and still counts as a successful source. A valid empty source also counts as success. If every requested source fails, the process exits with code 1. Invalid input and dataset/storage failures also fail the run.

`RUN_SUMMARY` in the default key-value store records status (`succeeded`, `partial` or `failed`), stop reason (`sourceExhausted`, `maxItems`, `budgetReached` or `error`), request/retry counts, fetched/selected/delivered counts, duplicates removed, and each requested platform's method, warnings, error and counts. Skipped platforms are marked `skipped` when a cap or budget prevents requests. An exhausted available feed window is not a complete historical archive.

Use fresh storage for every local run. A nonempty default dataset is rejected to avoid repeated output and billing after a restart. Preparation happens before delivery. A failure during storage or charging can leave partial dataset output; inspect it and the summary before starting a new run.

### Run and test locally

Requires Node.js 22. From this folder:

```sh
npm ci
npm test
npm run local
```

In this workspace, use the supplied Node and npm without changing other folders:

```sh
export PATH="$PWD/../.tools/node-v22.23.4-darwin-arm64/bin:$PATH"
node ../.tools/npm/bin/npm-cli.js ci --cache .npm-cache
node ../.tools/npm/bin/npm-cli.js test --cache .npm-cache
node scripts/local-run.js live
node scripts/local-run.js partial
node scripts/local-run.js content
node scripts/local-run.js budget
node scripts/local-run.js zero-budget
node scripts/local-run.js all-failed
node ../.tools/npm/bin/npm-cli.js run check --cache .npm-cache
node ../.tools/npm/bin/npm-cli.js run validate --cache .npm-cache
```

`npm test` uses only JSON/XML fixtures and offline SDK simulations. `local-run.js` launches the real Actor in a fresh `storage-local-*` directory **inside this folder**, with an environment allowlist that excludes Apify credentials and cloud flags. The live run asserts that all four platforms contribute results, saves [SAMPLE_OUTPUT.json](./SAMPLE_OUTPUT.json), and writes `research/local-live.json`. Partial mode deliberately supplies a nonexistent Medium username, checks its HTTP failure and verifies the other three sources. Budget mode uses the SDK's synthetic $1/event simulation with a three-dollar local limit: one start and two posts, no real money. Exact $0.005/$0.002 boundaries are verified separately with the real `ChargingManager` and a stubbed cloud client in unit tests.

`content`, `zero-budget` and `all-failed` verify public content, no-work stopping and the fatal all-sources-failed path. Reports are under `research/`. `npm run validate` checks metadata against the official Apify schemas copied from the neighboring reference Actor, verifies file links and validates all real sample rows. The lockfile is included and the Docker image installs production dependencies only.

For custom input, create a fresh local `storage/key_value_stores/default/INPUT.json` and run `npm start`, leaving `APIFY_IS_AT_HOME` unset. The pinned SDK treats any nonempty value, including `false`, as cloud mode. Test results and local-run evidence are in [STATUS.md](./STATUS.md).

### FAQ

**Can I collect all posts ever published by an author?** dev.to and the legacy Hashnode GraphQL adapter paginate. RSS feeds expose a recent window only. A large cap or old `dateFrom` cannot restore missing RSS history.

**Does full content include paywalled articles?** Only content returned publicly by the API/feed is included. A teaser remains a teaser. The Actor does not log in or bypass subscriptions. HTML is source content; sanitize it before rendering in your own application.

**Why are claps, likes or authors null?** Public feeds often omit them. The Actor preserves missing values and does not invent metrics, authors or reading times.

**Why do fewer platforms appear in the dataset?** Fair allocation includes every platform with distinct eligible posts when `maxItems` has enough slots. A smaller total cap favors newer posts in the first round. A source can also be empty, filtered out, merged into original-platform rows, or fail, and a billing budget can stop delivery early. Check its fetched and delivered counts in `RUN_SUMMARY`.

**How does this compare with a generic RSS reader or a single-platform scraper?** This Actor combines four platform inputs with unified metadata, API engagement fields where published, original-platform deduplication, one date filter, a per-platform limit and a fairly shared total cap. The supplied [competitor snapshot](./COMPETITORS.md) includes cheaper generic RSS and dev.to tools, Medium tools around $0.002/post, and Substack tools around $0.005/post. The tradeoff is convenience and consistent output; individual public APIs/feeds can also be read directly.

**Does this write to my blogs?** Collection uses public read endpoints only. Hashnode GraphQL requests contain queries, not mutations. This project has not been pushed, deployed or published.

### Source references

- [Forem/dev.to API documentation](https://developers.forem.com/api/v1)
- [Hashnode GraphQL API paid-access announcement](https://hashnode.com/changelog/2026-05-13-graphql-api-paid-access)
- [Public Hashnode RSS example](https://townhall.hashnode.com/rss.xml)
- [Public Medium RSS example](https://medium.com/feed/@netflixtechblog)
- [Public Substack RSS example](https://on.substack.com/feed)
- [Apify SDK ChargingManager](https://docs.apify.com/sdk/js/reference/class/ChargingManager)

Public availability does not transfer copyright. Preserve source links and obtain any rights needed for redistributing article content. This is an independent integration with no platform endorsement.

# Actor input Schema

## `devtoUsername` (type: `string`):

Handle or dev.to profile URL, for example ben or https://dev.to/ben.

## `hashnode` (type: `string`):

Publication host, username or profile URL. GraphQL falls back to the public RSS feed when unavailable; RSS history and metadata are limited.

## `mediumUsername` (type: `string`):

Medium handle, @handle or profile URL. Public RSS usually exposes only the latest 10 posts.

## `substack` (type: `string`):

Publication subdomain or https://name.substack.com URL. Public RSS exposes only recent posts and may contain previews.

## `maxItems` (type: `integer`):

Total dataset cap after filtering and deduplication, shared fairly across platforms with unused shares redistributed. Results are sorted newest-first. Zero skips source requests.

## `maxItemsPerPlatform` (type: `integer`):

Maximum newest matching posts contributed by each platform before combining and sharing the total maxItems budget. Public feeds may return fewer posts.

## `dateFrom` (type: `string`):

Inclusive ISO date (YYYY-MM-DD, UTC) or timezone-qualified timestamp. Posts without dates are excluded when this filter is set.

## `includeContent` (type: `boolean`):

Include available HTML or markdown in content. Otherwise return a short plain-text summary. RSS may provide previews; paywalls are not bypassed.

## Actor input object example

```json
{
  "devtoUsername": "ben",
  "hashnode": "townhall.hashnode.com",
  "mediumUsername": "@netflixtechblog",
  "substack": "on",
  "maxItems": 100,
  "maxItemsPerPlatform": 25,
  "includeContent": false
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "devtoUsername": "ben",
    "hashnode": "townhall.hashnode.com",
    "mediumUsername": "@netflixtechblog",
    "substack": "on"
};

// Run the Actor and wait for it to finish
const run = await client.actor("resourcefully_dialect/blog-aggregator").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "devtoUsername": "ben",
    "hashnode": "townhall.hashnode.com",
    "mediumUsername": "@netflixtechblog",
    "substack": "on",
}

# Run the Actor and wait for it to finish
run = client.actor("resourcefully_dialect/blog-aggregator").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "devtoUsername": "ben",
  "hashnode": "townhall.hashnode.com",
  "mediumUsername": "@netflixtechblog",
  "substack": "on"
}' |
apify call resourcefully_dialect/blog-aggregator --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,resourcefully_dialect/blog-aggregator"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Bb9WkK3fIMaJaCpMZ/builds/2HLGF403fZTfqsOs9/openapi.json