# Dev.to & Forem Content Scraper (`claudethegoat/devto-forem-content-scraper`) Actor Scrape Dev.to / Forem articles, full markdown bodies, nested comments, tags, users and organizations via the official public API. Keyless — no login, no token, no proxy. Public data only. - **URL**: https://apify.com/claudethegoat/devto-forem-content-scraper.md - **Developed by:** [Manik Anand](https://apify.com/claudethegoat) (community) - **Categories:** Developer tools, News - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing $1.00 / 1,000 result scrapeds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Dev.to & Forem Content Scraper Extract **articles, full post bodies, nested comments, tags, users and organizations** from **Dev.to** (and any Forem-powered community) using the official **public API** — no account, no API token, no proxy spend. One actor covers the whole Dev.to content surface, with clean flattened output ready for analytics, RAG pipelines, and content research. ### What it does - **Articles** — pull posts by **tag**, by **author**, or from the **global feed**, sorted `popular` / `latest`, or ranked as **top of the last N days** - **Full article bodies** — optionally enrich every article with its complete `body_markdown` + `body_html` (most competitors only give you the list view) - **Comments** — the **full nested comment tree** for any article, flattened with `depth` + `parent_id_code` so you can rebuild the thread - **Tags catalog** — the Dev.to tag taxonomy with colors and summaries, for discovery - **Users** — public profile lookup by username - **Organizations** — org profiles, plus optionally all of an org's articles - **Unified, flat output** — consistent snake_case fields across every mode; a `type` field labels each row ### Why this actor | Other Dev.to actors | Gap | This actor | |---|---|---| | Most "Dev.to articles" scrapers (≤42 users) | Articles list only — **no full body**, no comments | Optional full `body_markdown`, plus a dedicated comments mode | | Article-only scrapers | No users / tags / organizations | **6 modes** in one actor | | Thin/unmaintained clones (~2 users each) | Crash on a bad tag or rate-limit, flagged "under maintenance" | **Never-crash**: retries transient errors, degrades gracefully, always exits clean | | — | Nested comments returned as raw JSON | Comments **flattened** with `depth` + `parent_id_code` + plain-text `body_text` | Dev.to hosts **millions of developer articles** and is one of the most-linked technical content platforms on the web — a rich, permissively-public source for developer-trend analysis, competitive content research, and AI training/RAG corpora. ### Modes & inputs | Mode | Uses | Returns | |---|---|---| | `articles` | `tags`, `usernames`, `sortBy`, `topDays`, `maxResults`, `fetchFullContent` | Article rows (optionally with full body) | | `article_detail` | `articleIds` | Full article incl. `body_markdown` / `body_html` | | `comments` | `articleIds` | Flattened nested comment tree | | `tags` | `maxResults` | Tag catalog | | `users` | `usernames` | Public profiles | | `organizations` | `usernames`, `includeOrgArticles`, `maxResults` | Org profile (+ optional articles) | #### Example input ```json { "mode": "articles", "tags": ["javascript", "webdev", "react"], "sortBy": "popular", "topDays": 7, "maxResults": 200, "fetchFullContent": true } ```` Get every comment on two articles: ```json { "mode": "comments", "articleIds": ["4100283", "4123925"] } ``` ### Output fields #### Article | Field | Type | Description | |---|---|---| | `id` | integer | Dev.to article ID (use in `article_detail` / `comments`) | | `title` | string | Article title | | `description` | string | Short summary | | `url` | string | Canonical dev.to permalink | | `tags` | array | Tag names | | `published_at` | string | ISO 8601 publish timestamp | | `reading_time_minutes` | integer | Estimated read time | | `comments_count` | integer | Number of comments | | `public_reactions_count` | integer | Total reactions | | `author_username` / `author_name` | string | Author identity | | `author_github` / `author_twitter` | string | Author social handles | | `organization_username` / `organization_name` | string | Publishing org (if any) | | `cover_image` / `social_image` | string | Image URLs | | `body_markdown` / `body_html` | string | Full post content (only with `fetchFullContent` or `article_detail`) | #### Comment | Field | Type | Description | |---|---|---| | `id_code` | string | Comment ID | | `article_id` | integer | Parent article | | `depth` | integer | Nesting depth (0 = top-level) | | `parent_id_code` | string | Parent comment (null at top level) | | `body_html` / `body_text` | string | Comment body (HTML + plain text) | | `author_username` / `author_name` | string | Commenter identity | | `created_at` | string | ISO 8601 timestamp | #### User / Organization `id`, `username`, `name`, `url`, `summary`, `location`, `website_url`, `joined_at`, `twitter_username`, `github_username`, `profile_image` (organizations also include `tag_line`, `tech_stack`, `story`). #### Tag `id`, `name`, `url`, `bg_color_hex`, `text_color_hex`, `short_summary`. ### Common use cases - **Developer-trend analysis** — what topics/tags are surging this week (`topDays`) - **Content research & SEO** — mine top-performing posts and titles by tag - **RAG / LLM training corpora** — full markdown bodies + threaded comments as clean text - **Competitive intelligence** — track a company's dev-relations org and its authors - **Community analytics** — comment-thread structure, engagement, and contributor mapping - **Newsletter / digest automation** — pull the latest or top articles for a niche on a schedule ### Pricing **Pay per result:** **$0.001** per dataset item (article, comment, user, tag, or organization row). Example: 5,000 articles ≈ **$5**. Comments and tags are billed the same per-row rate. ### Reliability - **Keyless & proxy-free** — Dev.to's public REST API is designed for logged-out reads; zero proxy cost - **Never-crash design** — transient errors (429 / 5xx / network) are retried with exponential backoff (honoring `Retry-After`); a bad tag or missing user is skipped, not fatal - **Deliver-then-charge** — results are written to the dataset before any charge; you never pay for rows you didn't receive - **Respects your limit** — stops cleanly when the run's charge limit is reached - **Polite pacing** — gentle request spacing so runs stay well-behaved against the API ### Data compliance This actor reads only **public, logged-out data** through Dev.to / Forem's official public API. No credentials are used and no private or gated content is accessed. Suitable for research and business use under the platform's terms for public data. # Actor input Schema ## `mode` (type: `string`): What to scrape. articles = list by tag/author/global feed. article\_detail = full markdown body for specific article IDs. comments = full nested comment tree for article IDs. tags = the Dev.to tag catalog. users = public profile lookup. organizations = org profile (+ optional org articles). ## `tags` (type: `array`): Dev.to tag names to pull articles for — each runs as a separate source (e.g. javascript, react, webdev, python, ai). Leave empty to scrape the global feed. ## `usernames` (type: `array`): In articles mode: authors whose articles to pull. In users mode: profiles to look up. In organizations mode: org slugs (e.g. heroku). Accepts @handle or bare name. ## `articleIds` (type: `array`): Numeric Dev.to article IDs. Used by article\_detail (full body) and comments (comment tree). Find IDs in the output of articles mode. ## `sortBy` (type: `string`): popular = Dev.to's default feed ordering. latest = strict newest-first (chronological). Ignored when 'Top: last N days' is set. ## `topDays` (type: `integer`): When > 0, return the most-reacted articles from the last N days (overrides Sort). 0 = disabled. Try 7 for the weekly top, 30 for monthly. ## `maxResults` (type: `integer`): Cap on articles per tag/author (articles mode), tags (tags mode), or articles per org. Higher values increase cost and run time. ## `fetchFullContent` (type: `boolean`): When on, enrich each article with its full body\_markdown + body\_html (one extra request per article — slower, but gives complete post content). Off = fast list view only. ## `includeOrgArticles` (type: `boolean`): When on, also pull each organization's published articles (up to Max results per source). ## Actor input object example ```json { "mode": "articles", "tags": [ "javascript", "webdev" ], "usernames": [], "articleIds": [], "sortBy": "popular", "topDays": 0, "maxResults": 100, "fetchFullContent": false, "includeOrgArticles": false } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "tags": [ "javascript", "webdev" ], "usernames": [], "articleIds": [] }; // Run the Actor and wait for it to finish const run = await client.actor("claudethegoat/devto-forem-content-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "tags": [ "javascript", "webdev", ], "usernames": [], "articleIds": [], } # Run the Actor and wait for it to finish run = client.actor("claudethegoat/devto-forem-content-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "tags": [ "javascript", "webdev" ], "usernames": [], "articleIds": [] }' | apify call claudethegoat/devto-forem-content-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=claudethegoat/devto-forem-content-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Dev.to & Forem Content Scraper", "description": "Scrape Dev.to / Forem articles, full markdown bodies, nested comments, tags, users and organizations via the official public API. Keyless — no login, no token, no proxy. Public data only.", "version": "1.0", "x-build-id": "zaL3aY05fJgqQuqTM" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/claudethegoat~devto-forem-content-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-claudethegoat-devto-forem-content-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/claudethegoat~devto-forem-content-scraper/runs": { "post": { "operationId": "runs-sync-claudethegoat-devto-forem-content-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/claudethegoat~devto-forem-content-scraper/run-sync": { "post": { "operationId": "run-sync-claudethegoat-devto-forem-content-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "mode": { "title": "Mode", "enum": [ "articles", "article_detail", "comments", "tags", "users", "organizations" ], "type": "string", "description": "What to scrape. articles = list by tag/author/global feed. article_detail = full markdown body for specific article IDs. comments = full nested comment tree for article IDs. tags = the Dev.to tag catalog. users = public profile lookup. organizations = org profile (+ optional org articles).", "default": "articles" }, "tags": { "title": "Tags (articles mode)", "type": "array", "description": "Dev.to tag names to pull articles for — each runs as a separate source (e.g. javascript, react, webdev, python, ai). Leave empty to scrape the global feed.", "items": { "type": "string" } }, "usernames": { "title": "Usernames / org slugs", "type": "array", "description": "In articles mode: authors whose articles to pull. In users mode: profiles to look up. In organizations mode: org slugs (e.g. heroku). Accepts @handle or bare name.", "items": { "type": "string" } }, "articleIds": { "title": "Article IDs (article_detail & comments modes)", "type": "array", "description": "Numeric Dev.to article IDs. Used by article_detail (full body) and comments (comment tree). Find IDs in the output of articles mode.", "items": { "type": "string" } }, "sortBy": { "title": "Sort (articles mode)", "enum": [ "popular", "latest" ], "type": "string", "description": "popular = Dev.to's default feed ordering. latest = strict newest-first (chronological). Ignored when 'Top: last N days' is set.", "default": "popular" }, "topDays": { "title": "Top: last N days (articles mode)", "minimum": 0, "maximum": 365, "type": "integer", "description": "When > 0, return the most-reacted articles from the last N days (overrides Sort). 0 = disabled. Try 7 for the weekly top, 30 for monthly.", "default": 0 }, "maxResults": { "title": "Max results per source", "minimum": 1, "maximum": 10000, "type": "integer", "description": "Cap on articles per tag/author (articles mode), tags (tags mode), or articles per org. Higher values increase cost and run time.", "default": 100 }, "fetchFullContent": { "title": "Fetch full article body (articles mode)", "type": "boolean", "description": "When on, enrich each article with its full body_markdown + body_html (one extra request per article — slower, but gives complete post content). Off = fast list view only.", "default": false }, "includeOrgArticles": { "title": "Include org articles (organizations mode)", "type": "boolean", "description": "When on, also pull each organization's published articles (up to Max results per source).", "default": false } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```