Pricing

from $0.01 / 1,000 results

Dev.to Articles & Developer Leads Scraper

Scrape Dev.to articles by tag, author, top, or latest and extract developer leads: author name, GitHub, X/Twitter, website, and location. No login or API key. Export to JSON, CSV, or Excel.

Pricing

from $0.01 / 1,000 results

Rating

0.0

(0)

Developer

Mangudäi

Actor stats

Bookmarked

Total users

Monthly active users

4 days ago

Last modified

.actor/actor.json

{
    "actorSpecification": 1,
    "name": "devto-articles-developer-leads-scraper",
    "title": "Dev.to Articles & Developer Leads Scraper",
    "description": "Scrape Dev.to (Forem) articles by tag, author, top, or latest and extract developer leads: author name, GitHub, X/Twitter, website, and location. No login or API key.",
    "version": "0.0",
    "buildTag": "latest",
    "dockerfile": "./Dockerfile",
    "input": {
        "title": "Dev.to scraper input",
        "type": "object",
        "schemaVersion": 1,
        "properties": {
            "mode": {
                "title": "Search mode",
                "type": "string",
                "description": "How to select articles: by tag, by author username, by top-ranked over a period, or the latest feed.",
                "enum": ["tag", "author", "top", "latest"],
                "enumTitles": ["By tag", "By author", "Top (by period)", "Latest"],
                "default": "tag"
            },
            "tag": {
                "title": "Tag",
                "type": "string",
                "description": "Tag to scrape when mode is 'By tag' (for example python, javascript, webdev).",
                "editor": "textfield",
                "default": "javascript"
            },
            "username": {
                "title": "Author username",
                "type": "string",
                "description": "Dev.to username to scrape when mode is 'By author' (for example ben).",
                "editor": "textfield"
            },
            "topDays": {
                "title": "Top period",
                "type": "string",
                "description": "Ranking window when mode is 'Top'.",
                "enum": ["1", "7", "30", "365"],
                "enumTitles": ["Last day", "Last week", "Last month", "Last year"],
                "default": "7"
            },
            "maxItems": {
                "title": "Max articles",
                "type": "integer",
                "description": "Maximum number of articles to return.",
                "default": 100,
                "minimum": 1,
                "maximum": 100000
            },
            "enrichAuthors": {
                "title": "Enrich author leads",
                "type": "boolean",
                "description": "Fetch each unique author's public profile to add name, GitHub, X/Twitter, website, and location.",
                "default": true
            }
        },
        "required": []
    },
    "storages": {
        "dataset": {
            "actorSpecification": 1,
            "title": "Dev.to articles and developer leads",
            "views": {
                "overview": {
                    "title": "Overview",
                    "transformation": {
                        "fields": [
                            "title", "url", "publishedAt", "tags",
                            "positiveReactions", "comments",
                            "authorName", "authorUsername", "authorProfileUrl",
                            "authorGithub", "authorTwitter", "authorWebsite", "authorLocation"
                        ]
                    },
                    "display": {
                        "component": "table",
                        "properties": {
                            "title": {"label": "Title", "format": "text"},
                            "url": {"label": "URL", "format": "link"},
                            "publishedAt": {"label": "Published", "format": "text"},
                            "tags": {"label": "Tags", "format": "array"},
                            "positiveReactions": {"label": "Reactions", "format": "number"},
                            "comments": {"label": "Comments", "format": "number"},
                            "authorName": {"label": "Author", "format": "text"},
                            "authorUsername": {"label": "Username", "format": "text"},
                            "authorProfileUrl": {"label": "Profile", "format": "link"},
                            "authorGithub": {"label": "GitHub", "format": "text"},
                            "authorTwitter": {"label": "X/Twitter", "format": "text"},
                            "authorWebsite": {"label": "Website", "format": "link"},
                            "authorLocation": {"label": "Location", "format": "text"}
                        }
                    }
                }
            }
        }
    },
    "output": {
        "actorOutputSchemaVersion": 1,
        "title": "Dev.to scraper output",
        "properties": {
            "results": {
                "type": "string",
                "title": "Results",
                "template": "{{links.apiDefaultDatasetUrl}}/items"
            }
        }
    }
}

my_actor/init.py

my_actor/main.py

1import asyncio
2
3from .main import main
4
5if __name__ == '__main__':
6    asyncio.run(main())

my_actor/main.py

1"""Dev.to (Forem) articles and developer leads scraper."""
2from __future__ import annotations
3
4import json
5import time
6import urllib.parse
7import urllib.request
8
9from apify import Actor
10
11BASE = "https://dev.to/api"
12UA = "Mozilla/5.0 (compatible; devto-leads-scraper/1.0; +https://apify.com)"
13MAX_PER_PAGE = 1000
14
15
16def _get(path: str, params: dict, retries: int = 4):
17    query = urllib.parse.urlencode({k: v for k, v in params.items() if v not in (None, "")})
18    url = f"{BASE}{path}?{query}" if query else f"{BASE}{path}"
19    last_exc = None
20    for attempt in range(retries):
21        try:
22            req = urllib.request.Request(url, headers={"User-Agent": UA, "Accept": "application/json"})
23            with urllib.request.urlopen(req, timeout=45) as resp:
24                return json.loads(resp.read().decode("utf-8"))
25        except Exception as exc:  # noqa: BLE001
26            last_exc = exc
27            time.sleep(1.5 * (attempt + 1))
28    raise last_exc
29
30
31def _clean(article: dict, author: dict | None) -> dict:
32    user = article.get("user") or {}
33    username = user.get("username")
34    org = article.get("organization") or {}
35    prof = author or {}
36    return {
37        "id": article.get("id"),
38        "title": article.get("title"),
39        "url": article.get("url"),
40        "slug": article.get("slug"),
41        "description": article.get("description"),
42        "publishedAt": article.get("published_at"),
43        "tags": article.get("tag_list"),
44        "positiveReactions": article.get("positive_reactions_count"),
45        "comments": article.get("comments_count"),
46        "readingTimeMinutes": article.get("reading_time_minutes"),
47        "coverImage": article.get("cover_image"),
48        "canonicalUrl": article.get("canonical_url"),
49        "organization": org.get("name"),
50        "authorName": prof.get("name") or user.get("name"),
51        "authorUsername": username,
52        "authorProfileUrl": f"https://dev.to/{username}" if username else None,
53        "authorGithub": prof.get("github_username") or user.get("github_username"),
54        "authorTwitter": prof.get("twitter_username") or user.get("twitter_username"),
55        "authorWebsite": prof.get("website_url"),
56        "authorLocation": prof.get("location"),
57        "authorSummary": prof.get("summary"),
58        "authorJoinedAt": prof.get("joined_at"),
59    }
60
61
62async def main() -> None:
63    async with Actor:
64        inp = await Actor.get_input() or {}
65        mode = inp.get("mode", "tag")
66        tag = (inp.get("tag") or "").strip().lstrip("#").lower()
67        username = (inp.get("username") or "").strip().lstrip("@")
68        top_days = str(inp.get("topDays", "7"))
69        max_items = int(inp.get("maxItems", 100) or 100)
70        enrich = bool(inp.get("enrichAuthors", True))
71
72        Actor.log.info(f"Mode={mode} tag={tag!r} username={username!r} maxItems={max_items} enrich={enrich}")
73
74        articles: list[dict] = []
75        page = 1
76        per = min(MAX_PER_PAGE, max_items)
77        while len(articles) < max_items:
78            params: dict = {"per_page": per, "page": page}
79            if mode == "tag" and tag:
80                params["tag"] = tag
81            elif mode == "author" and username:
82                params["username"] = username
83            elif mode == "top":
84                params["top"] = top_days
85            path = "/articles/latest" if mode == "latest" else "/articles"
86            batch = _get(path, params)
87            if not batch:
88                break
89            articles.extend(batch)
90            Actor.log.info(f"Fetched page {page}: {len(batch)} articles (total {len(articles)})")
91            if len(batch) < per:
92                break
93            page += 1
94            time.sleep(0.4)
95
96        articles = articles[:max_items]
97
98        profile_cache: dict[str, dict] = {}
99        rows = []
100        for art in articles:
101            author_profile = None
102            uname = (art.get("user") or {}).get("username")
103            if enrich and uname:
104                if uname not in profile_cache:
105                    try:
106                        profile_cache[uname] = _get("/users/by_username", {"url": uname})
107                    except Exception:  # noqa: BLE001
108                        profile_cache[uname] = {}
109                    time.sleep(0.25)
110                author_profile = profile_cache[uname]
111            rows.append(_clean(art, author_profile))
112
113        await Actor.push_data(rows)
114        Actor.log.info(f"Done. Pushed {len(rows)} items.")

my_actor/py.typed

.dockerignore

.git
.mise.toml
.nvim.lua
storage

# The rest is copied from https://github.com/github/gitignore/blob/main/Python.gitignore

# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# PyInstaller
#  Usually these files are written by a python script from a template
#  before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
.pybuilder/
target/

# Jupyter Notebook
.ipynb_checkpoints

# IPython
profile_default/
ipython_config.py

# pyenv
#   For a library or package, you might want to ignore these files since the code is
#   intended to run in multiple environments; otherwise, check them in:
.python-version

# pdm
#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
#   in version control.
#   https://pdm.fming.dev/latest/usage/project/#working-with-version-control
.pdm.toml
.pdm-python
.pdm-build/

# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/

# Celery stuff
celerybeat-schedule
celerybeat.pid

# SageMath parsed files
*.sage.py

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/
.dmypy.json
dmypy.json

# Pyre type checker
.pyre/

# pytype static type analyzer
.pytype/

# Cython debug symbols
cython_debug/

# PyCharm
#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
#  and can be added to the global gitignore or merged into this file.  For a more nuclear
#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
.idea/

# Visual Studio Code
#  Ignores the folder created by VS Code when changing workspace settings, doing debugger
#  configuration, etc. Can be commented out to share Workspace Settings within a team
.vscode

# Zed editor
#  Ignores the folder created when setting Project Settings in the Zed editor. Can be commented out
#  to share Project Settings within a team
.zed

.gitignore

.mise.toml
.nvim.lua
storage

# The rest is copied from https://github.com/github/gitignore/blob/main/Python.gitignore

# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# PyInstaller
#  Usually these files are written by a python script from a template
#  before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
.pybuilder/
target/

# Jupyter Notebook
.ipynb_checkpoints

# IPython
profile_default/
ipython_config.py

# pyenv
#   For a library or package, you might want to ignore these files since the code is
#   intended to run in multiple environments; otherwise, check them in:
.python-version

# pdm
#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
#   in version control.
#   https://pdm.fming.dev/latest/usage/project/#working-with-version-control
.pdm.toml
.pdm-python
.pdm-build/

# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/

# Celery stuff
celerybeat-schedule
celerybeat.pid

# SageMath parsed files
*.sage.py

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/
.dmypy.json
dmypy.json

# Pyre type checker
.pyre/

# pytype static type analyzer
.pytype/

# Cython debug symbols
cython_debug/

# PyCharm
#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
#  and can be added to the global gitignore or merged into this file.  For a more nuclear
#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
.idea/

# Visual Studio Code
#  Ignores the folder created by VS Code when changing workspace settings, doing debugger
#  configuration, etc. Can be commented out to share Workspace Settings within a team
.vscode

# Zed editor
#  Ignores the folder created when setting Project Settings in the Zed editor. Can be commented out
#  to share Project Settings within a team
.zed

AGENTS.md

1# Apify Actors Development Guide
2
3Important: Before you begin, fill in the `generatedBy` property in the meta section of `.actor/actor.json`. Replace it with the tool and model you're currently using, such as "Claude Code with Claude Sonnet 4.5". This helps Apify monitor and improve AGENTS.md for specific AI tools and models.
4
5## What are Apify Actors?
6
7- Actors are serverless programs that run in the cloud. They're inspired by the UNIX philosophy - programs that do one thing well and can be easily combined to build complex systems.
8- Actors are programs packaged as Docker images that run in isolated containers
9
10## Core Concepts
11
12- Accept well-defined JSON input
13- Perform isolated tasks (web scraping, automation, data processing)
14- Produce structured JSON output to datasets and/or store data in key-value stores
15- Can run from seconds to hours or even indefinitely
16- Persist state and can be restarted
17
18## Do
19
20- accept well-defined JSON input and produce structured JSON output
21- use Apify SDK (`apify`) for code running ON Apify platform
22- validate input early with proper error handling and fail gracefully
23- use CheerioCrawler for static HTML content (10x faster than browsers)
24- use PlaywrightCrawler only for JavaScript-heavy sites and dynamic content
25- use router pattern (createCheerioRouter/createPlaywrightRouter) for complex crawls
26- implement retry strategies with exponential backoff for failed requests
27- use proper concurrency settings (HTTP: 10-50, Browser: 1-5)
28- set sensible defaults in `.actor/input_schema.json` for all optional fields
29- set up output schema in `.actor/output_schema.json`
30- clean and validate data before pushing to dataset
31- use semantic CSS selectors and fallback strategies for missing elements
32- respect robots.txt, ToS, and implement rate limiting with delays
33- check which tools (cheerio/playwright/crawlee) are installed before applying guidance
34- use `Actor.log` for logging (censors sensitive data)
35- implement readiness probe handler for standby Actors
36- handle the `aborting` event to gracefully shut down when Actor is stopped
37
38## Don't
39
40- do not rely on `Dataset.getInfo()` for final counts on Cloud platform
41- do not use browser crawlers when HTTP/Cheerio works (massive performance gains with HTTP)
42- do not hard code values that should be in input schema or environment variables
43- do not skip input validation or error handling
44- do not overload servers - use appropriate concurrency and delays
45- do not scrape prohibited content or ignore Terms of Service
46- do not store personal/sensitive data unless explicitly permitted
47- do not use deprecated options like `requestHandlerTimeoutMillis` on CheerioCrawler (v3.x)
48- do not use `additionalHttpHeaders` - use `preNavigationHooks` instead
49- do not assume that local storage is persistent or automatically synced to Apify Console - when running locally with `apify run`, the `storage/` directory is local-only and is NOT pushed to the Cloud
50- do not disable standby mode (`usesStandbyMode: false`) without explicit permission
51
52## Logging
53
54- **ALWAYS use `Actor.log` for logging** - This logger contains critical security logic including censoring sensitive data (Apify tokens, API keys, credentials) to prevent accidental exposure in logs
55
56### Available Log Levels
57
58The Apify Actor logger provides the following methods for logging:
59
60- `Actor.log.debug()` - Debug level logs (detailed diagnostic information)
61- `Actor.log.info()` - Info level logs (general informational messages)
62- `Actor.log.warning()` - Warning level logs (warning messages for potentially problematic situations)
63- `Actor.log.error()` - Error level logs (error messages for failures)
64- `Actor.log.exception()` - Exception level logs (for exceptions with stack traces)
65
66**Best practices:**
67
68- Use `Actor.log.debug()` for detailed operation-level diagnostics (inside functions)
69- Use `Actor.log.info()` for general informational messages (API requests, successful operations)
70- Use `Actor.log.warning()` for potentially problematic situations (validation failures, unexpected states)
71- Use `Actor.log.error()` for actual errors and failures
72- Use `Actor.log.exception()` for caught exceptions with stack traces
73
74## Graceful Abort Handling
75
76Handle the `aborting` event to terminate the Actor quickly when stopped by user or platform, minimizing costs especially for PPU/PPE+U billing.
77
78```python
79import asyncio
80
81async def on_aborting() -> None:
82    # Persist any state, do any cleanup you need, and terminate the Actor using `await Actor.exit()` explicitly as soon as possible
83    # This will help ensure that the Actor is doing best effort to honor any potential limits on costs of a single run set by the user
84    # Wait 1 second to allow Crawlee/SDK state persistence operations to complete
85    # This is a temporary workaround until SDK implements proper state persistence in the aborting event
86    await asyncio.sleep(1)
87    await Actor.exit()
88
89Actor.on('aborting', on_aborting)
90```
91
92## Standby Mode
93
94- **NEVER disable standby mode (`usesStandbyMode: false`) in `.actor/actor.json` without explicit permission** - Actor Standby mode solves this problem by letting you have the Actor ready in the background, waiting for the incoming HTTP requests. In a sense, the Actor behaves like a real-time web server or standard API server instead of running the logic once to process everything in batch. Always keep `usesStandbyMode: true` unless there is a specific documented reason to disable it
95- **ALWAYS implement readiness probe handler for standby Actors** - Handle the `x-apify-container-server-readiness-probe` header at GET / endpoint to ensure proper Actor lifecycle management
96
97You can recognize a standby Actor by checking the `usesStandbyMode` property in `.actor/actor.json`. Only implement the readiness probe if this property is set to `true`.
98
99### Readiness Probe Implementation Example
100
101```python
102# Apify standby readiness probe
103from http.server import SimpleHTTPRequestHandler
104
105class GetHandler(SimpleHTTPRequestHandler):
106    def do_GET(self):
107        # Handle Apify standby readiness probe
108        if 'x-apify-container-server-readiness-probe' in self.headers:
109            self.send_response(200)
110            self.end_headers()
111            self.wfile.write(b'Readiness probe OK')
112            return
113
114        self.send_response(200)
115        self.end_headers()
116        self.wfile.write(b'Actor is ready')
117```
118
119Key points:
120
121- Detect the `x-apify-container-server-readiness-probe` header in incoming requests
122- Respond with HTTP 200 status code for both readiness probe and normal requests
123- This enables proper Actor lifecycle management in standby mode
124
125## Commands
126
127```bash
128# Bootstrap & local development
129apify create [name]                    # Create new Actor project from a template
130apify init                             # Initialize Actor in current directory
131apify run                              # Run Actor locally with simulated platform env
132apify run --purge                      # Run after clearing previous local storage
133apify validate-schema                  # Validate .actor/input_schema.json
134
135# Authentication & account
136apify login                            # Authenticate account (token stored in ~/.apify)
137apify logout                           # Remove stored credentials
138apify info                             # Print currently authenticated account info
139
140# Deployment & remote execution
141apify push                             # Deploy Actor to platform per .actor/actor.json
142apify pull <actor>                     # Download Actor code from the platform
143apify call <actor>                     # Execute Actor remotely on the platform
144apify actors build <actor>             # Create a new build of an Actor
145apify runs ls                          # List recent runs
146
147# Discovery (search Apify Store for community Actors)
148apify actors search "<query>" --user-agent <your-agent-name>
149apify actors info <actor>              # Get details about a specific Actor
150
151# Secrets (referenced from actor.json via "@mySecret")
152apify secrets add <name> <value>       # Store a secret locally; uploaded on push
153apify secrets ls                       # List stored secret keys
154
155# Direct API access
156apify api <endpoint>                   # Send an authenticated HTTP request to Apify API
157
158# Help
159apify help                             # List all commands
160apify <command> --help                 # Get help for a specific command
161```
162
163Note: If no dedicated Actor exists for your target, search Apify Store for community options with `apify actors search "<query>" --user-agent <your-agent-name>` before building from scratch.
164
165Tip: Inside a running Actor, prefer the SDK (`Actor.get_input()`, `Actor.push_data()`, `Actor.set_value()`) over the equivalent `apify actor` runtime subcommands.
166
167## Apify Platform Environment
168
169When the Actor runs on the Apify platform, the API token is automatically available via the `APIFY_TOKEN` environment variable (note: the variable is `APIFY_TOKEN`, not `APIFY_API_TOKEN`). The Apify SDK reads it automatically, so you do not need to pass it explicitly. Locally, run `apify login` once and the SDK will use your stored credentials.
170
171## Safety and Permissions
172
173Allowed without prompt:
174
175- read files with `Actor.get_value()`
176- push data with `Actor.push_data()`
177- set values with `Actor.set_value()`
178- enqueue requests to RequestQueue
179- run locally with `apify run`
180
181Ask first:
182
183- npm/pip package installations
184- apify push (deployment to cloud)
185- proxy configuration changes (requires paid plan)
186- Dockerfile changes affecting builds
187- deleting datasets or key-value stores
188
189## Project Structure
190
191.actor/
192├── actor.json # Actor config: name, version, env vars, runtime settings
193├── input_schema.json # Input validation & Console form definition
194└── output_schema.json # Specifies where an Actor stores its output
195src/
196└── main.js # Actor entry point and orchestrator
197storage/ # Local-only storage for development (NOT synced to Cloud)
198├── datasets/ # Output items (JSON objects)
199├── key_value_stores/ # Files, config, INPUT
200└── request_queues/ # Pending crawl requests
201Dockerfile # Container image definition
202AGENTS.md # AI agent instructions (this file)
203
204## Local vs Cloud Storage
205
206When running locally with `apify run`, the Apify SDK emulates Cloud storage APIs using the local `storage/` directory. This local storage behaves differently from Cloud storage:
207
208- **Local storage is NOT persistent** - The `storage/` directory is meant for local development and testing only. Data stored there (datasets, key-value stores, request queues) exists only on your local disk.
209- **Local storage is NOT automatically pushed to Apify Console** - Running `apify run` does not upload any storage data to the Apify platform. The data stays local.
210- **Each local run may overwrite previous data** - The local `storage/` directory is reused between runs, but this is local-only behavior, not Cloud persistence.
211- **Cloud storage only works when running on Apify platform** - After deploying with `apify push` and running the Actor in the Cloud, storage calls (`Actor.push_data()`, `Actor.set_value()`, etc.) interact with real Apify Cloud storage, which is then visible in the Apify Console.
212- **To verify Actor output, deploy and run in Cloud** - Do not rely on local `storage/` contents as proof that data will appear in the Apify Console. Always test by deploying (`apify push`) and running the Actor on the platform.
213
214## Actor Input Schema
215
216The input schema defines the input parameters for an Actor. It's a JSON object comprising various field types supported by the Apify platform.
217
218### Structure
219
220```json
221{
222    "title": "<INPUT-SCHEMA-TITLE>",
223    "type": "object",
224    "schemaVersion": 1,
225    "properties": {/* define input fields here */},
226    "required": []
227}
228```
229
230### Example
231
232```json
233{
234    "title": "E-commerce Product Scraper Input",
235    "type": "object",
236    "schemaVersion": 1,
237    "properties": {
238        "startUrls": {
239            "title": "Start URLs",
240            "type": "array",
241            "description": "URLs to start scraping from (category pages or product pages)",
242            "editor": "requestListSources",
243            "default": [{ "url": "https://example.com/category" }],
244            "prefill": [{ "url": "https://example.com/category" }]
245        },
246        "followVariants": {
247            "title": "Follow Product Variants",
248            "type": "boolean",
249            "description": "Whether to scrape product variants (different colors, sizes)",
250            "default": true
251        },
252        "maxRequestsPerCrawl": {
253            "title": "Max Requests per Crawl",
254            "type": "integer",
255            "description": "Maximum number of pages to scrape (0 = unlimited)",
256            "default": 1000,
257            "minimum": 0
258        },
259        "proxyConfiguration": {
260            "title": "Proxy Configuration",
261            "type": "object",
262            "description": "Proxy settings for anti-bot protection",
263            "editor": "proxy",
264            "default": { "useApifyProxy": false }
265        },
266        "locale": {
267            "title": "Locale",
268            "type": "string",
269            "description": "Language/country code for localized content",
270            "default": "cs",
271            "enum": ["cs", "en", "de", "sk"],
272            "enumTitles": ["Czech", "English", "German", "Slovak"]
273        }
274    },
275    "required": ["startUrls"]
276}
277```
278
279## Actor Output Schema
280
281The Actor output schema builds upon the schemas for the dataset and key-value store. It specifies where an Actor stores its output and defines templates for accessing that output. Apify Console uses these output definitions to display run results.
282
283### Structure
284
285```json
286{
287    "actorOutputSchemaVersion": 1,
288    "title": "<OUTPUT-SCHEMA-TITLE>",
289    "properties": {/* define your outputs here */}
290}
291```
292
293### Example
294
295```json
296{
297    "actorOutputSchemaVersion": 1,
298    "title": "Output schema of the files scraper",
299    "properties": {
300        "files": {
301            "type": "string",
302            "title": "Files",
303            "template": "{{links.apiDefaultKeyValueStoreUrl}}/keys"
304        },
305        "dataset": {
306            "type": "string",
307            "title": "Dataset",
308            "template": "{{links.apiDefaultDatasetUrl}}/items"
309        }
310    }
311}
312```
313
314### Output Schema Template Variables
315
316- `links` (object) - Contains quick links to most commonly used URLs
317- `links.publicRunUrl` (string) - Public run url in format `https://console.apify.com/view/runs/:runId`
318- `links.consoleRunUrl` (string) - Console run url in format `https://console.apify.com/actors/runs/:runId`
319- `links.apiRunUrl` (string) - API run url in format `https://api.apify.com/v2/actor-runs/:runId`
320- `links.apiDefaultDatasetUrl` (string) - API url of default dataset in format `https://api.apify.com/v2/datasets/:defaultDatasetId`
321- `links.apiDefaultKeyValueStoreUrl` (string) - API url of default key-value store in format `https://api.apify.com/v2/key-value-stores/:defaultKeyValueStoreId`
322- `links.containerRunUrl` (string) - URL of a webserver running inside the run in format `https://<containerId>.runs.apify.net/`
323- `run` (object) - Contains information about the run same as it is returned from the `GET Run` API endpoint
324- `run.defaultDatasetId` (string) - ID of the default dataset
325- `run.defaultKeyValueStoreId` (string) - ID of the default key-value store
326
327## Dataset Schema Specification
328
329The dataset schema defines how your Actor's output data is structured, transformed, and displayed in the Output tab in the Apify Console.
330
331### Example
332
333Consider an example Actor that calls `Actor.pushData()` to store data into dataset:
334
335```python
336# Dataset push example (Python)
337import asyncio
338from datetime import datetime
339from apify import Actor
340
341async def main():
342    await Actor.init()
343
344    # Actor code
345    await Actor.push_data({
346        'numericField': 10,
347        'pictureUrl': 'https://www.google.com/images/branding/googlelogo/2x/googlelogo_color_92x30dp.png',
348        'linkUrl': 'https://google.com',
349        'textField': 'Google',
350        'booleanField': True,
351        'dateField': datetime.now().isoformat(),
352        'arrayField': ['#hello', '#world'],
353        'objectField': {},
354    })
355
356    # Exit successfully
357    await Actor.exit()
358
359if __name__ == '__main__':
360    asyncio.run(main())
361```
362
363To set up the Actor's output tab UI, reference a dataset schema file in `.actor/actor.json`:
364
365```json
366{
367    "actorSpecification": 1,
368    "name": "book-library-scraper",
369    "title": "Book Library Scraper",
370    "version": "1.0.0",
371    "storages": {
372        "dataset": "./dataset_schema.json"
373    }
374}
375```
376
377Then create the dataset schema in `.actor/dataset_schema.json`:
378
379```json
380{
381    "actorSpecification": 1,
382    "fields": {},
383    "views": {
384        "overview": {
385            "title": "Overview",
386            "transformation": {
387                "fields": [
388                    "pictureUrl",
389                    "linkUrl",
390                    "textField",
391                    "booleanField",
392                    "arrayField",
393                    "objectField",
394                    "dateField",
395                    "numericField"
396                ]
397            },
398            "display": {
399                "component": "table",
400                "properties": {
401                    "pictureUrl": {
402                        "label": "Image",
403                        "format": "image"
404                    },
405                    "linkUrl": {
406                        "label": "Link",
407                        "format": "link"
408                    },
409                    "textField": {
410                        "label": "Text",
411                        "format": "text"
412                    },
413                    "booleanField": {
414                        "label": "Boolean",
415                        "format": "boolean"
416                    },
417                    "arrayField": {
418                        "label": "Array",
419                        "format": "array"
420                    },
421                    "objectField": {
422                        "label": "Object",
423                        "format": "object"
424                    },
425                    "dateField": {
426                        "label": "Date",
427                        "format": "date"
428                    },
429                    "numericField": {
430                        "label": "Number",
431                        "format": "number"
432                    }
433                }
434            }
435        }
436    }
437}
438```
439
440### Structure
441
442```json
443{
444    "actorSpecification": 1,
445    "fields": {},
446    "views": {
447        "<VIEW_NAME>": {
448            "title": "string (required)",
449            "description": "string (optional)",
450            "transformation": {
451                "fields": ["string (required)"],
452                "unwind": ["string (optional)"],
453                "flatten": ["string (optional)"],
454                "omit": ["string (optional)"],
455                "limit": "integer (optional)",
456                "desc": "boolean (optional)"
457            },
458            "display": {
459                "component": "table (required)",
460                "properties": {
461                    "<FIELD_NAME>": {
462                        "label": "string (optional)",
463                        "format": "text|number|date|link|boolean|image|array|object (optional)"
464                    }
465                }
466            }
467        }
468    }
469}
470```
471
472**Dataset Schema Properties:**
473
474- `actorSpecification` (integer, required) - Specifies the version of dataset schema structure document (currently only version 1)
475- `fields` (JSONSchema object, required) - Schema of one dataset object (use JsonSchema Draft 2020-12 or compatible)
476- `views` (DatasetView object, required) - Object with API and UI views description
477
478**DatasetView Properties:**
479
480- `title` (string, required) - Visible in UI Output tab and API
481- `description` (string, optional) - Only available in API response
482- `transformation` (ViewTransformation object, required) - Data transformation applied when loading from Dataset API
483- `display` (ViewDisplay object, required) - Output tab UI visualization definition
484
485**ViewTransformation Properties:**
486
487- `fields` (string[], required) - Fields to present in output (order matches column order)
488- `unwind` (string[], optional) - Deconstructs nested children into parent object
489- `flatten` (string[], optional) - Transforms nested object into flat structure
490- `omit` (string[], optional) - Removes specified fields from output
491- `limit` (integer, optional) - Maximum number of results (default: all)
492- `desc` (boolean, optional) - Sort order (true = newest first)
493
494**ViewDisplay Properties:**
495
496- `component` (string, required) - Only `table` is available
497- `properties` (Object, optional) - Keys matching `transformation.fields` with ViewDisplayProperty values
498
499**ViewDisplayProperty Properties:**
500
501- `label` (string, optional) - Table column header
502- `format` (string, optional) - One of: `text`, `number`, `date`, `link`, `boolean`, `image`, `array`, `object`
503
504## Key-Value Store Schema Specification
505
506The key-value store schema organizes keys into logical groups called collections for easier data management.
507
508### Example
509
510Consider an example Actor that calls `Actor.setValue()` to save records into the key-value store:
511
512```python
513# Key-Value Store set example (Python)
514import asyncio
515from apify import Actor
516
517async def main():
518    await Actor.init()
519
520    # Actor code
521    await Actor.set_value('document-1', 'my text data', content_type='text/plain')
522
523    image_id = '123'          # example placeholder
524    image_buffer = b'...'     # bytes buffer with image data
525    await Actor.set_value(f'image-{image_id}', image_buffer, content_type='image/jpeg')
526
527    # Exit successfully
528    await Actor.exit()
529
530if __name__ == '__main__':
531    asyncio.run(main())
532```
533
534To configure the key-value store schema, reference a schema file in `.actor/actor.json`:
535
536```json
537{
538    "actorSpecification": 1,
539    "name": "data-collector",
540    "title": "Data Collector",
541    "version": "1.0.0",
542    "storages": {
543        "keyValueStore": "./key_value_store_schema.json"
544    }
545}
546```
547
548Then create the key-value store schema in `.actor/key_value_store_schema.json`:
549
550```json
551{
552    "actorKeyValueStoreSchemaVersion": 1,
553    "title": "Key-Value Store Schema",
554    "collections": {
555        "documents": {
556            "title": "Documents",
557            "description": "Text documents stored by the Actor",
558            "keyPrefix": "document-"
559        },
560        "images": {
561            "title": "Images",
562            "description": "Images stored by the Actor",
563            "keyPrefix": "image-",
564            "contentTypes": ["image/jpeg"]
565        }
566    }
567}
568```
569
570### Structure
571
572```json
573{
574    "actorKeyValueStoreSchemaVersion": 1,
575    "title": "string (required)",
576    "description": "string (optional)",
577    "collections": {
578        "<COLLECTION_NAME>": {
579            "title": "string (required)",
580            "description": "string (optional)",
581            "key": "string (conditional - use key OR keyPrefix)",
582            "keyPrefix": "string (conditional - use key OR keyPrefix)",
583            "contentTypes": ["string (optional)"],
584            "jsonSchema": "object (optional)"
585        }
586    }
587}
588```
589
590**Key-Value Store Schema Properties:**
591
592- `actorKeyValueStoreSchemaVersion` (integer, required) - Version of key-value store schema structure document (currently only version 1)
593- `title` (string, required) - Title of the schema
594- `description` (string, optional) - Description of the schema
595- `collections` (Object, required) - Object where each key is a collection ID and value is a Collection object
596
597**Collection Properties:**
598
599- `title` (string, required) - Collection title shown in UI tabs
600- `description` (string, optional) - Description appearing in UI tooltips
601- `key` (string, conditional\*) - Single specific key for this collection
602- `keyPrefix` (string, conditional\*) - Prefix for keys included in this collection
603- `contentTypes` (string[], optional) - Allowed content types for validation
604- `jsonSchema` (object, optional) - JSON Schema Draft 07 format for `application/json` content type validation
605
606\*Either `key` or `keyPrefix` must be specified for each collection, but not both.
607
608## Actor README
609
610**Always generate a README.md file as part of Actor development.** The README is the Actor's public landing page on Apify Store - it serves as SEO, first impression, documentation, and support page combined.
611
612### Required: Generate README automatically
613
614When building an Actor, always create a `README.md` in the project root. Do not wait for the user to ask for it. The README is a critical part of a complete Actor.
615
616### README structure
617
618Write in Markdown. Use H2 (`##`) for main sections (these become the table of contents) and H3 (`###`) for subsections. Do not use H1 - the Actor name is automatically the H1. Aim for at least 300 words.
619
620Include these sections in order:
621
6221. **What does [Actor name] do?** - 2-3 sentences explaining what it does, what data it extracts, and how to try it. Link to the target website. Mention Apify platform advantages (API access, scheduling, integrations, proxy rotation, monitoring).
6232. **Why use [Actor name]?** - Business use cases and benefits.
6243. **How to use [Actor name]** - Numbered step-by-step tutorial. Keep it simple and reassuring.
6254. **Input** - Describe input fields. Reference the Input tab. Optionally include a screenshot or JSON example of the input schema.
6265. **Output** - Show a simplified JSON output example. Mention "You can download the dataset in various formats such as JSON, HTML, CSV, or Excel."
6276. **Data table** - If the Actor extracts data, include a table of the main data fields it outputs.
6287. **Pricing / Cost estimation** - Set expectations on cost. Mention free tier limits if applicable. Frame as "How much does it cost to scrape [target site]?"
6298. **Tips or Advanced options** - How to optimize runs, limit compute units, improve speed or accuracy.
6309. **FAQ, disclaimers, and support** - Legality disclaimer for scrapers, known limitations, link to Issues tab for feedback, mention custom solution availability.
631
632### README best practices
633
634- Write SEO-friendly headings with relevant keywords (e.g., "How to scrape [site] data" not just "Tutorial")
635- Bold the most important words in the intro
636- The first 25% of the README matters most - front-load the value proposition
637- Match the tone to the target audience: simple language for no-code users, technical details for developers
638- Include a JSON output example showing 1-2 representative items
639- Reference these top Actors for README best practices: https://apify.com/apify/instagram-scraper and https://apify.com/compass/crawler-google-places
640- Embed YouTube video URLs on their own line (Apify Console auto-renders them)
641- Use HTML for image sizing if needed; CSS is not supported
642
643## MCP Tools
644
645### Apify MCP
646
647If the Apify MCP server is configured, use these tools for documentation:
648
649- `search-apify-docs` - Search documentation
650- `fetch-apify-docs` - Get full doc pages
651
652Otherwise, reference: `@https://mcp.apify.com/`
653
654### Playwright MCP (debugging)
655
656The Playwright MCP server is a useful tool for debugging Actors that interact with the web - it lets the agent drive a real browser to inspect pages, capture selectors, and reproduce issues.
657
658Install with the Claude Code CLI:
659
660```bash
661claude mcp add playwright npx @playwright/mcp@latest
662```
663
664Or add it manually to your MCP config:
665
666```json
667{
668    "mcpServers": {
669        "playwright": {
670            "command": "npx",
671            "args": ["@playwright/mcp@latest"]
672        }
673    }
674}
675```
676
677## Resources
678
679- [docs.apify.com/llms.txt](https://docs.apify.com/llms.txt) - Quick reference
680- [docs.apify.com/llms-full.txt](https://docs.apify.com/llms-full.txt) - Complete docs
681- [crawlee.dev](https://crawlee.dev) - Crawlee documentation
682- [whitepaper.actor](https://raw.githubusercontent.com/apify/actor-whitepaper/refs/heads/master/README.md) - Complete Actor specification

CLAUDE.md

1@AGENTS.md

Dockerfile

# First, specify the base Docker image.
# You can see the Docker images from Apify at https://hub.docker.com/r/apify/.
# You can also use any other image from Docker Hub.
FROM apify/actor-python:3.14

USER myuser

# Second, copy just requirements.txt into the Actor image,
# since it should be the only file that affects the dependency install in the next step,
# in order to speed up the build
COPY --chown=myuser:myuser requirements.txt ./

# Install the packages specified in requirements.txt,
# Print the installed Python version, pip version
# and all installed packages with their versions for debugging
RUN echo "Python version:" \
 && python --version \
 && echo "Pip version:" \
 && pip --version \
 && echo "Installing dependencies:" \
 && pip install -r requirements.txt \
 && echo "All installed Python packages:" \
 && pip freeze

# Next, copy the remaining files and directories with the source code.
# Since we do this after installing the dependencies, quick build will be really fast
# for most source file changes.
COPY --chown=myuser:myuser . ./

# Use compileall to ensure the runnability of the Actor Python code.
RUN python -m compileall -q my_actor/

# Specify how to launch the source code of your Actor.
CMD ["python", "-m", "my_actor"]

requirements.txt

1# Feel free to add your Python dependencies below. For formatting guidelines, see:
2# https://pip.pypa.io/en/latest/reference/requirements-file-format/
3
4apify >= 3.0.0, < 4.0.0

Dev.to Scraper - Articles by Tag & Author

logiover/devto-articles-scraper

Scrape Dev.to articles by tag or author. A no-API-key Dev.to API alternative; export article data to CSV and JSON. Forem articles scraper, no login.

Logiover

Dev.to Scraper - Articles & Developer Content

pear_fight/devto-scraper

Scrape Dev.to articles, author profiles, tags & comments. Extract titles, content, reactions, reading time, dates. Search by topic, tag, or author. Track developer trends. Pay per result. Export JSON/CSV.

Harald

Dev.to Scraper

leftwinglautus/dev-to-scraper

Scrape articles from Dev.to via the official Dev.to API. Fetch latest articles, articles by tag, or articles by user.

Moeeze Hassan

DEV.to Scraper — Articles by Tag & Author

devilscrapes/dev-to-articles-scraper

Pull articles from DEV.to by tag, author, or feed via the official dev.to v1 API — title, body (Markdown), author, tags, reading time, reactions, comment count, publish timestamp — export to JSON or CSV. Free, no key needed for read access.

DevilScrapes

DEV / Forem Intel: Articles, Dev Leads & Tags

obsidian937/devto-intel

Turn DEV (dev.to) into structured B2B data. Monitor articles by tag, author or top window, build developer and author lead lists with socials and location, or browse the tag catalog. Official public Forem API, no proxy, no auth. Delta mode for recurring monitoring.

Obsidian IT Consulting SRL

Devto Scraper

fortuitous_pirate/devto-scraper

Scrape developer articles from Dev.to: trending posts, articles by tag, top posts, and author profiles. Free public API.

Fortuitous Pirate

DEV.to Scraper

plantane/devto-scraper

Scrape articles, search results, and tags from DEV.to (dev.to). Supports latest, top, by tag, by username, search, and tag listing. Uses the official Forem API for fast, reliable data extraction.

Daniel

Dev.to Articles Scraper

ef12/devto-articles-scraper

Scrape Dev.to articles by tag, username, or trending. Get titles, descriptions, authors, tags, reactions, reading time, and comment counts via the free Dev.to API.

Daniel Wilson

DEV.to Article Scraper

dami_studio/devto-scraper

Scrape DEV.to articles via the public Forem API — by tag, author, or sort (top/latest/rising). Get clean structured data: title, author, tags, reactions, comments, reading time, cover image, and optional full body. No key, no login.

Dami's Studio

Dev.to Article Scraper – DEV Community Posts & Data

ninhothedev/devto-scraper

$0.5/1K 🔥 Fast DEV.to scraper! Articles — title, author, tags, reactions, comments & content. JSON, CSV, Excel or API in seconds. Search by tag or user & pull thousands of dev posts for content & trend research ⚡

ninhothedev

Dev.to Articles & Developer Leads Scraper

.actor/actor.json

my_actor/__init__.py

my_actor/__main__.py

my_actor/main.py

my_actor/py.typed

.dockerignore

.gitignore

AGENTS.md

CLAUDE.md

Dockerfile

requirements.txt

You might also like

Dev.to Scraper - Articles by Tag & Author

Dev.to Scraper - Articles & Developer Content

Dev.to Scraper

DEV.to Scraper — Articles by Tag & Author

DEV / Forem Intel: Articles, Dev Leads & Tags

Devto Scraper

DEV.to Scraper

Dev.to Articles Scraper

DEV.to Article Scraper

Dev.to Article Scraper – DEV Community Posts & Data

.actor/actor.json

my_actor/__init__.py

my_actor/__main__.py

my_actor/main.py

my_actor/py.typed

.dockerignore

.gitignore

AGENTS.md

CLAUDE.md

Dockerfile

requirements.txt

my_actor/init.py

my_actor/main.py

my_actor/init.py

my_actor/main.py