⚡ DEV.to Article Search Scraper (Fast & AI-Ready)

Extract, clean, and structure DEV.to articles in bulk using parallel full-text search indexing. Engineered specifically for AI Agents, RAG Ingestion Pipelines, LLM Training Workflows, Content Researchers, and DevRel Strategists.

---

🎯 Quick Overview

The DEV.to Article Search Scraper queries DEV Community's deep search index to extract articles, author metadata, reading metrics, body text, tags, and engagement scores—without requiring API keys, login credentials, or complex setups.

• 🚀 Bulk Keyword & Tag Search: Pass multiple search queries, topics, or hashtags to fetch matching articles concurrently.
• 🧹 Linear & Clean Data Output: Automatically strips Algolia markup (<em>), flattens nested JSON user objects, and normalizes date structures for frictionless export.
• 🤖 AI Agent Native (x402 Protocol): Native support for Apify’s x402 agentic payment protocol (USDC micropayments on Base) for autonomous AI workflows.
• 💰 Pay-Per-Event (PPE): Extremely cost-effective pricing—pay strictly for the article results you fetch.

---

💡 Key Use Cases

• RAG Pipeline & Vector DB Ingestion: Extract clean, unformatted plain-text developer articles for vector embeddings (Pinecone, Weaviate, Qdrant).
• Developer Relations & Tech Trend Monitoring: Track mentions, developer sentiment, and popular tutorials across tags like #webdev, #python, #ai, and #react.
• SEO & Content Strategy Analysis: Analyze top-performing articles on DEV Community by reading time, reaction counts, and tag distribution.
• LLM Fine-Tuning Datasets: Build high-quality tech datasets for fine-tuning open-source code models and technical chatbots.

---

📥 Input Configuration

Configure your inputs via the Apify Console UI or JSON payload:

Field	Key	Type	Default	Description
Search Queries	queries	Array[String]	Required	Keywords, hashtags, or topic strings (e.g., ["react 19", "python", "ai agents"]).
Max Results	max_results	Integer	10	Maximum number of articles to extract per query.

Example Input JSON

{
  "queries": ["react 19", "twitter algorithm"],
  "max_results": 20
}

---

📤 Output Dataset Format

The dataset returns flattened, structured JSON records ready for immediate conversion to CSV, Excel, pandas DataFrames, or direct vector database ingestion.

[
  {
    "searchQuery": "example-query",
    "id": "1234567",
    "title": "How to Build Modern Web Applications with Example Framework",
    "url": "https://dev.to/johndoe/how-to-build-modern-web-applications-abc12",
    "publishedAtISO": "2026-01-15T12:00:00Z",
    "readablePublishDate": "Jan 15, 2026",
    "timestamp": 1768478400,
    "readingTimeMinutes": 5,
    "score": 42,
    "commentsCount": 8,
    "publicReactionsCount": 25,
    "positiveReactionsCount": 25,
    "reactionCategories": [
      {
        "slug": "like",
        "count": 15
      },
      {
        "slug": "unicorn",
        "count": 10
      }
    ],
    "tags": [
      "webdev",
      "javascript",
      "tutorial",
      "programming"
    ],
    "featured": false,
    "mainImage": "https://example.com/images/cover-image.png",
    "authorId": "9876543",
    "authorName": "John Doe",
    "authorUsername": "johndoe",
    "authorProfileURL": "https://dev.to/johndoe",
    "authorProfileImage": "https://example.com/images/john-doe-avatar.jpg",
    "bodyText": "Building modern web applications requires a solid understanding of fundamental concepts.\n\n\n\nIn this article, we will explore the core principles of software architecture, data management, and performance optimization.\n\n\n\n\n  \n  \n  Getting Started\n\n\nFirst, set up your development environment by installing the necessary dependencies. Make sure your runtime version is up to date.\n\n\n\nNext, configure your project settings and define your application pipeline.\n\n\n\nThis setup ensures scalable and maintainable code for your team.",
    "bodyHtml": "<p>Building modern web applications requires a solid understanding of fundamental concepts.</p>\n\n<p>In this article, we will explore the core principles of software architecture, data management, and performance optimization.</p>\n\n<h2>\n  <a name=\"getting-started\" href=\"#getting-started\">\n  </a>\n  Getting Started\n</h2>\n\n<p>First, set up your development environment by installing the necessary dependencies. Make sure your runtime version is up to date.</p>\n\n<p>Next, configure your project settings and define your application pipeline.</p>\n\n<p>This setup ensures scalable and maintainable code for your team.</p>"
  }
]

---

🤖 Executing via Autonomous AI Agents (x402 Protocol)

This Actor supports direct autonomous execution by AI agents using the HTTP 402 protocol with USDC on the Base network:

npx awal x402 pay '[https://api.apify.com/v2/actors/devil_port369-owner~devto-article-search-scraper/run-sync-get-dataset-items](https://api.apify.com/v2/actors/devil_port369-owner~devto-article-search-scraper/run-sync-get-dataset-items)' \
  --data '{"queries": ["react 19", "ai agents"], "max_results": 10}'

---

❓ Frequently Asked Questions (FAQ)

How to extract DEV.to articles without an API key?

This Actor uses direct full-text search endpoints to scrape public DEV.to posts and metadata without requiring a DEV API key or user authentication.

Can I export DEV Community data to CSV or Excel?

Yes. Apify automatically enables data downloads in JSON, CSV, Excel, XML, or HTML table formats directly from the run dataset panel or via API.

Is this DEV.to scraper suitable for building LLM training datasets?

Yes. Output items are flattened and pre-cleaned (stripping HTML tags and search markup), making them immediately usable for RAG chunking and LLM training.