🇮🇳 India Government Data (data.gov.in) Scraper

Overview

India Government Data Scraper turns any dataset on data.gov.in into clean, structured JSON via the platform's official OGD API. Foreign trade export and import numbers, mandi commodity prices, census records, agriculture, health, energy, transport. If it lives on data.gov.in, this actor pulls it. Give it a resource ID, add optional field filters, and receive flat records ready for your pipeline, dashboard, or LLM.

Because it wraps the government's own endpoint, there is no anti-bot fragility, no headless browser, and no rate-limit war. Bring the free key you can sign up for in a minute on data.gov.in, or use the shared sample key for testing.

Reliability posture: blocked, empty, or failed runs are never charged. You only pay for a record that was actually written to the dataset.

✅ No login | ✅ No scraping fragility | ✅ Any of thousands of datasets | ✅ MCP-ready for AI agents

Features

Any dataset supported. Trade, prices, census, agriculture, health, transport, energy. Server-side field filters. Narrow by state, commodity, year, or any exposed field. Full pagination. Sample a slice or pull the entire dataset in one run. Official government API. Stable, legitimate, and free of anti-bot cat and mouse. Bring your own free key. Shared sample key for testing, personal key for production.

How it works

India publishes an enormous open-data catalogue on data.gov.in through its OGD platform, but the API is per-dataset, returns inconsistent envelopes, paginates by offset, and requires an API key. This actor wraps all of that behind one input: paste the resource ID of the dataset you want, optionally add filters, and receive flat JSON records with two audit fields appended, _resource_id and _scraped_at.

To find a resource ID, open a dataset on data.gov.in, click its API tab, and copy the UUID from the URL. Drop it into resourceId and run. A final summary record at the end of every run reports dataset title, records returned, and total records available.

🧾 Input configuration

{
  "resourceId": "9ef84268-d588-465a-a308-a864a43d0070",
  "filters": ["state=Maharashtra", "commodity=Onion"],
  "maxResults": 500,
  "apiKey": "your-free-data-gov-in-key"
}

📤 Output format

{
  "state": "Keralam",
  "district": "Idukki",
  "market": "Kattappana Market",
  "commodity": "Water Melon",
  "variety": "Other",
  "grade": "Medium",
  "arrival_date": "15/07/2026",
  "min_price": 2000,
  "max_price": 2600,
  "modal_price": 2300,
  "_resource_id": "9ef84268-d588-465a-a308-a864a43d0070",
  "_scraped_at": "2026-07-15T04:14:45.668Z"
}

Every record follows the dataset's native schema plus the two audit fields:

Field	Description
🆔 _resource_id	The data.gov.in resource UUID for the source dataset.
🕒 _scraped_at	ISO timestamp when the record was pulled.
🏷️ (dataset fields)	Every native field the dataset exposes, preserved as returned.

Field names and types vary by dataset. A mandi-price dataset returns commodity and price columns; a foreign-trade dataset returns HS code, country, and value. The actor never renames or drops native fields.

💼 Common use cases

Trade and economics research Pull DGCIS or DGFT foreign-trade series to model export and import trends by commodity and country. Combine multiple resource IDs for time-series analysis without touching a single HTML page.

Commodity price monitoring Feed daily mandi arrivals and modal prices into a dashboard for buyers, exporters, and agri-analytics teams. Filter by state and commodity to keep runs cheap.

Public-sector analytics Ingest census, health, and transport datasets for civic dashboards, policy research, and academic studies. Ideal for anyone building on top of India's open-data infrastructure.

AI agents and LLM tools Expose the actor over MCP so Claude or ChatGPT can answer natural-language questions like "what were onion prices in Nashik last week" against live government data.

🚀 Getting started

1. Open the actor on Apify and paste your target resourceId. Find IDs on data.gov.in under the dataset's API tab.
2. Add filters as field=value strings to narrow the slice (optional).
3. Set maxResults to cap volume and cost, or leave the default of 1000.
4. Add your free data.gov.in API key for production, or leave blank to use the shared sample key.
5. Click Start, then download as JSON, CSV, or Excel, or pull via API or MCP.

FAQ

Do I need an API key? A free personal key from data.gov.in is recommended for production runs (sign up instantly under My Account). The actor falls back to a shared sample key for testing, which is heavily rate-limited.

Can I get India's export and import data? Yes. The DGCIS and DGFT foreign-trade datasets are published on data.gov.in. Search "foreign trade" or "export import," copy the resource ID, and pass it in.

Which datasets are supported? All of them. data.gov.in hosts hundreds of thousands of resources across every ministry, and this actor works against any of them.

Is this legal and reliable? Yes. It uses the government's own official OGD API. No scraping of rendered pages, no anti-bot circumvention, no ToS grey zone.

Use in Claude, ChatGPT & any MCP agent

https://mcp.apify.com/?tools=themineworks/india-data-gov-scraper

Or call it programmatically with the Apify client:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });

const run = await client.actor('themineworks/india-data-gov-scraper').call({
  resourceId: '9ef84268-d588-465a-a308-a864a43d0070',
  filters: ['state=Maharashtra', 'commodity=Onion'],
  maxResults: 500,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

🛠️ Complete your India data pipeline

Pair this with the rest of the suite for a complete open-data stack:

• RAG Website Crawler: crawl ministry websites for context that isn't published as a dataset.
• Google Maps Leads Scraper: enrich mandi and market records with on-ground business details.
• B2B Leads Finder: turn government contractor and exporter lists into contactable leads.

Questions or need a custom field set? Reach out through the Apify profile.