Dataset Sharder

Split one large Apify dataset into evenly sized child datasets without downloading, scripting, or re-uploading records.

Dataset Sharder creates clean batch boundaries for downstream Actors, API limits, parallel processing, per-client deliveries, manual review queues, and automation tools. Every child dataset contains the original records unchanged, while the default dataset provides a manifest with IDs, URLs, source-index ranges, and completion status.

What it does

• Reads any Apify dataset you authorize.
• Splits records in their original order by a configurable maximum shard size.
• Creates up to nine per-run child datasets (shard1 through shard9).
• Copies each original record unchanged for immediate downstream use.
• Returns a manifest with dataset IDs, API URLs, item counts, and source ranges.
• Stops safely at the run spending limit and marks incomplete shards explicitly.

The source dataset is read-only and is never modified.

Example

{
  "sourceDatasetId": "YOUR_DATASET_ID",
  "itemsPerShard": 10000,
  "maxShards": 9,
  "maxItems": 100000
}

A 23,500-record source with itemsPerShard: 10000 creates:

• shard1: source indexes 0–9,999
• shard2: source indexes 10,000–19,999
• shard3: source indexes 20,000–23,499

Output

The default dataset is the shard manifest. Each row contains:

• shard number and alias
• output dataset ID and API URL
• first and last source indexes
• planned and copied record counts
• completion status

Each aliased child dataset contains the original source objects without wrappers or added fields. The OUTPUT key-value-store record contains aggregate totals and the full manifest.

Pricing

Pay per copied record:

• Actor start: Apify's standard low-cost start event
• sharded-record: $0.00001 per copied record ($0.01 per 1,000)

Records are charged before being written. If a spending limit is reached, only paid records are copied and the manifest clearly marks the partial shard.

Limits and safety

• Up to nine child datasets per run (Apify's declared-storage limit leaves one dataset for the manifest).
• Up to 1,000,000 source records, controlled by maxItems.
• The run fails before copying if the selected shard size would require more than maxShards datasets.
• No proxy, browser, LLM, or external API is used.
• Runs under Apify limited permissions and accesses only the selected source dataset plus its own run storages.

Local development

npm install
npm test
npm start