# REST API to Dataset: Auto Pagination, Auth & Retries (`genesis.labs.co/rest-api-to-dataset`) Actor

Fetch any JSON REST or GraphQL API into a clean dataset: auto pagination (next links, cursors, pages, offsets), API key auth, retries, rate limits, records extraction and per-row enrichment. Failed requests are free.

- **URL**: https://apify.com/genesis.labs.co/rest-api-to-dataset.md
- **Developed by:** [Nicholas Powers](https://apify.com/genesis.labs.co) (community)
- **Categories:** Developer tools, Integrations, Automation
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.80 / 1,000 api requests

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## REST API to Dataset: Auto Pagination, Auth & Retries

Turn **any JSON REST or GraphQL API into a clean Apify dataset** in one run. Paste an endpoint, add your API key if it needs one, and the Actor follows every page, splits the response into one row per record, retries rate limits and server errors, and gives you CSV, Excel or JSON. **Failed requests are free.**

It works with your own APIs, SaaS APIs (with your key), open data APIs and GraphQL endpoints. It is also an **API enrichment** tool: feed it a dataset and it calls the API once per row (`https://api.example.com/users/{{id}}`), adding the results to each row.

### What it does

- 🔁 **Auto pagination** that detects the common styles: `Link: rel="next"` headers (GitHub and others), next URLs in the body (`next`, `links.next`, `_links.next.href`, `paging.next`, `@odata.nextLink`, `info.next`...), cursors and page tokens (`next_cursor`, `meta.next_cursor`, `response_metadata.next_cursor`, `nextPageToken`), `has_more` + `starting_after`, and `current_page` / `last_page` counts. You can also choose **page number**, **offset + limit**, **cursor** (in the query or in a JSON body for GraphQL) or **next URL at a path** yourself.
- 📦 **Records extraction**: finds the list of records automatically (`data`, `results`, `items`, `hits.hits`, `message.items`, GraphQL `edges`/`nodes`...), or you set a dot path like `data.items`. Each record becomes one dataset row.
- 🔐 **Auth**: Bearer token, API key in a header or query string, or Basic auth. The key is stored as an **encrypted secret input**, hidden in logs and summaries, and only sent to the host of your API URL, never to other hosts that appear in pagination links or redirects.
- 🧰 **Reliable fetching**: retries with exponential backoff for timeouts, connection resets, `408`, `425`, `429` and `5xx`; honors `Retry-After`; one global **requests-per-second** limit across all parallel requests; per-request timeout; response size cap; redirects handled safely.
- 🧩 **Enrich a dataset**: one request per input row with `{{field}}` placeholders in the URL, query, headers and JSON body. Copy fields such as `id` or `email` from the input row into the output to join results back.
- ✂️ **Shape the output**: keep only some fields, rename fields (`price.amount` → `price`), flatten nested objects for spreadsheets, add the source URL and page number.
- 📄 **Per-response mode**: one item per HTTP response with status, final URL, timing, headers and the raw body (works for XML, HTML, text or any other format).
- 🛡️ **Safe by default**: private, internal and cloud-metadata addresses are blocked (including via DNS or redirects), and the Actor runs with **limited permissions**, so it can only read the dataset you pick.
- 💸 **Failures are free** and the Actor stops before going over your **Max cost per run**.

### Pricing

Pay per event. Compute and storage are included.

| Event | Free & Bronze | Silver | Gold & above |
|---|---|---|---|
| Successful API request (one page) | $0.001 ($1 per 1,000) | $0.0009 | $0.0008 |
| Record saved to the dataset | $0.00005 ($0.05 per 1,000) | $0.000045 | $0.00004 |
| Run start | $0.00005 per GB of memory (256 MB default = 1 event) | same | same |

Failed requests (4xx/5xx after retries, timeouts, blocked addresses, non-JSON in records mode), retries and redirects are **not charged**. In per-response mode only the request is charged, not a record.

**Examples (Free/Bronze prices):**

| Job | Requests | Records | Cost |
|---|---|---|---|
| Every Pokémon from PokeAPI (`limit=100`, next URLs) | 14 | 1,351 | about **$0.08** |
| GraphQL query returning 250 countries | 1 | 250 | about **$0.013** |
| Daily sync of 50,000 records at 100 per page | 500 | 50,000 | about **$3.00** |
| Enrich 10,000 CRM rows, one API call each | 10,000 | 10,000 | about **$10.50** |
| Hourly health check of 1 endpoint, per-response mode | 24 a day | – | about **$0.024 a day** |

Set **Max cost per run** in the run options: the Actor checks the budget before every request and stops cleanly when it would go over.

Running with no API URL starts a free offline demo (10 sample records in 3 pages, nothing charged), so you can see the output format first.

### How to use it

#### Fetch a paginated API

1. Paste the endpoint in **API URL**, e.g. `https://api.github.com/repos/apify/crawlee/issues?state=all&per_page=100`.
2. If the API needs a key, pick **Auth type** and paste it in **API key / token**.
3. Leave **Pagination** on *Auto-detect* and **Records path** empty. Start the run.
4. Check the **Output** tab, or the `OUTPUT` record for what was detected (`recordsPathDetected`, `paginationUsed`). If auto-detection guesses wrong, set the mode and paths yourself.

#### Page-number and offset APIs

```json
{
  "url": "https://hn.algolia.com/api/v1/search_by_date?query=web%20scraping&tags=story&hitsPerPage=100",
  "pagination": "page",
  "pageParam": "page",
  "startPage": 0,
  "totalPagesPath": "nbPages"
}
```

```json
{
  "url": "https://openlibrary.org/search.json?q=climate",
  "pagination": "offset",
  "offsetParam": "offset",
  "limitParam": "limit",
  "pageSize": 100,
  "totalPath": "numFound",
  "maxRecords": 5000
}
```

#### Cursor APIs

```json
{
  "url": "https://api.crossref.org/works?rows=200&select=DOI,title,published&cursor=*",
  "pagination": "cursor",
  "cursorPath": "message.next-cursor",
  "cursorParam": "cursor",
  "maxRecords": 10000
}
```

#### GraphQL

```json
{
  "url": "https://api.example.com/graphql",
  "method": "POST",
  "authType": "bearer",
  "authToken": "<your token>",
  "body": {
    "query": "query($after: String) { products(first: 100, after: $after) { pageInfo { hasNextPage endCursor } nodes { id title price } } }",
    "variables": {}
  },
  "pagination": "cursor",
  "cursorPath": "data.products.pageInfo.endCursor",
  "cursorParam": "variables.after",
  "paginationParamIn": "body",
  "hasMorePath": "data.products.pageInfo.hasNextPage",
  "recordsPath": "data.products.nodes"
}
```

#### Enrich a dataset (one API call per row)

1. Pick the dataset in **Input dataset**, or in an integration ("Run another Actor when a run succeeds") use `{{resource.defaultDatasetId}}`.
2. Put placeholders in the URL, query, headers or body: `https://pokeapi.co/api/v2/pokemon/{{name}}`, or `{{url}}` to call a URL stored in each row.
3. List the input fields to keep in **Copy fields from the input row**, e.g. `crm_id`.

```json
{
  "datasetId": "<your dataset ID>",
  "url": "https://api.example.com/v1/companies/{{domain}}",
  "authType": "header",
  "authName": "X-API-Key",
  "authToken": "<your key>",
  "pagination": "none",
  "recordsPath": ".",
  "inputFields": ["crm_id"],
  "requestsPerSecond": 5
}
```

Rows with a missing placeholder field fail with `TEMPLATE_FIELD_MISSING` (free) and the rest continue. Values in URLs are URL-encoded. A JSON body value that is exactly `"{{field}}"` keeps its type (number, object, array).

### Output

One dataset row per record, with the fields the API returns:

```json
{ "number": 3172, "title": "Add proxy rotation docs", "state": "open", "author": "octocat", "created_at": "2026-10-20T08:11:02Z" }
```

The `OUTPUT` record in the key-value store has the run summary:

```json
{
  "requests": 14, "requestsSucceeded": 14, "requestsFailed": 0, "records": 1351,
  "recordsPathDetected": ["results"], "paginationUsed": { "next URL in response": 13 },
  "stoppedAtMaxCost": false, "failureCodes": {}, "seconds": 1.4,
  "exports": { "csv": "https://api.apify.com/v2/datasets/<id>/items?format=csv", "xlsx": "..." }
}
```

Failed requests are listed in the `ERRORS` record with URL (secrets hidden), input row index, page, error code, HTTP status and message. The run **succeeds when at least one request works** and fails with the first error message when every request fails. Turn on **Fail the run on any error** for strict pipelines.

### Error codes

| Code | Meaning |
|---|---|
| `INPUT` | Invalid input (the message says which field and why) |
| `TEMPLATE_FIELD_MISSING` | A `{{field}}` is missing in an input row |
| `DATASET_ACCESS` | The input dataset can't be read (pick it in "Input dataset") |
| `BAD_URL` | Not an http(s) URL |
| `BLOCKED_HOST` | Private, internal or cloud-metadata address |
| `DNS` | Domain not found |
| `TIMEOUT` | No complete response within the request timeout (retried) |
| `CONNECTION` | Connection refused or reset (retried) |
| `TLS` | Invalid HTTPS certificate |
| `PROXY` | Proxy error (retried) |
| `TOO_LARGE` | Response bigger than "Max response size (MB)" |
| `TOO_MANY_REDIRECTS` | More than 5 redirects |
| `HTTP_401`, `HTTP_404`, `HTTP_429`, ... | The API answered with that status after retries; the message includes the start of the API's error body |
| `NOT_JSON` | Records mode got HTML, XML or text; use per-response mode for those |
| `EMPTY_RESPONSE` | The first page had an empty body |
| `RECORDS_PATH_NOT_FOUND` | The "Records path" isn't in the response; the message lists the top-level keys |

### Tips

- **Rate limits**: keep **Requests per second** under the API's limit. A `429` pauses all parallel requests for the `Retry-After` time.
- **Big jobs**: use the API's largest page size (`per_page=100`, `limit=1000`): you pay per request, so bigger pages cost less.
- **Only new records** after each scheduled run: chain this Actor into a dedupe Actor with a memory of seen IDs.
- **Proxy**: most APIs work without one. Turn on Apify Proxy only for APIs that block datacenter IPs or need a specific country (proxy usage is billed by Apify separately).
- Use APIs you're allowed to use, with your own keys, and follow each API's terms and rate limits.

### FAQ

**Is my API key safe?** It's a secret input: encrypted at rest, not shown in logs, summaries or error messages (query parameters that look like keys are masked as `***`), and sent only to the host of your API URL.

**Does it work with XML, CSV or HTML APIs?** Use per-response mode to get the raw body of each response. Records mode needs JSON or NDJSON.

**Can I call internal or localhost APIs?** No. Private and internal addresses are blocked for safety. The API must be reachable from the internet.

**Does it need special permissions?** No. It runs with limited permissions and reads only the dataset you pick in "Input dataset".

# Actor input Schema

## `url` (type: `string`):

The JSON API endpoint to fetch, e.g. https://api.example.com/v1/items?status=active. With an Input dataset this is a template: https://api.example.com/users/{{id}}, or {{url}} to call a URL stored in each row. Leave empty to run a free demo.

## `urls` (type: `array`):

Optional extra endpoints, fetched with the same settings. Each one is paginated on its own.

## `method` (type: `string`):

GET for most APIs; POST for GraphQL and search endpoints.

## `queryParams` (type: `object`):

Added to every first request, e.g. {"per_page": 100, "state": "open"}. Values can use {{field}} placeholders.

## `headers` (type: `object`):

Extra request headers, e.g. {"Accept": "application/vnd.github+json"}. Put API keys in "API key / token" below instead, which is stored encrypted.

## `body` (type: `object`):

For POST/PUT/PATCH, e.g. a GraphQL {"query": "...", "variables": {}} or a search filter. Values can use {{field}} placeholders (a value that is exactly "{{field}}" keeps its type).

## `bodyFormat` (type: `string`):

How the request body is encoded.

## `authType` (type: `string`):

How your API key or token is sent.

## `authToken` (type: `string`):

The token or API key. For Basic auth use username:password.

## `authName` (type: `string`):

Header name for "API key in a header" (default X-API-Key) or query parameter for "API key in the query string" (default api_key). For Bearer, overrides the Authorization header name.

## `recordsPath` (type: `string`):

Where the list of records is in the response, as a dot path, e.g. data.items or results. Leave empty to auto-detect (top-level arrays, data, results, items, hits.hits, GraphQL edges/nodes...). Use . to save the whole response as one record.

## `outputMode` (type: `string`):

Records: the record list is split into rows (best for CSV/Excel). Per response: one item per HTTP response, for non-JSON or when you want the raw body.

## `fields` (type: `array`):

Optional. Dot paths like id, name, price.amount. Leave empty to keep every field.

## `renameFields` (type: `object`):

Optional map of old name or dot path to new name, e.g. {"price.amount": "price"}.

## `flatten` (type: `boolean`):

Turn {"a": {"b": 1}} into {"a.b": 1} for spreadsheets. Arrays of values are joined with commas.

## `includeRequestInfo` (type: `boolean`):

Adds \_sourceUrl and \_page (and \_inputIndex with an Input dataset) to every record.

## `includeResponseHeaders` (type: `boolean`):

Per-response output only: adds the response headers to each item.

## `pagination` (type: `string`):

How to find the next page. Auto-detect works for most APIs.

## `nextUrlPath` (type: `string`):

For "Next URL": dot path of the next page's URL, e.g. links.next or paging.next.

## `cursorPath` (type: `string`):

For "Cursor": dot path of the next cursor in the response, e.g. meta.next_cursor or data.search.pageInfo.endCursor.

## `cursorParam` (type: `string`):

For "Cursor": the query parameter (or body path, e.g. variables.after) that receives the cursor.

## `pageParam` (type: `string`):

For "Page number". Default page.

## `startPage` (type: `integer`):

For "Page number". Default 1 (use 0 for zero-based APIs).

## `offsetParam` (type: `string`):

For "Offset". Default offset (e.g. skip, start).

## `startOffset` (type: `integer`):

For "Offset". Default 0.

## `limitParam` (type: `string`):

For "Page number" or "Offset": the parameter that sets page size, e.g. limit or per_page. Sent with Page size.

## `pageSize` (type: `integer`):

Records per page. When set, a shorter page means it was the last one.

## `paginationParamIn` (type: `string`):

Where the page, offset or cursor parameter is sent.

## `hasMorePath` (type: `string`):

Optional dot path of a true/false flag; false stops paging (e.g. has_more, pageInfo.hasNextPage).

## `totalPagesPath` (type: `string`):

Optional, for "Page number": dot path of the page count, e.g. meta.total_pages or nbPages.

## `totalPath` (type: `string`):

Optional, for "Offset": dot path of the total record count, e.g. total or numFound.

## `stopOnEmptyPage` (type: `boolean`):

Stop paging when a page has no records.

## `maxPagesPerUrl` (type: `integer`):

Safety limit per endpoint (or per input row).

## `datasetId` (type: `string`):

Optional. Calls the API once per row, filling {{field}} placeholders in the URL, query, headers and body from that row. In an integration use {{resource.defaultDatasetId}}.

## `inputFields` (type: `array`):

Fields of the input row to add to every output record, e.g. id or email, so you can join results back.

## `maxInputItems` (type: `integer`):

0 = all rows.

## `maxRecords` (type: `integer`):

Stop after this many records in total. 0 = no limit.

## `maxRequests` (type: `integer`):

Safety limit on API requests per run (retries not counted).

## `maxConcurrency` (type: `integer`):

Endpoints or input rows fetched in parallel. Pages of one endpoint are always fetched in order.

## `requestsPerSecond` (type: `number`):

Global rate limit across all parallel requests. 0 = no limit. A 429 pauses everyone for the Retry-After time.

## `timeoutSecs` (type: `integer`):

Per request, including reading the body.

## `maxRetries` (type: `integer`):

Retries for timeouts, connection errors, 408, 425, 429 and 5xx, with exponential backoff and Retry-After.

## `maxResponseMB` (type: `integer`):

Larger responses fail with TOO_LARGE instead of exhausting memory.

## `proxyConfiguration` (type: `object`):

Optional. Most APIs don't need one. Use it for APIs that block datacenter IPs or need a specific country.

## `failOnRequestError` (type: `boolean`):

By default the run succeeds if at least one request worked, and failures are listed in the ERRORS record. Turn on to fail the run instead (useful in pipelines).

## Actor input object example

```json
{
  "url": "https://rickandmortyapi.com/api/episode",
  "method": "GET",
  "bodyFormat": "json",
  "authType": "none",
  "outputMode": "records",
  "flatten": false,
  "includeRequestInfo": false,
  "includeResponseHeaders": false,
  "pagination": "auto",
  "paginationParamIn": "query",
  "stopOnEmptyPage": true,
  "maxPagesPerUrl": 1000,
  "maxRequests": 10000,
  "maxConcurrency": 5,
  "requestsPerSecond": 10,
  "timeoutSecs": 30,
  "maxRetries": 3,
  "maxResponseMB": 20,
  "proxyConfiguration": {
    "useApifyProxy": false
  },
  "failOnRequestError": false
}
```

# Actor output Schema

## `records` (type: `string`):

Records saved from the API.

## `csv` (type: `string`):

All records as CSV.

## `xlsx` (type: `string`):

All records as an Excel file.

## `summary` (type: `string`):

No description

## `errors` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "url": "https://rickandmortyapi.com/api/episode"
};

// Run the Actor and wait for it to finish
const run = await client.actor("genesis.labs.co/rest-api-to-dataset").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "url": "https://rickandmortyapi.com/api/episode" }

# Run the Actor and wait for it to finish
run = client.actor("genesis.labs.co/rest-api-to-dataset").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "url": "https://rickandmortyapi.com/api/episode"
}' |
apify call genesis.labs.co/rest-api-to-dataset --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,genesis.labs.co/rest-api-to-dataset"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/askw4f4zH9twgUUIc/builds/8nXbp0aeMdyHPS7Ok/openapi.json