REST API to Dataset: Auto Pagination, Auth & Retries

Turn any JSON REST or GraphQL API into a clean Apify dataset in one run. Paste an endpoint, add your API key if it needs one, and the Actor follows every page, splits the response into one row per record, retries rate limits and server errors, and gives you CSV, Excel or JSON. Failed requests are free.

It works with your own APIs, SaaS APIs (with your key), open data APIs and GraphQL endpoints. It is also an API enrichment tool: feed it a dataset and it calls the API once per row (https://api.example.com/users/{{id}}), adding the results to each row.

What it does

• 🔁 Auto pagination that detects the common styles: Link: rel="next" headers (GitHub and others), next URLs in the body (next, links.next, _links.next.href, paging.next, @odata.nextLink, info.next...), cursors and page tokens (next_cursor, meta.next_cursor, response_metadata.next_cursor, nextPageToken), has_more + starting_after, and current_page / last_page counts. You can also choose page number, offset + limit, cursor (in the query or in a JSON body for GraphQL) or next URL at a path yourself.
• 📦 Records extraction: finds the list of records automatically (data, results, items, hits.hits, message.items, GraphQL edges/nodes...), or you set a dot path like data.items. Each record becomes one dataset row.
• 🔐 Auth: Bearer token, API key in a header or query string, or Basic auth. The key is stored as an encrypted secret input, hidden in logs and summaries, and only sent to the host of your API URL, never to other hosts that appear in pagination links or redirects.
• 🧰 Reliable fetching: retries with exponential backoff for timeouts, connection resets, 408, 425, 429 and 5xx; honors Retry-After; one global requests-per-second limit across all parallel requests; per-request timeout; response size cap; redirects handled safely.
• 🧩 Enrich a dataset: one request per input row with {{field}} placeholders in the URL, query, headers and JSON body. Copy fields such as id or email from the input row into the output to join results back.
• ✂️ Shape the output: keep only some fields, rename fields (price.amount → price), flatten nested objects for spreadsheets, add the source URL and page number.
• 📄 Per-response mode: one item per HTTP response with status, final URL, timing, headers and the raw body (works for XML, HTML, text or any other format).
• 🛡️ Safe by default: private, internal and cloud-metadata addresses are blocked (including via DNS or redirects), and the Actor runs with limited permissions, so it can only read the dataset you pick.
• 💸 Failures are free and the Actor stops before going over your Max cost per run.

Pricing

Pay per event. Compute and storage are included.

Event	Free & Bronze	Silver	Gold & above
Successful API request (one page)	$0.001 ($1 per 1,000)	$0.0009	$0.0008
Record saved to the dataset	$0.00005 ($0.05 per 1,000)	$0.000045	$0.00004
Run start	$0.00005 per GB of memory (256 MB default = 1 event)	same	same

Failed requests (4xx/5xx after retries, timeouts, blocked addresses, non-JSON in records mode), retries and redirects are not charged. In per-response mode only the request is charged, not a record.

Examples (Free/Bronze prices):

Job	Requests	Records	Cost
Every Pokémon from PokeAPI (limit=100, next URLs)	14	1,351	about $0.08
GraphQL query returning 250 countries	1	250	about $0.013
Daily sync of 50,000 records at 100 per page	500	50,000	about $3.00
Enrich 10,000 CRM rows, one API call each	10,000	10,000	about $10.50
Hourly health check of 1 endpoint, per-response mode	24 a day	–	about $0.024 a day

Set Max cost per run in the run options: the Actor checks the budget before every request and stops cleanly when it would go over.

Running with no API URL starts a free offline demo (10 sample records in 3 pages, nothing charged), so you can see the output format first.

How to use it

Fetch a paginated API

1. Paste the endpoint in API URL, e.g. https://api.github.com/repos/apify/crawlee/issues?state=all&per_page=100.
2. If the API needs a key, pick Auth type and paste it in API key / token.
3. Leave Pagination on Auto-detect and Records path empty. Start the run.
4. Check the Output tab, or the OUTPUT record for what was detected (recordsPathDetected, paginationUsed). If auto-detection guesses wrong, set the mode and paths yourself.

Page-number and offset APIs

{
  "url": "https://hn.algolia.com/api/v1/search_by_date?query=web%20scraping&tags=story&hitsPerPage=100",
  "pagination": "page",
  "pageParam": "page",
  "startPage": 0,
  "totalPagesPath": "nbPages"
}

{
  "url": "https://openlibrary.org/search.json?q=climate",
  "pagination": "offset",
  "offsetParam": "offset",
  "limitParam": "limit",
  "pageSize": 100,
  "totalPath": "numFound",
  "maxRecords": 5000
}

Cursor APIs

{
  "url": "https://api.crossref.org/works?rows=200&select=DOI,title,published&cursor=*",
  "pagination": "cursor",
  "cursorPath": "message.next-cursor",
  "cursorParam": "cursor",
  "maxRecords": 10000
}

GraphQL

{
  "url": "https://api.example.com/graphql",
  "method": "POST",
  "authType": "bearer",
  "authToken": "<your token>",
  "body": {
    "query": "query($after: String) { products(first: 100, after: $after) { pageInfo { hasNextPage endCursor } nodes { id title price } } }",
    "variables": {}
  },
  "pagination": "cursor",
  "cursorPath": "data.products.pageInfo.endCursor",
  "cursorParam": "variables.after",
  "paginationParamIn": "body",
  "hasMorePath": "data.products.pageInfo.hasNextPage",
  "recordsPath": "data.products.nodes"
}

Enrich a dataset (one API call per row)

1. Pick the dataset in Input dataset, or in an integration ("Run another Actor when a run succeeds") use {{resource.defaultDatasetId}}.
2. Put placeholders in the URL, query, headers or body: https://pokeapi.co/api/v2/pokemon/{{name}}, or {{url}} to call a URL stored in each row.
3. List the input fields to keep in Copy fields from the input row, e.g. crm_id.

{
  "datasetId": "<your dataset ID>",
  "url": "https://api.example.com/v1/companies/{{domain}}",
  "authType": "header",
  "authName": "X-API-Key",
  "authToken": "<your key>",
  "pagination": "none",
  "recordsPath": ".",
  "inputFields": ["crm_id"],
  "requestsPerSecond": 5
}

Rows with a missing placeholder field fail with TEMPLATE_FIELD_MISSING (free) and the rest continue. Values in URLs are URL-encoded. A JSON body value that is exactly "{{field}}" keeps its type (number, object, array).

Output

One dataset row per record, with the fields the API returns:

{ "number": 3172, "title": "Add proxy rotation docs", "state": "open", "author": "octocat", "created_at": "2026-10-20T08:11:02Z" }

The OUTPUT record in the key-value store has the run summary:

{
  "requests": 14, "requestsSucceeded": 14, "requestsFailed": 0, "records": 1351,
  "recordsPathDetected": ["results"], "paginationUsed": { "next URL in response": 13 },
  "stoppedAtMaxCost": false, "failureCodes": {}, "seconds": 1.4,
  "exports": { "csv": "https://api.apify.com/v2/datasets/<id>/items?format=csv", "xlsx": "..." }
}

Failed requests are listed in the ERRORS record with URL (secrets hidden), input row index, page, error code, HTTP status and message. The run succeeds when at least one request works and fails with the first error message when every request fails. Turn on Fail the run on any error for strict pipelines.

Error codes

Code	Meaning
INPUT	Invalid input (the message says which field and why)
TEMPLATE_FIELD_MISSING	A {{field}} is missing in an input row
DATASET_ACCESS	The input dataset can't be read (pick it in "Input dataset")
BAD_URL	Not an http(s) URL
BLOCKED_HOST	Private, internal or cloud-metadata address
DNS	Domain not found
TIMEOUT	No complete response within the request timeout (retried)
CONNECTION	Connection refused or reset (retried)
TLS	Invalid HTTPS certificate
PROXY	Proxy error (retried)
TOO_LARGE	Response bigger than "Max response size (MB)"
TOO_MANY_REDIRECTS	More than 5 redirects
HTTP_401, HTTP_404, HTTP_429, ...	The API answered with that status after retries; the message includes the start of the API's error body
NOT_JSON	Records mode got HTML, XML or text; use per-response mode for those
EMPTY_RESPONSE	The first page had an empty body
RECORDS_PATH_NOT_FOUND	The "Records path" isn't in the response; the message lists the top-level keys

Tips

• Rate limits: keep Requests per second under the API's limit. A 429 pauses all parallel requests for the Retry-After time.
• Big jobs: use the API's largest page size (per_page=100, limit=1000): you pay per request, so bigger pages cost less.
• Only new records after each scheduled run: chain this Actor into a dedupe Actor with a memory of seen IDs.
• Proxy: most APIs work without one. Turn on Apify Proxy only for APIs that block datacenter IPs or need a specific country (proxy usage is billed by Apify separately).
• Use APIs you're allowed to use, with your own keys, and follow each API's terms and rate limits.

FAQ

Is my API key safe? It's a secret input: encrypted at rest, not shown in logs, summaries or error messages (query parameters that look like keys are masked as ***), and sent only to the host of your API URL.

Does it work with XML, CSV or HTML APIs? Use per-response mode to get the raw body of each response. Records mode needs JSON or NDJSON.

Can I call internal or localhost APIs? No. Private and internal addresses are blocked for safety. The API must be reachable from the internet.

Does it need special permissions? No. It runs with limited permissions and reads only the dataset you pick in "Input dataset".