# Paginated REST API Fetcher - Fetch All Pages (`automa-flow/paginated-rest-api-fetcher`) Actor

Export a REST API into one Apify Dataset using page-number, offset, cursor or next-link pagination. Set page and item limits, then check the run summary to see whether the export finished or stopped early.

- **URL**: https://apify.com/automa-flow/paginated-rest-api-fetcher.md
- **Developed by:** [Vadim Bezrukov](https://apify.com/automa-flow) (community)
- **Categories:** Automation, Developer tools, Other
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 page fetcheds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Paginated REST API Fetcher - Fetch All Pages

Export a REST API into one Apify Dataset without fetching each page yourself.
Choose page-number, offset, cursor or next-link pagination, then set page and
item limits. The run summary tells you whether the export finished or stopped early.

### What it does

You provide one REST request plus the API's pagination pattern. The Actor walks pages sequentially, pushes each item into the Dataset as it goes, and stops with a clear reason: complete, empty, limit reached, pagination loop, or partial source failure.

### First-run demo

The Console prefill calls the public PokéAPI list endpoint with `nextUrl` pagination, capped at 2 pages / 40 items. Expect ~40 `ITEM` rows and a `RUN_SUMMARY` with `status=LIMIT_REACHED` or `SUCCESS` depending on how many pages the source returns under the cap.

### Related tools after your API export

- **Enrich selected records:** [HTTP Request - Batch API Calls](https://apify.com/automa-flow/http-request-batch) sends independent requests you construct from exported record IDs or URLs. It does not infer the endpoint or pagination configuration for you.
- **Download public attachments:** [Parallel Bulk File Downloader](https://apify.com/automa-flow/parallel-bulk-file-downloader) stores files from an explicit URL list. Select permitted public attachment URLs from successful `ITEM` rows and map them into the downloader's `urls` input; private API credentials are not transferred.

Check `RUN_SUMMARY` for completeness before using the export. These are separate
Actors with separate pricing; build the mapping and next run in your own workflow.

### Input

| Field | Purpose |
| --- | --- |
| `url` | Public `http`/`https` API endpoint (ports 80/443 only) |
| `method` | `GET` (default) or `POST` |
| `headers` / `query` / `body` | Request extras (`body` only with `POST`) |
| `itemsPath` | Dotted path to the items array; empty string = root array |
| `pagination.mode` | `page` | `offset` | `cursor` | `nextUrl` |
| `maxPages` / `maxItems` | Hard bounds; stop as `LIMIT_REACHED` |
| `timeoutSeconds` / `maxRetries` | Per-attempt timeout and GET retry budget; POST is never retried automatically |
| `dedupeIdPath` | Optional required scalar id path inside each item (off by default) |

#### Pagination modes

1. **page** - increment a page query parameter; optional page size stops on a short final page.
2. **offset** - advance `offset` by returned count (or a fixed `offsetStep`) until a short/empty page.
3. **cursor** - send `cursorRequestParam` from `nextCursorPath`; stop on missing cursor; fail on a repeated cursor.
4. **nextUrl** - follow `nextUrlPath` and/or RFC 5988 `Link: rel=next`; fail on a repeated URL. Relative next links are resolved against the page URL that returned them; every next URL is SSRF-validated.

Dotted paths support object keys with hyphens/colons/`@` (for example `data.next-cursor`, `hydra:member`). A missing *intermediate* path segment is `SOURCE_FAILED`; a missing/null *leaf* cursor or next URL is a normal end of pagination.

#### Example (cursor)

```json
{
  "url": "https://api.example.com/v1/items",
  "method": "GET",
  "headers": {
    "Authorization": "Bearer <your-token>"
  },
  "query": {
    "status": "active"
  },
  "itemsPath": "data.items",
  "pagination": {
    "mode": "cursor",
    "cursorRequestParam": "after",
    "nextCursorPath": "data.next_cursor"
  },
  "maxPages": 1000,
  "maxItems": 100000,
  "timeoutSeconds": 30,
  "maxRetries": 2
}
```

Put API tokens in headers. User-provided header values are sent only to
the configured URL's origin and are always redacted from Actor logs. Cross-origin
GET redirects/next links receive no user headers; cross-origin requests that
would carry a POST body are rejected.

### Output

Each Dataset `ITEM` row:

```json
{
  "record_type": "ITEM",
  "source_url": "https://api.example.com/v1/items",
  "page_index": 17,
  "item_index": 1604,
  "data": { "id": "abc123" },
  "fetched_at": "2026-09-17T21:00:00Z",
  "schema_version": 1
}
```

`source_url` retains host/path provenance but redacts every query value.
`RUN_SUMMARY` (Key-Value Store) includes `pagesRequested`, `pagesSucceeded`,
`itemsFetched`, `itemsEmitted`, `duplicateItemsSkipped`, `stopReason`, billing
fields, Dataset/run identifiers and `nextAction`.

#### Failure semantics

| Situation | Status / stop |
| --- | --- |
| Valid first page, zero items | `NO_RESULTS` |
| Natural end of pagination | `SUCCESS` |
| `maxPages` / `maxItems` hit | `LIMIT_REACHED` |
| Repeated cursor / next URL | `PAGINATION_LOOP` (run fails) |
| 429/5xx/timeout after retries, bad JSON | `SOURCE_FAILED` or `PARTIAL` if earlier pages succeeded |
| Empty result list from HTTP errors | Never - transport failures are not empty success |

`NO_RESULTS` is never collapsed into `SOURCE_FAILED`.

### Pricing (PPE)

- `apify-actor-start`: $0.01
- `page-fetched`: $0.002 per non-empty page persisted to the Dataset

Not charged: empty or duplicate-only pages, retries, failed pages, malformed
responses, diagnostics. Platform compute/transfer/storage are separate.

### Security

- Only public HTTP/HTTPS URLs on ports 80/443
- Localhost, private, link-local, metadata and reserved targets blocked
- Every redirect and every `next` URL re-validated (including DNS)
- All user-provided header values are redacted from logs
- User headers stay on the configured origin; cross-origin GETs receive no user headers
- POST bodies are never sent cross-origin and POST requests are never retried automatically
- URL query values and secrets are never written into Dataset rows

### Why run it again

- The upstream API changed and you need a fresh full export
- You raised `maxPages` / `maxItems` after a `LIMIT_REACHED` stop
- You schedule a recurring sync into a warehouse or n8n/Make workflow

### MCP and agents

**Select this Actor when** an agent must export every page of a caller-authorized
public REST JSON API (page / offset / cursor / next-link) into one Dataset with
hard `maxPages` / `maxItems` caps and an explicit stop reason.

**Do not select it for** HTML scraping, browser pages, GraphQL `pageInfo`, OAuth
login flows, private/localhost URLs, or unbounded “crawl the whole API” without
ceilings.

Direct Actor tool (Streamable HTTP; execution requires the caller’s Apify OAuth
or API token - never the developer’s token):

`https://mcp.apify.com?tools=automa-flow/paginated-rest-api-fetcher`

Anonymous discovery uses `search-actors` / `fetch-actor-details` only after the
Actor is public on the Store. Running the Actor always needs authentication.
Limited permissions + PPE keep this Actor eligible for MCP agent workflows
(full-permission and rental Actors are excluded from MCP search/execution).

Example agent ask: “Fetch all pages from this cursor-paginated REST API into a
Dataset using maxPages=50 and maxTotalChargeUsd=0.12.”

#### Estimate your run cost

| Example | Pages | Approximate event charge |
| --- | ---: | ---: |
| Demo | 2 (prefill) | $0.01 + 2×$0.002 = **$0.014** |
| Three-page export | 3 | **$0.016** |
| Your page limit | set `maxPages` | $0.01 + $0.002 × pages |

Always set `maxPages`, `maxItems`, and run `maxTotalChargeUsd` (≥ $0.012). At the
charge limit the run stops, already persisted pages stay in the Dataset, and
`RUN_SUMMARY` explains the stop. Retries and source failures do not add
`page-fetched` charges. Platform usage is billed separately.

#### What to read first after a run

1. Key-Value Store `RUN_SUMMARY` - `status`, `stopReason`, `itemsEmitted`,
   `billablePages`, `chargedEventCounts`, `nextAction`
2. Dataset view `items` for payload rows (`record_type=ITEM`)
3. Full Dataset via MCP `get-actor-output` / storage tools when the preview truncates

Partial and limit-reached runs do not expose a resumable cursor in v1. Re-running
starts pagination from the beginning and can bill the same pages again.

### Limitations

- No GraphQL connection/`pageInfo` helpers in v1
- No OAuth flows or secret vault
- No HTML scraping, browsers, or user-supplied scripts
- Pages are fetched sequentially (correctness over throughput)
- Not every API shape is supported - configure the matching mode and paths

# Actor input Schema

## `url` (type: `string`):

Public http(s) REST endpoint to fetch. Localhost, private, link-local and cloud-metadata targets are rejected. Only ports 80 and 443 are allowed.

## `method` (type: `string`):

GET (default) or POST. Body is only allowed with POST.

## `headers` (type: `object`):

Optional request headers. Do not put secrets in logs; Authorization/Cookie/API-key values are redacted from Actor logs but still sent to the API. Prefer Apify secret inputs where possible.

## `query` (type: `object`):

Base query string parameters merged with query parameters already present in the configured URL for the first and parameter-based page requests. Follow-up nextUrl pages use the returned link.

## `body` (type: `object`):

Optional JSON body for POST requests.

## `itemsPath` (type: `string`):

Dotted path to the items array (for example data.items). Use an empty string when the response root is already a JSON array.

## `pagination` (type: `object`):

Pagination mode and fields for the target API.

## `maxPages` (type: `integer`):

Hard stop after this many successful or attempted page requests. Stop reason LIMIT\_PAGES (run status LIMIT\_REACHED).

## `maxItems` (type: `integer`):

Hard stop after this many Dataset ITEM rows. Stop reason LIMIT\_ITEMS (run status LIMIT\_REACHED).

## `timeoutSeconds` (type: `integer`):

Timeout applied to each HTTP attempt.

## `maxRetries` (type: `integer`):

Extra GET attempts after the first try for timeouts, network errors and retryable HTTP statuses (408/425/429/5xx). POST is never retried automatically because it may be non-idempotent. Deterministic 4xx responses are never retried. Retries are not billed.

## `dedupeIdPath` (type: `string`):

Optional dotted path inside each item for stable-ID deduplication. Off by default. Never invents an id from arbitrary JSON.

## Actor input object example

```json
{
  "url": "https://pokeapi.co/api/v2/pokemon",
  "method": "GET",
  "headers": {},
  "query": {
    "limit": "20"
  },
  "itemsPath": "results",
  "pagination": {
    "mode": "nextUrl",
    "nextUrlPath": "next"
  },
  "maxPages": 2,
  "maxItems": 40,
  "timeoutSeconds": 30,
  "maxRetries": 2
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `runSummary` (type: `string`):

No description

## `billingReceipt` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "url": "https://pokeapi.co/api/v2/pokemon",
    "method": "GET",
    "headers": {},
    "query": {
        "limit": "20"
    },
    "itemsPath": "results",
    "pagination": {
        "mode": "nextUrl",
        "nextUrlPath": "next"
    },
    "maxPages": 2,
    "maxItems": 40,
    "timeoutSeconds": 30,
    "maxRetries": 2
};

// Run the Actor and wait for it to finish
const run = await client.actor("automa-flow/paginated-rest-api-fetcher").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "url": "https://pokeapi.co/api/v2/pokemon",
    "method": "GET",
    "headers": {},
    "query": { "limit": "20" },
    "itemsPath": "results",
    "pagination": {
        "mode": "nextUrl",
        "nextUrlPath": "next",
    },
    "maxPages": 2,
    "maxItems": 40,
    "timeoutSeconds": 30,
    "maxRetries": 2,
}

# Run the Actor and wait for it to finish
run = client.actor("automa-flow/paginated-rest-api-fetcher").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "url": "https://pokeapi.co/api/v2/pokemon",
  "method": "GET",
  "headers": {},
  "query": {
    "limit": "20"
  },
  "itemsPath": "results",
  "pagination": {
    "mode": "nextUrl",
    "nextUrlPath": "next"
  },
  "maxPages": 2,
  "maxItems": 40,
  "timeoutSeconds": 30,
  "maxRetries": 2
}' |
apify call automa-flow/paginated-rest-api-fetcher --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,automa-flow/paginated-rest-api-fetcher"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/JJkZM9o3rYGA0IKHS/builds/xjNCH50yyewW8x7LV/openapi.json