Paginated REST API Fetcher - Fetch All Pages

Export a REST API into one Apify Dataset without fetching each page yourself. Choose page-number, offset, cursor or next-link pagination, then set page and item limits. The run summary tells you whether the export finished or stopped early.

What it does

You provide one REST request plus the API's pagination pattern. The Actor walks pages sequentially, pushes each item into the Dataset as it goes, and stops with a clear reason: complete, empty, limit reached, pagination loop, or partial source failure.

First-run demo

The Console prefill calls the public PokéAPI list endpoint with nextUrl pagination, capped at 2 pages / 40 items. Expect ~40 ITEM rows and a RUN_SUMMARY with status=LIMIT_REACHED or SUCCESS depending on how many pages the source returns under the cap.

Related tools after your API export

• Enrich selected records: HTTP Request - Batch API Calls sends independent requests you construct from exported record IDs or URLs. It does not infer the endpoint or pagination configuration for you.
• Download public attachments: Parallel Bulk File Downloader stores files from an explicit URL list. Select permitted public attachment URLs from successful ITEM rows and map them into the downloader's urls input; private API credentials are not transferred.

Check RUN_SUMMARY for completeness before using the export. These are separate Actors with separate pricing; build the mapping and next run in your own workflow.

Input

Field	Purpose
url	Public http/https API endpoint (ports 80/443 only)
method	GET (default) or POST
headers / query / body	Request extras (body only with POST)
itemsPath	Dotted path to the items array; empty string = root array
pagination.mode	page | offset | cursor | nextUrl
maxPages / maxItems	Hard bounds; stop as LIMIT_REACHED
timeoutSeconds / maxRetries	Per-attempt timeout and GET retry budget; POST is never retried automatically
dedupeIdPath	Optional required scalar id path inside each item (off by default)

Pagination modes

1. page - increment a page query parameter; optional page size stops on a short final page.
2. offset - advance offset by returned count (or a fixed offsetStep) until a short/empty page.
3. cursor - send cursorRequestParam from nextCursorPath; stop on missing cursor; fail on a repeated cursor.
4. nextUrl - follow nextUrlPath and/or RFC 5988 Link: rel=next; fail on a repeated URL. Relative next links are resolved against the page URL that returned them; every next URL is SSRF-validated.

Dotted paths support object keys with hyphens/colons/@ (for example data.next-cursor, hydra:member). A missing intermediate path segment is SOURCE_FAILED; a missing/null leaf cursor or next URL is a normal end of pagination.

Example (cursor)

{
  "url": "https://api.example.com/v1/items",
  "method": "GET",
  "headers": {
    "Authorization": "Bearer <your-token>"
  },
  "query": {
    "status": "active"
  },
  "itemsPath": "data.items",
  "pagination": {
    "mode": "cursor",
    "cursorRequestParam": "after",
    "nextCursorPath": "data.next_cursor"
  },
  "maxPages": 1000,
  "maxItems": 100000,
  "timeoutSeconds": 30,
  "maxRetries": 2
}

Put API tokens in headers. User-provided header values are sent only to the configured URL's origin and are always redacted from Actor logs. Cross-origin GET redirects/next links receive no user headers; cross-origin requests that would carry a POST body are rejected.

Output

Each Dataset ITEM row:

{
  "record_type": "ITEM",
  "source_url": "https://api.example.com/v1/items",
  "page_index": 17,
  "item_index": 1604,
  "data": { "id": "abc123" },
  "fetched_at": "2026-09-17T21:00:00Z",
  "schema_version": 1
}

source_url retains host/path provenance but redacts every query value. RUN_SUMMARY (Key-Value Store) includes pagesRequested, pagesSucceeded, itemsFetched, itemsEmitted, duplicateItemsSkipped, stopReason, billing fields, Dataset/run identifiers and nextAction.

Failure semantics

Situation	Status / stop
Valid first page, zero items	NO_RESULTS
Natural end of pagination	SUCCESS
maxPages / maxItems hit	LIMIT_REACHED
Repeated cursor / next URL	PAGINATION_LOOP (run fails)
429/5xx/timeout after retries, bad JSON	SOURCE_FAILED or PARTIAL if earlier pages succeeded
Empty result list from HTTP errors	Never - transport failures are not empty success

NO_RESULTS is never collapsed into SOURCE_FAILED.

Pricing (PPE)

• apify-actor-start: $0.01
• page-fetched: $0.002 per non-empty page persisted to the Dataset

Not charged: empty or duplicate-only pages, retries, failed pages, malformed responses, diagnostics. Platform compute/transfer/storage are separate.

Security

• Only public HTTP/HTTPS URLs on ports 80/443
• Localhost, private, link-local, metadata and reserved targets blocked
• Every redirect and every next URL re-validated (including DNS)
• All user-provided header values are redacted from logs
• User headers stay on the configured origin; cross-origin GETs receive no user headers
• POST bodies are never sent cross-origin and POST requests are never retried automatically
• URL query values and secrets are never written into Dataset rows

Why run it again

• The upstream API changed and you need a fresh full export
• You raised maxPages / maxItems after a LIMIT_REACHED stop
• You schedule a recurring sync into a warehouse or n8n/Make workflow

MCP and agents

Select this Actor when an agent must export every page of a caller-authorized public REST JSON API (page / offset / cursor / next-link) into one Dataset with hard maxPages / maxItems caps and an explicit stop reason.

Do not select it for HTML scraping, browser pages, GraphQL pageInfo, OAuth login flows, private/localhost URLs, or unbounded “crawl the whole API” without ceilings.

Direct Actor tool (Streamable HTTP; execution requires the caller’s Apify OAuth or API token - never the developer’s token):

https://mcp.apify.com?tools=automa-flow/paginated-rest-api-fetcher

Anonymous discovery uses search-actors / fetch-actor-details only after the Actor is public on the Store. Running the Actor always needs authentication. Limited permissions + PPE keep this Actor eligible for MCP agent workflows (full-permission and rental Actors are excluded from MCP search/execution).

Example agent ask: “Fetch all pages from this cursor-paginated REST API into a Dataset using maxPages=50 and maxTotalChargeUsd=0.12.”

Estimate your run cost

Example	Pages	Approximate event charge
Demo	2 (prefill)	$0.01 + 2×$0.002 = $0.014
Three-page export	3	$0.016
Your page limit	set maxPages	$0.01 + $0.002 × pages

Always set maxPages, maxItems, and run maxTotalChargeUsd (≥ $0.012). At the charge limit the run stops, already persisted pages stay in the Dataset, and RUN_SUMMARY explains the stop. Retries and source failures do not add page-fetched charges. Platform usage is billed separately.

What to read first after a run

1. Key-Value Store RUN_SUMMARY - status, stopReason, itemsEmitted, billablePages, chargedEventCounts, nextAction
2. Dataset view items for payload rows (record_type=ITEM)
3. Full Dataset via MCP get-actor-output / storage tools when the preview truncates

Partial and limit-reached runs do not expose a resumable cursor in v1. Re-running starts pagination from the beginning and can bill the same pages again.

Limitations

• No GraphQL connection/pageInfo helpers in v1
• No OAuth flows or secret vault
• No HTML scraping, browsers, or user-supplied scripts
• Pages are fetched sequentially (correctness over throughput)
• Not every API shape is supported - configure the matching mode and paths