# Hidden API Finder: Capture XHR/JSON Endpoints (`leadsbrary/hidden-api-finder-network-capture`) Actor

Find the hidden API of a website: records XHR/fetch JSON calls and returns ranked endpoints with URL template, params, pagination hints, response sample, curl and fetch snippets. Export JSON, CSV, Excel.

- **URL**: https://apify.com/leadsbrary/hidden-api-finder-network-capture.md
- **Developed by:** [Alexandre Manguis](https://apify.com/leadsbrary) (community)
- **Categories:** Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 results

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Find Hidden API of Website: Capture XHR/JSON Endpoints

**Find hidden API of website pages without opening DevTools.** Give this Actor a public page URL: it loads the page in a headless browser, records the XHR/fetch JSON calls, drops analytics and ad noise, and returns a ranked list of candidate endpoints with URL template, query parameters, pagination hints, a response sample and ready-to-run `curl` and `fetch` snippets. Export as JSON, CSV, Excel, XML or HTML.

### What does Hidden API Finder do?

Many modern pages load their data from a JSON endpoint behind the scenes. Scraping that endpoint is faster and more stable than parsing HTML, but finding it means manually digging through the browser's Network tab. This Actor does that reverse-engineering step for you:

1. Opens each page in a real headless browser (optionally clicking elements and scrolling to trigger more calls).
2. Records the JSON requests the page makes.
3. Filters out known ad, analytics and consent calls.
4. Ranks the remaining calls, giving a higher score to responses that contain texts you can see on the page (a product name, a price) and that hold arrays of items.
5. Outputs one row per candidate endpoint, ready to reuse in your own scraper.

### Who is it for / use cases

- **Scraper developers** who want to replace a slow browser scraper by a direct API call.
- **Data teams** who need to know where a page's data comes from before building a pipeline.
- **No-code users** who want the endpoint and its parameters without reading network logs.
- **QA and integration engineers** who want to inventory the JSON calls a page makes.

### Features

- Ranked candidate endpoints per page with a numeric `score`.
- `urlTemplate` with variable query parameters (e.g. `?ajax={ajax}&year={year}`).
- Detection of the array of items in the response (`itemArrayPath`, `itemCount`, `itemFields`).
- Pagination hints when present in the call.
- Optional text matching to rank the call that carries the data you actually see.
- Optional CSS selectors to click (tabs, "load more", year links) and optional page scrolling.
- Ready-to-run `curlSnippet` and `fetchSnippet`.
- Sensitive header values such as cookies are redacted in the output.
- Pages that give no result, or that are blocked, produce an error row (not billed).

### Input

| Field | Type | Description | Default |
|---|---|---|---|
| `startUrls` (required) | array | Public pages to analyse. | – |
| `matchTexts` | string list | Texts visible on the page; responses containing them rank higher. | empty |
| `clickSelectors` | string list | CSS selectors to click after load. IDs starting with a digit need the form `a[id="2015"]`. | empty |
| `scrollPage` | boolean | Scroll a few times to trigger lazy-loaded data. | `true` |
| `waitAfterLoadSecs` | integer 0–30 | Extra wait for late calls. | `2` |
| `includeThirdParty` | boolean | Keep endpoints on other domains (Algolia, GraphQL gateways…). | `true` |
| `maxEndpointsPerPage` | integer 1–20 | Max ranked endpoints per page. | `5` |
| `responseSampleMaxChars` | integer 0–20000 | Characters of each response kept (0 to omit). | `1500` |
| `maxResults` | integer | Stop after this many billed endpoint rows. | `100` |
| `proxyConfiguration` | object | Datacenter proxy is enough for most pages. | Apify Proxy |

Example input:

```json
{
  "startUrls": [{ "url": "https://www.scrapethissite.com/pages/ajax-javascript/" }],
  "clickSelectors": ["a.year-link[id=\"2015\"]"],
  "maxResults": 10
}
```

### Output

Each row is one candidate endpoint. Real sample (response sample shortened):

```json
{
  "pageUrl": "https://www.scrapethissite.com/pages/ajax-javascript/",
  "pageTitle": "Oscar Winning Films: AJAX and Javascript | Scrape This Site | A public sandbox for learning web scraping",
  "rank": 1,
  "score": 89,
  "method": "GET",
  "endpointUrl": "https://www.scrapethissite.com/pages/ajax-javascript/?ajax=true&year=2015",
  "urlTemplate": "https://www.scrapethissite.com/pages/ajax-javascript/?ajax={ajax}&year={year}",
  "host": "www.scrapethissite.com",
  "isThirdParty": false,
  "resourceType": "xhr",
  "status": 200,
  "contentType": "application/json",
  "queryParams": { "ajax": "true", "year": "2015" },
  "requestBody": null,
  "requestHeaders": { "accept": "*/*", "cookie": "[REDACTED]", "x-requested-with": "XMLHttpRequest" },
  "usedCredentials": true,
  "responseSizeBytes": 1570,
  "responseSample": "[ { \"title\": \"Spotlight\", \"year\": 2015, \"awards\": 2, ... } ]",
  "jsonTopLevelKeys": ["[array]"],
  "itemArrayPath": "$",
  "itemCount": 16,
  "itemFields": ["title", "year", "awards", "nominations", "best_picture"],
  "matchedTexts": ["spotlight", "mad max: fury road", "the revenant"],
  "paginationHints": [],
  "curlSnippet": "curl -X GET 'https://www.scrapethissite.com/pages/ajax-javascript/?ajax=true&year=2015' -H 'accept: */*' ...",
  "fetchSnippet": "const res = await fetch(\"https://www.scrapethissite.com/...\", {...}); const data = await res.json();",
  "capturedAt": "2026-10-04T11:15:43.292Z",
  "error": null
}
```

#### Output fields & fill rates

Fill rates measured on an 11-row medium test run (share of rows where the field is filled):

| Field | Fill rate | Notes |
|---|---|---|
| `pageUrl`, `pageTitle`, `rank`, `score`, `method`, `endpointUrl`, `urlTemplate`, `host`, `isThirdParty`, `resourceType`, `status`, `contentType`, `requestHeaders`, `usedCredentials`, `responseSizeBytes`, `responseSample`, `jsonTopLevelKeys`, `itemArrayPath`, `itemCount`, `itemFields`, `curlSnippet`, `fetchSnippet`, `capturedAt` | 100% | |
| `queryParams` | 89% | Empty when the call has no query string. |
| `matchedTexts` | 56% | Filled when you provide `matchTexts` or visible texts match. |
| `paginationHints` | 44% | Only when the call exposes page/offset/cursor parameters. |
| `requestBody` | 0% | Only for POST calls with a body; the test set contained none. |
| `error` | 0% | `null` on normal rows; filled on error rows (none in this run). |

#### Error rows (not billed)

When a page yields no usable endpoint, you get one error row instead of an empty run. Verified on a real run with `https://example.com` (server-rendered, no XHR): the run succeeded with 1 row, `rank` and `endpointUrl` are `null`, and `error` is `"no XHR/fetch calls found (page may be server-rendered)"`.

```json
{
  "pageUrl": "https://example.com/",
  "pageTitle": "Example Domain",
  "rank": null,
  "endpointUrl": null,
  "capturedAt": "2026-10-04T11:25:25.727Z",
  "error": "no XHR/fetch calls found (page may be server-rendered)"
}
```

#### Mixed inputs

A run with three pages (one with `clickSelectors`, one infinite-scroll page, one server-rendered page) succeeded with 5 rows: 4 endpoint rows and 1 error row for the server-rendered page. A click that times out does not stop the run: it is written as a warning in the run log ("Click selector problems on …"). It only appears in a row's `error` text when the page produced no endpoint row.

### How to use via API

**curl**

```bash
curl -X POST "https://api.apify.com/v2/acts/ACTOR_ID/run-sync-get-dataset-items?token=YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://www.scrapethissite.com/pages/ajax-javascript/"}],"maxResults":10}'
```

**JavaScript**

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });
const run = await client.actor('ACTOR_ID').call({
  startUrls: [{ url: 'https://www.scrapethissite.com/pages/ajax-javascript/' }],
  maxResults: 10,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items[0].endpointUrl);
```

**Python**

```python
from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")
run = client.actor("ACTOR_ID").call(run_input={
    "startUrls": [{"url": "https://www.scrapethissite.com/pages/ajax-javascript/"}],
    "scrollPage": True,
    "maxResults": 10,
})
items = client.dataset(run["defaultDatasetId"]).list_items().items
print(items[0]["endpointUrl"])
```

Replace `ACTOR_ID` with this Actor's ID or `username~actor-name`.

### Pricing

Pay per result: you are billed for each endpoint row delivered (error rows are not billed). Price per 1,000 results depends on your Apify plan:

| Plan | Per 1,000 results |
|---|---|
| FREE | $1.50 |
| BRONZE | $1.30 |
| SILVER | $1.15 |
| GOLD and above | $1.00 |

Platform usage (browser compute, proxy) is charged separately by Apify. In the 11-row medium test run (3 pages), platform usage was about $0.0045. Platform usage mostly depends on how many pages you submit and how long each stays open (`waitAfterLoadSecs`, `scrollPage`, `clickSelectors`). Use `maxResults` to cap what you pay for results.

### Troubleshooting

- **No endpoint found:** the data may be in the HTML itself, or load only after an interaction. Add `clickSelectors`, keep `scrollPage` on, increase `waitAfterLoadSecs`, or add `matchTexts`.
- **Blocked or challenge page:** the row says "blocked or challenge page". This Actor does not bypass logins, paywalls or CAPTCHAs; try another page or proxy settings.
- **Click selector fails:** the problem is logged as a warning in the run log; the error row lists the click errors only when no endpoint was found for that page. IDs starting with a digit must be written `a[id="2015"]`.
- **Replayed endpoint returns an error:** some endpoints depend on cookies, tokens or headers set by the page (see `usedCredentials`, `requestHeaders`). Cookie values are redacted.
- **Run refused at start ("Field input.startUrls is required"):** `startUrls` is mandatory. Apify rejects an input without it before any run starts (no run, no charge); an empty list is rejected by the Actor with a clear message.
- **Empty `requestBody`:** normal for GET calls.

### FAQ

**Does it work on any website?** It works on public pages that load JSON via XHR/fetch. It cannot access pages behind a login and does not solve CAPTCHAs.

**Is it legal to use the endpoints I find?** The Actor only observes calls your browser would make on a public page. Check the site's terms and robots.txt, and handle personal data in line with GDPR, before reusing an endpoint.

**Does it return the full data?** No, it returns a candidate endpoint and a response sample (up to `responseSampleMaxChars`), so you can build a direct scraper.

**Does it capture WebSocket or HTML calls?** It targets XHR/fetch JSON responses.

### Keywords

find hidden API of website, hidden api finder, xhr capture, network requests scraper, reverse engineer website api, capture fetch requests, json endpoint finder, api discovery, website api finder, scraper developer tool

### Hashtags

\#hiddenapi #xhr #apidiscovery #reverseengineering #webscraping #json #fetch #devtools #scraping #network

# Changelog

This Actor's version history is a separate document: https://apify.com/leadsbrary/hidden-api-finder-network-capture/changelog.md

# Actor input Schema

## `startUrls` (type: `array`):

Public web pages to analyse. Each page is opened once in a headless browser and its XHR/fetch JSON calls are recorded.

## `matchTexts` (type: `array`):

Optional texts you can see on the page (a product name, a price). Responses containing them rank higher.

## `clickSelectors` (type: `array`):

Optional elements to click after load (tabs, 'load more', year links) to trigger more calls. IDs starting with a digit need the form a\[id="2015"].

## `scrollPage` (type: `boolean`):

Scroll down a few times to trigger lazy-loaded data.

## `waitAfterLoadSecs` (type: `integer`):

Extra time to wait for late calls after loading and scrolling.

## `includeThirdParty` (type: `boolean`):

Keep endpoints on other domains (Algolia, GraphQL gateways, CDNs). Known ad/analytics/consent calls are always dropped.

## `maxEndpointsPerPage` (type: `integer`):

Maximum ranked endpoints returned for each page.

## `responseSampleMaxChars` (type: `integer`):

Characters of each response body kept in the output (0 to omit).

## `maxResults` (type: `integer`):

Stop after this many billed endpoint rows (also protects your spending).

## `proxyConfiguration` (type: `object`):

Datacenter proxy is enough for most pages.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://www.scrapethissite.com/pages/ajax-javascript/"
    }
  ],
  "matchTexts": [],
  "clickSelectors": [
    "a.year-link[id=\"2015\"]"
  ],
  "scrollPage": true,
  "waitAfterLoadSecs": 2,
  "includeThirdParty": true,
  "maxEndpointsPerPage": 5,
  "responseSampleMaxChars": 1500,
  "maxResults": 10,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://www.scrapethissite.com/pages/ajax-javascript/"
        }
    ],
    "clickSelectors": [
        "a.year-link[id=\"2015\"]"
    ],
    "maxResults": 10,
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("leadsbrary/hidden-api-finder-network-capture").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://www.scrapethissite.com/pages/ajax-javascript/" }],
    "clickSelectors": ["a.year-link[id=\"2015\"]"],
    "maxResults": 10,
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("leadsbrary/hidden-api-finder-network-capture").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://www.scrapethissite.com/pages/ajax-javascript/"
    }
  ],
  "clickSelectors": [
    "a.year-link[id=\\"2015\\"]"
  ],
  "maxResults": 10,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call leadsbrary/hidden-api-finder-network-capture --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,leadsbrary/hidden-api-finder-network-capture"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/ytuQgNpj4Kow1PoZQ/builds/TczVk0tGxzLke616Z/openapi.json