Find Hidden API of Website: Capture XHR/JSON Endpoints

Find hidden API of website pages without opening DevTools. Give this Actor a public page URL: it loads the page in a headless browser, records the XHR/fetch JSON calls, drops analytics and ad noise, and returns a ranked list of candidate endpoints with URL template, query parameters, pagination hints, a response sample and ready-to-run curl and fetch snippets. Export as JSON, CSV, Excel, XML or HTML.

What does Hidden API Finder do?

Many modern pages load their data from a JSON endpoint behind the scenes. Scraping that endpoint is faster and more stable than parsing HTML, but finding it means manually digging through the browser's Network tab. This Actor does that reverse-engineering step for you:

1. Opens each page in a real headless browser (optionally clicking elements and scrolling to trigger more calls).
2. Records the JSON requests the page makes.
3. Filters out known ad, analytics and consent calls.
4. Ranks the remaining calls, giving a higher score to responses that contain texts you can see on the page (a product name, a price) and that hold arrays of items.
5. Outputs one row per candidate endpoint, ready to reuse in your own scraper.

Who is it for / use cases

• Scraper developers who want to replace a slow browser scraper by a direct API call.
• Data teams who need to know where a page's data comes from before building a pipeline.
• No-code users who want the endpoint and its parameters without reading network logs.
• QA and integration engineers who want to inventory the JSON calls a page makes.

Features

• Ranked candidate endpoints per page with a numeric score.
• urlTemplate with variable query parameters (e.g. ?ajax={ajax}&year={year}).
• Detection of the array of items in the response (itemArrayPath, itemCount, itemFields).
• Pagination hints when present in the call.
• Optional text matching to rank the call that carries the data you actually see.
• Optional CSS selectors to click (tabs, "load more", year links) and optional page scrolling.
• Ready-to-run curlSnippet and fetchSnippet.
• Sensitive header values such as cookies are redacted in the output.
• Pages that give no result, or that are blocked, produce an error row (not billed).

Input

Field	Type	Description	Default
startUrls (required)	array	Public pages to analyse.	–
matchTexts	string list	Texts visible on the page; responses containing them rank higher.	empty
clickSelectors	string list	CSS selectors to click after load. IDs starting with a digit need the form a[id="2015"].	empty
scrollPage	boolean	Scroll a few times to trigger lazy-loaded data.	true
waitAfterLoadSecs	integer 0–30	Extra wait for late calls.	2
includeThirdParty	boolean	Keep endpoints on other domains (Algolia, GraphQL gateways…).	true
maxEndpointsPerPage	integer 1–20	Max ranked endpoints per page.	5
responseSampleMaxChars	integer 0–20000	Characters of each response kept (0 to omit).	1500
maxResults	integer	Stop after this many billed endpoint rows.	100
proxyConfiguration	object	Datacenter proxy is enough for most pages.	Apify Proxy

Example input:

{
  "startUrls": [{ "url": "https://www.scrapethissite.com/pages/ajax-javascript/" }],
  "clickSelectors": ["a.year-link[id=\"2015\"]"],
  "maxResults": 10
}

Output

Each row is one candidate endpoint. Real sample (response sample shortened):

{
  "pageUrl": "https://www.scrapethissite.com/pages/ajax-javascript/",
  "pageTitle": "Oscar Winning Films: AJAX and Javascript | Scrape This Site | A public sandbox for learning web scraping",
  "rank": 1,
  "score": 89,
  "method": "GET",
  "endpointUrl": "https://www.scrapethissite.com/pages/ajax-javascript/?ajax=true&year=2015",
  "urlTemplate": "https://www.scrapethissite.com/pages/ajax-javascript/?ajax={ajax}&year={year}",
  "host": "www.scrapethissite.com",
  "isThirdParty": false,
  "resourceType": "xhr",
  "status": 200,
  "contentType": "application/json",
  "queryParams": { "ajax": "true", "year": "2015" },
  "requestBody": null,
  "requestHeaders": { "accept": "*/*", "cookie": "[REDACTED]", "x-requested-with": "XMLHttpRequest" },
  "usedCredentials": true,
  "responseSizeBytes": 1570,
  "responseSample": "[ { \"title\": \"Spotlight\", \"year\": 2015, \"awards\": 2, ... } ]",
  "jsonTopLevelKeys": ["[array]"],
  "itemArrayPath": "$",
  "itemCount": 16,
  "itemFields": ["title", "year", "awards", "nominations", "best_picture"],
  "matchedTexts": ["spotlight", "mad max: fury road", "the revenant"],
  "paginationHints": [],
  "curlSnippet": "curl -X GET 'https://www.scrapethissite.com/pages/ajax-javascript/?ajax=true&year=2015' -H 'accept: */*' ...",
  "fetchSnippet": "const res = await fetch(\"https://www.scrapethissite.com/...\", {...}); const data = await res.json();",
  "capturedAt": "2026-10-04T11:15:43.292Z",
  "error": null
}

Output fields & fill rates

Fill rates measured on an 11-row medium test run (share of rows where the field is filled):

Field	Fill rate	Notes
pageUrl, pageTitle, rank, score, method, endpointUrl, urlTemplate, host, isThirdParty, resourceType, status, contentType, requestHeaders, usedCredentials, responseSizeBytes, responseSample, jsonTopLevelKeys, itemArrayPath, itemCount, itemFields, curlSnippet, fetchSnippet, capturedAt	100%
queryParams	89%	Empty when the call has no query string.
matchedTexts	56%	Filled when you provide matchTexts or visible texts match.
paginationHints	44%	Only when the call exposes page/offset/cursor parameters.
requestBody	0%	Only for POST calls with a body; the test set contained none.
error	0%	null on normal rows; filled on error rows (none in this run).

Error rows (not billed)

When a page yields no usable endpoint, you get one error row instead of an empty run. Verified on a real run with https://example.com (server-rendered, no XHR): the run succeeded with 1 row, rank and endpointUrl are null, and error is "no XHR/fetch calls found (page may be server-rendered)".

{
  "pageUrl": "https://example.com/",
  "pageTitle": "Example Domain",
  "rank": null,
  "endpointUrl": null,
  "capturedAt": "2026-10-04T11:25:25.727Z",
  "error": "no XHR/fetch calls found (page may be server-rendered)"
}

Mixed inputs

A run with three pages (one with clickSelectors, one infinite-scroll page, one server-rendered page) succeeded with 5 rows: 4 endpoint rows and 1 error row for the server-rendered page. A click that times out does not stop the run: it is written as a warning in the run log ("Click selector problems on …"). It only appears in a row's error text when the page produced no endpoint row.

How to use via API

curl

curl -X POST "https://api.apify.com/v2/acts/ACTOR_ID/run-sync-get-dataset-items?token=YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://www.scrapethissite.com/pages/ajax-javascript/"}],"maxResults":10}'

JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });
const run = await client.actor('ACTOR_ID').call({
  startUrls: [{ url: 'https://www.scrapethissite.com/pages/ajax-javascript/' }],
  maxResults: 10,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items[0].endpointUrl);

Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")
run = client.actor("ACTOR_ID").call(run_input={
    "startUrls": [{"url": "https://www.scrapethissite.com/pages/ajax-javascript/"}],
    "scrollPage": True,
    "maxResults": 10,
})
items = client.dataset(run["defaultDatasetId"]).list_items().items
print(items[0]["endpointUrl"])

Replace ACTOR_ID with this Actor's ID or username~actor-name.

Pricing

Pay per result: you are billed for each endpoint row delivered (error rows are not billed). Price per 1,000 results depends on your Apify plan:

Plan	Per 1,000 results
FREE	$1.50
BRONZE	$1.30
SILVER	$1.15
GOLD and above	$1.00

Platform usage (browser compute, proxy) is charged separately by Apify. In the 11-row medium test run (3 pages), platform usage was about $0.0045. Platform usage mostly depends on how many pages you submit and how long each stays open (waitAfterLoadSecs, scrollPage, clickSelectors). Use maxResults to cap what you pay for results.

Troubleshooting

• No endpoint found: the data may be in the HTML itself, or load only after an interaction. Add clickSelectors, keep scrollPage on, increase waitAfterLoadSecs, or add matchTexts.
• Blocked or challenge page: the row says "blocked or challenge page". This Actor does not bypass logins, paywalls or CAPTCHAs; try another page or proxy settings.
• Click selector fails: the problem is logged as a warning in the run log; the error row lists the click errors only when no endpoint was found for that page. IDs starting with a digit must be written a[id="2015"].
• Replayed endpoint returns an error: some endpoints depend on cookies, tokens or headers set by the page (see usedCredentials, requestHeaders). Cookie values are redacted.
• Run refused at start ("Field input.startUrls is required"): startUrls is mandatory. Apify rejects an input without it before any run starts (no run, no charge); an empty list is rejected by the Actor with a clear message.
• Empty requestBody: normal for GET calls.

FAQ

Does it work on any website? It works on public pages that load JSON via XHR/fetch. It cannot access pages behind a login and does not solve CAPTCHAs.

Is it legal to use the endpoints I find? The Actor only observes calls your browser would make on a public page. Check the site's terms and robots.txt, and handle personal data in line with GDPR, before reusing an endpoint.

Does it return the full data? No, it returns a candidate endpoint and a response sample (up to responseSampleMaxChars), so you can build a direct scraper.

Does it capture WebSocket or HTML calls? It targets XHR/fetch JSON responses.

Keywords

find hidden API of website, hidden api finder, xhr capture, network requests scraper, reverse engineer website api, capture fetch requests, json endpoint finder, api discovery, website api finder, scraper developer tool

Hashtags

#hiddenapi #xhr #apidiscovery #reverseengineering #webscraping #json #fetch #devtools #scraping #network