# Website to OpenAPI Spec (Hidden API Reverse-Engineering) (`jixiaohao/network-to-openapi`) Actor

Browse any website or web app in a real browser, capture its XHR, fetch and GraphQL calls, and get an OpenAPI 3.1 / Swagger spec of the hidden API behind it: endpoints, parameters, JSON schemas, auth and curl examples. Also converts HAR files. Works as an MCP server for AI agents.

- **URL**: https://apify.com/jixiaohao/network-to-openapi.md
- **Developed by:** [Ji](https://apify.com/jixiaohao) (community)
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $5.00 / 1,000 page captureds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to OpenAPI Spec (Hidden API Reverse-Engineering)

Point it at a website or web app. It opens the pages in a real Chromium browser, records every API call the front end makes (XHR, fetch, GraphQL), and writes an **OpenAPI 3.1 spec of the API behind the site**, ready for Swagger UI, Postman, Insomnia, code generators or an AI agent.

What you get for each endpoint:

- **Path templates** with named parameters: `/api/v1/products/8812/reviews` and `/api/v1/products/9031/reviews` become `/api/v1/products/{productId}/reviews`; slug URLs like `/catalog/blue-cotton-shirt` are recognized too
- **Query, header and path parameters**, typed (integer, boolean, date, uuid…) and marked required when every call had them
- **Request and response JSON Schemas** merged across every call seen: optional vs required fields, nullable fields, string formats (date-time, email, uri, uuid), id-keyed maps
- **Every status code** seen (200, 201, 404, 401…) with its own schema
- **Auth** detected and declared as security schemes: Bearer/JWT, Basic, API keys in headers or query, session cookies
- **GraphQL**: each operation (name, query, variables schema, result schema), including persisted queries
- **Redacted examples** and a ready-to-run **curl** command; tokens, passwords, API keys, cookies, JWTs and emails are masked

Also converts **HAR files** (DevTools > Network > Save all as HAR, Charles, Proxyman, mitmproxy) into the same spec, with no browsing needed.

### What it's for

- **Documenting an undocumented or legacy API** your client's front end already uses, as a Swagger spec in minutes instead of days of DevTools work
- **Building an integration or scraper** against the JSON API behind a page instead of parsing HTML
- **Competitive and technical research**: what endpoints, backends (Algolia, Supabase, Shopify, Contentful, Firebase…) and auth a product uses
- **Giving an AI agent a tool definition** for a site: the spec drops straight into agent frameworks that load OpenAPI tools

### How it compares

| | This actor | API & Endpoint Discovery (Store) | mitmproxy2swagger / har-to-openapi (CLI) | Manual DevTools |
| --- | --- | --- | --- | --- |
| Output | **OpenAPI 3.1 spec** + endpoint table + curl | Endpoint list with field names | OpenAPI spec | Notes |
| Captures traffic for you | **Yes, real browser, scrolls and clicks "Load more"** | Yes | No, you record it yourself | You click around |
| Path parameters | **Automatic, with names** (`{productId}`) | No | Manual: you mark them in a config and run it again | Manual |
| Request/response schemas | **Full JSON Schema, merged across calls** | Response field names only | Basic | Manual |
| Auth, GraphQL operations | **Detected** | Auth flag only | No | Manual |
| Secrets in output | **Redacted** | n/a | Up to you | n/a |
| Runs in the cloud / MCP | **Yes, both** | Cloud | Local | Local |

### Pricing

Pay per event, only for work that produced something:

| Event | Price | When |
| --- | --- | --- |
| `page-captured` | $0.005 | One page opened and browsed (failed page loads are free) |
| `endpoint-documented` | $0.01 | One distinct endpoint (method + path template) in the spec |

A typical run (10 pages of a web app, 15 endpoints found) costs **$0.20**. HAR conversion has no page charges: a HAR with 40 endpoints costs $0.40.

Runs default to 4 GB of memory (2 GB minimum): Chromium needs it, and more memory also means more CPU on Apify. Compute is included in the event prices.

If you set a maximum charge for the run, browsing stops when it's reached and the spec keeps the endpoints that were called most often. Free-plan runs browse up to 10 pages.

### Input

| Field | Default | Notes |
| --- | --- | --- |
| `startUrls` | | Pages to start from |
| `maxPages` | 10 | Pages to open in total. Links on the same host are followed, **preferring pages of a kind not seen yet** (after one product page, the next pick is a category or account page, not another product) |
| `crawlScope` | `sameHostname` | `sameSite` also follows subdomains; `none` opens only the start URLs |
| `harUrls` | | HAR files to convert, by URL |
| `apiScope` | `site` | `site`: the site's own API hosts and hosted backends its front end calls directly. `all`: every host except analytics, ads, chat widgets and other trackers |
| `includeUrlPatterns`, `excludeUrlPatterns` | | Regexes to narrow down which API URLs are documented |
| `clickSelectors` | | CSS selectors to click on every page (tabs, filters, "details") to trigger API calls that only happen on interaction |
| `clickLoadMore` | true | Clicks up to 2 "Load more / Show more / Next" buttons per page. Never clicks anything that looks like delete, buy, submit or log out; log-out and delete links are never followed |
| `scrolls`, `waitSecs` | 3, 3 | Scroll steps (lazy loading, infinite scroll) and extra wait per page |
| `cookies`, `headers` | | For apps behind a login: cookies exported with Cookie-Editor (JSON), or an `Authorization` header |
| `title`, `includeExamples`, `redactPersonalData` | | Spec title; examples on/off; email redaction (secrets are always redacted) |
| `proxyConfiguration` | None | Pages load fastest from Apify's own IPs. Turn on a proxy (residential for strict sites) if the site blocks the run; if nothing comes through the proxy, the run browses again without it |

#### Example: a public web app

```json
{
  "startUrls": [{ "url": "https://crates.io/crates?sort=downloads" }],
  "maxPages": 8
}
```

This run opens the listing and six crate pages and documents the API behind crates.io: `GET /api/v1/crates` (with `page`, `per_page`, `sort`), `/api/v1/crates/{crateId}`, `/api/v1/crates/{crateId}/downloads`, `/api/v1/crates/{crateId}/owners`, `/api/v1/summary` and `/api/v1/site_metadata`, for $0.10.

#### Example: an app behind a login, only its `/api/` calls

```json
{
  "startUrls": [{ "url": "https://app.example.com/dashboard" }, { "url": "https://app.example.com/settings" }],
  "maxPages": 20,
  "cookies": [{ "name": "session", "value": "…", "domain": ".example.com" }],
  "includeUrlPatterns": ["/api/"],
  "clickSelectors": ["[role=tab]", ".filters button"]
}
```

#### Example: convert a HAR file

```json
{ "harUrls": ["https://api.apify.com/v2/key-value-stores/<store>/records/my-session.har"] }
```

### Output

- **`openapi.yaml`** and **`openapi.json`** in the run's key-value store (linked from the run's Output tab)
- **Dataset**: one row per endpoint, for a quick look or a spreadsheet
- **`SUMMARY`**: pages browsed, API calls per page, warnings

Dataset row:

```json
{
  "method": "GET",
  "path": "/api/v1/products/{productId}",
  "url": "https://shop.example.com/api/v1/products/{productId}",
  "operationId": "getProductsByProductId",
  "calls": 6,
  "statusCodes": [200, 404],
  "pathParams": ["productId"],
  "queryParams": [],
  "auth": ["Bearer token (JWT)"],
  "responseContentType": "application/json",
  "responseFields": ["id", "slug", "name", "price", "tags", "updatedAt", "discount", "stock"],
  "graphqlOperations": [],
  "exampleUrl": "https://shop.example.com/api/v1/products/101",
  "curl": "curl -X GET 'https://shop.example.com/api/v1/products/101' \\\n  -H 'authorization: Bearer <redacted>' \\\n  -H 'x-client-version: 4.2.0'",
  "seenOnPages": ["https://shop.example.com/product/blue-cotton-shirt"]
}
```

Spec excerpt:

```yaml
paths:
  /api/v1/products/{productId}:
    get:
      tags: [products]
      operationId: getProductsByProductId
      parameters:
        - { name: productId, in: path, required: true, schema: { type: integer }, example: 101 }
        - { name: x-client-version, in: header, required: true, schema: { type: string }, example: 4.2.0 }
      security:
        - bearerAuth: []
      responses:
        "200":
          description: OK
          content:
            application/json:
              schema: { $ref: "#/components/schemas/GetProductsByProductIdResponse" }
        "404":
          description: Not Found
          content:
            application/json:
              schema:
                type: object
                properties: { error: { type: string }, message: { type: string } }
                required: [error, message]
components:
  securitySchemes:
    bearerAuth: { type: http, scheme: bearer, bearerFormat: JWT }
```

The spec documents what the site was **seen** doing: an endpoint the pages never called, or a field that never appeared, is not in it, and `required` means "present in every observed response". Browse more pages, click more (`clickSelectors`), or merge in a HAR of a manual session to cover more.

### For AI agents (Standby + MCP)

The actor also runs as a Standby web server:

- **MCP:** `https://jixiaohao--network-to-openapi.apify.actor/mcp` with the tool `synthesize_openapi` (`url`, `maxPages` default 3, `harUrl`, `apiScope`, `clickSelectors`, `cookies`, `includeUrlPatterns`). It returns the endpoint list and the spec as YAML. Add it to Claude or any MCP client with your Apify token (`?token=...` or a Bearer header).
- **REST:** `GET /openapi?url=https://example.com&maxPages=3` (endpoints + spec as JSON), `&format=yaml` for the spec alone, or `POST /openapi` with a JSON body.

### How it works

1. **Capture.** Each page loads in headless Chromium (images, video, fonts and tracker scripts blocked to save time; a page that's slow to finish loading is still captured). After the network goes quiet the actor scrolls, clicks "Load more" and any selectors you gave, and records every request and response the page's scripts make. Service workers are bypassed so no call is hidden.
2. **Filter.** Analytics, ads, consent banners, chat widgets, error trackers, feature-flag polling, static assets and CORS preflights are dropped; first-party API hosts and known hosted backends are kept.
3. **Template paths.** Segments that are clearly ids (numbers, UUIDs, hashes, version numbers like `1.0.229`, mixed ids like `B08N5WRWNW`) become parameters. Sibling URLs that differ in one segment are merged when the values look like slugs or the responses share a structure, but kept apart when they are different resources (`/api/users` vs `/api/orders`).
4. **Infer schemas.** All bodies for an endpoint are merged into one JSON Schema per status code and media type.
5. **Redact and write** the spec, the endpoint table and curl commands.

Only document APIs you are allowed to access, and respect the site's terms. The actor sends the same requests a normal visitor's browser would; it does not fuzz or probe endpoints.

### Development

```bash
npm install
npm test        # unit tests plus an end-to-end run in Chromium against the demo shop in test/demo-app.mjs
node test/demo-app.mjs 4321   # the demo shop on its own
```

# Actor input Schema

## `startUrls` (type: `array`):

Pages of the website or web app whose API you want documented. Links on the same host are followed up to Max pages, preferring pages of a kind not seen yet (one product page teaches as much as ten).

## `maxPages` (type: `integer`):

How many pages to open, start URLs included. Each page is billed as page-captured. Free-plan runs browse at most 10.

## `crawlScope` (type: `string`):

Which links to follow from the start pages.

## `harUrls` (type: `array`):

URLs of HAR files to convert (Chrome/Firefox DevTools > Network > Save all as HAR, or Charles, Proxyman, mitmproxy). Upload one to a key-value store or any file host and paste its URL. Works with or without pages to browse; no page charges for HAR traffic.

## `apiScope` (type: `string`):

Site: the site's own API hosts plus hosted backends its front end calls directly (Algolia, Supabase, Firebase, Shopify, Contentful, AWS API Gateway...). All: every host except analytics, ads, chat widgets and other trackers.

## `includeUrlPatterns` (type: `array`):

Regular expressions; when set, only API calls whose URL matches one are documented (and matching calls are kept even if they would be filtered out otherwise), e.g. "/api/v2/".

## `excludeUrlPatterns` (type: `array`):

Regular expressions for API calls to leave out, e.g. "/feature-flags".

## `clickSelectors` (type: `array`):

CSS selectors clicked on every page (first 3 matches each) to trigger API calls that only happen on interaction: tabs, filters, "show details". Buttons that could change data (delete, buy, submit, log out) should not be listed here.

## `clickLoadMore` (type: `boolean`):

Click up to 2 visible buttons labelled Load more, Show more, Next and similar on each page. Anything that looks like delete, buy, submit or log out is never clicked.

## `scrolls` (type: `integer`):

Scroll down this many screens on each page to trigger lazy loading and infinite scroll.

## `waitSecs` (type: `integer`):

Extra time on each page after it loads and the network goes quiet, for late API calls.

## `cookies` (type: `array`):

To document the API behind a login: cookies exported from your browser with an extension such as Cookie-Editor (JSON export). Their values are redacted in the output.

## `headers` (type: `object`):

Headers sent with every request, e.g. {"Authorization": "Bearer ..."} for an app that keeps its token outside cookies. Values are redacted in the output.

## `title` (type: `string`):

info.title of the generated spec. Default: "<host> API (inferred)".

## `includeExamples` (type: `boolean`):

Put a redacted example request and response for each endpoint in the spec.

## `redactPersonalData` (type: `boolean`):

Replace email addresses in examples. Tokens, passwords, API keys, cookies and JWTs are always redacted.

## `blockMedia` (type: `boolean`):

Faster and cheaper; turn off only if the site's API calls depend on media loading.

## `maxConcurrency` (type: `integer`):

Browser tabs open at once.

## `proxyConfiguration` (type: `object`):

No proxy by default: pages load fastest from Apify's own IPs. Turn on a proxy (residential for strict sites) if the site blocks the run. If nothing is captured through the proxy, the run browses again without it.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://crates.io/crates?sort=downloads"
    }
  ],
  "maxPages": 8,
  "crawlScope": "sameHostname",
  "apiScope": "site",
  "clickLoadMore": true,
  "scrolls": 3,
  "waitSecs": 3,
  "includeExamples": true,
  "redactPersonalData": true,
  "blockMedia": true,
  "maxConcurrency": 3,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `openapiYaml` (type: `string`):

No description

## `openapiJson` (type: `string`):

No description

## `endpoints` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://crates.io/crates?sort=downloads"
        }
    ],
    "maxPages": 8,
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("jixiaohao/network-to-openapi").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://crates.io/crates?sort=downloads" }],
    "maxPages": 8,
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("jixiaohao/network-to-openapi").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://crates.io/crates?sort=downloads"
    }
  ],
  "maxPages": 8,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call jixiaohao/network-to-openapi --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,jixiaohao/network-to-openapi"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/rmIxp3vaW9tOY2Lae/builds/Skex7om8cBJgEtse0/openapi.json