OpenAPI Operation Metadata Scraper

Extract paths, methods, parameters and response descriptions from public OpenAPI JSON.

Example.com inputs are placeholders, not verified compatible content. Replace them with your permitted source. Run npm run smoke for an account-free deterministic example. JSON Feed, RSS and table examples use the separately recorded small live-test sources; they do not imply compatibility with every site.

What it does

Produces api-operation records for api directory maintainers. Values are extracted deterministically from public pages; unavailable fields remain null.

Common use cases

Extract paths, methods, parameters and response descriptions from public OpenAPI JSON. Use the resulting dataset in scheduled tasks, API pipelines, spreadsheets, or agent workflows.

Input

Field	Type	Description
startUrls	array	Public HTTP(S) URLs. No credentials or local-network targets.
maxPages	integer	Maximum pages scheduled in this bounded run.
maxResults	integer	Maximum visible api-operation records returned.
respectRobotsTxt	boolean	Honor robots rules and supported crawl delays. Unavailable rules fail closed.
keywords	array	Keywords. See README for extraction behavior.
excludeKeywords	array	Exclude Keywords. See README for extraction behavior.
concurrency	integer	Maximum concurrent page handlers. Per-origin delays still apply.
requestDelayMs	integer	Minimum spacing between request starts to the same origin.
timeoutSecs	integer	Maximum individual request duration in seconds.
retries	integer	Retries for page handlers. Access blocks are not retried.
proxyConfiguration	object	Optional Apify or public custom proxy. Direct connections are the default.

Output

Field	Type
recordId	string or null
sourceUrl	string or null
extractedAt	string or null
method	string or null
path	string or null
operationId	string or null
summary	string or null
description	string or null
tags	array or null
parameters	array or null
responses	array or null
securitySchemeNames	array or null
serverUrls	array or null
deprecated	boolean or null

Results are in the default Dataset. RUN_SUMMARY, FAILURES and DIAGNOSTICS are available in the default key-value store. Diagnostics are not billed entity events. JSON, CSV and spreadsheet exports use Apify Dataset.

Example

Replace example.com with a public source relevant to this Actor. Plain example.com has no products, jobs or opportunities; an empty feed there is expected. Run the included local smoke test for a deterministic working example.

{
  "startUrls": [
    "https://example.com/openapi.json"
  ],
  "maxPages": 1,
  "maxResults": 10
}

Example output

The following records come from synthetic local fixtures, not a live website or claimed customer data.

[
  {
    "recordId": "38eea35fa2dac449b5a3af3c28a04520d76bd3866ccff6393a0970ecf244d274",
    "sourceUrl": "https://fixture.example/index",
    "extractedAt": "2026-09-29T23:35:03.158Z",
    "method": "GET",
    "path": "/things",
    "operationId": "listThings",
    "summary": "Fixture Alpha",
    "description": null,
    "tags": [],
    "parameters": [
      {
        "name": "q",
        "in": "query",
        "required": false,
        "description": null,
        "schemaType": "string",
        "reference": null
      }
    ],
    "responses": [
      {
        "status": "200",
        "description": "Success",
        "contentTypes": [
          "application/json"
        ],
        "reference": null
      }
    ],
    "securitySchemeNames": [
      "apiKey"
    ],
    "serverUrls": [
      "https://api.example.test/"
    ],
    "deprecated": false
  }
]

Pricing model

One api-operation event per visible result record. Event prices are configured in Apify Console, never in extraction code. The SDK enforces the run's maxTotalChargeUsd. Failed pages and diagnostic-only messages are not billed. Do not configure an additional automatic default-dataset-item event.

How it works

Crawlee BasicCrawler fetches explicitly supplied public URLs through a DNS-pinned, bounded HTTP transport. The dedicated json parser reads the response without a browser. No linked resources, external references or media are downloaded. No external paid API is required.

Limits

• OpenAPI 3.x JSON only, not YAML or Swagger 2. External references are not fetched. Parameter examples/defaults and credential values are omitted.
• Public GET/HEAD content only; no login, form submission, CAPTCHA solving or browser stealth.
• Responses are capped at 2 MiB decoded. Browser mode caps requests per page and blocks media, fonts, downloads, service workers and WebSockets.
• Missing/blocked robots rules fail closed. Robots crawl delays above 60 seconds are not supported.
• Native-format Actor: no browser mode, no browser installation in its Dockerfile. Begin at 512 MB and measure representative input sizes.

Responsible use

Use only public content you are authorized to access. Respect source terms, licenses and applicable law. No private-account extraction, personal-data enrichment, credential input or access-control bypass is provided. Cookie values are never returned.

Local development

Requires Node.js 24 LTS. All commands below run from this Actor directory.

npm ci --ignore-scripts
npm run input:example
# Edit storage/key_value_stores/default/INPUT.json with your public URLs.
npm run start:dev
npm test
npm run typecheck
npm run lint
npm run validate
npm run build
npm start
npm run smoke
npx apify run
docker build -t openapi-operation-scraper .

Browser testing: set PLAYWRIGHT_BROWSERS_PATH to an Actor-local .cache/browsers directory, then run npx playwright install chromium. Docker installs its own matching browser.

Deployment

Deploy only after validating the Docker build and a representative target. Deployment is not performed by installation or tests.

npx apify login
npx apify push

Then configure the exact event in Console, set pricing and spending limits, test a private run, complete PUBLICATION_CHECKLIST.md, and deliberately enable Store visibility.

API usage

Replace YOUR_USERNAME with your Apify account name. Keep the token in an environment variable rather than input JSON.

$curl -X POST "https://api.apify.com/v2/acts/YOUR_USERNAME~openapi-operation-scraper/runs?maxTotalChargeUsd=1" -H "Authorization: Bearer $APIFY_TOKEN" -H "Content-Type: application/json" --data-binary @examples/basic-input.json

Monitor the returned run ID and read its default Dataset. Store/API names and price configuration are account-side settings.

Verified platform example

The saved Console input performs a bounded HTTP run with one page and at most two result records. It fetches an explicitly synthetic demonstration fixture from a public Apify key-value store maintained for these examples. These records demonstrate extraction and billing, and are not live commercial data or evidence of compatibility with blocked sites.

Use the saved default input in Apify Console, or the repository examples/basic-input.json. Replace its source with your own permitted URL and keep limits small for the first run.

The following unmodified record was returned by a successful Apify platform run on 2026-09-30:

[
  {
    "recordId": "ed0d0a86b182b3387e21c90d98605bcb018ad2eb77fe3dcb2656c970ab4114b7",
    "sourceUrl": "https://api.apify.com/v2/key-value-stores/oa2wsF7nVnW6shckK/records/demo-beb3fe8184704b9d9e64",
    "extractedAt": "2026-09-30T15:32:24.402Z",
    "method": "GET",
    "path": "/things",
    "operationId": "listThings",
    "summary": "Fixture Alpha",
    "description": null,
    "tags": [],
    "parameters": [
      {
        "name": "q",
        "in": "query",
        "required": false,
        "description": null,
        "schemaType": "string",
        "reference": null
      }
    ],
    "responses": [
      {
        "status": "200",
        "description": "Success",
        "contentTypes": [
          "application/json"
        ],
        "reference": null
      }
    ],
    "securitySchemeNames": [
      "apiKey"
    ],
    "serverUrls": [
      "https://api.example.test/"
    ],
    "deprecated": false
  }
]

Introductory pricing: $0.50 per 1,000 api-operation results, plus Apify platform usage. The standard Actor start event costs $0.00005 per GB (minimum one event). Empty output and duplicate records incur no primary result event; the start fee and any platform usage still apply. The SDK enforces the event spending limit; platform usage is billed separately.