# Katana Web Crawler (ProjectDiscovery) (`rl1987/pd-katana`) Actor Crawl websites with ProjectDiscovery Katana and stream results as structured JSONL into an Apify dataset. - **URL**: https://apify.com/rl1987/pd-katana.md - **Developed by:** [R.L.](https://apify.com/rl1987) (community) - **Categories:** Developer tools, Automation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.01 / actor start This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Katana Web Crawler – Fast Website URL & Endpoint Extractor **Katana Web Crawler** is a fast, no-code **website crawler and URL extractor** built on [**ProjectDiscovery Katana**](https://github.com/projectdiscovery/katana), the popular open-source crawling and spidering framework. Point it at any website, and it **crawls every page, link, JavaScript file, form, and API endpoint**, then delivers the results as a clean, **structured dataset you can download as JSON, CSV, Excel, or XML** — or pull straight from the [Apify API](https://docs.apify.com/api/v2). Run it in the cloud with **zero setup**: no Go install, no binaries, no servers. You get **headless Chromium crawling, JavaScript endpoint discovery, proxy rotation, scheduling, and integrations** (Make, Zapier, Google Sheets, Slack, and more) out of the box. > **Try it now:** paste a URL like `https://example.com`, hit **Start**, and watch discovered URLs stream into your dataset in seconds. ### What is Katana Web Crawler? Katana Web Crawler is a serverless [Apify Actor](https://apify.com/actors) that **scrapes all URLs from a website** and maps its complete structure. It's the fastest way to **crawl a site, list every reachable link, and export the URLs** without writing code. Use it for **web crawling, site mapping, URL discovery, attack-surface enumeration, SEO audits, and API endpoint discovery**. ### What does Katana Web Crawler do? - 🕷️ **Crawls entire websites** in standard (fast HTTP) or **headless Chromium** mode. - 🔗 **Extracts every URL** — links, assets, scripts, forms, and inputs. - 🧩 **Discovers hidden API endpoints** parsed from **JavaScript files** (`js-crawl` + `jsluice`). - 📡 **Captures XHR / fetch API calls** made by JavaScript-heavy single-page apps. - 📂 **Finds known files** such as `robots.txt` and `sitemap.xml`. - 🎯 **Filters precisely** with scope, regex, extension, and DSL conditions. - 🛠️ **Detects technologies** and extracts forms while it crawls. - 📊 **Exports flat, ready-to-use data** — every field is its own column. ### Why use Katana Web Crawler? Whether you're a **security researcher, penetration tester, SEO specialist, or developer**, this crawler turns any website into structured data in minutes: - **Attack-surface mapping** — enumerate every reachable endpoint of a web app before a pentest or bug-bounty hunt. - **Security reconnaissance** — feed the URL list into scanners like **Nuclei, httpx, or ffuf**. - **API endpoint discovery** — surface undocumented REST/GraphQL endpoints referenced in JavaScript. - **SEO & site audits** — inventory pages, assets, redirects, and broken links at scale. - **Content & asset discovery** — find every script, image, PDF, and download link on a domain. - **Migration & QA** — snapshot a site's full URL structure before and after a release. ### How to use Katana Web Crawler to crawl a website No coding required — you can run your first crawl in under a minute: 1. Click **Try for free** / open the Actor and go to the **Input** tab. 2. Add one or more **Target URLs** (e.g. `https://example.com`). 3. Set the crawl **depth**, and optionally enable **JavaScript crawling** or **headless** mode for modern web apps. 4. (Optional) Add **scope, regex, or extension filters** to collect only what you need. 5. Click **Start** and watch URLs stream into the **Output** tab in real time. 6. **Download** the dataset as JSON, CSV, Excel, or XML — or fetch it via the [Apify API](https://docs.apify.com/api/v2). Prefer automation? **Schedule** recurring crawls or trigger the Actor from your own app, Make, or Zapier. ### Features at a glance | Capability | Supported | |------------|-----------| | Standard (HTTP) crawling | ✅ | | Headless Chromium crawling | ✅ | | JavaScript endpoint parsing (js-crawl, jsluice) | ✅ | | XHR / fetch API extraction | ✅ | | Form extraction & auto-fill | ✅ | | Technology detection | ✅ | | robots.txt & sitemap.xml crawling | ✅ | | Scope, regex, extension & DSL filters | ✅ | | Proxy support (HTTP/SOCKS5) | ✅ | | Rate limiting & concurrency control | ✅ | | JSON / CSV / Excel / XML export | ✅ | | API access, scheduling & integrations | ✅ | ### Input All fields are configured on the **Input** tab, grouped into sections. The only required field is `urls` — everything else has a sensible default. Almost the entire Katana CLI is exposed; the tables below group the main options. **Targets** | Field | Type | Description | |-------|------|-------------| | `urls` | array | Target URLs / domains to crawl (required). | | `exclude` | array | Exclude hosts by `cdn`, `private-ips`, CIDR, IP, or regex. | **Configuration** | Field | Type | Description | |-------|------|-------------| | `depth` | integer | Maximum crawl depth (default `3`). | | `crawlDuration` | integer | Max crawl time per target in seconds (`0` = unlimited). | | `strategy` | string | `depth-first` or `breadth-first`. | | `jsCrawl` / `jsluice` | boolean | Parse endpoints in JavaScript (jsluice = deeper, memory-heavy). | | `knownFiles` | string | Crawl `robots.txt` / `sitemap.xml` (`none`/`all`/`robotstxt`/`sitemapxml`). | | `automaticFormFill` / `formExtraction` | boolean | Auto-fill forms / extract form elements. | | `techDetect` / `knowledgeBase` | boolean | Technology detection / knowledge-base classification. | | `pathClimb` | boolean | Auto-crawl parent paths. | | `ignoreQueryParams` / `filterSimilar` / `filterSimilarThreshold` | mixed | Collapse duplicate/similar URLs. | | `disableRedirects` / `tlsImpersonate` | boolean | Stop following redirects / JA3 TLS randomization. | | `retry` / `timeout` / `timeStable` / `maxResponseSize` | integer | Request tuning. | | `headers` / `resolvers` | array | Custom headers/cookies / DNS resolvers. | **Headless (Chromium)** | Field | Type | Description | |-------|------|-------------| | `headless` / `hybrid` | boolean | Headless / hybrid Chromium crawling. | | `xhrExtraction` | boolean | Capture XHR request URL & method. | | `noIncognito` / `headlessOptions` / `chromeWsUrl` | mixed | Chrome behaviour, extra flags, remote Chrome. | | `captchaSolverProvider` / `captchaSolverKey` | string | Captcha solver integration (key is secret). | **Scope** | Field | Type | Description | |-------|------|-------------| | `fieldScope` | string | `dn`, `rdn`, `fqdn`, or a custom regex. | | `crawlScope` / `crawlOutScope` | array | In-scope / out-of-scope URL regexes. | | `noScope` / `displayOutScope` | boolean | Disable scoping / show external endpoints. | **Filters** | Field | Type | Description | |-------|------|-------------| | `matchRegex` / `filterRegex` | array | Match or drop output URLs by regex. | | `extensionMatch` / `extensionFilter` / `noDefaultExtFilter` | mixed | Filter by extension. | | `matchCondition` / `filterCondition` | string | DSL-based response conditions. | | `filterPageType` | array | Drop `error` / `captcha` / `parked` pages. | | `disableUniqueFilter` | boolean | Disable duplicate-content filtering. | **Rate limiting** | Field | Type | Description | |-------|------|-------------| | `concurrency` / `parallelism` | integer | Concurrent fetchers / inputs. | | `rateLimit` / `rateLimitMinute` / `delay` | integer | Requests per second / minute / inter-request delay. | **Output & advanced** | Field | Type | Description | |-------|------|-------------| | `flattenOutput` | boolean | Flatten nested fields into columns (default `true`). | | `proxyUrl` | string | HTTP/SOCKS5 proxy URL. | | `omitBody` / `omitRaw` | boolean | Trim response body / raw dumps (default `true`). | | `storeField` / `excludeOutputFields` | mixed | Add / remove output fields. | | `verbose` | boolean | Verbose Katana logs (to run log, not dataset). | | `extraArgs` | array | Any remaining raw Katana flags, passed verbatim. | Example input: ```json { "urls": ["https://example.com"], "depth": 2, "jsCrawl": true, "knownFiles": "all" } ```` ### Output Each dataset item represents one crawled URL. By default (`flattenOutput: true`) the nested `request` / `response` objects are flattened into dotted top-level fields, so **every field — including each response header — is its own column**, ready for the table view and CSV/Excel exports: ```json { "timestamp": "2026-07-03T12:00:00.000000Z", "request.method": "GET", "request.endpoint": "https://example.com/about", "request.tag": "a", "request.attribute": "href", "request.source": "https://example.com/", "response.status_code": 200, "response.content_length": 1256, "response.headers.Content-Type": "text/html; charset=UTF-8", "response.technologies": ["Nginx"] } ``` Set `flattenOutput: false` to keep the raw nested Katana structure instead. **Note on field counts:** Katana emits a row per discovered endpoint. Pages it actually fetches carry full response fields (status, headers, technologies, and — unless `omitBody`/`omitRaw` are on — the body and raw request/response). Endpoints it only discovers (child links, JS/XHR endpoints) are lighter, request-only rows. So some rows legitimately have fewer fields than others. You can download the dataset in various formats such as **JSON, CSV, Excel, or XML**. #### Data table | Field | Description | |-------|-------------| | `timestamp` | When the endpoint was crawled. | | `request.method` | HTTP method. | | `request.endpoint` | The discovered URL. | | `request.source` | Page the URL was found on. | | `response.status_code` | HTTP status returned. | | `response.content_length` | Response size in bytes. | | `response.technologies` | Detected technologies (with `techDetect`). | ### How much does it cost to crawl a website? This Actor uses **pay-per-event** pricing: - **$0.01 flat fee per run** — charged once when the crawl starts, **no matter how many URLs you crawl**. Crawl 10 pages or 100,000; the Actor fee is the same. - **Plus Apify platform usage** (compute, proxy, data transfer) — passed through at cost, so you only pay for the resources your crawl actually consumes. New Apify accounts include **free monthly usage credits**, enough to run plenty of crawls at no cost. **Tips to control cost:** start with a low `depth`, set a bounded `crawlDuration`, and reserve `headless` mode for sites that truly need JavaScript rendering. ### Tips and advanced options - Keep `omitBody` and `omitRaw` enabled (default) to keep dataset items small; disable them to capture full response bodies and raw request/response dumps. - Output is flattened by default so all fields appear as columns; set `flattenOutput: false` for raw nested JSON. - For JavaScript-heavy sites and SPAs, enable `headless` + `jsCrawl` + `xhrExtraction` and give the crawl a longer `crawlDuration` so pages have time to render and get fetched. - Use `crawlScope` / `fieldScope` to keep the crawl on-domain and avoid wandering off to external sites. - Set `crawlDuration` to cap runtime and cost on large targets. - Use `extraArgs` for any flag not exposed in the schema, e.g. `["-headers", "Cookie: session=abc"]`. ### Frequently asked questions **How do I crawl all URLs on a website?** Add the site to `urls`, set a `depth` (2–3 is a good start), and run the Actor. Every discovered URL lands in the dataset, ready to export. **Can it crawl JavaScript-heavy or single-page apps?** Yes. Enable `headless` for real Chromium rendering, plus `jsCrawl`/`jsluice` to parse endpoints out of JavaScript and `xhrExtraction` to capture API calls. **Can it discover API endpoints?** Yes — `jsCrawl` and `jsluice` extract REST/GraphQL endpoints referenced in JavaScript, and `xhrExtraction` records live XHR/fetch requests. **What output formats are supported?** JSON, CSV, Excel (XLSX), XML, JSONL, and RSS — via the Output tab or the Apify API. **Can I schedule recurring crawls?** Yes. Use Apify **Schedules**, or trigger the Actor from Make, Zapier, or your own code through the API. **Is web crawling legal?** Crawling publicly available data is generally legal, but you are responsible for complying with the target site's Terms of Service, `robots.txt`, and applicable laws. Only crawl sites you own or are authorized to test. **Which Katana version is used?** The Docker image pins a specific Katana release (see `KATANA_VERSION` in the Dockerfile). ### Support Found a bug or want a feature? Open an issue on the Actor's **Issues** tab — we actively maintain this Actor and welcome feedback. ### Resources - [Katana on GitHub](https://github.com/projectdiscovery/katana) - [Apify SDK for Python](https://docs.apify.com/sdk/python) - [Apify Platform documentation](https://docs.apify.com/platform) - [Apify integrations](https://apify.com/integrations) # Actor input Schema ## `urls` (type: `array`): One or more target URLs / domains to crawl. Each entry is passed to Katana as a seed (-u). ## `exclude` (type: `array`): Exclude hosts matching a filter: 'cdn', 'private-ips', a CIDR, an IP, or a regex (Katana -exclude). ## `depth` (type: `integer`): Maximum depth to crawl (Katana -depth). ## `crawlDuration` (type: `integer`): Maximum time to crawl per target in seconds (Katana -crawl-duration). 0 = unlimited. ## `strategy` (type: `string`): Traversal order (Katana -strategy). ## `jsCrawl` (type: `boolean`): Parse & crawl endpoints inside JavaScript files (Katana -js-crawl). ## `jsluice` (type: `boolean`): Enable jsluice parsing in JavaScript files. Memory intensive (Katana -jsluice). ## `knownFiles` (type: `string`): Crawl known files such as robots.txt and sitemap.xml (requires depth >= 3). ## `automaticFormFill` (type: `boolean`): Automatically fill and submit forms while crawling. Experimental (Katana -automatic-form-fill). ## `formExtraction` (type: `boolean`): Extract form, input, textarea & select elements into output (Katana -form-extraction). ## `techDetect` (type: `boolean`): Enable technology detection on crawled responses (Katana -tech-detect). ## `knowledgeBase` (type: `boolean`): Enable knowledge base classification (Katana -knowledge-base). ## `pathClimb` (type: `boolean`): Auto-crawl parent paths of discovered URLs (Katana -path-climb). ## `ignoreQueryParams` (type: `boolean`): Ignore crawling the same path with different query-param values (Katana -ignore-query-params). ## `filterSimilar` (type: `boolean`): Filter crawling of similar-looking URLs, e.g. /users/123 and /users/456 (Katana -filter-similar). ## `filterSimilarThreshold` (type: `integer`): Distinct values before a path position is treated as a parameter (Katana -filter-similar-threshold). 0 = default. ## `disableRedirects` (type: `boolean`): Do not follow HTTP redirects (Katana -disable-redirects). ## `tlsImpersonate` (type: `boolean`): Enable experimental client-hello (JA3) TLS randomization (Katana -tls-impersonate). ## `retry` (type: `integer`): Number of times to retry a failed request (Katana -retry). 0 = Katana default. ## `timeout` (type: `integer`): Time to wait for a request in seconds (Katana -timeout). ## `timeStable` (type: `integer`): Time to wait until the page is stable, headless mode (Katana -time-stable). 0 = default. ## `maxResponseSize` (type: `integer`): Maximum response size to read (Katana -max-response-size). 0 = Katana default (4 MB). ## `headers` (type: `array`): Custom headers/cookies for every request, in 'Name: value' format (Katana -headers). ## `resolvers` (type: `array`): Custom DNS resolvers, e.g. 1.1.1.1, 8.8.8.8 (Katana -resolvers). ## `headless` (type: `boolean`): Enable headless Chromium crawling. Slower but handles JS-heavy sites (Katana -headless). ## `hybrid` (type: `boolean`): Enable headless hybrid crawling combining standard and headless (Katana -hybrid). ## `xhrExtraction` (type: `boolean`): Extract XHR request URL & method into output (Katana -xhr-extraction). ## `noIncognito` (type: `boolean`): Start headless Chrome without incognito mode (Katana -no-incognito). ## `headlessOptions` (type: `array`): Additional flags passed to headless Chrome (Katana -headless-options). ## `chromeWsUrl` (type: `string`): Connect to a Chrome instance launched elsewhere via its debugger WebSocket URL (Katana -chrome-ws-url). ## `captchaSolverProvider` (type: `string`): Captcha solver provider, e.g. capsolver (Katana -captcha-solver-provider). ## `captchaSolverKey` (type: `string`): API key for the captcha solver provider (Katana -captcha-solver-key). ## `fieldScope` (type: `string`): Pre-defined scope (dn, rdn, fqdn) or a custom regex (Katana -field-scope). ## `crawlScope` (type: `array`): Regexes of URLs that are in scope and should be followed (Katana -crawl-scope). ## `crawlOutScope` (type: `array`): Regexes of URLs that are out of scope and should not be followed (Katana -crawl-out-scope). ## `noScope` (type: `boolean`): Disable the host-based default scope, crawling everything reachable (Katana -no-scope). ## `displayOutScope` (type: `boolean`): Display external endpoints found during scoped crawling (Katana -display-out-scope). ## `matchRegex` (type: `array`): Only output URLs matching these regexes (Katana -match-regex). ## `filterRegex` (type: `array`): Drop output URLs matching these regexes (Katana -filter-regex). ## `extensionMatch` (type: `array`): Only output URLs with these extensions, e.g. php, html, js (Katana -extension-match). ## `extensionFilter` (type: `array`): Drop output URLs with these extensions, e.g. png, css (Katana -extension-filter). ## `noDefaultExtFilter` (type: `boolean`): Remove the built-in default extension filter list (Katana -no-default-ext-filter). ## `matchCondition` (type: `string`): Match responses with a DSL-based condition (Katana -match-condition). ## `filterCondition` (type: `string`): Filter responses with a DSL-based condition (Katana -filter-condition). ## `filterPageType` (type: `array`): Filter responses by page type: error, captcha, parked (Katana -filter-page-type). ## `disableUniqueFilter` (type: `boolean`): Disable duplicate-content filtering (Katana -disable-unique-filter). ## `concurrency` (type: `integer`): Number of concurrent fetchers (Katana -concurrency). ## `parallelism` (type: `integer`): Number of inputs processed in parallel (Katana -parallelism). ## `rateLimit` (type: `integer`): Maximum requests to send per second (Katana -rate-limit). ## `rateLimitMinute` (type: `integer`): Maximum requests to send per minute (Katana -rate-limit-minute). 0 = disabled. ## `delay` (type: `integer`): Delay between each request in seconds (Katana -delay). 0 = none. ## `proxyUrl` (type: `string`): HTTP/SOCKS5 proxy, e.g. http://user:pass@host:port (Katana -proxy). Leave empty for none. ## `flattenOutput` (type: `boolean`): Flatten each record's nested request/response objects into dotted top-level fields (e.g. response.status\_code) so every field shows as its own dataset column. Disable to keep the raw nested Katana JSON. ## `omitBody` (type: `boolean`): Omit the response body from JSONL output to reduce dataset size (Katana -omit-body). ## `omitRaw` (type: `boolean`): Omit raw request/response dumps from JSONL output (Katana -omit-raw). ## `storeField` (type: `string`): Additional per-host field to store, e.g. url,path,fqdn,rdn (Katana -store-field). ## `excludeOutputFields` (type: `array`): Fields to exclude from the JSONL output records (Katana -exclude-output-fields). ## `verbose` (type: `boolean`): Enable verbose Katana logging (Katana -verbose). Logs go to the run log, not the dataset. ## `extraArgs` (type: `array`): Additional raw Katana CLI flags for anything not exposed above, passed verbatim, e.g. \["-time-stable", "2"]. ## Actor input object example ```json { "urls": [ "https://example.com", "https://test.com" ], "depth": 3, "crawlDuration": 0, "strategy": "depth-first", "jsCrawl": false, "jsluice": false, "knownFiles": "none", "automaticFormFill": false, "formExtraction": false, "techDetect": false, "knowledgeBase": false, "pathClimb": false, "ignoreQueryParams": false, "filterSimilar": false, "filterSimilarThreshold": 0, "disableRedirects": false, "tlsImpersonate": false, "retry": 0, "timeout": 10, "timeStable": 0, "maxResponseSize": 0, "headers": [ "Cookie: session=abc", "User-Agent: my-crawler" ], "headless": false, "hybrid": false, "xhrExtraction": false, "noIncognito": false, "fieldScope": "rdn", "noScope": false, "displayOutScope": false, "noDefaultExtFilter": false, "disableUniqueFilter": false, "concurrency": 10, "parallelism": 10, "rateLimit": 150, "rateLimitMinute": 0, "delay": 0, "flattenOutput": true, "omitBody": true, "omitRaw": true, "verbose": false } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "urls": [ "https://example.com" ] }; // Run the Actor and wait for it to finish const run = await client.actor("rl1987/pd-katana").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "urls": ["https://example.com"] } # Run the Actor and wait for it to finish run = client.actor("rl1987/pd-katana").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "urls": [ "https://example.com" ] }' | apify call rl1987/pd-katana --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=rl1987/pd-katana", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Katana Web Crawler (ProjectDiscovery)", "description": "Crawl websites with ProjectDiscovery Katana and stream results as structured JSONL into an Apify dataset.", "version": "0.1", "x-build-id": "crzy1B656oo2eMUYr" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/rl1987~pd-katana/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-rl1987-pd-katana", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/rl1987~pd-katana/runs": { "post": { "operationId": "runs-sync-rl1987-pd-katana", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/rl1987~pd-katana/run-sync": { "post": { "operationId": "run-sync-rl1987-pd-katana", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "urls" ], "properties": { "urls": { "title": "Target URLs", "type": "array", "description": "One or more target URLs / domains to crawl. Each entry is passed to Katana as a seed (-u).", "items": { "type": "string" } }, "exclude": { "title": "Exclude hosts", "type": "array", "description": "Exclude hosts matching a filter: 'cdn', 'private-ips', a CIDR, an IP, or a regex (Katana -exclude).", "items": { "type": "string" } }, "depth": { "title": "Crawl depth", "minimum": 1, "maximum": 100, "type": "integer", "description": "Maximum depth to crawl (Katana -depth).", "default": 3 }, "crawlDuration": { "title": "Max crawl duration (seconds)", "minimum": 0, "type": "integer", "description": "Maximum time to crawl per target in seconds (Katana -crawl-duration). 0 = unlimited.", "default": 0 }, "strategy": { "title": "Visit strategy", "enum": [ "depth-first", "breadth-first" ], "type": "string", "description": "Traversal order (Katana -strategy).", "default": "depth-first" }, "jsCrawl": { "title": "JavaScript crawling", "type": "boolean", "description": "Parse & crawl endpoints inside JavaScript files (Katana -js-crawl).", "default": false }, "jsluice": { "title": "Jsluice parsing", "type": "boolean", "description": "Enable jsluice parsing in JavaScript files. Memory intensive (Katana -jsluice).", "default": false }, "knownFiles": { "title": "Crawl known files", "enum": [ "none", "all", "robotstxt", "sitemapxml" ], "type": "string", "description": "Crawl known files such as robots.txt and sitemap.xml (requires depth >= 3).", "default": "none" }, "automaticFormFill": { "title": "Automatic form fill", "type": "boolean", "description": "Automatically fill and submit forms while crawling. Experimental (Katana -automatic-form-fill).", "default": false }, "formExtraction": { "title": "Extract forms", "type": "boolean", "description": "Extract form, input, textarea & select elements into output (Katana -form-extraction).", "default": false }, "techDetect": { "title": "Technology detection", "type": "boolean", "description": "Enable technology detection on crawled responses (Katana -tech-detect).", "default": false }, "knowledgeBase": { "title": "Knowledge base classification", "type": "boolean", "description": "Enable knowledge base classification (Katana -knowledge-base).", "default": false }, "pathClimb": { "title": "Path climb", "type": "boolean", "description": "Auto-crawl parent paths of discovered URLs (Katana -path-climb).", "default": false }, "ignoreQueryParams": { "title": "Ignore query params", "type": "boolean", "description": "Ignore crawling the same path with different query-param values (Katana -ignore-query-params).", "default": false }, "filterSimilar": { "title": "Filter similar URLs", "type": "boolean", "description": "Filter crawling of similar-looking URLs, e.g. /users/123 and /users/456 (Katana -filter-similar).", "default": false }, "filterSimilarThreshold": { "title": "Similar filter threshold", "minimum": 0, "type": "integer", "description": "Distinct values before a path position is treated as a parameter (Katana -filter-similar-threshold). 0 = default.", "default": 0 }, "disableRedirects": { "title": "Disable redirects", "type": "boolean", "description": "Do not follow HTTP redirects (Katana -disable-redirects).", "default": false }, "tlsImpersonate": { "title": "TLS impersonation (JA3)", "type": "boolean", "description": "Enable experimental client-hello (JA3) TLS randomization (Katana -tls-impersonate).", "default": false }, "retry": { "title": "Retries", "minimum": 0, "type": "integer", "description": "Number of times to retry a failed request (Katana -retry). 0 = Katana default.", "default": 0 }, "timeout": { "title": "Request timeout (seconds)", "minimum": 1, "type": "integer", "description": "Time to wait for a request in seconds (Katana -timeout).", "default": 10 }, "timeStable": { "title": "Page stable wait (seconds)", "minimum": 0, "type": "integer", "description": "Time to wait until the page is stable, headless mode (Katana -time-stable). 0 = default.", "default": 0 }, "maxResponseSize": { "title": "Max response size (bytes)", "minimum": 0, "type": "integer", "description": "Maximum response size to read (Katana -max-response-size). 0 = Katana default (4 MB).", "default": 0 }, "headers": { "title": "Custom headers", "type": "array", "description": "Custom headers/cookies for every request, in 'Name: value' format (Katana -headers).", "items": { "type": "string" } }, "resolvers": { "title": "Custom DNS resolvers", "type": "array", "description": "Custom DNS resolvers, e.g. 1.1.1.1, 8.8.8.8 (Katana -resolvers).", "items": { "type": "string" } }, "headless": { "title": "Headless mode", "type": "boolean", "description": "Enable headless Chromium crawling. Slower but handles JS-heavy sites (Katana -headless).", "default": false }, "hybrid": { "title": "Hybrid crawling", "type": "boolean", "description": "Enable headless hybrid crawling combining standard and headless (Katana -hybrid).", "default": false }, "xhrExtraction": { "title": "XHR extraction", "type": "boolean", "description": "Extract XHR request URL & method into output (Katana -xhr-extraction).", "default": false }, "noIncognito": { "title": "No incognito", "type": "boolean", "description": "Start headless Chrome without incognito mode (Katana -no-incognito).", "default": false }, "headlessOptions": { "title": "Headless Chrome options", "type": "array", "description": "Additional flags passed to headless Chrome (Katana -headless-options).", "items": { "type": "string" } }, "chromeWsUrl": { "title": "Remote Chrome WS URL", "type": "string", "description": "Connect to a Chrome instance launched elsewhere via its debugger WebSocket URL (Katana -chrome-ws-url)." }, "captchaSolverProvider": { "title": "Captcha solver provider", "type": "string", "description": "Captcha solver provider, e.g. capsolver (Katana -captcha-solver-provider)." }, "captchaSolverKey": { "title": "Captcha solver API key", "type": "string", "description": "API key for the captcha solver provider (Katana -captcha-solver-key)." }, "fieldScope": { "title": "Field scope", "type": "string", "description": "Pre-defined scope (dn, rdn, fqdn) or a custom regex (Katana -field-scope).", "default": "rdn" }, "crawlScope": { "title": "In-scope regexes", "type": "array", "description": "Regexes of URLs that are in scope and should be followed (Katana -crawl-scope).", "items": { "type": "string" } }, "crawlOutScope": { "title": "Out-of-scope regexes", "type": "array", "description": "Regexes of URLs that are out of scope and should not be followed (Katana -crawl-out-scope).", "items": { "type": "string" } }, "noScope": { "title": "Disable scope", "type": "boolean", "description": "Disable the host-based default scope, crawling everything reachable (Katana -no-scope).", "default": false }, "displayOutScope": { "title": "Display out-of-scope", "type": "boolean", "description": "Display external endpoints found during scoped crawling (Katana -display-out-scope).", "default": false }, "matchRegex": { "title": "Match output regexes", "type": "array", "description": "Only output URLs matching these regexes (Katana -match-regex).", "items": { "type": "string" } }, "filterRegex": { "title": "Filter output regexes", "type": "array", "description": "Drop output URLs matching these regexes (Katana -filter-regex).", "items": { "type": "string" } }, "extensionMatch": { "title": "Match extensions", "type": "array", "description": "Only output URLs with these extensions, e.g. php, html, js (Katana -extension-match).", "items": { "type": "string" } }, "extensionFilter": { "title": "Filter extensions", "type": "array", "description": "Drop output URLs with these extensions, e.g. png, css (Katana -extension-filter).", "items": { "type": "string" } }, "noDefaultExtFilter": { "title": "No default extension filter", "type": "boolean", "description": "Remove the built-in default extension filter list (Katana -no-default-ext-filter).", "default": false }, "matchCondition": { "title": "Match condition (DSL)", "type": "string", "description": "Match responses with a DSL-based condition (Katana -match-condition)." }, "filterCondition": { "title": "Filter condition (DSL)", "type": "string", "description": "Filter responses with a DSL-based condition (Katana -filter-condition)." }, "filterPageType": { "title": "Filter page types", "type": "array", "description": "Filter responses by page type: error, captcha, parked (Katana -filter-page-type).", "items": { "type": "string" } }, "disableUniqueFilter": { "title": "Disable unique filter", "type": "boolean", "description": "Disable duplicate-content filtering (Katana -disable-unique-filter).", "default": false }, "concurrency": { "title": "Concurrency", "minimum": 1, "type": "integer", "description": "Number of concurrent fetchers (Katana -concurrency).", "default": 10 }, "parallelism": { "title": "Parallelism", "minimum": 1, "type": "integer", "description": "Number of inputs processed in parallel (Katana -parallelism).", "default": 10 }, "rateLimit": { "title": "Rate limit (req/sec)", "minimum": 1, "type": "integer", "description": "Maximum requests to send per second (Katana -rate-limit).", "default": 150 }, "rateLimitMinute": { "title": "Rate limit (req/min)", "minimum": 0, "type": "integer", "description": "Maximum requests to send per minute (Katana -rate-limit-minute). 0 = disabled.", "default": 0 }, "delay": { "title": "Request delay (seconds)", "minimum": 0, "type": "integer", "description": "Delay between each request in seconds (Katana -delay). 0 = none.", "default": 0 }, "proxyUrl": { "title": "Proxy URL", "type": "string", "description": "HTTP/SOCKS5 proxy, e.g. http://user:pass@host:port (Katana -proxy). Leave empty for none." }, "flattenOutput": { "title": "Flatten output", "type": "boolean", "description": "Flatten each record's nested request/response objects into dotted top-level fields (e.g. response.status_code) so every field shows as its own dataset column. Disable to keep the raw nested Katana JSON.", "default": true }, "omitBody": { "title": "Omit response body", "type": "boolean", "description": "Omit the response body from JSONL output to reduce dataset size (Katana -omit-body).", "default": true }, "omitRaw": { "title": "Omit raw request/response", "type": "boolean", "description": "Omit raw request/response dumps from JSONL output (Katana -omit-raw).", "default": true }, "storeField": { "title": "Store field", "type": "string", "description": "Additional per-host field to store, e.g. url,path,fqdn,rdn (Katana -store-field)." }, "excludeOutputFields": { "title": "Exclude output fields", "type": "array", "description": "Fields to exclude from the JSONL output records (Katana -exclude-output-fields).", "items": { "type": "string" } }, "verbose": { "title": "Verbose logs", "type": "boolean", "description": "Enable verbose Katana logging (Katana -verbose). Logs go to the run log, not the dataset.", "default": false }, "extraArgs": { "title": "Extra Katana arguments", "type": "array", "description": "Additional raw Katana CLI flags for anything not exposed above, passed verbatim, e.g. [\"-time-stable\", \"2\"].", "items": { "type": "string" } } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```