Katana Web Crawler – Fast Website URL & Endpoint Extractor

Katana Web Crawler is a fast, no-code website crawler and URL extractor built on ProjectDiscovery Katana, the popular open-source crawling and spidering framework. Point it at any website, and it crawls every page, link, JavaScript file, form, and API endpoint, then delivers the results as a clean, structured dataset you can download as JSON, CSV, Excel, or XML — or pull straight from the Apify API.

Run it in the cloud with zero setup: no Go install, no binaries, no servers. You get headless Chromium crawling, JavaScript endpoint discovery, proxy rotation, scheduling, and integrations (Make, Zapier, Google Sheets, Slack, and more) out of the box.

Try it now: paste a URL like https://example.com, hit Start, and watch discovered URLs stream into your dataset in seconds.

What is Katana Web Crawler?

Katana Web Crawler is a serverless Apify Actor that scrapes all URLs from a website and maps its complete structure. It's the fastest way to crawl a site, list every reachable link, and export the URLs without writing code. Use it for web crawling, site mapping, URL discovery, attack-surface enumeration, SEO audits, and API endpoint discovery.

What does Katana Web Crawler do?

• 🕷️ Crawls entire websites in standard (fast HTTP) or headless Chromium mode.
• 🔗 Extracts every URL — links, assets, scripts, forms, and inputs.
• 🧩 Discovers hidden API endpoints parsed from JavaScript files (js-crawl + jsluice).
• 📡 Captures XHR / fetch API calls made by JavaScript-heavy single-page apps.
• 📂 Finds known files such as robots.txt and sitemap.xml.
• 🎯 Filters precisely with scope, regex, extension, and DSL conditions.
• 🛠️ Detects technologies and extracts forms while it crawls.
• 📊 Exports flat, ready-to-use data — every field is its own column.

Why use Katana Web Crawler?

Whether you're a security researcher, penetration tester, SEO specialist, or developer, this crawler turns any website into structured data in minutes:

• Attack-surface mapping — enumerate every reachable endpoint of a web app before a pentest or bug-bounty hunt.
• Security reconnaissance — feed the URL list into scanners like Nuclei, httpx, or ffuf.
• API endpoint discovery — surface undocumented REST/GraphQL endpoints referenced in JavaScript.
• SEO & site audits — inventory pages, assets, redirects, and broken links at scale.
• Content & asset discovery — find every script, image, PDF, and download link on a domain.
• Migration & QA — snapshot a site's full URL structure before and after a release.

How to use Katana Web Crawler to crawl a website

No coding required — you can run your first crawl in under a minute:

1. Click Try for free / open the Actor and go to the Input tab.
2. Add one or more Target URLs (e.g. https://example.com).
3. Set the crawl depth, and optionally enable JavaScript crawling or headless mode for modern web apps.
4. (Optional) Add scope, regex, or extension filters to collect only what you need.
5. Click Start and watch URLs stream into the Output tab in real time.
6. Download the dataset as JSON, CSV, Excel, or XML — or fetch it via the Apify API.

Prefer automation? Schedule recurring crawls or trigger the Actor from your own app, Make, or Zapier.

Features at a glance

Capability	Supported
Standard (HTTP) crawling	✅
Headless Chromium crawling	✅
JavaScript endpoint parsing (js-crawl, jsluice)	✅
XHR / fetch API extraction	✅
Form extraction & auto-fill	✅
Technology detection	✅
robots.txt & sitemap.xml crawling	✅
Scope, regex, extension & DSL filters	✅
Proxy support (HTTP/SOCKS5)	✅
Rate limiting & concurrency control	✅
JSON / CSV / Excel / XML export	✅
API access, scheduling & integrations	✅

Input

All fields are configured on the Input tab, grouped into sections. The only required field is urls — everything else has a sensible default. Almost the entire Katana CLI is exposed; the tables below group the main options.

Targets

Field	Type	Description
urls	array	Target URLs / domains to crawl (required).
exclude	array	Exclude hosts by cdn, private-ips, CIDR, IP, or regex.

Configuration

Field	Type	Description
depth	integer	Maximum crawl depth (default 3).
crawlDuration	integer	Max crawl time per target in seconds (0 = unlimited).
strategy	string	depth-first or breadth-first.
jsCrawl / jsluice	boolean	Parse endpoints in JavaScript (jsluice = deeper, memory-heavy).
knownFiles	string	Crawl robots.txt / sitemap.xml (none/all/robotstxt/sitemapxml).
automaticFormFill / formExtraction	boolean	Auto-fill forms / extract form elements.
techDetect / knowledgeBase	boolean	Technology detection / knowledge-base classification.
pathClimb	boolean	Auto-crawl parent paths.
ignoreQueryParams / filterSimilar / filterSimilarThreshold	mixed	Collapse duplicate/similar URLs.
disableRedirects / tlsImpersonate	boolean	Stop following redirects / JA3 TLS randomization.
retry / timeout / timeStable / maxResponseSize	integer	Request tuning.
headers / resolvers	array	Custom headers/cookies / DNS resolvers.

Headless (Chromium)

Field	Type	Description
headless / hybrid	boolean	Headless / hybrid Chromium crawling.
xhrExtraction	boolean	Capture XHR request URL & method.
noIncognito / headlessOptions / chromeWsUrl	mixed	Chrome behaviour, extra flags, remote Chrome.
captchaSolverProvider / captchaSolverKey	string	Captcha solver integration (key is secret).

Scope

Field	Type	Description
fieldScope	string	dn, rdn, fqdn, or a custom regex.
crawlScope / crawlOutScope	array	In-scope / out-of-scope URL regexes.
noScope / displayOutScope	boolean	Disable scoping / show external endpoints.

Filters

Field	Type	Description
matchRegex / filterRegex	array	Match or drop output URLs by regex.
extensionMatch / extensionFilter / noDefaultExtFilter	mixed	Filter by extension.
matchCondition / filterCondition	string	DSL-based response conditions.
filterPageType	array	Drop error / captcha / parked pages.
disableUniqueFilter	boolean	Disable duplicate-content filtering.

Rate limiting

Field	Type	Description
concurrency / parallelism	integer	Concurrent fetchers / inputs.
rateLimit / rateLimitMinute / delay	integer	Requests per second / minute / inter-request delay.

Output & advanced

Field	Type	Description
flattenOutput	boolean	Flatten nested fields into columns (default true).
proxyUrl	string	HTTP/SOCKS5 proxy URL.
omitBody / omitRaw	boolean	Trim response body / raw dumps (default true).
storeField / excludeOutputFields	mixed	Add / remove output fields.
verbose	boolean	Verbose Katana logs (to run log, not dataset).
extraArgs	array	Any remaining raw Katana flags, passed verbatim.

Example input:

{
    "urls": ["https://example.com"],
    "depth": 2,
    "jsCrawl": true,
    "knownFiles": "all"
}

Output

Each dataset item represents one crawled URL. By default (flattenOutput: true) the nested request / response objects are flattened into dotted top-level fields, so every field — including each response header — is its own column, ready for the table view and CSV/Excel exports:

{
    "timestamp": "2026-07-03T12:00:00.000000Z",
    "request.method": "GET",
    "request.endpoint": "https://example.com/about",
    "request.tag": "a",
    "request.attribute": "href",
    "request.source": "https://example.com/",
    "response.status_code": 200,
    "response.content_length": 1256,
    "response.headers.Content-Type": "text/html; charset=UTF-8",
    "response.technologies": ["Nginx"]
}

Set flattenOutput: false to keep the raw nested Katana structure instead.

Note on field counts: Katana emits a row per discovered endpoint. Pages it actually fetches carry full response fields (status, headers, technologies, and — unless omitBody/omitRaw are on — the body and raw request/response). Endpoints it only discovers (child links, JS/XHR endpoints) are lighter, request-only rows. So some rows legitimately have fewer fields than others.

You can download the dataset in various formats such as JSON, CSV, Excel, or XML.

Data table

Field	Description
timestamp	When the endpoint was crawled.
request.method	HTTP method.
request.endpoint	The discovered URL.
request.source	Page the URL was found on.
response.status_code	HTTP status returned.
response.content_length	Response size in bytes.
response.technologies	Detected technologies (with techDetect).

How much does it cost to crawl a website?

This Actor uses pay-per-event pricing:

• $0.01 flat fee per run — charged once when the crawl starts, no matter how many URLs you crawl. Crawl 10 pages or 100,000; the Actor fee is the same.
• Plus Apify platform usage (compute, proxy, data transfer) — passed through at cost, so you only pay for the resources your crawl actually consumes.

New Apify accounts include free monthly usage credits, enough to run plenty of crawls at no cost.

Tips to control cost: start with a low depth, set a bounded crawlDuration, and reserve headless mode for sites that truly need JavaScript rendering.

Tips and advanced options

• Keep omitBody and omitRaw enabled (default) to keep dataset items small; disable them to capture full response bodies and raw request/response dumps.
• Output is flattened by default so all fields appear as columns; set flattenOutput: false for raw nested JSON.
• For JavaScript-heavy sites and SPAs, enable headless + jsCrawl + xhrExtraction and give the crawl a longer crawlDuration so pages have time to render and get fetched.
• Use crawlScope / fieldScope to keep the crawl on-domain and avoid wandering off to external sites.
• Set crawlDuration to cap runtime and cost on large targets.
• Use extraArgs for any flag not exposed in the schema, e.g. ["-headers", "Cookie: session=abc"].

Frequently asked questions

How do I crawl all URLs on a website? Add the site to urls, set a depth (2–3 is a good start), and run the Actor. Every discovered URL lands in the dataset, ready to export.

Can it crawl JavaScript-heavy or single-page apps? Yes. Enable headless for real Chromium rendering, plus jsCrawl/jsluice to parse endpoints out of JavaScript and xhrExtraction to capture API calls.

Can it discover API endpoints? Yes — jsCrawl and jsluice extract REST/GraphQL endpoints referenced in JavaScript, and xhrExtraction records live XHR/fetch requests.

What output formats are supported? JSON, CSV, Excel (XLSX), XML, JSONL, and RSS — via the Output tab or the Apify API.

Can I schedule recurring crawls? Yes. Use Apify Schedules, or trigger the Actor from Make, Zapier, or your own code through the API.

Is web crawling legal? Crawling publicly available data is generally legal, but you are responsible for complying with the target site's Terms of Service, robots.txt, and applicable laws. Only crawl sites you own or are authorized to test.

Which Katana version is used? The Docker image pins a specific Katana release (see KATANA_VERSION in the Dockerfile).

Support

Found a bug or want a feature? Open an issue on the Actor's Issues tab — we actively maintain this Actor and welcome feedback.

Resources

• Katana on GitHub
• Apify SDK for Python
• Apify Platform documentation
• Apify integrations