View all ideas
Katana Web Crawler
STATUS
Open to developCATEGORIES
SUBMITTED
Aug 27, 2025
Fast and configurable web crawling for security researchers and SEO professionals. This Actor discovers all endpoints, assets, and URLs on a website with advanced filtering and customizable depth settings.
Source: Katana on GitHub
how does the Katana Web Crawler work?
- Input parameters such as:
- Target website URL(s)
- Crawl depth limit
- Allowed and disallowed domains or paths
- Filters for file types, query params, or keywords
- Rate limiting or concurrency settings
- Run the Actor
- The crawler will:
- Parse HTML, JavaScript, and APIs to discover links and endpoints
- Collect internal and external assets
- Respect configured depth and filtering rules
- Export results in JSON, CSV, or Excel for further analysis
Katana Web Crawler API
The Katana Web Crawler API delivers structured crawl data including URLs, endpoints, and discovered assets. You can integrate it with JavaScript, Node.js, Python, or curl to automate crawling tasks in your Apify environment.
use cases of the Katana Web Crawler
- Security researchers mapping endpoints and hidden assets
- SEO professionals analyzing site structure and discoverability
- Developers monitoring large websites for broken or outdated links
- Data teams collecting URLs for downstream scraping and analysis
