# gau - Get All URLs (`rl1987/gau-wrapper`) Actor Fetch known URLs from the Wayback Machine, Common Crawl, AlienVault OTX, and URLScan for any domain. A wrapper around the gau OSINT tool for attack-surface and data-pipeline use. - **URL**: https://apify.com/rl1987/gau-wrapper.md - **Developed by:** [R.L.](https://apify.com/rl1987) (community) - **Categories:** Other - **Stats:** 3 total users, 1 monthly users, 91.7% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.01 / actor run This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ### What does gau - Get All URLs do? **gau - Get All URLs** is an Apify Actor that wraps the popular open-source OSINT tool [`gau`](https://github.com/lc/gau) (Get All URLs). For any domain you give it, it collects **every URL ever seen** by major web archives and threat-intelligence feeds — the [Wayback Machine](https://web.archive.org), [Common Crawl](https://commoncrawl.org), [AlienVault OTX](https://otx.alienvault.com), and [URLScan](https://urlscan.io) — and returns them as a clean, structured dataset. Running it on the Apify platform gives you API access, scheduling, monitoring, proxy rotation, and easy integration with the rest of your data pipeline — no need to install Go or manage the binary yourself. ### Why use gau - Get All URLs? - **Attack-surface mapping** — enumerate historical and current endpoints, parameters, and forgotten paths for a target domain during authorized security assessments. - **Content & SEO audits** — discover every URL that archives know about, including pages no longer linked from the live site. - **Pipeline integration** — feed the resulting URLs into other Apify Actors (HTTP scrapers, vulnerability scanners, link checkers) straight from the dataset. - **No local setup** — schedule recurring runs, call it from the API, and connect it to Make, Zapier, n8n, Google Drive, and more. ### How to use gau - Get All URLs 1. Open the Actor and go to the **Input** tab. 2. Enter one or more **domains** (bare hostnames such as `example.com` — no `https://` and no path). 3. Optionally pick the **providers**, enable **subdomains**, set date ranges, or filter by extension / status code / MIME type. 4. Click **Start** and watch URLs stream into the **Output** tab. 5. Download the dataset in JSON, CSV, or Excel — or fetch a plain newline-delimited URL list directly from the dataset API with `?fields=url&format=csv&clean=true`. ### Input Configure the run from the **Input** tab or via JSON. Key fields: | Field | Type | Description | |-------|------|-------------| | `domains` | array | **Required.** Bare domains to query, e.g. `["example.com"]`. | | `providers` | array | Sources to use: `wayback`, `commoncrawl`, `otx`, `urlscan`. | | `includeSubdomains` | boolean | Include subdomains of the target (`--subs`). | | `fromDate` / `toDate` | string | Limit by first-seen month, format `YYYYMM`. | | `blacklistExtensions` | array | Extensions to skip, e.g. `["png","jpg","woff"]`. | | `matchStatusCodes` / `filterStatusCodes` | array | Keep / drop by archived HTTP status. | | `matchMimeTypes` / `filterMimeTypes` | array | Keep / drop by archived MIME type. | | `removeDuplicateParams` | boolean | Collapse endpoints that differ only in parameter values (`--fp`). | | `threads`, `timeout`, `retries` | integer | HTTP client tuning. | | `maxResults` | integer | Stop after N URLs (`0` = unlimited). | | `proxyConfiguration` | object | Route requests through an Apify or custom proxy. | Example input: ```json { "domains": ["example.com"], "providers": ["wayback", "commoncrawl", "otx"], "includeSubdomains": true, "blacklistExtensions": ["png", "jpg", "css"], "maxResults": 5000 } ```` ### Output Each discovered URL becomes one dataset item, **pushed to the dataset the instant gau yields it** — so rows stream into the Output tab live while the run is still in progress, ready for downstream consumers to pick up immediately. You can download the dataset in various formats such as JSON, HTML, CSV, or Excel. Need just a plain newline-delimited URL list (e.g. to pipe into `httpx` or `nuclei`)? Hit the dataset API with `?fields=url&format=csv&clean=true`. ```json { "url": "https://www.example.com/login?next=/account", "domain": "example.com", "host": "www.example.com", "scheme": "https", "path": "/login", "query": "next=/account", "fileExtension": null, "provider": "wayback" } ``` #### Data table | Field | Description | |-------|-------------| | `url` | The full discovered URL. | | `domain` | Which input domain the URL belongs to. | | `host` | Hostname of the URL (may be a subdomain). | | `scheme` | `http` or `https`. | | `path` | URL path component. | | `query` | Raw query string, if any. | | `fileExtension` | Lower-cased file extension of the path, if any. | | `provider` | Source provider when a single provider is selected, otherwise `null`. | ### How much does it cost? The Actor is lightweight — it streams text from public archives and does no browser rendering, and it never buffers the full result set in memory, so it runs comfortably at the modest **512 MB default** memory regardless of how many URLs a domain has. Most runs finish in seconds to a few minutes. Cost scales with how many URLs a domain has archived and how many providers you query. Large, popular domains can return hundreds of thousands of URLs; use `maxResults`, `blacklistExtensions`, and date ranges to keep runs bounded. ### Tips and advanced options - **Speed vs. completeness** — querying all four providers is the most thorough but slowest; pick a subset for quick runs. - **Rate limits** — if a provider throttles you, enable `proxyConfiguration` and/or raise `retries` and `timeout`. - **Noise reduction** — blacklist static asset extensions (`png,jpg,css,woff,svg`) and use `removeDuplicateParams` to shrink the result set. - **Subdomains** — `includeSubdomains` greatly increases coverage (and volume) for an organization. ### FAQ, disclaimers, and support **Is this legal?** The Actor only reads from public archives and threat-intel feeds; it does not touch the target's own servers. Use it only against domains you own or are explicitly authorized to test, and comply with the providers' and Apify's Terms of Service. **Why are some URLs dead or old?** Results come from historical archives, so they include URLs that may no longer exist. That is by design for OSINT and attack-surface work. **Found a bug or need a custom version?** Open an issue from the Actor's **Issues** tab — feedback and custom-solution requests are welcome. This Actor wraps the open-source [`gau`](https://github.com/lc/gau) tool by [@lc](https://github.com/lc), distributed under the MIT license. # Actor input Schema ## `domains` (type: `array`): List of domains (or apex domains) to fetch known URLs for, e.g. `example.com`. Do not include a scheme (`https://`) or path. ## `providers` (type: `array`): Which URL sources to query. Leave empty to use gau's defaults (wayback, commoncrawl, otx). ## `includeSubdomains` (type: `boolean`): Include URLs from subdomains of the target domain (gau `--subs`). ## `fromDate` (type: `string`): Only fetch URLs first seen on or after this month. Format: YYYYMM (e.g. `202101`). ## `toDate` (type: `string`): Only fetch URLs first seen on or before this month. Format: YYYYMM (e.g. `202412`). ## `blacklistExtensions` (type: `array`): File extensions to skip, without the dot (gau `--blacklist`), e.g. `png`, `jpg`, `woff`. ## `matchStatusCodes` (type: `array`): Only keep URLs whose archived HTTP status code is in this list (gau `--mc`), e.g. `200`, `301`. ## `filterStatusCodes` (type: `array`): Drop URLs whose archived HTTP status code is in this list (gau `--fc`), e.g. `404`, `302`. ## `matchMimeTypes` (type: `array`): Only keep URLs whose archived MIME type is in this list (gau `--mt`), e.g. `text/html`. ## `filterMimeTypes` (type: `array`): Drop URLs whose archived MIME type is in this list (gau `--ft`), e.g. `text/plain`, `image/png`. ## `removeDuplicateParams` (type: `boolean`): Keep only one representative URL per endpoint that differs only in parameter values (gau `--fp`). ## `threads` (type: `integer`): Number of worker threads gau spawns. ## `timeout` (type: `integer`): Timeout for gau's HTTP client, in seconds. ## `retries` (type: `integer`): Number of retries for gau's HTTP client. ## `maxResults` (type: `integer`): Stop after pushing this many URLs to the dataset. Set 0 for unlimited. ## `proxyConfiguration` (type: `object`): Route gau's requests through an Apify or custom HTTP proxy. Useful when archive providers rate-limit your runs. ## Actor input object example ```json { "domains": [ "trickster.dev" ], "providers": [ "wayback", "commoncrawl", "otx" ], "includeSubdomains": true, "blacklistExtensions": [], "matchStatusCodes": [], "filterStatusCodes": [], "matchMimeTypes": [], "filterMimeTypes": [], "removeDuplicateParams": false, "threads": 5, "timeout": 45, "retries": 3, "maxResults": 0, "proxyConfiguration": { "useApifyProxy": false } } ``` # Actor output Schema ## `dataset` (type: `string`): No description ## `urlsCsv` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "domains": [ "trickster.dev" ] }; // Run the Actor and wait for it to finish const run = await client.actor("rl1987/gau-wrapper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "domains": ["trickster.dev"] } # Run the Actor and wait for it to finish run = client.actor("rl1987/gau-wrapper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "domains": [ "trickster.dev" ] }' | apify call rl1987/gau-wrapper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=rl1987/gau-wrapper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "gau - Get All URLs", "description": "Fetch known URLs from the Wayback Machine, Common Crawl, AlienVault OTX, and URLScan for any domain. A wrapper around the gau OSINT tool for attack-surface and data-pipeline use.", "version": "1.0", "x-build-id": "061W49x1sz05YEIhI" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/rl1987~gau-wrapper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-rl1987-gau-wrapper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/rl1987~gau-wrapper/runs": { "post": { "operationId": "runs-sync-rl1987-gau-wrapper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/rl1987~gau-wrapper/run-sync": { "post": { "operationId": "run-sync-rl1987-gau-wrapper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "domains" ], "properties": { "domains": { "title": "Domains", "type": "array", "description": "List of domains (or apex domains) to fetch known URLs for, e.g. `example.com`. Do not include a scheme (`https://`) or path.", "default": [ "trickster.dev" ], "items": { "type": "string" } }, "providers": { "title": "Providers", "type": "array", "description": "Which URL sources to query. Leave empty to use gau's defaults (wayback, commoncrawl, otx).", "items": { "type": "string", "enum": [ "wayback", "commoncrawl", "otx", "urlscan" ], "enumTitles": [ "Wayback Machine", "Common Crawl", "AlienVault OTX", "URLScan" ] }, "default": [ "wayback", "commoncrawl", "otx" ] }, "includeSubdomains": { "title": "Include subdomains", "type": "boolean", "description": "Include URLs from subdomains of the target domain (gau `--subs`).", "default": true }, "fromDate": { "title": "From date (YYYYMM)", "pattern": "^[0-9]{6}$", "type": "string", "description": "Only fetch URLs first seen on or after this month. Format: YYYYMM (e.g. `202101`)." }, "toDate": { "title": "To date (YYYYMM)", "pattern": "^[0-9]{6}$", "type": "string", "description": "Only fetch URLs first seen on or before this month. Format: YYYYMM (e.g. `202412`)." }, "blacklistExtensions": { "title": "Blacklist extensions", "type": "array", "description": "File extensions to skip, without the dot (gau `--blacklist`), e.g. `png`, `jpg`, `woff`.", "default": [], "items": { "type": "string" } }, "matchStatusCodes": { "title": "Match status codes", "type": "array", "description": "Only keep URLs whose archived HTTP status code is in this list (gau `--mc`), e.g. `200`, `301`.", "default": [], "items": { "type": "string" } }, "filterStatusCodes": { "title": "Filter status codes", "type": "array", "description": "Drop URLs whose archived HTTP status code is in this list (gau `--fc`), e.g. `404`, `302`.", "default": [], "items": { "type": "string" } }, "matchMimeTypes": { "title": "Match MIME types", "type": "array", "description": "Only keep URLs whose archived MIME type is in this list (gau `--mt`), e.g. `text/html`.", "default": [], "items": { "type": "string" } }, "filterMimeTypes": { "title": "Filter MIME types", "type": "array", "description": "Drop URLs whose archived MIME type is in this list (gau `--ft`), e.g. `text/plain`, `image/png`.", "default": [], "items": { "type": "string" } }, "removeDuplicateParams": { "title": "Collapse duplicate-parameter endpoints", "type": "boolean", "description": "Keep only one representative URL per endpoint that differs only in parameter values (gau `--fp`).", "default": false }, "threads": { "title": "Threads", "minimum": 1, "maximum": 50, "type": "integer", "description": "Number of worker threads gau spawns.", "default": 5 }, "timeout": { "title": "HTTP timeout (seconds)", "minimum": 1, "maximum": 600, "type": "integer", "description": "Timeout for gau's HTTP client, in seconds.", "default": 45 }, "retries": { "title": "HTTP retries", "minimum": 0, "maximum": 20, "type": "integer", "description": "Number of retries for gau's HTTP client.", "default": 3 }, "maxResults": { "title": "Max results", "minimum": 0, "type": "integer", "description": "Stop after pushing this many URLs to the dataset. Set 0 for unlimited.", "default": 0 }, "proxyConfiguration": { "title": "Proxy configuration", "type": "object", "description": "Route gau's requests through an Apify or custom HTTP proxy. Useful when archive providers rate-limit your runs.", "default": { "useApifyProxy": false } } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```