gau - Get All URLs
Pricing
from $0.01 / actor run
Fetch known URLs from the Wayback Machine, Common Crawl, AlienVault OTX, and URLScan for any domain. A wrapper around the gau OSINT tool for attack-surface and data-pipeline use.
List of domains (or apex domains) to fetch known URLs for, e.g. example.com. Do not include a scheme (https://) or path.
[ "trickster.dev"]Which URL sources to query. Leave empty to use gau's defaults (wayback, commoncrawl, otx).
[ "wayback", "commoncrawl", "otx"]Include URLs from subdomains of the target domain (gau --subs).
Only fetch URLs first seen on or after this month. Format: YYYYMM (e.g. 202101).
Only fetch URLs first seen on or before this month. Format: YYYYMM (e.g. 202412).
File extensions to skip, without the dot (gau --blacklist), e.g. png, jpg, woff.
[]Only keep URLs whose archived HTTP status code is in this list (gau --mc), e.g. 200, 301.
[]Drop URLs whose archived HTTP status code is in this list (gau --fc), e.g. 404, 302.
[]Only keep URLs whose archived MIME type is in this list (gau --mt), e.g. text/html.
[]Drop URLs whose archived MIME type is in this list (gau --ft), e.g. text/plain, image/png.
[]Keep only one representative URL per endpoint that differs only in parameter values (gau --fp).
Timeout for gau's HTTP client, in seconds.
Number of retries for gau's HTTP client.
Stop after pushing this many URLs to the dataset. Set 0 for unlimited.