Website URL Finder (Recent & Historical)

Discover website URLs from recent and historical web records using a domain, subdomain, or path pattern. Build clean URL lists for active sites while also uncovering older pages, orphan URLs, and content that is no longer linked.

🤖 What does Website URL Finder do?

Enter patterns such as example.com/*, example.com/blog/*, or *.example.com/* and receive a deduplicated list of matching URLs. Each result was successfully captured at the reported time, making the dataset useful for both recent website discovery and historical research.

• Find website URLs across an entire domain or a specific directory
• Discover subdomains with a single wildcard pattern
• Search recent records using the 10 newest web snapshots automatically
• Explore URL history by searching every available snapshot from a chosen year
• Reveal orphan and legacy pages that may no longer appear in navigation or sitemaps
• Export unique results as JSON, CSV, Excel, XML, or other Apify dataset formats

Great for: SEO audits, site migrations, content inventories, competitor research, lead generation, redirect maps, and data pipelines.

SEO keywords: website URL finder, find all website URLs, URL discovery tool, historical URL finder, orphan page finder, domain URL extractor, website URL list, Apify Actor.

💡 Why discover website URLs?

• 🔎 Inventory an existing website — collect known recent URLs without depending on its current navigation
• 🧭 Find orphan pages — reveal URLs that are hard to reach from the live site
• 🚚 Plan site migrations — build redirect maps and reduce lost traffic during redesigns
• 📈 Improve SEO audits — compare recent and historical paths to identify cleanup opportunities
• 🗂️ Create datasets — export focused URL lists for enrichment, monitoring, or research
• 🎯 Research competitors — discover product, category, campaign, and content structures

🧠 How history works

By default, the Actor searches the 10 newest web snapshots. This is the best starting point for finding URLs associated with an existing website while retaining some recent history.

Set historyYear when you need a deeper historical view. The Actor then searches all available snapshots from that year, newest first. Results are deduplicated by exact URL across every pattern and snapshot.

The capture time describes when a URL was successfully recorded. It does not guarantee that the URL is still live or returns the same content today.

🚀 How to use

1. Add the Actor – Open Website URL Finder (Recent & Historical) in Apify Store.
2. Enter URL patterns – Add one or more values to urlPatterns.
3. Choose the time range – Leave historyYear empty for recent discovery, or select a year for a historical search.
4. Set a result limit – Use maxResults for a controlled run, or keep 0 to return every unique match.
5. Run and download – Export the dataset as JSON, CSV, Excel, XML, or another supported format.

💳 Pricing

Apify's Free plan includes monthly credits suitable for small searches. Broad whole-domain patterns or full-year history searches can return many URLs, so start with maxResults before running an unlimited job.

📝 Input parameters

Parameter	Type	Required	Description
urlPatterns	Array of strings	Yes	Domain, subdomain, or path patterns such as example.com/*, example.com/blog/*, or *.example.com/*.
maxResults	Integer	No	Maximum number of unique URLs returned. Default 0 returns every match.
historyYear	Integer	No	Search every available web snapshot from this year. When omitted, the 10 newest snapshots are searched. Minimum 2008.

Example inputs

Find up to 1,000 URLs for an existing website

{
  "urlPatterns": ["example.com/*"],
  "maxResults": 1000
}

Build a historical content inventory for selected paths

{
  "urlPatterns": [
    "example.com/blog/*",
    "example.com/products/*"
  ],
  "historyYear": 2024,
  "maxResults": 5000
}

Discover URLs across subdomains

{
  "urlPatterns": ["*.example.com/*"],
  "maxResults": 2500
}

📊 Output results

Each unique URL produces one dataset item:

• matchedPattern (string) — the first input pattern that matched the URL
• url (string) — the discovered absolute URL
• capturedAt (string, optional) — when the URL was successfully recorded, in ISO 8601 format

{
  "matchedPattern": "example.com/blog/*",
  "url": "https://example.com/blog/website-migration-checklist",
  "capturedAt": "2025-07-18T14:23:09Z"
}

🧭 Patterns and common recipes

• Whole website: use example.com/* to discover paths associated with the main domain.
• One directory: use example.com/blog/* or example.com/products/* for a focused inventory.
• Subdomain homepages: use *.example.com to return only the root URL of each subdomain.
• All subdomains: use *.example.com/* to find URLs on hosts below the domain.
• One path across subdomains: use *.example.com/blog/* to keep only /blog/ URLs on subdomains.
• Multiple areas: add several patterns; maxResults applies to the combined, deduplicated output.
• Recent discovery: omit historyYear to use the 10 newest snapshots.
• Historical research: set historyYear: 2022 to search all snapshots available from 2022.

Patterns support subdomain wildcards and path prefixes, not arbitrary regular expressions.

For bare subdomain patterns, capturedAt is the capture time of the page that proved the subdomain existed; the derived root URL itself may not have been captured.

⚙️ Best practices

• Start with a limit such as maxResults: 1000, then expand after checking relevance.
• Use narrow path patterns when a full domain could contain hundreds of thousands of URLs.
• Treat results as discoveries, not live checks because older URLs may now redirect, fail, or contain different content.
• Run separate years for comparisons when researching how a site's URL structure changed over time.
• Expect exact-URL deduplication across all input patterns and searched snapshots.

🌍 Proxy usage

Direct access works for most runs. Proxy settings are available for environments that require them, but they are usually unnecessary for normal URL discovery.

⚖️ Legal and ethical considerations

• Respect website terms, robots rules, and applicable data-use restrictions.
• Handle personal information responsibly and comply with privacy laws such as GDPR and CCPA.
• Keep search scopes reasonable and avoid unnecessarily broad automated collection.
• Preserve attribution and ownership when publishing or redistributing derived URL datasets.
• Review usage rights before using discovered data in commercial products.

You are responsible for ensuring that your use of the results complies with applicable laws and third-party rights.

❓ FAQ

Can it find URLs that currently exist?

Yes. The newest snapshots are useful for finding URLs associated with existing websites. Because this is snapshot-based discovery rather than a live crawl, verify critical URLs before using them operationally.

Can it find deleted or unlinked pages?

Yes. A historical year can reveal older and orphan URLs, including pages that are no longer linked or live today.

Does it download page content?

No. It returns a focused list of discovered URLs and their capture times.

Are duplicate URLs returned?

No. Each exact URL is emitted once, even if it appears in multiple patterns or snapshots.

Does *.example.com/* include the root domain?

No. It returns only subdomain URLs such as www.example.com and docs.example.com. Add example.com/* as another pattern when you also want root-domain URLs.

What does maxResults: 0 mean?

It removes the result-count limit. The Actor searches every selected pattern across the chosen recent or historical scope.

Why might a discovered URL not work today?

The URL was successfully recorded at capturedAt, but websites change. It may now redirect, return an error, or have been removed.

🛟 Support

Need another output field, pattern option, or specialized URL-discovery workflow? Open an issue or contact me at https://igolaizola.com/#contact