URL Extractor & Website Crawler – All Links, Sitemaps
Pricing
from $9.00 / 1,000 results
URL Extractor & Website Crawler – All Links, Sitemaps
URL extractor and website crawler: get every URL from a site with its link text, depth and parent page. Read all sitemap URLs or find broken links. Map site structure and audit internal links for SEO. Works on static and JavaScript sites. Export to CSV, JSON, Excel or API.
Pricing
from $9.00 / 1,000 results
Rating
0.0
(0)
Developer
Maged
Maintained by CommunityActor stats
0
Bookmarked
241
Total users
9
Monthly active users
6 days ago
Last modified
Categories
Features
Share
What does URL Extractor & Website Crawler do?
URL Extractor & Website Crawler starts from any page, follows its links and gives you every URL it finds, one row each, with the link text, how deep it was and the page it was found on. So you get the list of links and the shape of the site in one go.
It also has two extra modes: pull every URL from a site's sitemaps, or check a site for broken links. It works on normal sites and on JavaScript apps (React, Vue, Angular). It runs on Apify, so you can schedule it, call it from the API, or send the results to Google Sheets, Make, Zapier or n8n.
What people use it for
- Getting every URL of a site: for a migration, a redesign, a content inventory or a scraping job.
- SEO site structure: see how deep your pages sit and which pages link to which.
- Sitemap work: pull the URLs a site's sitemap lists, with last-modified dates, or build a list for a site that has no sitemap.
- Broken link checks: find 404s and dead external links with the page they're on, before your visitors or Google do.
- Competitor research: map a competitor's site, sections and landing pages.
- Feeding other tools: get the URLs here, then turn the pages into text or pull contacts with the related tools below.
Why use this one?
- Three jobs in one actor: crawl links, read sitemaps, check broken links.
- The hierarchy comes with it: every row has its
depthandparentUrl, not just a flat list. - You control the size: max depth, max links followed per page, same-domain only, and file extensions to skip.
- JavaScript sites: turn on JavaScript rendering for sites whose links only appear after the page loads.
- No duplicates by default: each URL shows up once unless you ask for every occurrence.
How to use it
- Click Try for free.
- Enter the start URL.
- Set Max depth and Max children per link to decide how far it goes.
- Click Start. URLs appear in the Output tab as they're found.
- Download as JSON, CSV or Excel, or use the API.
Input (crawl mode)
{"startUrl": "https://example.com","maxDepth": 3,"maxChildrenPerLink": 20,"sameDomainOnly": true,"renderJavaScript": false,"allowDuplicates": false,"ignoredExtensions": ["pdf", "jpg", "png", "zip"]}
| Field | Type | Description | Default |
|---|---|---|---|
startUrl | string | The page to start from | required |
mode | string | crawl, sitemap or brokenLinks. Empty = crawl | crawl |
maxDepth | integer | How many clicks deep to follow links (1–30) | 3 |
maxChildrenPerLink | integer | Links followed from each page (1–100) | 3 |
sameDomainOnly | boolean | Stay on the start URL's domain | true |
renderJavaScript | boolean | Load pages in a browser for JavaScript sites. Slower | false |
allowDuplicates | boolean | List a URL every time it's found, not just once | false |
ignoredExtensions | array | File extensions to skip, like pdf or jpg | [] |
Sitemap mode
Every URL listed in a site's sitemaps (found through robots.txt or the usual locations, including sitemap indexes and .gz files) with lastModified, changeFrequency and priority, plus one summary row per site. You can do several sites in one run.
{"startUrl": "https://apify.com","mode": "sitemap","websites": ["wordpress.org"],"maxUrlsPerSite": 1000}
Broken links mode
Crawls the site and checks every link, and images too if you want. Each problem link comes with the page it was found on, its text, the status code and whether it's internal or external. Login walls and rate limits (401/403/429) count as restricted, not broken. One summary row per site has the pages crawled, links checked and the broken count.
{"startUrl": "https://the-internet.herokuapp.com","mode": "brokenLinks","maxPagesPerSite": 100,"checkImages": true,"onlyBrokenLinks": true}
| Field | Mode | Description | Default |
|---|---|---|---|
websites | sitemap, brokenLinks | Extra sites to do in the same run | — |
maxUrlsPerSite | sitemap | Stop after this many URLs per site, 0 = no limit | 1000 |
maxPagesPerSite | brokenLinks | Internal pages to visit per site | 100 |
checkExternalLinks | brokenLinks | Also check links to other sites | true |
checkImages | brokenLinks | Also check image sources | false |
onlyBrokenLinks | brokenLinks | Turn off to get every checked link with its status | true |
Sites with no sitemap, or that can't be reached, are listed in the run's status message and aren't charged.
Output (crawl mode)
[{ "url": "https://example.com", "name": null, "query": "", "depth": 0, "parentUrl": null },{ "url": "https://example.com/about", "name": "About Us", "query": "", "depth": 1, "parentUrl": "https://example.com" },{ "url": "https://example.com/blog?page=2", "name": "Older posts", "query": "page=2", "depth": 2, "parentUrl": "https://example.com/blog" }]
| Field | Type | Description |
|---|---|---|
url | string | The full URL |
name | string | The link text it was found with (empty for the start URL) |
query | string | The URL's query string, if it has one |
depth | number | Clicks from the start URL (0 = the start URL) |
parentUrl | string | The page this link was found on |
How many results will I get?
You pay per row, so the bill follows the number of URLs found. Your plan's rates are on the Pricing tab.
| Input | Results |
|---|---|
| Max depth 1 | The start URL plus the links followed from it |
| Max depth 3, max children 3 (the defaults) | Up to about 40 rows |
| Max depth 3, max children 20 | Up to about 8,400 rows |
| Sitemap mode, max 1,000 per site | Up to 1,000 rows per site, plus a summary row |
The count grows fast with depth × children, so start small and raise it. Keep JavaScript rendering off unless the site needs it, since it's much slower.
FAQ
Crawl, sitemap or broken links, which do I need? Want every URL of a site and the site has a sitemap? Sitemap mode is the fastest. Want to see how pages link to each other, or the site has no sitemap? Use crawl. Want to find dead links? Broken links.
What's the difference between normal mode and JavaScript rendering? Normal mode reads the page's HTML and is much faster. JavaScript rendering opens each page in a browser, for sites whose links only show up after scripts run.
Can I do several sites in one run?
In sitemap and broken links mode, yes, with websites. Crawl mode starts from one URL, so start one run per site (the API makes that easy).
Will it follow links to other websites? Only if you turn off Same domain only.
Is it maintained? Yes, I keep it running. If something breaks, open an issue in the Issues tab and I'll look at it.
Need a custom feature? Open an issue or message me on Discord: maged03211
⭐ Found it useful? A short review on the Store helps other people find it, and it tells me what to improve next.