URL Extractor & Website Crawler – All Links, Sitemaps avatar

URL Extractor & Website Crawler – All Links, Sitemaps

Pricing

from $9.00 / 1,000 results

Go to Apify Store
URL Extractor & Website Crawler – All Links, Sitemaps

URL Extractor & Website Crawler – All Links, Sitemaps

URL extractor and website crawler: get every URL from a site with its link text, depth and parent page. Read all sitemap URLs or find broken links. Map site structure and audit internal links for SEO. Works on static and JavaScript sites. Export to CSV, JSON, Excel or API.

Pricing

from $9.00 / 1,000 results

Rating

0.0

(0)

Developer

Maged

Maged

Maintained by Community

Actor stats

0

Bookmarked

241

Total users

9

Monthly active users

6 days ago

Last modified

Features

Limited permissions

Share

What does URL Extractor & Website Crawler do?

URL Extractor & Website Crawler starts from any page, follows its links and gives you every URL it finds, one row each, with the link text, how deep it was and the page it was found on. So you get the list of links and the shape of the site in one go.

It also has two extra modes: pull every URL from a site's sitemaps, or check a site for broken links. It works on normal sites and on JavaScript apps (React, Vue, Angular). It runs on Apify, so you can schedule it, call it from the API, or send the results to Google Sheets, Make, Zapier or n8n.

What people use it for

  • Getting every URL of a site: for a migration, a redesign, a content inventory or a scraping job.
  • SEO site structure: see how deep your pages sit and which pages link to which.
  • Sitemap work: pull the URLs a site's sitemap lists, with last-modified dates, or build a list for a site that has no sitemap.
  • Broken link checks: find 404s and dead external links with the page they're on, before your visitors or Google do.
  • Competitor research: map a competitor's site, sections and landing pages.
  • Feeding other tools: get the URLs here, then turn the pages into text or pull contacts with the related tools below.

Why use this one?

  • Three jobs in one actor: crawl links, read sitemaps, check broken links.
  • The hierarchy comes with it: every row has its depth and parentUrl, not just a flat list.
  • You control the size: max depth, max links followed per page, same-domain only, and file extensions to skip.
  • JavaScript sites: turn on JavaScript rendering for sites whose links only appear after the page loads.
  • No duplicates by default: each URL shows up once unless you ask for every occurrence.

How to use it

  1. Click Try for free.
  2. Enter the start URL.
  3. Set Max depth and Max children per link to decide how far it goes.
  4. Click Start. URLs appear in the Output tab as they're found.
  5. Download as JSON, CSV or Excel, or use the API.

Input (crawl mode)

{
"startUrl": "https://example.com",
"maxDepth": 3,
"maxChildrenPerLink": 20,
"sameDomainOnly": true,
"renderJavaScript": false,
"allowDuplicates": false,
"ignoredExtensions": ["pdf", "jpg", "png", "zip"]
}
FieldTypeDescriptionDefault
startUrlstringThe page to start fromrequired
modestringcrawl, sitemap or brokenLinks. Empty = crawlcrawl
maxDepthintegerHow many clicks deep to follow links (1–30)3
maxChildrenPerLinkintegerLinks followed from each page (1–100)3
sameDomainOnlybooleanStay on the start URL's domaintrue
renderJavaScriptbooleanLoad pages in a browser for JavaScript sites. Slowerfalse
allowDuplicatesbooleanList a URL every time it's found, not just oncefalse
ignoredExtensionsarrayFile extensions to skip, like pdf or jpg[]

Sitemap mode

Every URL listed in a site's sitemaps (found through robots.txt or the usual locations, including sitemap indexes and .gz files) with lastModified, changeFrequency and priority, plus one summary row per site. You can do several sites in one run.

{
"startUrl": "https://apify.com",
"mode": "sitemap",
"websites": ["wordpress.org"],
"maxUrlsPerSite": 1000
}

Crawls the site and checks every link, and images too if you want. Each problem link comes with the page it was found on, its text, the status code and whether it's internal or external. Login walls and rate limits (401/403/429) count as restricted, not broken. One summary row per site has the pages crawled, links checked and the broken count.

{
"startUrl": "https://the-internet.herokuapp.com",
"mode": "brokenLinks",
"maxPagesPerSite": 100,
"checkImages": true,
"onlyBrokenLinks": true
}
FieldModeDescriptionDefault
websitessitemap, brokenLinksExtra sites to do in the same run—
maxUrlsPerSitesitemapStop after this many URLs per site, 0 = no limit1000
maxPagesPerSitebrokenLinksInternal pages to visit per site100
checkExternalLinksbrokenLinksAlso check links to other sitestrue
checkImagesbrokenLinksAlso check image sourcesfalse
onlyBrokenLinksbrokenLinksTurn off to get every checked link with its statustrue

Sites with no sitemap, or that can't be reached, are listed in the run's status message and aren't charged.

Output (crawl mode)

[
{ "url": "https://example.com", "name": null, "query": "", "depth": 0, "parentUrl": null },
{ "url": "https://example.com/about", "name": "About Us", "query": "", "depth": 1, "parentUrl": "https://example.com" },
{ "url": "https://example.com/blog?page=2", "name": "Older posts", "query": "page=2", "depth": 2, "parentUrl": "https://example.com/blog" }
]
FieldTypeDescription
urlstringThe full URL
namestringThe link text it was found with (empty for the start URL)
querystringThe URL's query string, if it has one
depthnumberClicks from the start URL (0 = the start URL)
parentUrlstringThe page this link was found on

How many results will I get?

You pay per row, so the bill follows the number of URLs found. Your plan's rates are on the Pricing tab.

InputResults
Max depth 1The start URL plus the links followed from it
Max depth 3, max children 3 (the defaults)Up to about 40 rows
Max depth 3, max children 20Up to about 8,400 rows
Sitemap mode, max 1,000 per siteUp to 1,000 rows per site, plus a summary row

The count grows fast with depth × children, so start small and raise it. Keep JavaScript rendering off unless the site needs it, since it's much slower.

FAQ

Crawl, sitemap or broken links, which do I need? Want every URL of a site and the site has a sitemap? Sitemap mode is the fastest. Want to see how pages link to each other, or the site has no sitemap? Use crawl. Want to find dead links? Broken links.

What's the difference between normal mode and JavaScript rendering? Normal mode reads the page's HTML and is much faster. JavaScript rendering opens each page in a browser, for sites whose links only show up after scripts run.

Can I do several sites in one run? In sitemap and broken links mode, yes, with websites. Crawl mode starts from one URL, so start one run per site (the API makes that easy).

Will it follow links to other websites? Only if you turn off Same domain only.

Is it maintained? Yes, I keep it running. If something breaks, open an issue in the Issues tab and I'll look at it.

Need a custom feature? Open an issue or message me on Discord: maged03211


⭐ Found it useful? A short review on the Store helps other people find it, and it tells me what to improve next.