🔗 Website Link Crawler

Website Link Crawler lets you crawl websites and extract structured SEO link data from every page.

Find broken links, analyze internal linking, detect redirects, inspect anchor text, identify follow/nofollow links, review crawl depth, and export clean source → target link relationships without manually checking pages one by one.

• 🔎 Discover website links: extract internal and external links from crawled pages
• 🚨 Find broken links: detect URLs returning HTTP errors or request failures
• ↪️ Detect redirects: capture redirect status and destination URLs
• 🧭 Analyze internal linking: preserve the exact page where every link was found
• 🏷 Inspect anchor text: extract the visible text attached to each link
• 🔗 Check follow status: distinguish normal follow links from nofollow links
• 📏 Track crawl depth: understand how deep each source page is from the starting URL
• 📄 Inspect page metadata: collect page titles and canonical URLs
• 🧹 Get cleaner results: normalize fragments and remove duplicate target links on the same source page
• ⚡ Automate SEO audits: export structured results to Apify Dataset, API, integrations, or downstream applications

---

📦 What data does Website Link Crawler extract?

📄 Source URL	📝 Source page title
🔖 Canonical URL	🎯 Target URL
🏷 Anchor text	🔗 Internal / external classification
👁 Follow / nofollow status	📏 Crawl depth
🌐 HTTP status code	🚨 Broken-link status
↪️ Redirect status	🎯 Redirect destination
⚠️ Request error information	🧹 Normalized target URL

Unavailable values are returned as null. The Actor does not invent metadata that the website does not expose.

---

🚀 Features

Website Link Crawler is designed for technical SEO audits, website maintenance, site migrations, and link analysis.

• Multiple start URLs: crawl one or more websites or sections in a single Actor run
• Configurable crawl depth: control how many internal-link levels the crawler follows
• Configurable page limit: prevent unexpectedly large crawls
• Internal link discovery: capture links pointing to the same hostname
• External link discovery: capture links pointing to other hostnames
• External-link safety: external links are analyzed but are not recursively crawled
• HTTP status checking: inspect the response status of discovered targets
• Broken-link detection: mark HTTP errors and unreachable targets
• Redirect detection: identify HTTP redirects without automatically hiding the original response
• Redirect destination extraction: preserve the URL returned in the redirect Location header
• Follow / nofollow classification
• Anchor-text extraction
• Source page title extraction
• Canonical URL extraction
• Crawl-depth tracking
• Fragment normalization: URLs such as /about and /about#team are treated as the same HTTP resource
• Per-page duplicate removal: repeated target links on the same source page are not emitted repeatedly
• Same-hostname recursive crawling
• API-friendly output: every Dataset row follows a predictable source → target relationship
• HTTP-first architecture: normal crawling does not require a browser

---

⬇️ Input

You can configure Website Link Crawler directly from the Apify Console or send the input programmatically through the Apify API.

---

🔗 Start URLs

Add one or more URLs where crawling should begin.

Example:

https://example.com
https://example.org

Each starting URL begins at:

crawlDepth: 0

The Actor can then follow internal links depending on the configured maximum crawl depth.

---

📏 Max Crawl Depth

Controls how many levels of internal links the crawler follows.

Minimum: 0
Default: 2
Maximum: 20

Depth 0

https://example.com

Only the starting page is crawled.

Links found on that page are still analyzed and written to the Dataset, but the linked pages are not recursively crawled.

---

Depth 1

Example:

Start:
https://example.com

Links to:
https://example.com/about
https://example.com/services

The Actor crawls:

Depth 0 → Homepage
Depth 1 → About
Depth 1 → Services

but does not continue to depth 2.

---

Depth 2

Example:

Homepage
   ↓
Services
   ↓
Plumbing

becomes:

Homepage → depth 0
Services → depth 1
Plumbing → depth 2

The crawler stops adding deeper requests after the configured limit is reached.

---

🔢 Maximum Pages to Crawl

Limits how many pages the crawler may process during the Actor run.

Minimum: 1
Default: 100
Maximum: 100000

For example:

{
  "maxRequestsPerCrawl": 500
}

allows the crawler to process up to 500 pages.

The requested number is a maximum, not a guarantee.

A crawl may naturally finish earlier when no new internal pages remain.

---

🧪 Example input

{
  "startUrls": [
    {
      "url": "https://example.com"
    }
  ],
  "maxCrawlDepth": 2,
  "maxRequestsPerCrawl": 100
}

---

⬆️ Output

Results are stored in the Actor's default Apify Dataset.

You can access the results from:

Output
Storage
Apify API
Integrations

Each Dataset row represents:

one discovered target URL
+
the source page where it was found

---

📊 Example result

{
  "sourceUrl": "https://example.com/",
  "sourceTitle": "Example Company",
  "canonicalUrl": "https://example.com/",
  "targetUrl": "https://example.com/services",
  "anchorText": "Our Services",
  "linkType": "internal",
  "followType": "follow",
  "crawlDepth": 0,
  "statusCode": 200,
  "isBroken": false,
  "isRedirect": false,
  "redirectUrl": null,
  "statusError": null
}

---

📄 Source URL

The sourceUrl field tells you exactly where a link was discovered.

Example:

sourceUrl:
https://example.com/services

targetUrl:
https://example.com/contact

This preserves the relationship:

/services → /contact

instead of returning only a flat list of URLs.

This is useful for technical SEO because you can identify which page contains the problematic link.

---

📝 Source page title

Each result includes:

sourceTitle

when the source page contains an HTML <title> element.

Example:

{
  "sourceUrl": "https://example.com/services",
  "sourceTitle": "Home Services | Example Company"
}

This makes large Dataset exports easier to understand without manually opening each source page.

---

🔖 Canonical URL

When the source page contains:

<link rel="canonical" href="https://example.com/preferred-page">

the Actor returns:

{
  "canonicalUrl": "https://example.com/preferred-page"
}

Relative canonical URLs are resolved against the source page.

Example:

<link rel="canonical" href="/services">

becomes:

https://example.com/services

If no canonical tag is available:

{
  "canonicalUrl": null
}

---

🔗 Internal and external links

Every target is classified as either:

internal
external

---

🏠 Internal links

A link is considered internal when its hostname matches the source page hostname.

Example:

Source:
https://example.com/

Target:
https://example.com/about

Output:

{
  "linkType": "internal"
}

---

🌍 External links

Example:

Source:
https://example.com/

Target:
https://wikipedia.org/

Output:

{
  "linkType": "external"
}

External links are still:

extracted
status checked
classified
stored

but the crawler does not recursively crawl the external website.

This prevents a normal website audit from accidentally expanding across the entire web.

---

🏷 Anchor text

The Actor extracts the visible text inside each link.

Example HTML:

<a href="/pricing">View our pricing</a>

Output:

{
  "anchorText": "View our pricing"
}

This can help with:

Internal anchor-text audits
SEO optimization
Navigation analysis
Generic anchor detection
Site migration reviews

---

👁 Follow and nofollow links

Website Link Crawler checks the link's rel attribute.

Normal link:

<a href="/about">About</a>

Output:

{
  "followType": "follow"
}

Nofollow link:

<a href="/partner" rel="nofollow">Partner</a>

Output:

{
  "followType": "nofollow"
}

A link containing multiple rel values is still recognized.

Example:

<a href="/partner" rel="nofollow sponsored">

returns:

nofollow

---

🚨 Broken-link detection

One of the main uses of Website Link Crawler is finding links that no longer work.

The Actor checks each discovered target URL and returns:

statusCode
isBroken
statusError

---

✅ Working URL

Example:

{
  "statusCode": 200,
  "isBroken": false,
  "statusError": null
}

---

❌ Broken URL

Example:

{
  "statusCode": 404,
  "isBroken": true,
  "statusError": null
}

HTTP error responses such as:

400
401
403
404
410
429
500
503

are represented by their actual response status.

The Actor currently marks responses with status code 400 or higher as broken.

---

⚠️ Request failure

If the target cannot be checked because of a network/request failure:

{
  "statusCode": null,
  "isBroken": true,
  "statusError": "Request error information"
}

This distinguishes:

HTTP error response

from:

No valid HTTP response received

---

↪️ Redirect detection

Redirecting links are preserved instead of automatically hiding the original response.

For example:

https://example.com/old-page

returns:

301

with:

Location: /new-page

The Actor can return:

{
  "targetUrl": "https://example.com/old-page",
  "statusCode": 301,
  "isBroken": false,
  "isRedirect": true,
  "redirectUrl": "https://example.com/new-page"
}

This is useful because SEO audits often need to identify pages that are still linking to redirected URLs.

---

Why redirects matter

Imagine:

Homepage
   ↓
/old-services
   ↓ 301
/new-services

The website technically works, but the internal link is outdated.

Website Link Crawler lets you identify that relationship so the original link can be updated directly to:

/new-services

---

📏 Crawl-depth tracking

Every result contains:

crawlDepth

The value describes the depth of the source page where the link was found.

Example:

Homepage
Depth 0

↓ links to

Services
Depth 1

↓ links to

Roofing
Depth 2

A link discovered on the Roofing page would therefore contain:

{
  "crawlDepth": 2
}

This can help identify:

Deep pages
Site architecture problems
Important content buried too far from the homepage
Navigation complexity
Internal-link opportunities

---

🧹 URL normalization and duplicate handling

Website pages often contain repeated links.

For example:

<a href="/about">About</a>
<a href="/about">Company</a>
<a href="/about#team">Our Team</a>

Website Link Crawler normalizes URL fragments before duplicate detection.

These:

https://example.com/about
https://example.com/about#team

are treated as the same HTTP resource for target deduplication.

The crawler keeps only the first occurrence of the normalized target URL within that source page.

This avoids:

Duplicate Dataset rows
Duplicate HTTP status requests
Unnecessary processing
Messy exports

---

Important deduplication behavior

Deduplication is performed per source page.

This means:

Homepage → /contact
Services → /contact
Blog → /contact

remain three useful relationships.

That is intentional.

They represent three different internal links even though they point to the same target.

The Actor does not globally remove these relationships.

---

🌐 Same-hostname crawling

Recursive crawling uses a same-hostname strategy.

Example:

Start:
https://example.com

The crawler may follow:

https://example.com/about
https://example.com/services
https://example.com/blog

but it does not recursively begin crawling:

https://facebook.com/...
https://youtube.com/...
https://wikipedia.org/...

Those external targets can still appear in the Dataset and have their link status checked.

---

⚡ Performance

Website Link Crawler uses an HTTP-first architecture based on Crawlee and Cheerio.

It does not launch a browser for normal page crawling.

This generally means:

Lower memory use
Faster startup
Lower browser overhead
Lower compute requirements
Efficient HTML parsing

This design is especially useful for crawling ordinary server-rendered websites.

---

🛡 Reliability

The Actor is designed so discovered links are processed independently.

A target status check can return:

HTTP status
redirect information
request error

without requiring the Dataset to hide the underlying result.

The crawler also includes:

Configurable page limits
Configurable depth limits
Per-page URL deduplication
Protocol filtering
Same-hostname recursion
Invalid URL protection
Request timeout protection

Only:

http:
https:

links are analyzed as web targets.

Links using protocols such as:

mailto:
tel:
javascript:

are currently skipped.

---

🔌 API & integrations

Website Link Crawler can be used programmatically through the Apify API.

You can connect Dataset results to workflows such as:

• Google Sheets
• Make
• Zapier
• CRMs
• Internal APIs
• Data warehouses
• SEO reporting systems
• Website monitoring workflows
• AI agents
• MCP workflows

Each Actor run stores structured source → target link relationships in the default Dataset, making the output straightforward to process with another application.

---

💡 Example use cases

🚨 Broken-link audit

Run the crawler on your website and filter:

isBroken = true

to identify URLs that may need attention.

Useful output:

Source page
Broken target
Anchor text
HTTP status
Crawl depth

This tells you both what is broken and where the broken link exists.

---

↪️ Redirect cleanup

Filter:

isRedirect = true

to locate links pointing to redirects.

Example:

Source:
https://example.com/

Target:
https://example.com/old-pricing

Redirect:
https://example.com/pricing

You can then update the source page to point directly to the final URL.

---

🧭 Internal-link audit

Filter:

linkType = internal

to analyze relationships between pages on the website.

Use:

sourceUrl
targetUrl
anchorText
crawlDepth

to understand the internal linking structure.

---

🌍 External-link audit

Filter:

linkType = external

to review websites being linked to from your pages.

This can help identify:

Dead external references
Redirecting external links
Old resources
Unexpected outbound links

---

🏷 Anchor-text analysis

Export:

sourceUrl
targetUrl
anchorText

to inspect how internal links are labeled.

Example:

/services → /contact → Contact us
/blog/article → /contact → Get a quote
/pricing → /contact → Start today

This can support internal-link and SEO reviews.

---

📏 Site-depth analysis

Use:

crawlDepth

to find pages and links located deeper within the site's architecture.

For example:

Depth 0 → Homepage
Depth 1 → Main category
Depth 2 → Subcategory
Depth 3 → Individual page

Deep pages may be useful candidates for additional internal links.

---

🚚 Website migration auditing

Before or after a website migration, crawl the site and inspect:

404 links
301 redirects
Old internal URLs
Canonical URLs
Internal anchor text
Source → target relationships

This helps identify links that still point to outdated paths.

---

⚠️ Current limitations

The current version focuses on fast HTTP-based website link analysis.

It does not currently perform:

JavaScript/browser-rendered link discovery
Redirect-chain traversal
Internal-link graph visualization
Site-wide orphan-page detection
XML sitemap comparison
robots.txt auditing
Hreflang auditing
Meta robots analysis
Structured-data auditing
Page-speed analysis
Full page-content extraction
Domain-level link summaries
Site-wide anchor-text summaries
Automatic scheduled historical comparisons

These are intentionally separate from the current MVP.

---

JavaScript-rendered websites

The Actor currently uses:

HTTP
+
Cheerio

instead of Playwright or Puppeteer.

If a website creates links only after JavaScript executes in the browser, those links may not appear in the HTML received by the crawler.

For ordinary server-rendered HTML websites, this is not a problem.

Browser-rendered crawling can be introduced later as a separate mode if needed.

---

❓ FAQ

How does Website Link Crawler work?

The Actor starts from the URLs you provide.

It then:

Downloads the source page
Parses HTML
Finds <a href> links
Normalizes target URLs
Removes duplicate targets on that page
Extracts anchor text
Classifies internal/external
Detects follow/nofollow
Checks HTTP status
Detects redirects
Stores source → target relationships
Follows eligible internal links
Tracks crawl depth
Stops at configured limits

---

What is the difference between this and a normal URL crawler?

A normal crawler may return:

https://example.com
https://example.com/about
https://example.com/services

Website Link Crawler returns relationships such as:

Homepage
   ↓ "Our services"
/services
HTTP 200
Internal
Follow
Depth 0

This makes the output more useful for SEO audits and website maintenance.

---

Does the Actor find backlinks?

No.

This Actor discovers links on the website being crawled.

Example:

example.com/page-a
        ↓
example.com/page-b

A backlink tool would instead need to discover:

otherwebsite.com
        ↓
example.com

Those are different problems.

Website Link Crawler is an on-site link crawler, not a global backlink index.

---

Does it crawl external websites?

No.

External links are:

discovered
classified
status checked
stored

but they are not recursively crawled.

This prevents an audit from expanding uncontrollably across unrelated websites.

---

Does it detect broken links?

Yes.

The Actor marks:

HTTP status >= 400

as broken.

It also marks targets as broken when the status request cannot be completed.

---

Does a redirect count as a broken link?

No.

For example:

301
302
307
308

are treated as redirects rather than broken HTTP errors.

A redirecting URL can therefore return:

{
  "isBroken": false,
  "isRedirect": true
}

---

Does it follow redirects?

The current status checker preserves the original redirect response.

It returns:

Original status code
Redirect indicator
Redirect destination

Example:

301
https://example.com/new-page

It does not currently build a full multi-hop redirect chain.

---

Does it remove duplicate links?

Yes, within each source page.

For example:

/about
/about
/about#team

are normalized and treated as one target resource on that source page.

But:

Homepage → /about
Services → /about
Blog → /about

are all retained because each represents a different source → target relationship.

---

Does it extract canonical URLs?

Yes, when present in the source HTML.

Example:

<link rel="canonical" href="/preferred-page">

is resolved to an absolute URL.

If no canonical exists:

canonicalUrl: null

---

Can I control how deeply the website is crawled?

Yes.

Use:

maxCrawlDepth

Current limits:

Minimum: 0
Default: 2
Maximum: 20

---

Can I limit how many pages are crawled?

Yes.

Use:

maxRequestsPerCrawl

Current limits:

Minimum: 1
Default: 100
Maximum: 100000

---

Can I crawl multiple websites in one run?

Yes.

You can provide multiple:

startUrls

in the same Actor input.

Each start request begins at crawl depth 0.

---

Can I use Website Link Crawler through an API?

Yes.

Every Actor can be started through the Apify API, and the resulting Dataset can be consumed programmatically.

This makes Website Link Crawler suitable for:

Automated SEO audits
Website QA pipelines
Migration checks
Reporting systems
Internal tools
Scheduled workflows

---

Can I export the results?

Yes.

Apify Dataset results can be exported in supported formats such as:

JSON
CSV
Excel
XML
HTML

---

Is a browser required?

Not for the current workflow.

Website Link Crawler uses an HTTP-first approach with:

Crawlee
Cheerio

which avoids launching full browser automation during normal crawls.

---

Why is canonicalUrl sometimes null?

Because not every page contains a canonical tag.

The Actor returns:

null

instead of inventing one.

---

Why is statusCode sometimes null?

This means the status request did not receive a normal HTTP response.

The associated:

statusError

field may contain additional information about the request failure.

---

🧭 Recommended workflow

For a normal SEO website audit, start conservatively.

Example:

{
  "startUrls": [
    {
      "url": "https://example.com"
    }
  ],
  "maxCrawlDepth": 2,
  "maxRequestsPerCrawl": 500
}

First inspect:

isBroken
isRedirect
linkType
crawlDepth
canonicalUrl
anchorText

Then increase the crawl depth or maximum pages if additional coverage is needed.

For very large websites, avoid immediately using extremely high limits before understanding the site's linking structure.

---

💬 Feedback & support

Found a bug or have a feature request?

Use the Actor's Issues tab on Apify.

Useful future additions may include:

Redirect-chain detection
Internal-link graph
Pages with very few internal links
Pages with excessive outgoing links
External-domain summaries
Site-wide anchor-text summaries
Sitemap comparison
JavaScript rendering mode
Orphan-page analysis
Historical website audit comparisons

---

🚀 Website link auditing without manually checking every page

Enter your website URL, choose the crawl depth, run the Actor, and receive structured source → target link data ready for SEO audits, broken-link cleanup, migration analysis, APIs, and automation.