Link Extractor: Extract All Links, Internal & External, Bulk

This link extractor reads a list of web pages and returns one row per link: the absolute URL, anchor text, internal or external type, nofollow / sponsored / ugc flags, file extension, domain and the page it was found on. Use it to extract links from a website, build a URL extractor list, audit outbound links, or collect every PDF a site links to. Bulk input, flat output, no browser, no login, no API key.

Optional extras: check the HTTP status of every unique link (with the final URL after redirects), crawl 1-2 levels of the same site, and filter links with regular expressions.

What does this link extractor return

One dataset row per link:

• Where: sourcePage (the page), depth (0 = a page you listed, 1-2 = crawled).
• Link: href (absolute URL, relative links and <base href> resolved), text (anchor text, falling back to aria-label, title or image alt), domain, extension.
• Type: type is internal, external, mailto, tel, anchor, file (pdf, zip, docx, images, media and similar), image or other. isInternal is a plain true/false (null for mailto and tel).
• Rel flags: nofollow, sponsored, ugc and the full rel string.
• Images: isImageLink and imageSrc for links that wrap an image. Optional extra rows for every <img src>.
• Repeats: occurrences (how many times the link appears on the page when duplicates are removed).
• Status check (optional): statusCode, finalUrl, redirectCount, ok, checkError.

A per-page summary with counts by type is saved in the key-value store record SUMMARY (pages[]), failed pages in FAILED, skipped duplicates and caps in NOTES.

How to extract all links from a website page

1. Paste page URLs in Page URLs, one per line (a bare domain works too).
2. Pick Which links: internal, external or both. Optional: regex filters, anchors, image sources.
3. Run it and download the dataset as JSON, CSV or Excel. The Links table view shows the key columns.

How to extract only external links (or only PDFs)

Set Which links to external to get outbound links only. To collect files, set Only links matching (regex) to \.(pdf|docx?|xlsx?|zip)$. To drop noise use Skip links matching such as /(login|logout|cart).

How to check if links are broken

Turn on Check link status. Each unique http(s) link is requested once (HEAD first, GET if the server refuses HEAD) and every row gets statusCode, finalUrl, redirectCount and ok. Filter on ok = false or statusCode >= 400 to find dead links. Links that cannot be reached at all (DNS, timeout) get checkError. Use Max links to check to cap cost.

How to crawl a site for links

Set Crawl depth to 1 or 2 and Max pages. The crawler follows internal HTML links only (same site, www ignored; subdomains optional), fetches at most Parallel page requests pages at a time, and by default honours robots.txt for pages it discovers. Pages you list yourself are always fetched.

Input example

{
  "urls": ["https://www.python.org", "https://www.iana.org/domains/reserved"],
  "linkScope": "both",
  "dedupePerPage": true,
  "excludePatterns": ["/(login|logout)"],
  "checkStatus": false,
  "crawlDepth": 0,
  "maxLinks": 100
}

Output example

{
  "sourcePage": "https://www.python.org/",
  "href": "https://docs.python.org/3/",
  "text": "Documentation",
  "type": "external",
  "isInternal": false,
  "domain": "docs.python.org",
  "extension": null,
  "nofollow": false,
  "sponsored": false,
  "ugc": false,
  "rel": null,
  "isImageLink": false,
  "imageSrc": null,
  "source": "a",
  "occurrences": 1,
  "depth": 0,
  "statusCode": 200,
  "finalUrl": "https://docs.python.org/3/",
  "redirectCount": 0,
  "ok": true,
  "checked": true,
  "checkError": null
}

Without checkStatus the status fields are null and checked is false.

How much does it cost to extract links

Pay per event, no start fee:

• link-extracted: $0.0009 per link row ($0.90 per 1,000 links).
• link-checked: $0.0008 per link whose status was obtained ($0.80 per 1,000), only when Check link status is on. Links that do not answer at all are free.

Plus small Apify platform usage. Pages that fail (404, timeout, not HTML) produce no rows and cost nothing. Use Max links and Max links to check as spending caps; a maximum spending limit on the run also stops it cleanly.

Use cases

• SEO audits: outbound links, nofollow and sponsored usage, broken links.
• Link building and backlink research: who does a page link to?
• Content inventory: every PDF, document or download linked from a site.
• Migration checks: find links to old domains or paths with a regex.
• Lead lists: mailto and tel links from contact pages.

Limits (honest)

• No JavaScript rendering. It reads the raw HTML the server sends. Links added by JavaScript after load (single-page apps, lazy menus) are not seen.
• Only the first 5 MB of each page is parsed. Pages that block normal requests (403, bot walls, CAPTCHAs) are reported in FAILED; there is no bot-protection evasion.
• Crawling is shallow (depth 0-2), same site only, polite and low in concurrency; it is not a full-site spider. Use Max pages to bound it.
• Status checks use HEAD/GET from the cloud: some sites answer differently to bots, so a 403 or 999 may still work in a browser. Literal private addresses (127.0.0.1, 10.x, localhost) are never requested.
• javascript: and data: links are ignored; #anchor links are only returned if you ask for them.

FAQ

Does it execute JavaScript? No. It is fast and cheap because it reads HTML only.

What counts as internal? Same host as the page, ignoring www.. Turn on Treat subdomains as internal to include blog.example.com.

Are duplicates removed? Per page, yes by default (occurrences counts the repeats). Turn Remove duplicates per page off for every occurrence.

Does it respect robots.txt? For pages found by crawling, yes by default. Pages you list yourself are fetched as requested.

Are failed pages charged? No. Only link rows (and, if enabled, status checks that returned an HTTP status).

How many links per run? Up to 200,000 rows (default 10,000), from up to 5,000 listed URLs.

Related tools

• HTTP Status Code Checker: Bulk URL, Website & Redirect Chains for full redirect chains of a URL list.
• SEO Audit Tool & Meta Tag Checker: Crawler, Broken Links for scoring every page of a site.
• Sitemap Parser & Extractor: Sitemap to CSV, URLs & New Pages to list a site's pages from its sitemap.