🌐 Internal Link Architecture Verifier

Runs a bounded same-origin crawl (unioned with your sitemap), builds the internal link graph, and returns one evidenced verdict per reachable page: orphan-from-sitemap status, click depth from root, navigation-only-inlinks, nofollow conflicts, whether the canonical target receives links, inbound/outbound counts and anchor-text concentration. Blocked or unreachable nodes are reported UNKNOWN and never poison the graph verdicts of reachable pages.

Output is one page_architecture_verdict row per result; billing is pay-per-event, the value event being one page architecture check (a $0.02 start fee per run, then $0.05 per page architecture check).

No login, no API key and no CAPTCHA solving are involved: the source is read logged-out with an identified contact User-Agent.

📊 Sample Output

Real rows from run DJeuKnrBTN6Nyf4vq on build 0.1.4 (2026-09-17), the same input as the Quick start below — every value is as the source published it (emails masked, long text shortened):

page_url	final_url	graph_completeness	outcome	input	outcome_reason
https://www.gov.uk		partial: 1 node(s) UNKNOWN (blocked/unreachable) — metrics are lower b	graph_summary	https://www.gov.uk	crawled 1 page(s); 1 UNKNOWN node(s); sitemap listed 0 URL(s)
http://169.254.169.254/latest/meta-data		partial: 1 node(s) UNKNOWN (blocked/unreachable) — metrics are lower b	blocked	http://169.254.169.254/latest/meta-data	robots.txt for http://169.254.169.254 unavailable/forbidden — cannot c
https://www.gov.uk/foreign-travel-advice/france	https://www.gov.uk/foreign-travel-advice/france	partial: 1 node(s) UNKNOWN (blocked/unreachable) — metrics are lower b	answer	https://www.gov.uk/foreign-travel-advice/france

The run finished with the status message: NORMAL: 1 billable verdict(s), 0 blocked (unbilled status), 0 unreachable, 0 unparseable of 1 target(s) | page_architecture_check billable=1 | page_architecture_check charged=1

✅ What you get

Each row is flat JSON with these fields (from the dataset schema and the sample run; a field the source does not publish for a given row is null):

• page_url (string/null) — e.g. https://www.gov.uk
• final_url (string/null) — e.g. https://www.gov.uk/foreign-travel-advice/france
• http_status (integer/string/null) — e.g. 200
• in_sitemap (boolean/null) — e.g. False
• orphan_from_sitemap (boolean/null) — e.g. False
• click_depth (integer/string/null) — e.g. 0
• inbound_internal (integer/string/null) — e.g. 0
• outbound_internal (integer/string/null) — e.g. 0
• nav_only_inlinks (boolean/null) — e.g. False
• nofollow_conflict (boolean/null) — e.g. False
• canonical_url (string/null) — e.g. https://www.gov.uk/foreign-travel-advice/france
• canonical_receives_links (boolean/null) — e.g. False
• top_anchor_text (string/null) — null in every sample row
• anchor_text_concentration (string/null) — null in every sample row
• graph_completeness (string/null) — e.g. partial: 1 node(s) UNKNOWN (blocked/unreachable) — metrics are lower bounds over
• faults (string/null) — e.g. canonical target receives no internal links
• record_type (string) — e.g. page_architecture_verdict
• source (string/null) — e.g. buyer-submitted site root / sitemap URLs (bounded same-origin crawl)
• outcome (string/null) — e.g. graph_summary
• input (string/null) — e.g. https://www.gov.uk
• outcome_reason (string/null) — e.g. crawled 1 page(s); 1 UNKNOWN node(s); sitemap listed 0 URL(s)
• observed_at (string/null) — null in every sample row
• terminal (string/null) — null in every sample row

Output

One row per target. A complete evidenced verdict carries outcome: answer; blocked / unreachable / unparseable targets carry that status and are not billed.

⚙️ Sample inputs

1. Quick start — the Store example (this is what the sample above came from)

{
  "urls": [
    "https://www.gov.uk/foreign-travel-advice/france",
    "http://169.254.169.254/latest/meta-data"
  ],
  "maxPages": 1,
  "maxDepth": 1,
  "userAgent": "Mozilla/5.0 (compatible; NexGenWatchBot/1.0; +https://apify.com/nexgenwatch)"
}

The sample run charged exactly: 1 × $0.02 apify-actor-start + 1 × $0.05 page_architecture_check = $0.07 on the Free tier — every delivered row was billed.

2. A smaller, narrowed run

{
  "urls": [
    "https://www.gov.uk/foreign-travel-advice/france",
    "http://169.254.169.254/latest/meta-data"
  ],
  "maxPages": 1,
  "maxDepth": 1,
  "userAgent": "Mozilla/5.0 (compatible; NexGenWatchBot/1.0; +https://apify.com/nexgenwatch)"
}

A bounded run for checking the field shape before scheduling. userAgent narrows what the source is asked for.

3. A full-size run

{
  "urls": [
    "https://www.gov.uk/foreign-travel-advice/france",
    "http://169.254.169.254/latest/meta-data"
  ],
  "maxPages": 1,
  "maxDepth": 1,
  "userAgent": "Mozilla/5.0 (compatible; NexGenWatchBot/1.0; +https://apify.com/nexgenwatch)"
}

The same input run on a schedule; each run is billed only for what it delivers.

🧾 JSON sample record

One real record from run DJeuKnrBTN6Nyf4vq, exactly as it lands in the dataset (emails masked, long text shortened):

{
  "page_url": "https://www.gov.uk",
  "final_url": null,
  "http_status": null,
  "in_sitemap": null,
  "orphan_from_sitemap": null,
  "click_depth": null,
  "inbound_internal": null,
  "outbound_internal": null,
  "nav_only_inlinks": null,
  "nofollow_conflict": null,
  "canonical_url": null,
  "canonical_receives_links": null,
  "top_anchor_text": null,
  "anchor_text_concentration": null,
  "graph_completeness": "partial: 1 node(s) UNKNOWN (blocked/unreachable) — metrics are lower bounds over the 1 reachable page(s)",
  "faults": null,
  "record_type": "page_architecture_verdict",
  "source": "buyer-submitted site root / sitemap URLs (bounded same-origin crawl)",
  "outcome": "graph_summary",
  "input": "https://www.gov.uk",
  "outcome_reason": "crawled 1 page(s); 1 UNKNOWN node(s); sitemap listed 0 URL(s)"
}

🔧 How it works

Transport. Plain HTTPS from the Apify platform, no proxy. robots.txt is read first and a disallowed path is never fetched. Every request carries an identified contact User-Agent.

Terminal states. A run ends NORMAL, CAPPED (your cap was reached), PARTIAL (something was withheld and the message says what), GENUINE_EMPTY (the source was read and truly had nothing in scope) or BLOCKED (the source refused or changed shape — the run FAILS loud and bills nothing). A zero-row run is never reported as a silent success.

Charging. Each page architecture check is charged at the moment it is pushed (page_architecture_check); a row that fails to charge is not delivered, so the dataset count always equals the charged count.

What you submit

urls — site root or sitemap urls. A site root URL and/or sitemap URL(s). The actor runs a BOUNDED same-origin crawl (page + depth caps) unioned with the sitemap URLs, builds the internal link graph, and returns one evidenced verdict per reachable page: orphan-from-sitemap status, click depth from root, navigation-only-inlinks flag, nofollow conflicts, whether the canonical target receives internal links, inbound/outbound internal-link counts, and inbound anchor-text concentration. A blocked or unreachable node is reported UNKNOWN, never bills, and never poisons the graph verdicts of reachable pages (the graph is labeled partial and metrics are lower bounds over the reachable subgraph).

The runtime source gate (per submitted target)

Because you choose the targets, the source contract is enforced at run time, per origin, before any page is read:

• robots.txt is fetched once per origin and honored for our crawler. No robots / 404 = permitted; a disallowed path or an unavailable robots file = BLOCKED.
• SSRF guard — every host is resolved and must be a public address. Private, loopback, link-local, reserved and cloud-metadata addresses are refused before a socket opens.
• HTTP — bounded body reads; 403 / 429 / 5xx = BLOCKED; DNS / timeout / connection faults = UNREACHABLE.

A BLOCKED or UNREACHABLE target is delivered as an unbilled status row — never a broken-site verdict, and never charged.

Why this and not a generic crawler

site-level internal-link-GRAPH verdicts (orphans, click depth, nav-only inlinks, nofollow conflicts, anchor concentration) — not a one-page link array (the 117-user store comparator), and a different job from our own sitemap-indexability-auditor (per-URL indexability) and website-migration-link-auditor (old->new redirect acceptance).

What is not done. No login, no cookie or CAPTCHA bypass, no private or personal-account data, no browser automation.

💰 Pricing example

Event	Free	Bronze	Silver	Gold
Actor Start (apify-actor-start)	$0.02	$0.02	$0.02	$0.02
Page Architecture Check (page_architecture_check)	$0.05	$0.04	$0.04	$0.03

Worked at the live Free-tier price:

• 3 page architecture checks: $0.02 start + 3 × $0.05 = $0.17
• 25 page architecture checks: $0.02 start + 25 × $0.05 = $1.27
• 100 page architecture checks: $0.02 start + 100 × $0.05 = $5.02

A run that delivers zero rows charges the $0.02 start fee only. A BLOCKED run (source refused) fails loud and charges no value event. The start fee is charged once per GB of run memory; the default run memory is 1024 MB.

Yield on the sample run: NORMAL: 1 billable verdict(s), 0 blocked (unbilled status), 0 unreachable, 0 unparseable of 1 target(s) | page_architecture_check billable=1 | page_architecture_check charged=1.

⚖️ Legal & ToS

This actor reads public data only. It collects only what the source publishes to any visitor, keeps to the source's robots rules (checked on every run), identifies itself with a contact User-Agent, and does not access accounts, private data or anything behind authentication. Use the output in line with the source's terms and your local law; the intended use is B2B research and monitoring.

❓ FAQ

Q: Do I need an API key or a login?
A: No. the input schema has no key field and the actor carries no secrets.

Q: Why did my run return 0 rows?
A: Read the run's status message. GENUINE_EMPTY means the source was read and had nothing in scope for your input; BLOCKED means the source refused and the run failed without billing a value event — retry later or narrow the input. A zero-row run bills the start fee only.

Q: How fresh is the data?
A: Every run reads the source live at run time; nothing is cached between runs. Put it on a schedule for a continuous feed.

Q: What formats can I export?
A: The dataset downloads as JSON, CSV, Excel, XML or RSS from the run's Dataset tab or the Apify API, and any run can push to a webhook or integration.

Q: How is this different from the other web QA and enrichment tools actors?
A: Same output shape and billing model; this one covers Internal Link Architecture Verifier. The siblings under Related Actors cover the other sources or slices — run several on one schedule for a combined feed.

Q: Are there rate limits?
A: The actor paces itself against the source and honours its robots rules; there is no per-buyer limit beyond your Apify plan's concurrency.

🆘 Troubleshooting

• Run FAILED with BLOCKED → the source refused the request or changed its page shape → nothing was billed beyond the start fee; retry after a while, and if it persists open an Issue with the run id.
• Fewer rows than expected → the source had fewer items in scope → widen the input.
• Input validation error on start → a field is outside the schema's allowed values → start from the Quick start block and change one field at a time.
• Run TIMED-OUT → a very wide request on a slow day → raise the run timeout in Run options or narrow the input; what was delivered before the timeout is still in the dataset.

🔗 Related Actors

• Contact Evidence Enricher — Give it a list of company domains; it returns one CRM-ready record per domain describing
• Crossref DOI Metadata QA — A per-DOI QA verdict from official Crossref metadata — each DOI returns valid, not_found, or metadata_incomplete with every missing required field na…
• HTML to PDF Evidence Renderer — Renders each HTML document or permitted public URL to a hashed, print-pinned PDF evidence artifact: page count, resource-failure ledger, input+output…
• Marketing Stack Enricher — Give it a list of company domains; it returns one CRM-ready record per domain describing
• PDF Text Intake Auditor — Judges whether a PDF is ingestible before you spend on extraction: text-layer presence, page count, encryption, scan likelihood, per-page text densit…
• Responsive Visual QA Matrix — Runs a headless-Chromium responsive QA pass on each URL across desktop/tablet/mobile and returns one verdict per URL: per-viewport layout metrics, a…
• 🏢 About NexGenData — NexGen Watch is NexGenData's fleet of 256 public monitoring and lookup actors built on official sources, pay-per-result. Browse the catalog at apify.com/nexgenwatch.

⭐ Found this useful?

If this actor saved you a manual check, a quick review on the Apify Store helps other teams find it. Feature request or a source that changed? Open it from the Issues tab — every one is read.