Seo Broken Internal Links

SEO Broken Internal Links Checker

Checks a small set of internal anchor links on one user-supplied public HTML page, for independent site owners, content editors, and SEO auditors. This Actor does one job: test a bounded sample of same-host internal links for HTTP errors.

What it does

Fetches one supplied page, reads <a href> URLs, resolves relative URLs (and same-page <base href>), removes fragments, deduplicates targets, and tests up to maxLinks links. Each target is checked with HEAD, falling back to GET when the server responds 403, 405, or 501. Robots.txt is respected for the source page and each tested target. Links outside the original hostname, including www variations, are not followed.

Input example

{"url": "https://www.python.org/", "maxLinks": 5}

url is required. maxLinks is a whole number from 1–30, default 10. The Console prefill provides one URL. No sitemap, external domain crawling, login, search pages, browser rendering, or search-engine results are used.

Output example (illustrative)

{
  "sourceUrl": "https://example.org/guide",
  "targetUrl": "https://example.org/missing",
  "httpStatus": 404,
  "broken": true,
  "status": "broken",
  "error": null
}

Fields are always present: sourceUrl (string), targetUrl (string or null), httpStatus (integer or null, final HTTP status after permitted redirects), broken (true / false / null), status (ok, broken, unverified, no_links, page_error), error (plain-language string or null). HTTP >=400 is flagged broken; 2xx and other 3xx status codes are treated as reachable, while failures/robots-blocked targets are unverified with broken=null rather than falsely marked broken. This is HTTP-level availability, not proof that content is correct.

One record is emitted per distinct sampled internal target, in page order. A source that cannot be fetched produces one page_error record; a source with no eligible links produces one no_links record, both with targetUrl=null.

Execution and dataset storage

• Apify cloud: APIFY_TOKEN, ACTOR_DEFAULT_KEY_VALUE_STORE_ID, and ACTOR_DEFAULT_DATASET_ID are used to read input and append output over the Apify REST API; no SDK or external Python dependencies.
• Locally: set ACTOR_INPUT_JSON or create storage/key_value_stores/default/INPUT.json, then run python src/main.py. Output is numbered JSON files under storage/datasets/default/ (or APIFY_LOCAL_STORAGE_DIR/datasets/default/).
• Docker: docker build -t seo-broken-internal-links . and docker run --rm -e ACTOR_INPUT_JSON='{"url":"https://www.python.org/","maxLinks":5}' seo-broken-internal-links. Volume-mount storage if local results must persist.

Limits and safeguards

• Exactly one source page, up to 30 distinct internal targets, each HTTP request capped at 1 MB and timed out at 8 seconds. One target may involve HEAD plus GET fallback. robots.txt fetches capped at 250 KB, cached per origin. Max 3 same-host redirects; HTTPS-to-HTTP redirects are refused.
• Internal means same exact hostname as supplied input; cross-host redirects are not followed. Query strings are preserved, fragments removed. Non-HTTP schemes and fragment-only links are skipped. Only links on the input page are checked; this is not a site-wide crawler.
• Public HTTP(S) only; private/local/reserved IPs refused by default. DNS pre-check does not fully eliminate DNS-rebinding risks. robots.txt uses stdlib urllib.robotparser with incomplete support for modern directives. 404/410 robots.txt is treated as absent; other robots errors fail closed (401/403 disallow). No browser-rendered links, JavaScript, CSS, authenticated content, or server-side hidden link discovery.
• 403, 429, and 5xx can be transient or access-policy effects; a flagged link is an HTTP result, not a guarantee the destination is permanently gone. HEAD/GET differences may occur.
• ACTOR_TEST_ALLOW_LOOPBACK=1 is only for the localhost test fixture; do not enable it on Apify.

Declared category recommendations (not Store-configured)

SEO tools · Website monitoring · Developer tools (3 categories).

Suggested pricing (not configured)

Pay per run: USD $0.02 per completed run (up to 30 tested links). Proposed pricing only; nothing has been published or billed.

Reproducible verification

python tests/run_smoke.py
python -m unittest discover -s tests -v

Both scripts launch python src/main.py as a separate process, passing a valid ACTOR_INPUT_JSON and using a local HTTP test fixture with the test-only loopback environment flag. Smoke output, input, and produced dataset item(s) are saved to tests/evidence/. No import-only verification is used.