Website Sitemap Link Finder & Robots Reference Audit

Sitemap Extractor

apify/sitemap-extractor

This Apify Actor extracts all URLs from a website's sitemaps and checks their status codes via lightweight HTTP requests. It provides a clean list of valid links, acting as an ideal pre-processor to ensure your larger crawling projects target only active URLs.

Robots.txt & Sitemap Analyzer

automation-lab/robots-sitemap-analyzer

This actor fetches and parses robots.txt and sitemap.xml files for any list of websites. It extracts crawl directives (user-agent rules, allowed/disallowed paths, crawl-delay), discovers sitemap URLs, and counts the number of pages listed in each sitemap. Use it for SEO audits, competitive...

Pagination Link Checker: rel=next and rel=prev

practical_ai_lab_jp/pal-pagination-link-checker

Inspect rel=next and rel=prev navigation tags. Static HTML and JSON-LD only; public HTTPS URLs.

Social Media Profile Link Finder for Websites

practical_ai_lab_jp/pal-social-link-inventory

Inventory official social profile domains linked from a webpage. Static HTML and JSON-LD only; public HTTPS URLs.

PAL | Internal Link & URL Inventory

practical_ai_lab_jp/pal-internal-link-auditor

Export internal navigation links and duplicate or fragment patterns from public HTML pages. Try the Example task before checking your pages. This is not a full-site crawler.

HTML Lang Attribute Checker for Website Accessibility

practical_ai_lab_jp/pal-html-language-checker

Detect missing or malformed html lang attributes. Static HTML and JSON-LD only; public HTTPS URLs.

Mobile Viewport Tag Checker for Website SEO

practical_ai_lab_jp/pal-mobile-viewport-checker

Check responsive viewport meta declaration and common mistakes. Static HTML and JSON-LD only; public HTTPS URLs.

Website Privacy Policy Link Finder & Checker

practical_ai_lab_jp/pal-privacy-policy-presence

Locate visible privacy and personal-data policy links on storefront pages. Static HTML and JSON-LD only; public HTTPS URLs.

Duplicate Meta Tags Checker: Titles & Canonicals

practical_ai_lab_jp/pal-duplicate-meta-detector

Detect repeated title, canonical, description and robots tags on a page. Static HTML and JSON-LD only; public HTTPS URLs.

Structured Data Type Finder: Schema.org JSON-LD

practical_ai_lab_jp/pal-schema-type-inventory

Inventory schema.org JSON-LD types on any public web page. Static HTML and JSON-LD only; public HTTPS URLs.