Pricing

from $1.50 / 1,000 policy observations

Try for free

Go to Apify Store

Robots.txt Policy Evidence API

Try for free

Extract public robots.txt directives, policy hashes and declared sitemap URLs. Compare policy changes and inspect explicitly labelled Python parser evaluations.

Pricing

from $1.50 / 1,000 policy observations

Rating

0.0

(0)

Developer

Salih Can Kurnaz

Actor stats

Bookmarked

Total users

Monthly active users

4 days ago

Last modified

Robots.txt Policy Evidence API API through CLI

The Apify CLI is the official tool that allows you to use Robots.txt Policy Evidence API locally, providing convenience functions and automatic retries on errors.

Install Apify CLI

Using installation script (macOS/Linux):

$curl -fsSL https://apify.com/install-cli.sh | bash

Using installation script (Windows):

$irm https://apify.com/install-cli.ps1 | iex

Using Homebrew:

$brew install apify-cli

Using npm:

$npm install -g apify-cli

Other API clients include:

Robots.txt Policy Evidence API API in Python

Robots.txt Policy Evidence API API in JavaScript

Robots.txt Policy Evidence API OpenAPI definition

Robots.txt Policy Evidence API API

Website Freshness Detector - Last Update Signals + Stale Flag

santamaria-automations/website-freshness-detector

Probe any website for signals of when it was last updated: sitemap lastmod, Last-Modified header, footer copyright year, and RSS/Atom feed dates. Returns a stale_flag verdict (active / slow / stale / dead) so agencies can prioritize rebuild pitches. Pay-per-result.

NanoScrape

Robots.txt Analyzer

mahogany_songbird/robots-txt-analyzer

Read robots.txt disallow rules and sitemap declarations.

Britton Furness

6sense Pixel Checker

tonic_yaffle/6sense-pixel-checker

Checks if 6sense pixel is installed on a list of websites.

Mudit Korpaul

instagram-followers-scraper

nemxxo/instagram-followers-scraper

nemxxo

Web Drift Detector – Website Change Monitoring & Content Diff

muhammad-bilal/web-drift-detector

Detect website changes automatically. Monitor pricing, content, policies, and competitors using fast browserless web change detection. Structured diffs, severity scoring, historical snapshots, and webhook alerts. Ideal for compliance, SaaS, ecommerce, and monitoring workflows.

Muhammad Bilal

5.0

(2)

Website Health Checker

literal_jacktree/website-health-checker

Check website health: status codes, SSL certificates, response times, and redirect chains. Perfect for monitoring, SEO audits, and uptime verification.

Janice

HTTP Status Code Checker & Bulk URL Checker API

pink_comic/http-status-code-checker

Bulk URL checker and HTTP status scraper for SEO audits. Check status codes, 301/302 redirect chains, response times, content types, and broken URLs. Use the API for website status checks, migrations, monitoring workflows, and link QA.

Ava Torres

Sitemap URL Extractor & Monitor — Recursive

hipersoft/sitemap-extractor

Extract every URL from any website's sitemaps: loc, lastmod, changefreq and priority. Follows sitemap indexes recursively, discovers sitemaps from robots.txt, handles gzipped sitemaps, and can return only URLs changed since a date. No setup.

hiper soft

Sitemap URL Delta Monitor

stellar_ballet_0bu/sitemap-delta-monitor

Monitor public sitemap.xml files for new URLs, removed URLs, lastmod changes, and clean stateful deltas for SEO, RAG indexing, and content operations.