Pricing

from $1.50 / 1,000 policy observations

Try for free

Go to Apify Store

Robots.txt Policy Evidence API

Try for free

Extract public robots.txt directives, policy hashes and declared sitemap URLs. Compare policy changes and inspect explicitly labelled Python parser evaluations.

Pricing

from $1.50 / 1,000 policy observations

Rating

0.0

(0)

Developer

Salih Can Kurnaz

Actor stats

Bookmarked

Total users

Monthly active users

4 days ago

Last modified

Configure MCP server with Robots.txt Policy Evidence API

The configuration above connects to the hosted Apify MCP server with the Robots.txt Policy Evidence API Actor already loaded. It uses OAuth, so your client opens a browser window to sign you in to Apify on first use. No API token is needed. Clients without OAuth support can authenticate with an Authorization: Bearer header carrying an API token from API & Integrations in Apify Console.

Get a ready-to-use configuration for your MCP client with the Robots.txt Policy Evidence API Actor preconfigured at https://mcp.apify.com/?tools=fetch-actor-details,soilair/robots-txt-policy-evidence-api.

You can connect to the Apify MCP Server using clients like Tester MCP Client, or any other MCP client of your choice.

If you want to learn more about our Apify MCP implementation, check out our MCP documentation. To learn more about the Model Context Protocol in general, refer to the official MCP documentation or read our blog post.

Website Freshness Detector - Last Update Signals + Stale Flag

santamaria-automations/website-freshness-detector

Probe any website for signals of when it was last updated: sitemap lastmod, Last-Modified header, footer copyright year, and RSS/Atom feed dates. Returns a stale_flag verdict (active / slow / stale / dead) so agencies can prioritize rebuild pitches. Pay-per-result.

NanoScrape

Robots.txt Analyzer

mahogany_songbird/robots-txt-analyzer

Read robots.txt disallow rules and sitemap declarations.

Britton Furness

6sense Pixel Checker

tonic_yaffle/6sense-pixel-checker

Checks if 6sense pixel is installed on a list of websites.

Mudit Korpaul

instagram-followers-scraper

nemxxo/instagram-followers-scraper

nemxxo

Web Drift Detector – Website Change Monitoring & Content Diff

muhammad-bilal/web-drift-detector

Detect website changes automatically. Monitor pricing, content, policies, and competitors using fast browserless web change detection. Structured diffs, severity scoring, historical snapshots, and webhook alerts. Ideal for compliance, SaaS, ecommerce, and monitoring workflows.

Muhammad Bilal

5.0

(2)

Website Health Checker

literal_jacktree/website-health-checker

Check website health: status codes, SSL certificates, response times, and redirect chains. Perfect for monitoring, SEO audits, and uptime verification.

Janice

HTTP Status Code Checker & Bulk URL Checker API

pink_comic/http-status-code-checker

Bulk URL checker and HTTP status scraper for SEO audits. Check status codes, 301/302 redirect chains, response times, content types, and broken URLs. Use the API for website status checks, migrations, monitoring workflows, and link QA.

Ava Torres

Sitemap URL Extractor & Monitor — Recursive

hipersoft/sitemap-extractor

Extract every URL from any website's sitemaps: loc, lastmod, changefreq and priority. Follows sitemap indexes recursively, discovers sitemaps from robots.txt, handles gzipped sitemaps, and can return only URLs changed since a date. No setup.

hiper soft

Sitemap URL Delta Monitor

stellar_ballet_0bu/sitemap-delta-monitor

Monitor public sitemap.xml files for new URLs, removed URLs, lastmod changes, and clean stateful deltas for SEO, RAG indexing, and content operations.