Pricing

from $4.20 / 1,000 url checkeds

Try for free

Go to Apify Store

URL Access Checker - robots.txt, Bot Protection & Scrapability

Try for free

Before your AI agent fetches a URL: may it (robots.txt per crawler, RFC 9309) and can it (Cloudflare, DataDome, PerimeterX, AWS WAF, login, JavaScript)? One row per URL with a verdict, safeToFetch true/false and a one-sentence recommendation the agent can follow. It only looks, never bypasses.

Pricing

from $4.20 / 1,000 url checkeds

Rating

0.0

(0)

Developer

NeverEmpty

Actor stats

Bookmarked

Total users

Monthly active users

2 days ago

Last modified

Configure MCP server with robots.txt Checker for AI Agents: Can I Scrape This URL?

The configuration above connects to the hosted Apify MCP server with the robots.txt Checker for AI Agents: Can I Scrape This URL? Actor already loaded. It uses OAuth, so your client opens a browser window to sign you in to Apify on first use. No API token is needed. Clients without OAuth support can authenticate with an Authorization: Bearer header carrying an API token from API & Integrations in Apify Console.

Get a ready-to-use configuration for your MCP client with the robots.txt Checker for AI Agents: Can I Scrape This URL? Actor preconfigured at https://mcp.apify.com/?tools=fetch-actor-details,neverempty/url-access-checker.

You can connect to the Apify MCP Server using clients like Tester MCP Client, or any other MCP client of your choice.

If you want to learn more about our Apify MCP implementation, check out our MCP documentation. To learn more about the Model Context Protocol in general, refer to the official MCP documentation or read our blog post.

YouTube Most Replayed Scraper - Heatmap Peaks & Timestamps

neverempty/youtube-most-replayed-scraper

Find the most replayed moments of any YouTube video: the top peaks of YouTube's own Most replayed heatmap as timestamped links, the chapter each peak falls in and the caption line spoken there, plus the raw heatmap segments. Videos without a heatmap are listed free.

NeverEmpty

Agent URL & SSRF Guard MCP

q_services/agent-url-ssrf-guard-mcp

Validate URLs, DNS targets and redirect chains before AI agents access remote resources.

Q Services

App Store App Details Tracker - Ratings, Star Counts & Versions

neverempty/app-store-app-details-tracker

For ASO teams, app marketers and competitor trackers: exact rating count, count per star (5 to 1), rating, version, update date, price, size and chart rank of App Store apps. Matched the live store page on 5 of 5 apps checked 2026-09-21. Monitoring mode returns only apps that changed.

NeverEmpty

Sitemap & robots.txt URL Discovery (no crawling)

everyotherfriday/url-discovery

Feed it site roots, get back URLs from their sitemaps plus each site's disallow rules and crawl-delay. Streams sitemaps, follows nested indexes and handles gzip without crawling page bodies. For seeding crawlers, SEO audits and migration checks.

Paul Vasquez

Sitemap URL Extractor — robots.txt, Sitemap Index, gzip

chorelet/sitemap-url-extractor

Every URL of a website from its sitemaps: robots.txt discovery, sitemap indexes, gzip, lastmod, changefreq, priority, hreflang alternates, image/video/news extensions. Include/exclude patterns and date filter, CSV/JSON export and API.

Chorelet

URL Health Check

wavy_millpond/url-health-check

Checks HTTP status, response time and TLS validity for a list of URLs. Returns one dataset item per URL.

Westbrook

Data Table Accessibility Evidence Auditor

gifted_wagon/data-table-accessibility-evidence-auditor

Audit static HTML tables for captions, headers, scope, identifiers, presentation roles, and normalized table evidence.

Michael Olmos

YouTube Subscriber Count Scraper - Channel Statistics Monitor

neverempty/youtube-subscriber-count-scraper

Subscribers, total views, video count, join date and country for any YouTube channel from its About panel. Total views and video count are unrounded; the subscriber count is flagged as rounded. Monitoring mode returns a channel only when a count moves, with the change since last run.

NeverEmpty

Website Block Checker: Datacenter vs Real Mobile IP

xavierfok/website-block-checker

Find out whether a website blocks scrapers by IP type. Each URL is loaded from a datacenter IP and from a real Singapore 4G mobile IP, and you get which one was blocked or challenged (Cloudflare, Akamai, PerimeterX, DataDome, AWS WAF, Google, Reddit...).

Xavier Fok

Discord Server Member Count Tracker - Online & Total Members

neverempty/discord-server-member-count-tracker

For community managers and growth teams: total members, online members, boosts, name, tag, icon and creation date for every Discord server you name by its invite link. One measured server: 431,725 members, 45,600 online. Monitoring returns a server only when its member count moves.

NeverEmpty