Pricing

from $4.20 / 1,000 url checkeds

Try for free

Go to Apify Store

URL Access Checker - robots.txt, Bot Protection & Scrapability

Try for free

Before your AI agent fetches a URL: may it (robots.txt per crawler, RFC 9309) and can it (Cloudflare, DataDome, PerimeterX, AWS WAF, login, JavaScript)? One row per URL with a verdict, safeToFetch true/false and a one-sentence recommendation the agent can follow. It only looks, never bypasses.

Pricing

from $4.20 / 1,000 url checkeds

Rating

0.0

(0)

Developer

NeverEmpty

Actor stats

Bookmarked

Total users

Monthly active users

2 days ago

Last modified

Categories

Developer tools

Automation

You can access the URL Access Checker - robots.txt, Bot Protection & Scrapability programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1from apify_client import ApifyClient
2
3# Initialize the ApifyClient with your Apify API token
4# Replace '<YOUR_API_TOKEN>' with your token.
5client = ApifyClient("<YOUR_API_TOKEN>")
6
7# Prepare the Actor input
8run_input = {
9    "urls": [
10        "https://en.wikipedia.org/wiki/Robots.txt",
11        "https://www.instagram.com/nasa/",
12        "https://github.com/settings/profile",
13    ],
14    "userAgents": ["*"],
15}
16
17# Run the Actor and wait for it to finish
18run = client.actor("neverempty/url-access-checker").call(run_input=run_input)
19
20# Fetch and print Actor results from the run's dataset (if there are any)
21print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
22for item in client.dataset(run.default_dataset_id).iterate_items():
23    print(item)
24
25# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

robots.txt Checker for AI Agents: Can I Scrape This URL? API in Python

The Apify API client for Python is the official library that allows you to use URL Access Checker - robots.txt, Bot Protection & Scrapability API in Python, providing convenience functions and automatic retries on errors.

Install the apify-client

$pip install apify-client

Other API clients include:

URL Access Checker - robots.txt, Bot Protection & Scrapability API in JavaScript

URL Access Checker - robots.txt, Bot Protection & Scrapability API through CLI

URL Access Checker - robots.txt, Bot Protection & Scrapability OpenAPI definition

URL Access Checker - robots.txt, Bot Protection & Scrapability API

YouTube Most Replayed Scraper - Heatmap Peaks & Timestamps

neverempty/youtube-most-replayed-scraper

Find the most replayed moments of any YouTube video: the top peaks of YouTube's own Most replayed heatmap as timestamped links, the chapter each peak falls in and the caption line spoken there, plus the raw heatmap segments. Videos without a heatmap are listed free.

NeverEmpty

Agent URL & SSRF Guard MCP

q_services/agent-url-ssrf-guard-mcp

Validate URLs, DNS targets and redirect chains before AI agents access remote resources.

Q Services

App Store App Details Tracker - Ratings, Star Counts & Versions

neverempty/app-store-app-details-tracker

For ASO teams, app marketers and competitor trackers: exact rating count, count per star (5 to 1), rating, version, update date, price, size and chart rank of App Store apps. Matched the live store page on 5 of 5 apps checked 2026-09-21. Monitoring mode returns only apps that changed.

NeverEmpty

Sitemap & robots.txt URL Discovery (no crawling)

everyotherfriday/url-discovery

Feed it site roots, get back URLs from their sitemaps plus each site's disallow rules and crawl-delay. Streams sitemaps, follows nested indexes and handles gzip without crawling page bodies. For seeding crawlers, SEO audits and migration checks.

Paul Vasquez

Sitemap URL Extractor — robots.txt, Sitemap Index, gzip

chorelet/sitemap-url-extractor

Every URL of a website from its sitemaps: robots.txt discovery, sitemap indexes, gzip, lastmod, changefreq, priority, hreflang alternates, image/video/news extensions. Include/exclude patterns and date filter, CSV/JSON export and API.

Chorelet

URL Health Check

wavy_millpond/url-health-check

Checks HTTP status, response time and TLS validity for a list of URLs. Returns one dataset item per URL.

Westbrook

Data Table Accessibility Evidence Auditor

gifted_wagon/data-table-accessibility-evidence-auditor

Audit static HTML tables for captions, headers, scope, identifiers, presentation roles, and normalized table evidence.

Michael Olmos

YouTube Subscriber Count Scraper - Channel Statistics Monitor

neverempty/youtube-subscriber-count-scraper

Subscribers, total views, video count, join date and country for any YouTube channel from its About panel. Total views and video count are unrounded; the subscriber count is flagged as rounded. Monitoring mode returns a channel only when a count moves, with the change since last run.

NeverEmpty

Website Block Checker: Datacenter vs Real Mobile IP

xavierfok/website-block-checker

Find out whether a website blocks scrapers by IP type. Each URL is loaded from a datacenter IP and from a real Singapore 4G mobile IP, and you get which one was blocked or challenged (Cloudflare, Akamai, PerimeterX, DataDome, AWS WAF, Google, Reddit...).

Xavier Fok

Discord Server Member Count Tracker - Online & Total Members

neverempty/discord-server-member-count-tracker

For community managers and growth teams: total members, online members, boosts, name, tag, icon and creation date for every Discord server you name by its invite link. One measured server: 431,725 members, 45,600 online. Monitoring returns a server only when its member count moves.

NeverEmpty