Pricing

from $1.00 / 1,000 results

robots.txt Parser & URL Tester

Fetch and parse robots.txt for any site: user-agent rules, crawl-delay, and declared sitemaps. Optionally test whether specific URLs are allowed for a given user-agent, using correct longest-match rules.

Pricing

from $1.00 / 1,000 results

Rating

0.0

(0)

Developer

Nicolas van Arkens

Actor stats

Bookmarked

Total users

Monthly active users

3 days ago

Last modified

robots.txt Parser & URL Tester 🤖

Fetch and parse robots.txt for any site and get a clean, structured breakdown — per-user-agent allow/disallow rules, crawl-delay, and every declared sitemap. Optionally test whether specific URLs are allowed or blocked for a chosen crawler, using correct longest-match precedence.

Built for SEO audits, crawler and bot development, compliance checks, and anyone who needs to know what a site permits before crawling it.

Why use it

📋 Structured rules — allow/disallow lists per user-agent, not raw text
🤖 User-agent aware — see the rules that actually apply to Googlebot, bingbot, or *
✅ URL allow/deny testing — check exact paths against the rules with proper * wildcard, $ anchor, and longest-match logic
🐌 Crawl-delay — extracted per user-agent
🗺️ Sitemaps — every sitemap the site declares, ready to feed into a sitemap extractor
🌐 Batch — check many sites at once

Use cases

SEO audits — verify a site isn't accidentally blocking important pages
Crawler development — respect robots.txt correctly before scraping
Compliance — confirm what a site permits for your user-agent
Sitemap discovery — pull declared sitemaps to drive further crawling
Monitoring — track robots.txt changes over time

Input

Field	Description
Sites	List of sites/URLs; robots.txt is fetched at each root.
User-agent	Which crawler's rules to apply (e.g. `Googlebot`, or `*`).
Test paths	Optional paths/URLs to test for allowed/blocked.

Output

{
  "site": "https://example.com",
  "robotsUrl": "https://example.com/robots.txt",
  "success": true,
  "userAgentChecked": "*",
  "sitemaps": ["https://example.com/sitemap.xml"],
  "userAgentsDeclared": ["*", "googlebot", "badbot"],
  "appliedGroupDisallow": ["/private/", "/tmp/"],
  "appliedGroupAllow": ["/private/public-page"],
  "crawlDelay": 10,
  "testResults": [
    { "path": "/private/secret", "allowed": false },
    { "path": "/private/public-page", "allowed": true }
  ]
}

Export to JSON, CSV, or Excel, or pull via the Apify API.

Notes

Implements standard robots.txt semantics: longest-match wins between Allow and Disallow, with * wildcards and $ end-anchors (per Google's specification).
A site with no robots.txt (404) is reported as such — by convention, that means all crawling is allowed.
Independent tool. Always honor robots.txt in your own crawling.

Sitemap Walker Pro — Recursive URL Discovery

jungle_synthesizer/sitemap-url-extractor-pro

Walk sitemaps and sitemap-index trees recursively. robots.txt fallback. Include/exclude globs/regex, lastmodSince filter, priorityMin filter, optional chunked output. Per-URL rows with image / video / news / hreflang metadata. Pure HTTP, no browser.

BowTiedRaccoon

Robots.txt Validator - Check Rules, Sitemaps & Crawl Directives

scrappy_garden/robots-txt-validator

Validate robots.txt for one or more websites: fetches /robots.txt per host, parses directive groups (User-agent/Allow/Disallow/Crawl-delay/Sitemap), reports common errors and warnings, and can test URLs against the chosen User-Agent.

Bikram Adhikari

JSON-LD Schema & Meta Tag Extractor

logiover/json-ld-schema-meta-tag-extractor

Extract JSON-LD/Schema.org structured data, Meta tags, OpenGraph and Twitter Cards from any URL. Get page title + meta description with a clean JSON output for SEO audits, validation, competitor research and AI datasets. Proxy-ready for large crawls.

Logiover

Structured Data Extractor

automation-lab/structured-data-extractor

This actor extracts structured data markup from web pages. It parses all three major formats: JSON-LD (`<script type="application/ld+json">`), Microdata (`itemscope`/`itemprop`), and RDFa (`typeof`/`property`). For each page, it returns the full structured data objects, detected Schema.org...

Stas Persiianenko

McMaster-Carr Category Map Scraper

parseforge/mcmaster-carr-scraper

Explore McMaster-Carr categories with ease. Perfect for researchers, procurement specialists, and analysts, this scraper delivers clean structured category data including names, URLs, and keywords to map catalogs, analyze trends, and plan sourcing.

ParseForge

5.0

(1)

Sitemap Audit

apage/sitemap-audit

Get a Sitemap Health Score (0-100) for any website. Discover, parse, and validate XML sitemaps. Find 404s, redirects, canonical mismatches, noindex conflicts, hreflang issues, missing pages and estimate crawl budget waste.

Andy Page

Google Maps Email Extractor & Lead Scraper

highbrow_fame/google-maps-email-extractor

Google Maps email scraper with built-in MX/SPF/DMARC validation + catch-all detection. 50-70% email hit rate (vs 0-15% English-only). Multilingual crawl (10+ langs: kapcsolat/kontakt/contacto/contatti/...). Lead scoring 0-100, delta mode ($0 dupes). PAY_PER_EVENT: $0.005/lead.

yestrue

LabX Laboratory Equipment Scraper

parseforge/labx-com-scraper

Collect LabX laboratory equipment listings and product page details in a structured format. Use it to review offers, compare sellers, and analyze pricing and product data in your reports, dashboards, or procurement workflows with less manual collection work and more consistent records.

ParseForge

5.0

(1)

Ace Hardware Scraper

parseforge/ace-hardware-scraper

Scrape product listings from AceHardware.com. Extract prices, brands, specifications, inventory status, store availability, and fulfillment options. Filter by category or brand. Fast parallel page fetching with 100 items per page. Export as CSV or JSON.

ParseForge

WhatClinic.com Clinic Scraper

haketa/whatclinic-scraper

Scrape clinic listings from WhatClinic.com — global healthcare directory with 120K+ clinics across 160+ countries. Extract name, reviews, prices, treatments, service scores and contact info by category, country and city.