Pricing

from $10.00 / 1,000 results

Go to Apify Store

Robots Txt Parser

Try for free

Robots Txt Parser - Parse and extract structured data from various formats. Clean, reliable output.

Pricing

from $10.00 / 1,000 results

Rating

0.0

(0)

Developer

Donny

Actor stats

Bookmarked

Total users

Monthly active users

a day ago

Last modified

Parse Robots Txt — Data, Details & Metadata

tropical_quince/robots-txt-parser

Parse robots txt data at scale with this powerful Apify actor. Extracts data, details & metadata with automatic pagination and proxy rotation. Perfect for market research, competitive intelligence, and data-driven decision making.

Donny Nguyen

Scrape Bbb Business — Data, Details & Metadata

tropical_quince/bbb-business-scraper

Scrape bbb business data at scale with this powerful Apify actor. Extracts data, details & metadata with automatic pagination and proxy rotation. Perfect for market research, competitive intelligence, and data-driven decision making.

Donny Nguyen

Website Metadata Extractor(sitemap, socialLinks, robotsTxt)

codescraper/website-metadata-extractor

A very fast metadata extractor to get all meta tags, robots.txt, sitemaps, social links, H1s, word count, and JSON-LD data. Also provides technology detection for a full analysis. Get your data fast for just $3/month.

CodeScraper

5.0

(1)

Robots Txt Audit

apage/robots-txt-audit

Audit robots.txt files for AI crawler access. Get an AI Readiness Score (0-100), analyze 61+ AI crawlers (ChatGPT, Claude, Perplexity, Gemini), detect syntax errors, security concerns, and get actionable recommendations. Batch audit multiple domains at once with optional subdomain scanning.

Andy Page

Sitemap Audit

apage/sitemap-audit

Get a Sitemap Health Score (0-100) for any website. Discover, parse, and validate XML sitemaps. Find 404s, redirects, canonical mismatches, noindex conflicts, hreflang issues, missing pages and estimate crawl budget waste.

Andy Page

Extract Sitemap Parser Url — URLs, Structure & Metadata

tropical_quince/sitemap-parser-url-extractor

Extract sitemap parser url data at scale with this powerful Apify actor. Extracts urls, structure & metadata with automatic pagination and proxy rotation. Perfect for market research, competitive intelligence, and data-driven decision making.

Donny Nguyen

Levels Fyi Salary Parser

serebryakov7/levels-fyi-salary-parser

Ivan Serebryakov

UK Veterinarians Scraper - RCVS Directory

giovannibiancia/rcvs-actor

Extract UK veterinarians and veterinary practices from the official RCVS Register. Get names, qualifications, practice addresses, phone numbers, and registration status. 32,000+ verified veterinary professionals.