Pricing

from $12.00 / 1,000 site auditeds

Try for free

Go to Apify Store

AI Readiness Audit · llms.txt, AI Crawlers, Structured Data

Try for free

Audit any website for how well AI search engines and agents can use it: AI crawler access in robots.txt, llms.txt validity, structured data, content readable without JavaScript, Markdown delivery and agent discovery files. Score 0-100 with a fix list, 1,000 sites per run. First audit a month free.

Pricing

from $12.00 / 1,000 site auditeds

Rating

0.0

(0)

Developer

AgentsGearbox

Actor stats

Bookmarked

Total users

Monthly active users

an hour ago

Last modified

Categories

SEO tools

Developer tools

You can access the AI Readiness Audit · llms.txt, AI Crawlers, Structured Data programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1from apify_client import ApifyClient
2
3# Initialize the ApifyClient with your Apify API token
4# Replace '<YOUR_API_TOKEN>' with your token.
5client = ApifyClient("<YOUR_API_TOKEN>")
6
7# Prepare the Actor input
8run_input = { "urls": [
9        "https://agentsgearbox.com",
10        "https://docs.apify.com",
11        "https://llmstxt.org",
12    ] }
13
14# Run the Actor and wait for it to finish
15run = client.actor("agentsgearbox/ai-readiness-audit").call(run_input=run_input)
16
17# Fetch and print Actor results from the run's dataset (if there are any)
18print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
19for item in client.dataset(run.default_dataset_id).iterate_items():
20    print(item)
21
22# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

AI Readiness Audit: llms.txt, AI Crawlers, Structured Data API in Python

The Apify API client for Python is the official library that allows you to use AI Readiness Audit · llms.txt, AI Crawlers, Structured Data API in Python, providing convenience functions and automatic retries on errors.

Install the apify-client

$pip install apify-client

Other API clients include:

AI Readiness Audit · llms.txt, AI Crawlers, Structured Data API in JavaScript

AI Readiness Audit · llms.txt, AI Crawlers, Structured Data API through CLI

AI Readiness Audit · llms.txt, AI Crawlers, Structured Data OpenAPI definition

AI Readiness Audit · llms.txt, AI Crawlers, Structured Data API

Twitter (X.com) Scraper Unlimited: No Limits

apidojo/twitter-scraper-lite

Introducing Twitter Scraper Unlimited, the most comprehensive Twitter data extraction solution available. Our enterprise-grade scraper offers unmatched capabilities with a transparent event-based pricing model, making it perfect for both small-scale and large-scale data extraction needs.

API Dojo

36K

4.8

(100)

Tweet Scraper V2 - X Scraper / Twitter Scraper / X.com Scraper

apidojo/tweet-scraper

Lightning-fast search, URL, list, and profile scraping, with customizable filters. At $0.40 per 1000 tweets, and 30-80 tweets per second, it is ideal for researchers, entrepreneurs, and businesses! Get comprehensive insights from Twitter (X) now!

API Dojo

107K

3.9

(196)

Youtube Scraper (Pay Per Result)

apidojo/youtube-scraper

Experience unparalleled efficiency and depth with its ultra-fast capabilities in searching, URL, playlist, channel, and profile scraping. It comes equipped with a variety of filters to customize your data collection precisely. Plus, it's incredibly affordable at only $0.50 per 1000 videos!

API Dojo

31K

3.3

(21)

Facebook Search Scraper — People, Pages, Posts & More

memo23/facebook-search-scraper

The most complete Facebook search: people, pages, places, events, posts, videos and top — in one actor, no login and no cookies. Get names, URLs, profile pictures, verified badges, event dates, and post text with authors. Auto-paginates across results. Export to JSON, CSV or Excel.

Muhamed Didovic

796

5.0

(2)

TikTok Scraper

clockworks/tiktok-scraper

Extract data from TikTok videos, hashtags, and users. Use URLs or search queries to scrape TikTok profiles, hashtags, posts, URLs, shares, followers, hearts, names, video, and music-related data. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Clockworks

341K

4.7

(370)

Instagram Scraper

apify/instagram-scraper

Extract Instagram posts, reels, profiles, places, hashtags, carousels, and comments. Get data from Instagram using one or more Instagram URLs or search queries: content, context, metrics, metadata. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Apify

435K

4.7

(602)

YouTube Scraper

streamers/youtube-scraper

YouTube crawler and video scraper. Alternative YouTube API with no limits or quotas. Extract and download channel name, likes, number of views, and number of subscribers.

Streamers

169K

4.8

(196)

Search Threads By Keywords

watcher.data/search-threads-by-keywords

Fast and reliable scraper for searching posts on Threads by Meta. Perfect for social media monitoring, research, and business intelligence.

Giichi Arai

3.8K

3.4

(10)

Facebook Search Scraper

silentflow/facebook-search-scraper

Search Facebook by keyword and export posts, pages, places, events and people as clean structured data. Get text, author, reactions breakdown, comment and share counts, images, videos, links and timestamps. No login. Six search types in one actor. For lead generation, social listening, research.

SilentFlow

347

TikTok Search Scraper: Keywords, Hashtags, Creators

xmolodtsov/tiktok-search-scraper

Scrape TikTok search results by keyword or hashtag and discover creators. Pay only per result. Views, likes, comments, shares, bookmarks, channel stats, music, subtitles and direct video URLs as JSON. Output compatible with popular TikTok scrapers. No proxies or login needed. Works via API and MCP.