Pricing

from $3.00 / 1,000 audited pages

Try for free

Go to Apify Store

Website SEO Audit Crawler – Broken Links, Schema & llms.txt

Try for free

Website SEO audit crawler: on-page issues (title, meta description, H1, canonical, hreflang, noindex), broken internal and external links, schema.org JSON-LD errors, sitemap and robots.txt checks, AI-crawler rules (GPTBot, ClaudeBot…) and llms.txt. One row per page plus a site summary.

Pricing

from $3.00 / 1,000 audited pages

Rating

0.0

(0)

Developer

Enes Bulanık

Actor stats

Bookmarked

Total users

Monthly active users

2 days ago

Last modified

Categories

SEO tools

Developer tools

Websites to audit

startUrls

Optional

Homepage or any start URL of each website. example.com, https://www.example.com/ and https://example.com/blog/ all work. Each URL is audited as a separate site (same host only).

Type:array

Default:

[
  "https://crawlee.dev"
]

Example:

[
  "https://crawlee.dev",
  "https://www.python.org"
]

Max pages per site

maxPagesPerSite

Optional

Maximum number of HTML pages to crawl and audit per website. You pay per audited page, so this also caps your cost.

Type:integer

Minimum:1

Maximum:10000

Default:50

Example:

Check external links

checkExternalLinks

Optional

Check outgoing links to other domains for broken (4xx/5xx, dead domain), redirected, blocked and timed-out targets. Each checked link is a separate 'external-link' event.

Type:boolean

Default:true

Example:

true

Max external links to check per site

maxExternalLinksPerSite

Optional

Upper limit of unique external links checked per website (in the order they were found). Set to 0 to skip the check.

Type:integer

Minimum:0

Maximum:5000

Default:100

Example:

100

Use XML sitemap

useSitemap

Optional

Read the sitemap(s) listed in robots.txt (or /sitemap.xml). Sitemap URLs are queued right after the links on the start page, and every page is flagged when it is in the sitemap but returns an error, is noindex or canonicalised.

Type:boolean

Default:true

Example:

true

Respect robots.txt

respectRobotsTxt

Optional

Skip pages that robots.txt disallows for generic crawlers (User-agent: *). Turn off only for sites you own.

Type:boolean

Default:true

Example:

true

Include subdomains

includeSubdomains

Optional

Treat subdomains (e.g. blog.example.com when auditing example.com) as part of the site: crawl them and do not count links to them as external.

Type:boolean

Default:false

Example:

false

Max concurrency

maxConcurrency

Optional

Maximum parallel page requests (all sites together). Keep it low to be polite to the audited server.

Type:integer

Minimum:1

Maximum:20

Default:5

Example:

Proxy configuration

proxyConfiguration

Optional

Usually not needed. If a site answers HTTP 403 / Cloudflare challenges (reported as free BLOCKED rows), try Apify Proxy, ideally the RESIDENTIAL group.

Type:object

Default:

{
  "useApifyProxy": false
}

LinkedIn Profile Scraper + Email (No Cookies)

harvestapi/linkedin-profile-scraper

Extract detailed information from LinkedIn profiles in bulk, including complete work experience, education history, skills and more. No cookies or account required.

HarvestAPI

80K

4.6

(93)

Instagram Scraper

apify/instagram-scraper

Extract Instagram posts, reels, profiles, places, hashtags, carousels, and comments. Get data from Instagram using one or more Instagram URLs or search queries: content, context, metrics, metadata. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Apify

430K

4.7

(601)

Google Maps Scraper

compass/crawler-google-places

Extract data from thousands of Google Maps locations and businesses, including reviews, reviewer details, images, contact info, including full name, email, and job title, opening hours, prices & more. Export data, run via API, schedule and monitor runs, or integrate with other tools.

Compass

649K

4.7

(1,816)

TikTok Scraper

clockworks/tiktok-scraper

Extract data from TikTok videos, hashtags, and users. Use URLs or search queries to scrape TikTok profiles, hashtags, posts, URLs, shares, followers, hearts, names, video, and music-related data. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Clockworks

333K

4.8

(369)

Instagram Profile Scraper

apify/instagram-profile-scraper

Scrape all Instagram profile info. Just add Instagram usernames, IDs or URLs and extract name, join date, number of followers, location, bio, website, related profiles, video&post count, latest posts. Export scraped data, schedule scraper via API, and integrate with other tools or AI workflows.

Apify

249K

4.7

(175)

Facebook Search Posts

scrapeforge/facebook-search-posts

Search Facebook posts, pages, groups, people, videos and events by keyword. Returns detailed data including reactions breakdown, media URLs, and engagement metrics.

ScrapeForge

58K

3.6

(10)

YouTube Scraper

streamers/youtube-scraper

YouTube crawler and video scraper. Alternative YouTube API with no limits or quotas. Extract and download channel name, likes, number of views, and number of subscribers.

Streamers

163K

4.8

(196)

Google Search Results Scraper

apify/google-search-scraper

Scrape Google Search Engine Results Pages (SERPs). Select the country or language and extract organic and paid results, AI Mode, AI overviews, ads, queries, People Also Ask, prices, reviews, like a Google SERP API. Export data, run the scraper via API, schedule runs, or integrate with other tools.

Apify

205K

4.6

(219)

Instagram Post Scraper

apify/instagram-post-scraper

Scrape Instagram posts. Just add one or more Instagram usernames and get your data in seconds including caption, metrics, images, mentions, coauthors, recent comments, sponsored status, video duration, views. Export scraped data, schedule scraper via API, integrate with other tools or AI workflows.

Apify

141K

4.3

(138)

Instagram Hashtag Scraper

apify/instagram-hashtag-scraper

Scrape Instagram posts and reels by hashtags. Just add one or more hashtags and extract captions, locations, likes, plays, shares, comments count, images, timestamps, audio, and other hashtags. Export scraped hashtags, run the scraper via API, schedule and monitor runs or integrate with other tools.

Apify

129K

3.5

(91)