Pricing

$0.50 / 1,000 url processeds

Try for free

Go to Apify Store

Sitemap Extractor & Broken Link Checker

Try for free

Extract every URL from a website's XML sitemaps (robots.txt discovery, sitemap indexes, .gz) with lastmod, check HTTP status codes, and crawl pages to find broken internal and external links with source page and anchor text. Polite: respects robots.txt.

Pricing

$0.50 / 1,000 url processeds

Rating

0.0

(0)

Developer

Jean Swiegers

Actor stats

Bookmarked

Total users

Monthly active users

15 hours ago

Last modified

Categories

SEO tools

Developer tools

Websites or sitemap URLs

startUrls

Required

A homepage (sitemaps are discovered from robots.txt, /sitemap.xml and /sitemap_index.xml) or a direct sitemap URL (.xml, .xml.gz, sitemap index or plain-text sitemap).

Type:array

Default:

[
  {
    "url": "https://crawlee.dev"
  }
]

Max sitemap URLs per site

maxUrls

Optional

Stop extracting after this many unique URLs per site.

Type:integer

Minimum:1

Maximum:100000

Default:100

Check HTTP status of sitemap URLs

checkStatus

Optional

Request every extracted URL (HEAD, falling back to GET) and add status code, redirect target and response time. Finds 404s and redirects listed in your sitemap.

Type:boolean

Default:false

Crawl pages to find broken links

crawlForBrokenLinks

Optional

Crawl the site's internal pages starting from the start URL, check every link and report broken ones with the page they're on and their anchor text.

Type:boolean

Default:false

Max pages to crawl per site

maxCrawlPages

Optional

Limit of internal HTML pages crawled for links.

Type:integer

Minimum:1

Maximum:10000

Default:100

Check external links

checkExternalLinks

Optional

Also check links pointing to other domains.

Type:boolean

Default:true

Output working links too

includeOkLinks

Optional

Also output links that work (full link inventory). Off = only broken and restricted links.

Type:boolean

Default:false

Max requests per second (per host)

maxRequestsPerSecond

Optional

Rate limit per host. Keep low for small sites. 429 responses trigger automatic back-off.

Type:integer

Minimum:1

Maximum:20

Default:4

Request timeout (s)

timeoutSecs

Optional

Timeout per request.

Type:integer

Minimum:5

Maximum:60

Default:20

Instagram Scraper

apify/instagram-scraper

Extract Instagram posts, reels, profiles, places, hashtags, carousels, and comments. Get data from Instagram using one or more Instagram URLs or search queries: content, context, metrics, metadata. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Apify

435K

4.7

(602)

TikTok Scraper

clockworks/tiktok-scraper

Extract data from TikTok videos, hashtags, and users. Use URLs or search queries to scrape TikTok profiles, hashtags, posts, URLs, shares, followers, hearts, names, video, and music-related data. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Clockworks

341K

4.7

(370)

Instagram Profile Scraper

apify/instagram-profile-scraper

Scrape all Instagram profile info. Just add Instagram usernames, IDs or URLs and extract name, join date, number of followers, location, bio, website, related profiles, video&post count, latest posts. Export scraped data, schedule scraper via API, and integrate with other tools or AI workflows.

Apify

253K

4.7

(175)

LinkedIn Profile Scraper + Email (No Cookies)

harvestapi/linkedin-profile-scraper

Extract detailed information from LinkedIn profiles in bulk, including complete work experience, education history, skills and more. No cookies or account required.

HarvestAPI

81K

4.6

(93)

Google Maps Scraper

compass/crawler-google-places

Extract data from thousands of Google Maps locations and businesses, including reviews, reviewer details, images, contact info, including full name, email, and job title, opening hours, prices & more. Export data, run via API, schedule and monitor runs, or integrate with other tools.

Compass

654K

4.7

(1,818)

Instagram Reel Scraper

apify/instagram-reel-scraper

Scrape or download Instagram reels. Just add Instagram profile or reel URL, and get caption, timestamp, transcript, hashtags, mentions, tagged users, comments, likes, shares, views, duration, and downloaded video. Export scraped data, schedule via API, and integrate with other tools or AI workflows.

Apify

167K

4.4

(103)

Instagram Hashtag Scraper

apify/instagram-hashtag-scraper

Scrape Instagram posts and reels by hashtags. Just add one or more hashtags and extract captions, locations, likes, plays, shares, comments count, images, timestamps, audio, and other hashtags. Export scraped hashtags, run the scraper via API, schedule and monitor runs or integrate with other tools.

Apify

132K

3.5

(91)

RAG Web Browser

apify/rag-web-browser

Web search and fetch tool for AI agents and RAG pipelines. It queries Google Search, scrapes the top N pages using a full web browser, and returns their content as clean Markdown for further processing by an LLM. Can also fetch individual URLs.

Apify

196K

4.7

(35)

Linkedin Jobs Scraper

curious_coder/linkedin-jobs-scraper

Scrape Linkedin jobs from jobs search results using search URLs or filters. Get full job details, job poster and company details.

Curious Coder

173K

4.5

(142)

YouTube Scraper

streamers/youtube-scraper

YouTube crawler and video scraper. Alternative YouTube API with no limits or quotas. Extract and download channel name, likes, number of views, and number of subscribers.