Pricing

$15.00 / 1,000 results

Try for free

Go to Apify Store

Advanced Website Crawling Actor

Try for free

A fast and reliable scraper for any website that extracts clean HTML, Markdown, and text content. Provides clean, structured data with support for dynamic rendering, recursive sitemap discovery, SSL bypass, and easy API integration for your applications.

Pricing

$15.00 / 1,000 results

Rating

5.0

(2)

Developer

Techforce Global

Actor stats

Bookmarked

Total users

Monthly active users

a month ago

Last modified

Categories

Developer tools

Automation

Start URL

startUrl

Required

Target URL to crawl

Type:string

Default:https://www.apify.com

Max Pages

maxPages

Optional

Maximum number of pages to crawl.

Type:integer

Maximum:200

Default:20

Max Crawl Depth

maxDepth

Optional

Maximum depth from start URLs.

Type:integer

Default:5

Concurrency

concurrency

Optional

Number of concurrent crawler workers.

Type:integer

Default:20

Dataset Batch Size

batchSize

Optional

Number of records pushed to dataset per batch.

Type:integer

Default:20

Output Format

outputFormat

Optional

Content format to store.

Type:string

Default:text

Include URL Patterns

includePatterns

Optional

Glob patterns for URLs to include.

Type:string[]

Default:

[
  "*"
]

Exclude URL Patterns

excludePatterns

Optional

Glob patterns for URLs to exclude.

Type:string[]

Default:

[]

Max Requests Per Crawl

maxRequestsPerCrawl

Optional

Hard cap on total requests per run.

Type:integer

Default:1000

Max Retries

maxRetries

Optional

Maximum retries per request.

Type:integer

Default:2

Browser Concurrency

browserConcurrency

Optional

Maximum concurrent Playwright browser pages (applies only to JS-heavy pages that fail static fetch).

Type:integer

Default:8

Deduplicate Content

deduplicateContent

Optional

Skip duplicate content using SHA-256 hashing.

Type:boolean

Default:true

Respect robots.txt

respectRobotsTxt

Optional

Whether to respect robots.txt rules.

Type:boolean

Default:true

Apify Proxy Configuration

proxy

Optional

Apify Proxy settings.

Type:object

All Events Scraper

techforce.global/all-events-scraper

A fast and reliable scraper for AllEvents.in that extracts event titles, dates, locations, interested people count, and more. Provides clean, structured data with support for Markdown, pagination, and easy API integration for your applications.

Techforce Global

4.4

Meetup.com City Events Scraper

techforce.global/meetup-city-events-scraper

A fast and reliable scraper for meetup.com that extracts Event Name, Venue details, Price and much more. Provides clean, structured data with support for Markdown, pagination, and easy API integration for your applications.

Techforce Global

Events Eye Scraper

techforce.global/events-eye-scraper

A fast and reliable scraper for eventseye.com that extracts Exhibition Name, Description, Venue details, Organizer details and much more. Provides clean, structured data with support for Markdown, pagination, and easy API integration for your applications.

Techforce Global

5.0

Firecrawl Website Crawler

alizarin_refrigerator-owner/firecrawl-website-crawler

Enhanced Website Crawling with Superior JS Rendering Enhanced website crawler using Firecrawl's Crawl API for superior JavaScript rendering, smart rate limiting, anti-bot bypass, and clean markdown extraction.

The Howlers

Website to Markdown for LLM and RAG

jeweled_jockstrap/my-actor-3

Convert any URL to clean Markdown text for AI applications. Strips HTML extracts content. For LLM training RAG pipelines and vector databases. Free Firecrawl alternative.

Juan Triviño

Sitemap Scraper

scrapers-hub/sitemap-scraper

Sitemap scraper to crawl and extract URLs, pages, and structure from website sitemaps 🌐📊 Perfect for SEO analysis, website auditing, and data extraction. Fast, reliable, and scalable.

Scrapers Hub

Universal AI Web Scraper

stanvanrooy6/universal-ai-web-scraper

Turn any website into an API. Extract structured data using plain English. Features anti-bot bypass, dynamic rendering, and web search. No coding needed.

Stan Van Rooy

1.5

Website Content Crawler

crawlerbros/website-content-crawler

Crawls websites and extracts clean text, markdown, or HTML content. Ideal for LLM training data, RAG pipelines, and knowledge base building.

Crawler Bros

5.0

Find Sitemap from url

eesti/find-sitemap-from-url

A powerful [Apify Actor] that finds sitemap URLs for any website. This Actor helps you discover XML sitemaps by checking common locations, robots.txt files, and analyzing HTML content for sitemap links.

ando

210

1.0

Website Content Crawler

parseforge/website-content-crawler

Crawl any website and pull clean Markdown content ready for AI! Follow links across a whole domain and extract page text, titles, headings, images, and metadata. Perfect for building RAG pipelines, training datasets, knowledge bases, and vector databases. Start crawling content in minutes!