Under maintenance

Pricing

Pay per usage

Try for free

Go to Apify Store

Intelligent Website Crawler

Under maintenance

Try for free

Pricing

Pay per usage

Rating

0.0

(0)

Developer

HappiTap

Actor stats

Bookmarked

Total users

Monthly active users

4 days ago

Last modified

Categories

Automation

Start URLs

startUrls

Required

Array of URLs to scrape. Each URL should be an object with a 'url' property and optional 'method'.

Type:array

Crawler Type

crawlerType

Optional

Type of crawler to use

Type:string

Default:playwright:adaptive

Options:

playwright:adaptiveplaywright:firefoxplaywright:chromepuppeteer

Max Pages to Crawl

maxCrawlPages

Optional

Maximum number of pages to crawl

Type:integer

Minimum:1

Maximum:999999

Default:100

Max Crawl Depth

maxCrawlDepth

Optional

Maximum depth to crawl from start URLs

Type:integer

Minimum:0

Maximum:99

Default:3

Max Concurrency

maxConcurrency

Optional

Maximum concurrent browser instances. Higher values = faster crawling but more memory usage. Pattern learning is thread-safe for high concurrency.

Type:integer

Minimum:1

Maximum:200

Default:50

Max Request Retries

maxRequestRetries

Optional

Maximum number of retries for failed requests

Type:integer

Minimum:0

Maximum:10

Default:3

Request Timeout (seconds)

requestTimeoutSecs

Optional

Timeout for each request in seconds

Type:integer

Minimum:1

Maximum:300

Default:60

Dynamic Content Wait (seconds)

dynamicContentWaitSecs

Optional

Time to wait for dynamic content to load

Type:integer

Minimum:0

Maximum:60

Default:10

Include URL Globs

includeUrlGlobs

Optional

Glob patterns for URLs to include

Type:array

Default:

[]

Exclude URL Globs

excludeUrlGlobs

Optional

Glob patterns for URLs to exclude

Type:array

Default:

[]

Use Sitemaps

useSitemaps

Optional

Whether to automatically discover and use sitemaps

Type:boolean

Default:true

Aggressive Prune

aggressivePrune

Optional

More aggressively prune content to reduce token usage

Type:boolean

Default:false

Block Media

blockMedia

Optional

Block images, videos, and other media to speed up crawling

Type:boolean

Default:true

Save Markdown

saveMarkdown

Optional

Save page content as markdown

Type:boolean

Default:true

Save HTML

saveHtml

Optional

Save raw HTML content

Type:boolean

Default:false

Save Screenshots

saveScreenshots

Optional

Take and save screenshots of pages

Type:boolean

Default:false

Remove Elements CSS Selector

removeElementsCssSelector

Optional

CSS selector for elements to remove before processing

Type:string

Default:nav, footer, script, style, noscript, svg, img[src^='data:'], [role="alert"], [role="banner"], [role="dialog"], [role="alertdialog"], [role="region"][aria-label*="skip" i], [aria-modal="true"]

Click Elements CSS Selector

clickElementsCssSelector

Optional

CSS selector for elements to click (e.g., expand buttons)

Type:string

Default:[aria-expanded="false"]

Wait for Selector

waitForSelector

Optional

CSS selector to wait for before processing page

Type:string

Default:

Max Scroll Height (pixels)

maxScrollHeightPixels

Optional

Maximum height to scroll for infinite scroll pages

Type:integer

Minimum:0

Maximum:50000

Default:5000

Readable Text Character Threshold

readableTextCharThreshold

Optional

Minimum characters required for content to be considered readable

Type:integer

Minimum:0

Maximum:10000

Default:100

Remove Cookie Warnings

removeCookieWarnings

Optional

Automatically remove cookie consent banners

Type:boolean

Default:true

Respect robots.txt

respectRobotsTxtFile

Optional

Whether to respect robots.txt file

Type:boolean

Default:true

Proxy Configuration

proxyConfiguration

Optional

Proxy settings for requests

Type:object

Enable AI Processing

enableAI

Optional

Enable AI-powered page classification, smart pattern learning, and intelligent content extraction. Highly recommended for structured data extraction.

Type:boolean

Default:true

AI Model

aiModel

Optional

OpenAI model for AI processing. gpt-3.5-turbo is cost-effective for most use cases. Use gpt-4 for complex content extraction.

Type:string

Default:gpt-3.5-turbo

Options:

gpt-3.5-turbogpt-4gpt-4-turbo

Enable Smart Pattern Learning

enablePatternLearning

Optional

Enable smart DOM pattern learning to dramatically reduce AI API costs (90-99% savings). Learns page structures once and reuses for similar pages. Essential for large sites.

Type:boolean

Default:true

Pattern Similarity Threshold

patternSimilarityThreshold

Optional

Threshold for considering pages similar (0-1)

Type:number

Minimum:0

Maximum:1

Default:0.8

Max AI Calls Per Run

maxAICallsPerRun

Optional

Maximum AI API calls allowed per run. With pattern learning, you can crawl thousands of pages with just 10-50 AI calls. Start conservative and increase as needed.

Type:integer

Minimum:1

Maximum:1000

Default:50

Primary Task Type

taskType

Optional

Primary task for content processing (auto-detection recommended)

Type:string

Default:auto

Options:

autoSummarize contentExtract productsExtract servicesExtract FAQ contentExtract contact information

Use Website Sitemap

useSitemap

Optional

Automatically discover pages from website sitemap.xml. Combines with Start URLs for comprehensive crawling.

Type:boolean

Default:false

Sitemap URLs

sitemapUrls

Optional

URLs of sitemaps to crawl

Type:array

Max Pages from Sitemap

sitemapMaxPages

Optional

Maximum number of pages to extract from sitemaps. Use to limit crawl scope for large sites.

Type:integer

Minimum:1

Maximum:50000

Default:1000

Sitemap URL Filter Patterns

sitemapFilterPatterns

Optional

Filter sitemap URLs by patterns (e.g., '/product/', '/category/'). Include patterns you want, exclude with '!' prefix.

Type:string[]

Default:

[]

Website Content Crawler

rupom888/website-content-crawler

Syed Rupom

AI Website Content Crawler

ilborso/ai-website-content-crawler

A super fast website crawler for Agentic AI integration

Fabio Borsotti

5.0

Website Crawler API — Markdown for RAG

tugelbay/website-content-crawler

Website crawler API for public pages and clean Markdown, text, or HTML output for RAG pipelines, AI agents, documentation indexing, and monitoring. Guide: https://konabayev.com/tools/website-content-crawler/?utm_source=apify_info&utm_medium=referral&utm_campaign=website-content-crawler

Tugelbay Konabayev

Website Content Crawler

bhansalisoft/website-content-crawler

Website Content Crawler : scrap any website content with meta title and meta description and site logo

bhansalisoft

Intelligent Website Scrapper

happitap/intelligent-website-scrapper

An intelligent website scraper that uses LangChain and LLM to extract and process content based on high-level goals like summarization, product extraction, service extraction, and FAQ extraction.

HappiTap

287

5.0

Website Content Crawler — Extract Full Site Content

oneary/website-content-crawler

🌐 Full website crawler that extracts structured content (text, headings, metadata, links, images) from any domain. Free platform compute pricing.

Luan M.

Website Contacts Crawler

quaking_pail/contact-crawler

Scrap website searching for contact details, emails and phone numbers

AI_Builder

Domain Contact Scraper – Intelligent Email & Phone Extractor

emastra/website-contact-scraper

Intelligent website contact scraper that extracts emails, phone numbers, WhatsApp links, and social profiles from any domain. Uses priority-first crawling and returns one structured, consolidated result per start URL with transparent crawl metadata.