Ai Powered Scraper

Under maintenance

Pricing

Pay per usage

Try for free

Go to Apify Store

Ai Powered Scraper

Under maintenance

Try for free

AI Powered Scraper using LangChain and OpenAI.

Pricing

Pay per usage

Rating

0.0

(0)

Developer

Dev with Bobby

Maintained by Community

Actor stats

Bookmarked

Total users

Monthly active users

23 days ago

Last modified

Categories

Automation

Developer tools

Start URLs

startUrls

Optional

One or more URLs of pages where the crawler will start. Note that the Actor will additionally only crawl sub-pages of these URLs. For example, for start URL https://www.example.com/blog, it will crawl pages like https://example.com/blog/article-1, but will skip https://example.com/docs/something-else.

Type:array

Max pages

maxCrawlPages

Optional

The maximum number pages to crawl. It includes the start URLs, pagination pages, pages with no content, etc. The crawler will automatically finish after reaching this number. This setting is useful to prevent accidental crawler runaway.

Type:integer

Minimum:0

Default:9999999

OpenAI API key

openAIApiKey

Optional

Enter your OpenAI account and an API key. This is needed for vectorizing the data and also to be able to prompt the OpenAI model.

Type:string

Query

query

Optional

The query you want to ask the model about the crawled data.

Type:string

Re-crawl the data

forceRecrawl

Optional

If enabled, the data will be re-crawled even if cached vector index is available.

Type:boolean

Default:false

Load URLs from Sitemaps

loadUrlsFromSitemaps

Optional

If enabled, the scraper will automatically find and load URLs from sitemap.xml files.

Type:boolean

Default:false

Respect robots.txt file

respectRobotsTxt

Optional

If enabled, the scraper will respect the robots.txt file and avoid crawling disallowed pages.

Type:boolean

Default:true

Crawler type

crawlerType

Optional

Select the crawler type based on your needs

Type:string

Default:adaptive

Options:

adaptivecheerioplaywrightjsdom

User Agent

userAgent

Optional

Custom User-Agent string to use for requests

Type:string

Default:Mozilla/5.0 (compatible; AI-Powered-Scraper/1.0)

Request delay (ms)

requestDelay

Optional

Delay between requests in milliseconds to avoid overwhelming the server

Type:integer

Minimum:0

Maximum:10000

Default:1000

Max request retries

maxRequestRetries

Optional

Maximum number of times to retry failed requests

Type:integer

Minimum:0

Maximum:10

Default:3

Docs Markdown Rag Ready Crawler

devwithbobby/docs-markdown-rag-ready-crawler

Turn any documentation site or website into clean, structured markdown—ready for RAG, embeddings, and AI agents.

Dev with Bobby

Price Tracking

devwithbobby/price-tracking

Advanced price tracking actor that monitors product prices across multiple e-commerce platforms (Amazon, Walmart, Target, etc.) with real-time notifications via Slack/email and data export to Google Sheets.

Dev with Bobby

Api Endpoint Monitor

devwithbobby/api-endpoint-monitor

Never get surprised by API changes again. This Actor continuously monitors your public APIs, tracks endpoint changes, auto-generates OpenAPI documentation, and alerts you instantly when breaking changes occur.

Dev with Bobby

Web2APK | Turn Your Websites Into APK

hrmn/web2apk-turn-your-websites-into-apk

Do you have a website but need a mobile presence? Stop wasting thousands of dollars on app developers! 💸 With Web2Apk, you can transform any URL into a high-quality APK in just seconds. No coding, no hassle, just results.

Harmon Developer

FB Profile Scraper

devwithbobby/fb-profile-scraper

Scrape data from any public FB Profile

Dev with Bobby

410

LangChain.js template

ellustar/my-actor-17

The LangChain.js Template Actor is a ready-to-use foundation for building AI agents with LangChain.js. It supports LLM integration, chains, tools, and memory for fast prototyping and scalable deployment of chatbots, automation workflows, and AI apps.

Ellustar

AI Extraction Agent - Smart Scraper

alizarin_refrigerator-owner/ai-extraction-agent

AI-powered data extraction using natural language prompts. Describe what you need & let AI extract structured data from any webpage automatically.

John Rippy

News Intelligence Pro (AI Powered)

obsequious_ontologist/news-intelligence-pro

Turn news into business intelligence. AI-powered sentiment analysis, brand monitoring, and competitor tracking from 50+ sources. Get market insights, trend alerts, and reputation management in real-time.

Dave Thomson

Sora AI Video Scraper - OpenAI Text-to-Video

payai/sora-video-scraper

Extract AI-generated videos from Sora by OpenAI. Collect video URLs, thumbnails, prompts, and metadata. Perfect for AI researchers and content creators.

PayAI

1.0

Website Contact Scraper - AI-Powered Lead Finder

timo.sieber/website-lead-scraper

AI-powered website scraper that extracts real contact data from company sites! Finds people, positions, emails & phone numbers using LLM technology. Scans team pages, contact sections & company info. Perfect for B2B lead generation and sales research.