Pricing
$10.00 / 1,000 results
Pricing
$10.00 / 1,000 results
Rating
0.0
(0)
Developer
Dev with Bobby
Actor stats
1
Bookmarked
1
Total users
1
Monthly active users
a month ago
Last modified
Categories
Share
startUrlsarrayOptional
One or more URLs of pages where the crawler will start. Note that the Actor will additionally only crawl sub-pages of these URLs. For example, for start URL https://www.example.com/blog, it will crawl pages like https://example.com/blog/article-1, but will skip https://example.com/docs/something-else.
maxCrawlPagesintegerOptional
The maximum number pages to crawl. It includes the start URLs, pagination pages, pages with no content, etc. The crawler will automatically finish after reaching this number. This setting is useful to prevent accidental crawler runaway.
Default value of this property is 9999999
openAIApiKeystringOptional
Enter your OpenAI account and an API key. This is needed for vectorizing the data and also to be able to prompt the OpenAI model.
forceRecrawlbooleanOptional
If enabled, the data will be re-crawled even if cached vector index is available.
Default value of this property is false
loadUrlsFromSitemapsbooleanOptional
If enabled, the scraper will automatically find and load URLs from sitemap.xml files.
Default value of this property is false
respectRobotsTxtbooleanOptional
If enabled, the scraper will respect the robots.txt file and avoid crawling disallowed pages.
Default value of this property is true
crawlerTypeEnumOptional
Select the crawler type based on your needs
"adaptive": string"cheerio": string"playwright": string"jsdom": stringDefault value of this property is "adaptive"
userAgentstringOptional
Custom User-Agent string to use for requests
Default value of this property is "Mozilla/5.0 (compatible; AI-Powered-Scraper/1.0)"