No credit card required

Website Content Crawler

apify/website-content-crawler

No credit card required

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.

Do you want to learn more about this Actor?

Get a demo

Back to issues Create new issue

Crawling is taking lot of time because of some exception and credits are getting exhausted

Closed

atomicwork opened this issue

below is the exception that we get in the logs ReferenceError: fetch is not defined 2024-06-04T08:26:11.549Z at a. (https://www.browsealoud.com/modules/3.9.3/browsealoud.js:1:56663) 2024-06-04T08:26:11.552Z at Generator.next () 2024-06-04T08:26:11.554Z at p (https://www.browsealoud.com/modules/3.9.3/browsealoud.js:1:53571) 2024-06-04T08:26:11.555Z at n (https://www.browsealoud.com/modules/3.9.3/browsealoud.js:1:53755) 2024-06-04T08:26:11.557Z at https://www.browsealoud.com/modules/3.9.3/browsealoud.js:1:53833 2024-06-04T08:26:11.559Z at new Promise () 2024-06-04T08:26:11.561Z at a. (https://www.browsealoud.com/modules/3.9.3/browsealoud.js:1:53714) 2024-06-04T08:26:11.563Z at a. (https://www.browsealoud.com/modules/3.9.3/browsealoud.js:1:56842) 2024-06-04T08:26:11.565Z at a. (https://www.browsealoud.com/modules/3.9.3/browsealoud.js:1:58782) 2024-06-04T08:26:11.568Z at Generator.next () 2024-06-04T08:26:11.570Z

and the actor is stuck in fetching the content

Jindřich Bär (jindrich.bar)

Hello and thank you for your interest in this Actor!

The fetch is not defined error is caused by the JSDOM crawler type - it is one of the well-known issues with the jsdom library (see the tracking GitHub issue here). If you want to scrape worry-free, I suggest switching to one of the Playwright based crawlers. Or, if performance/economy is a priority for you, you can try the Raw HTTP Client (Cheerio). This is even faster than jsdom, but it doesn't load the on-page Javascript whatsoever.

Looking at the website you're trying to scrape, it seems that it might work with Cheerio (although you'd better check the output manually for the first few pages).

I'll close this issue now, but feel free to ask additional questions if you have any. Cheers!

Add comment

Developer

Apify

Actor metrics

3.8k monthly users
544 stars
99.9% runs succeeded
3.4 days response time
Created in Mar 2023
Modified 1 day ago

Categories

Developer tools

Business

AI Website Content Markdown Scraper

quaking_pail/ai-website-content-markdown-scraper

This Apify Actor, "Website Content Crawler with Markdown Extraction," is designed to perform a comprehensive crawl of specified websites, extract their text content, convert it into Markdown format, and store it in a structured dataset. The extracted content is suitable for feeding LLMs.

AI_Builder

138

RAG Web Browser

apify/rag-web-browser

Web browser for OpenAI Assistants API and RAG pipelines, similar to a web browser in ChatGPT. It queries Google Search, scrapes the top N pages from the results, and returns their cleaned content as Markdown for further processing by an LLM.

Apify

124

Google Maps Scraper

compass/crawler-google-places

Extract data from hundreds of Google Maps locations and businesses. Get Google Maps data including reviews, images, contact info, opening hours, location, popular times, prices & more. Export scraped data, run the scraper via API, schedule and monitor runs, or integrate with other tools.

Compass

76.6k

492

📩📍 Google Maps Email Extractor

lukaskrivka/google-maps-with-contact-details

Extract Google Maps contact details. Scrape websites of Google Maps places for contact details and get email addresses, website, location, address, zipcode, phone number, social media links. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Lukáš Křivka

6.6k

116

Amazon Reviews Scraper

junglee/amazon-reviews-scraper

Amazon scraper to extract reviews from Amazon products. Scrape and download detailed reviews without using the Amazon API, including rating score, review description, reactions and images. Download your data as HTML table, JSON, CSV, Excel, XML.

Junglee

2.9k

Google Maps Reviews Scraper

compass/Google-Maps-Reviews-Scraper

Extract all reviews of Google Maps places using place URLs. Get review text, published date, response from owner, review URL, and reviewer's details. Download scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Compass

3.6k

Tiktok Hashtag Scraper

clockworks/tiktok-hashtag-scraper

Scrape TikTok hashtag data. Just add one or more hashtags and extract TikTok videos with that hashtag: URLs, likes, country of creation, video and music metadata, TikTok creator data. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Clockworks

2.1k

TikTok Sound Scraper

clockworks/tiktok-sound-scraper

Scrape TikTok videos with a chosen sound. Just add one or more sound URLs and extract tiktoks that have it: URLs, likes, country of creation, video and music metadata, creator data. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Clockworks

179

Google Maps Extractor

compass/google-maps-extractor

Extract data from hundreds of places fast. Scrape Google Maps by keyword, category, location, URLs & other filters. Get addresses, contact info, opening hours, popular times, prices, menus & more. Export scraped data, run the scraper via API, schedule and monitor runs, or integrate with other tools.

Compass

13.4k

235