No credit card required

Website Content Crawler

apify/website-content-crawler

No credit card required

Automatically crawl and extract text content from websites with documentation, knowledge bases, help centers, or blogs. This Actor is designed to provide data to feed, fine-tune, or train large language models such as ChatGPT or LLaMA.

All issues Create new issue

KV Store isn't downloading file attachments correctly

Closed

embrace_ai opened this issue

We recently initiated a crawl and noticed that some of the file attachments aren't being saved to the KV Store correctly.

For example, the file encountered at the URL below is 50 MB but the corresponding KV store entry is reporting 1.5 MB. Do we need to increase our download timeout in order to download the entire file in time?

https://barbercosmo.ca.gov/schools/healthsafety_textbook_ko.pdf
dc9f565-healthsafety_textbook_ko.pdf

Jindřich Bär (jindrich.bar)

Hello and thank you for your interest in this Actor!

I don't think there is anything you can do with this right now - this is most likely caused by the way we handle file downloads in this Actor. I started looking into this and already have a promising proof-of-concept for the patch, but before releasing it, I need to test this thoroughly.

I'll keep you posted with updates!

Thank you again for the report and sorry for the momentary inconvenience. Cheers!

Jindřich Bär (jindrich.bar)

This is fixed in the latest build of this Actor (0.3.33). Give it a try and let us know how it went.

Thank you again for your patience!

Add comment

Developer

Apify

Actor metrics

2k monthly users
99.9% runs succeeded
2.9 days response time
Created in Mar 2023
Modified 3 days ago

Categories

Developer tools

Business

Web Scraper

apify/web-scraper

Crawls arbitrary websites using the Chrome browser and extracts data from pages using a provided JavaScript code. The actor supports both recursive crawling and lists of URLs and automatically manages concurrency for maximum performance. This is Apify's basic tool for web crawling and scraping.

Apify

63.5k

Google Maps Scraper

compass/crawler-google-places

Extract data from hundreds of Google Maps locations and businesses. Get Google Maps data including reviews, images, contact info, opening hours, location, popular times, prices & more. Export scraped data, run the scraper via API, schedule and monitor runs, or integrate with other tools.

Compass

63.5k

Google Search Results Scraper

apify/google-search-scraper

Scrape Google Search Engine Results Pages (SERPs). Select the country or language and extract organic and paid results, ads, queries, People Also Ask, prices, reviews, like a Google SERP API. Export scraped data, run the scraper via API, schedule and monitor runs, or integrate with other tools.

Apify

43.8k

GPT Scraper

drobnikj/gpt-scraper

Extract data from any website and feed it into GPT via the OpenAI API. Use ChatGPT to proofread content, analyze sentiment, summarize reviews, extract contact details, and much more.

Jakub Drobník

4.4k

AI Product Matcher

equidem/ai-product-matcher

Match products across multiple e-commerce websites. Use this AI product matching Actor whenever you need to find matching pairs of products from different online shops for dynamic pricing, competitor analysis or market research.

Matěj Sochor

318

Cheerio Scraper

apify/cheerio-scraper

Crawls websites using raw HTTP requests, parses the HTML with the Cheerio library, and extracts data from the pages using a Node.js code. Supports both recursive crawling and lists of URLs. This actor is a high-performance alternative to apify/web-scraper for websites that do not require JavaScript.

Apify

4.3k

Facebook Ads Scraper

apify/facebook-ads-scraper

Extract advertising data from one or multiple Facebook Pages. Get page details, reach estimates, publisher platforms, report count, number of impressions, ad IDs, timestamps, and more. Download Facebook ads data in JSON, CSV, and Excel and use it in apps, spreadsheets, and reports.

Apify

AI Web Agent

apify/ai-web-agent

Use natural language prompts to browse the web, click on elements, fill and submit forms, extract data, and take screenshots using the OpenAI API.

Apify

431

📩📍 Google Maps Email Extractor

lukaskrivka/google-maps-with-contact-details

Extract Google Maps contact details. Scrape websites of Google Maps places for contact details and get email addresses, website, location, address, zipcode, phone number, social media links. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.