No credit card required

Website Content Crawler

apify/website-content-crawler

No credit card required

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗LangChain, LlamaIndex, and the wider LLM ecosystem.

Do you want to learn more about this Actor?

Get a demo

Back to issues Create new issue

Actor sometimes grabs only one screenshot or repeated 'mismatched' screenshots in output

Closed

jjohnson-dev opened this issue

The last couple days I noticed I don't seem to be getting as many screenshotUrls or accurately matched screenshotUrls in the dataset.

This run only grabbed one - https://console.apify.com/view/runs/Iyav2BXfpIAakad69.

This run has the SAME screenshot posted in the payload for multiple distinct websites.

screenshotUrl https://api.apify.com/v2/key-value-stores/OkUy6AhFjH7Qlt0VO/records/SCREENSHOT--.jpg over 70 times in the results for multiple different businesses. That can't be right?

https://console.apify.com/view/runs/RRsGOekCU3p5ggKMU

My goal is to grab a screenshot of each businesses home page I'm passing through the crawler. Should depth 0 be sufficient? Should sitemap option be off or on?

Thanks for looking!

Jindřich Bär (jindrich.bar)

Hello again!

This is indeed an issue with Website Content Crawler - in certain cases, it stored the same screenshot for multiple crawled webpages. This is fixed in the latest version of this Actor (0.3.38).

Also make sure you're using the playwright:firefox crawler type - the Adaptive crawler cannot take the website screenshots properly (since it switches between actual browsers and raw HTTP clients). This behavior is now enforced in 0.3.38 - the Actor will print a warning message and will turn the saveScreenshots option off if you try to use it with an unsupported crawler type.

Regarding your other questions - if you're trying to only scrape the URLs from the Start URLs input, then yes - Maximum crawl depth set to 0 should be enough. In this case, you can also keep the Use Sitemaps option set to false.

I'll close this issue now, but feel free to ask additional questions, if you have any. Cheers!

Add comment

Developer

Apify

Actor metrics

3.4k monthly users
486 stars
99.9% runs succeeded
3.2 days response time
Created in Mar 2023
Modified 4 days ago

Categories

Developer tools

Business

AI Website Content Markdown Scraper

quaking_pail/ai-website-content-markdown-scraper

This Apify Actor, "Website Content Crawler with Markdown Extraction," is designed to perform a comprehensive crawl of specified websites, extract their text content, convert it into Markdown format, and store it in a structured dataset. The extracted content is suitable for feeding LLMs.

AI_Builder

Web Scraper

apify/web-scraper

Crawls arbitrary websites using the Chrome browser and extracts data from pages using JavaScript code. The Actor supports both recursive crawling and lists of URLs and automatically manages concurrency for maximum performance. This is Apify's basic tool for web crawling and scraping.

Apify

69k

179

Google Maps Scraper

compass/crawler-google-places

Extract data from hundreds of Google Maps locations and businesses. Get Google Maps data including reviews, images, contact info, opening hours, location, popular times, prices & more. Export scraped data, run the scraper via API, schedule and monitor runs, or integrate with other tools.

Compass

75.2k

437

Fast Facebook Pages Scraper

oussemafr/fast-facebook-pages-scraper

Use this fast and reliable unofficial Facebook API to efficiently scrape detailed Facebook page information. Extract valuable data such as page names, URLs, contact details, addresses, likes, and followers. Perfect for competitive analysis, market research, trend monitoring and social media analysis

Oussema FRIKHA

Facebook Posts Scraper

apify/facebook-posts-scraper

Extract data from hundreds of Facebook posts from one or multiple Facebook pages and profiles. Get post URL, post text, page or profile URL, timestamp, number of likes, shares, comments, and more. Download the data in JSON, CSV, and Excel and use it in apps, spreadsheets, and reports.

Apify

12.8k

138

Instagram Scraper

apify/instagram-scraper

Scrape and download Instagram posts, profiles, places, hashtags, photos, and comments. Get data from Instagram using one or more Instagram URLs or search queries. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Apify

52.2k

326

Amazon Scraper

junglee/free-amazon-product-scraper

Gets you product data from Amazon. Unofficial API. Scrapes and downloads product information without using the Amazon API, including reviews, prices, descriptions, and ASIN.

Junglee

3.2k

Google Maps Extractor

compass/google-maps-extractor

Extract data from hundreds of places fast. Scrape Google Maps by keyword, category, location, URLs & other filters. Get addresses, contact info, opening hours, popular times, prices, menus & more. Export scraped data, run the scraper via API, schedule and monitor runs, or integrate with other tools.

Compass

10.8k

176

RAG Web Browser

apify/rag-web-browser

Web browser for OpenAI Assistants API and RAG pipelines, similar to a web browser in ChatGPT. It queries Google Search, scrapes the top N pages from the results, and returns their cleaned content as Markdown for further processing by an LLM.

Apify