No credit card required

Website Content Crawler

apify/website-content-crawler

No credit card required

Automatically crawl and extract text content from websites with documentation, knowledge bases, help centers, or blogs. This Actor is designed to provide data to feed, fine-tune, or train large language models such as ChatGPT or LLaMA.

All issues Create new issue

Website Content Crawler missed most of website content

Closed

meditative_windcheater opened this issue

Hi,

I have an issue with the website crawler. While the removedElementsHtmlUrl contains all the content I need, all extractions afterwards miss most of the content. How can I tackle this? Here are the snapshot urls:

"originalHtmlUrl": "https://api.apify.com/v2/key-value-stores/ommLFcytptMQ3mMIO/records/https---www-otto-de-shoppages-service-faq-originalHtmlUrl"

"removedElementsHtmlUrl": "https://api.apify.com/v2/key-value-stores/ommLFcytptMQ3mMIO/records/https---www-otto-de-shoppages-service-faq-removedElementsHtmlUrl"

"extractusHtmlUrl": "https://api.apify.com/v2/key-value-stores/ommLFcytptMQ3mMIO/records/https---www-otto-de-shoppages-service-faq-extractusHtmlUrl"

"readableTextHtmlUrl": "https://api.apify.com/v2/key-value-stores/ommLFcytptMQ3mMIO/records/https---www-otto-de-shoppages-service-faq-readableTextHtmlUrl"

"readableTextIfPossibleHtmlUrl": "https://api.apify.com/v2/key-value-stores/ommLFcytptMQ3mMIO/records/https---www-otto-de-shoppages-service-faq-readableTextIfPossibleHtmlUrl"

Jindřich Bär (jindrich.bar)

Hello and thank you for your interest in this Actor!

This is a prime example of the text extractors being too eager to clean the website contents.

To get the full webpage content into your dataset, you can simply switch HTML Processing > HTML Transformer to None. This yields results equivalent to the removedElementsHtmlUrl debug option. If you end up with some extra content in your dataset (that you don't want to have there), you can use the Remove HTML elements (CSS selector) to remove those based on their CSS selectors.

I'll close this issue now, but feel free to ask additional questions if you have any. Cheers!

Add comment

Developer

Apify

Actor metrics

2k monthly users
99.9% runs succeeded
2.9 days response time
Created in Mar 2023
Modified 3 days ago

Categories

Developer tools

Business

Web Scraper

apify/web-scraper

Crawls arbitrary websites using the Chrome browser and extracts data from pages using a provided JavaScript code. The actor supports both recursive crawling and lists of URLs and automatically manages concurrency for maximum performance. This is Apify's basic tool for web crawling and scraping.

Apify

63.5k

Google Maps Scraper

compass/crawler-google-places

Extract data from hundreds of Google Maps locations and businesses. Get Google Maps data including reviews, images, contact info, opening hours, location, popular times, prices & more. Export scraped data, run the scraper via API, schedule and monitor runs, or integrate with other tools.

Compass

63.5k

Google Search Results Scraper

apify/google-search-scraper

Scrape Google Search Engine Results Pages (SERPs). Select the country or language and extract organic and paid results, ads, queries, People Also Ask, prices, reviews, like a Google SERP API. Export scraped data, run the scraper via API, schedule and monitor runs, or integrate with other tools.

Apify

43.8k

GPT Scraper

drobnikj/gpt-scraper

Extract data from any website and feed it into GPT via the OpenAI API. Use ChatGPT to proofread content, analyze sentiment, summarize reviews, extract contact details, and much more.

Jakub Drobník

4.4k

AI Product Matcher

equidem/ai-product-matcher

Match products across multiple e-commerce websites. Use this AI product matching Actor whenever you need to find matching pairs of products from different online shops for dynamic pricing, competitor analysis or market research.

Matěj Sochor

318

Cheerio Scraper

apify/cheerio-scraper

Crawls websites using raw HTTP requests, parses the HTML with the Cheerio library, and extracts data from the pages using a Node.js code. Supports both recursive crawling and lists of URLs. This actor is a high-performance alternative to apify/web-scraper for websites that do not require JavaScript.

Apify

4.3k

Facebook Ads Scraper

apify/facebook-ads-scraper

Extract advertising data from one or multiple Facebook Pages. Get page details, reach estimates, publisher platforms, report count, number of impressions, ad IDs, timestamps, and more. Download Facebook ads data in JSON, CSV, and Excel and use it in apps, spreadsheets, and reports.

Apify

AI Web Agent

apify/ai-web-agent

Use natural language prompts to browse the web, click on elements, fill and submit forms, extract data, and take screenshots using the OpenAI API.

Apify

431

📩📍 Google Maps Email Extractor

lukaskrivka/google-maps-with-contact-details

Extract Google Maps contact details. Scrape websites of Google Maps places for contact details and get email addresses, website, location, address, zipcode, phone number, social media links. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.