No credit card required
Website Content Crawler
No credit card required
Automatically crawl and extract text content from websites with documentation, knowledge bases, help centers, or blogs. This Actor is designed to provide data to feed, fine-tune, or train large language models such as ChatGPT or LLaMA.
Crawling works, but starts at a random location half way in on a a given site
Closed
When my crawler recorded this site https://developer.firmhouse.com/liquid/available-tags it started to fetch only half way in, even though there are no section breaks or similar on the site. It starts to fetch from subscription, and not from asset (which would be the first item as seen in screenshot)
Hello and thank you for your interest in this Actor!
This is a prime example of the text extractors being too eager to clean the website contents.
To get the full webpage content into your dataset, you can simply switch HTML Processing > HTML Transformer to None. If you end up with some extra content in your dataset (that you don't want to have there), you can use the Remove HTML elements (CSS selector) to remove those based on their CSS selectors.
I'll close this issue now, but feel free to ask additional questions if you have any. Cheers!
Apify- 2k monthly users
- 99.9% runs succeeded
- 2.9 days response time
- Created in Mar 2023
- Modified 3 days ago
