Website Content Crawler
No credit card required
Website Content Crawler
No credit card required
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
Do you want to learn more about this Actor?
Get a demoWebsite Scrape doesn't include Social Media Links
Closed
I ran the Playwright + Firefox crawler type (after the adaptive crawler type) on a website and it wasn't able to retrieve the social media links at the bottom of the Home page. I've attached an image of the section for your reference. Would love to know if this is possible using a different crawler type? Or if a modification to the actor can be made to retrieve these results.
Thanks for the attention, and looking forward to your input!
Hello and thank you for your interest in this Actor!
This is a common issue - the default Website Content Crawler settings use the Readable Text HTML extractor, which is sometimes too eager with removing content. Turning the HTML extractor off keeps more content in the page. To do this, go to HTML Processing and set HTML Transfomer to None.
In addition, I found out that our toolkit for removing cookie modals also removes parts of the page footer. To turn this off, go to HTML Processing again and set Remove cookie warnings to false.
So, all in all, two things:
- set the HTML transformer to
None, and - turn off the
Remove cookie warningsoption.
I'll close this issue now, but feel free to ask additional questions if you have any.
Cheers!
Apify- 3.8k monthly users
- 636 stars
- 100.0% runs succeeded
- 2.7 days response time
- Created in Mar 2023
- Modified 7 days ago
