![Website Content Crawler avatar](https://images.apifyusercontent.com/hchWWpDJQyd-Nh_2beH12X2c6MBfXEQc59HAA24sOAY/rs:fill:250:250/aHR0cHM6Ly9hcGlmeS1pbWFnZS11cGxvYWRzLXByb2QuczMuYW1hem9uYXdzLmNvbS9hWUcwbDlzN2RiQjdqM2diUy9QZlRvRU5rSlp4YWh6UER1My1DbGVhblNob3RfMjAyMy0wMy0yOF9hdF8xMC40MC4yMF8yeC5wbmc.webp)
No credit card required
![Website Content Crawler](https://images.apifyusercontent.com/hchWWpDJQyd-Nh_2beH12X2c6MBfXEQc59HAA24sOAY/rs:fill:250:250/aHR0cHM6Ly9hcGlmeS1pbWFnZS11cGxvYWRzLXByb2QuczMuYW1hem9uYXdzLmNvbS9hWUcwbDlzN2RiQjdqM2diUy9QZlRvRU5rSlp4YWh6UER1My1DbGVhblNob3RfMjAyMy0wMy0yOF9hdF8xMC40MC4yMF8yeC5wbmc.webp)
Website Content Crawler
No credit card required
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗LangChain, LlamaIndex, and the wider LLM ecosystem.
Do you want to learn more about this Actor?
Get a demobug: iframe contents don't get extracted properly
Open
I tried to scrape site, but i had only its header content because scrapper can wait only for content loading, but not for content execution. So please, add the simple delay setting
Hello and thank you for your interest in this Actor!
Website Content Crawler already waits for dynamic content to load - see the HTML Processing > Wait for dynamic content (seconds)
input option. The issue with your webpage is that the actual content is in an iframe at https://boards.greenhouse.io/embed/job_app?for=growthloop&token=4008881005&b=https%3A%2F%2Fwww.growthloop.com%2Fcompany%2Fjob-listing
.
WCC cannot currently expand iframe
contents because of security reasons, but we're working on a feature that should help fix this. In the meantime, you can scrape the URL above to get all the contents of the page (see my example run)
I'll keep this issue open to track the iframe issue. Feel free to ask additional questions regarding this or any related issues in the comments here.
Cheers!
bllndman
Thank you for the response, now I see
- 2.8k monthly users
- 434 stars
- 99.9% runs succeeded
- 2.9 days response time
- Created in Mar 2023
- Modified 3 days ago