![Website Content Crawler avatar](https://images.apifyusercontent.com/hchWWpDJQyd-Nh_2beH12X2c6MBfXEQc59HAA24sOAY/rs:fill:250:250/aHR0cHM6Ly9hcGlmeS1pbWFnZS11cGxvYWRzLXByb2QuczMuYW1hem9uYXdzLmNvbS9hWUcwbDlzN2RiQjdqM2diUy9QZlRvRU5rSlp4YWh6UER1My1DbGVhblNob3RfMjAyMy0wMy0yOF9hdF8xMC40MC4yMF8yeC5wbmc.webp)
No credit card required
![Website Content Crawler](https://images.apifyusercontent.com/hchWWpDJQyd-Nh_2beH12X2c6MBfXEQc59HAA24sOAY/rs:fill:250:250/aHR0cHM6Ly9hcGlmeS1pbWFnZS11cGxvYWRzLXByb2QuczMuYW1hem9uYXdzLmNvbS9hWUcwbDlzN2RiQjdqM2diUy9QZlRvRU5rSlp4YWh6UER1My1DbGVhblNob3RfMjAyMy0wMy0yOF9hdF8xMC40MC4yMF8yeC5wbmc.webp)
Website Content Crawler
No credit card required
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗LangChain, LlamaIndex, and the wider LLM ecosystem.
Do you want to learn more about this Actor?
Get a demoNot crawling all content
Closed
Hey! Just tried running it on https://www.lecreuset.co.uk/en_GB/p/3-piece-starter-set-with-silicone-tool/GS1000.html
Exported it into a JSON file but can't seem to find a lot of the content, eg. "This 3-Piece Set brings together..."
Please advise. Thank you!
Hi @samuelsander,
Thank you for reaching out!
It seems the default HTML transformer (Readable text) might be removing some of the content you need. To capture all the content, try the following:
- Go to the
HTML processing
input category. - Change the
HTML transformer
option fromReadable text
toNone
.
This should help keep all the necessary content (see my example run at https://console.apify.com/view/runs/N4OWuajgeo0n0NRMP).
I'll close this issue now, but feel free to let us know if you need further assistance. Cheers!
- 2.8k monthly users
- 434 stars
- 99.9% runs succeeded
- 2.9 days response time
- Created in Mar 2023
- Modified 3 days ago