Website Content Crawler avatar

Website Content Crawler

Try for free

No credit card required

Go to Store
Website Content Crawler

Website Content Crawler

apify/website-content-crawler
Try for free

No credit card required

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.

Do you want to learn more about this Actor?

Get a demo
Hung crawl

Opened 14 hours ago by mcantrell-owner, last comment 10 hours ago by Jiří Spilka (jiri.spilka)

Poor results

Opened 4 days ago by Digital_Mole, last comment 9 hours ago by Jiří Spilka (jiri.spilka)

Integration with openai’s assistant through their functions calling tool

Opened 5 days ago by Cirula, last comment 5 days ago by Jiří Spilka (jiri.spilka)

"Run actor synchronously with input and get dataset items" supported?

Opened 6 days ago by Kilian, last comment 6 days ago by Dušan Vystrčil (dusan.vystrcil)

Blocked by cookies

Opened 7 days ago by andrewtshannon, last comment 6 days ago by andrewtshannon

Not getting PDF Files

Opened 15 days ago by laius, last comment 15 days ago by Jiří Spilka (jiri.spilka)

Website crawler less outputs than requests

Opened 16 days ago by codewithcheese.com, last comment 15 days ago by Jiří Spilka (jiri.spilka)

I can't "get output"

Opened 20 days ago by nicco.teodori, last comment 19 days ago by Jiří Spilka (jiri.spilka)

crawled a website and it just returns messages about cookies, not the text of the site

Opened 23 days ago by jhonovich, last comment 20 days ago by Jiří Spilka (jiri.spilka)

The actor is trying to proceed too many requests

Opened 25 days ago by capola, last comment 22 days ago by Jiří Spilka (jiri.spilka)

I am having trouble putting in Cookies as input.

Opened a month ago by chirag_xcvii, last comment 23 days ago by Jiří Spilka (jiri.spilka)

Input url's path is re-encoded when it contains specific characters, breaking the url

Opened a month ago by iadvize, last comment a month ago by iadvize

Configuring the RAM limits

Opened a month ago by Robmobius, last comment a month ago by Dušan Vystrčil (dusan.vystrcil)

Why does this job take so long?

Opened a month ago by grossjo, last comment a month ago by Dušan Vystrčil (dusan.vystrcil)

Strictly Necessary cookies are absolutely necessary for core functions such as navigating the page or accessing secure areas.

Opened a month ago by lei.niu.bigco, last comment a month ago by Dušan Vystrčil (dusan.vystrcil)

Can't get globs to work

Opened a month ago by matt333, last comment a month ago by matt333

set the depth to 0

Opened a month ago by environmental_mammal, last comment a month ago by Jiří Spilka (jiri.spilka)

Scrapping only few elements on the page and save them in the separate fields

Opened a month ago by igocza, last comment a month ago by igocza

not the whole content is scraped

Opened a month ago by igocza, last comment a month ago by Jiří Spilka (jiri.spilka)

can't crawl the whole website

Opened a month ago by visable, last comment a month ago by Jiří Spilka (jiri.spilka)

Developer
Maintained by Apify

Actor Metrics

  • 4k monthly users

  • 839 stars

  • >99% runs succeeded

  • 1 days response time

  • Created in Mar 2023

  • Modified 17 hours ago