Website Content Crawler avatar
Website Content Crawler

Pricing

Pay per usage

Go to Store
Website Content Crawler

Website Content Crawler

Developed by

Apify

Apify

Maintained by Apify

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.

3.7 (41)

Pricing

Pay per usage

1501

Total users

58K

Monthly users

8.1K

Runs succeeded

>99%

Issues response

7.6 days

Last modified

3 hours ago

FQ

No text parsed from from webpage.

Closed

formidable_quagmire opened this issue
a month ago

Even though text is there on url for some of the url It's coming as a empty. even after removing css selector we are getting it. and the htmlSelector is none as well

https://diariodarepublica.pt/dr/detalhe/despacho/5942-2025-919521011

jindrich.bar avatar

Hello, and thank you for your interest in this Actor!

This website seems to load the content dynamically, a few seconds after the initial page load. The Actor captures this state of the webpage (i.e. while it's still blank) and stores this in the Dataset.

You can tell the Actor to wait for a specific CSS selector by setting the Wait for selector input option. This way, the Actor will wait for the specified selector for Wait for dynamic content seconds. If the selector doesn't appear in the given timespan, the request processing will fail, and the request will get retried.

See my example run where I set the Wait for selector option to .content: Link to run.

I'll close this issue now, but feel free to ask for additional questions if you have any. Cheers!