Website Content Crawler
No credit card required
Website Content Crawler
No credit card required
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
Do you want to learn more about this Actor?
Get a demoThe subpages are not crawled
Closed
I can't identify the reason why subpages are not crawled for instance the page https://www.royalambassadorbh.com/apartment/studio-apartment-for-sale-seef-avenue
Hi, thank you for using the Website Content Crawler!
It seems there’s a small typo in your globs settings.
You need to specify includeUrlGlobs as follows:
1"includeUrlGlobs": [ 2 { 3 "glob": "https://www.royalambassadorbh.com/ar/apartment/*/**for-sale**" 4 } 5]
Additionally, make sure to enable the Consider URLs from sitemaps option. Otherwise, these for-sale pages won’t be discovered if only the home page is being crawled.
I hope this helps! I’m going to close this issue now, but feel free to ask another question or raise a new issue if needed.
ApifyActor Metrics
3.9k monthly users
-
718 stars
>99% runs succeeded
2.2 days response time
Created in Mar 2023
Modified 16 hours ago
