No credit card required

Website Content Crawler

apify/website-content-crawler

No credit card required

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗LangChain, LlamaIndex, and the wider LLM ecosystem.

Back to issues Create new issue

Crawling does not work for custom sitemaps

Closed

sai_sampath opened this issue

I want the Apify to take in and use custom sitemaps. We have a use case for this. For example, Take this sitemap: https://customgpt-streamlit.s3.amazonaws.com/customgpt-streamlit/f1357fcd-a8cb-4498-a641-ec541a999fbc.xml

It's failing to crawl on it, I've tried multiple actors and different settings like sitemaps on/not. It's still not working.

These are the runs:

RUN 1: https://console.apify.com/actors/runs/BVzk3hXVg4bC8HNh2

RUN 2: https://console.apify.com/actors/runs/KFJmq9MqQKPRMHn2c

RUN 3: https://console.apify.com/actors/runs/EBjTGlHXS2CDIzzW1

RUN 4: https://console.apify.com/actors/runs/MLrPsygiOhTnyaKnZ

Can you please look into this?

unified_antechamber

Jindřich Bär (jindrich.bar)

Hello and thank you for your interest in this Actor!

We're sorry about the inconvenience caused by this issue. Currently, WCC only supports automatic scanning for sitemaps (by reading the robots.txt on a given domain and downloading the sitemap automatically). Passing the actual sitemap file as a start URL causes issues, as the start URLs are only expected to be valid web pages.

Our team will look into this use case and we'll try to come up with a way of supporting this.

Thank you for your patience, I'll get back to you once we figure out the best way about this. Cheers!

sai_sampath

Thank you for the response, Please try to get this done as this is one of our major use cases. Hope to hear fruitful news from you!

Vlad Frangu (vladfrangu_apify)

Hey! Following up from above, we've detected that this should have worked (passing in a sitemap xml with the Use sitemaps option turned on), and the fact it doesn't is a bug. We will follow-up once we release the bug fix for this! :D

Thanks a bunch for reporting this!

sai_sampath

Thank you for updating this, Will be looking forward for good news soon!

Jindřich Bär (jindrich.bar)

We've finally released a patch that fixes this issue. When processing a URL leading to a sitemap, the crawler now processes the sitemap as a list of links to enqueue.

I'll close this issue now, but make sure to let us know if anything breaks for you.

Thank you for your patience!

Add comment

Developer

Apify

Actor metrics

2.8k monthly users
317 stars
100.0% runs succeeded
4 days response time
Created in Mar 2023
Modified 1 day ago

Categories

Developer tools

Business

AI Content Markdown Scraper

quaking_pail/ai-content-markdown-scraper

This Apify Actor, "Website Content Crawler with Markdown Extraction," is designed to perform a comprehensive crawl of specified websites, extract their text content, convert it into Markdown format, and store it in a structured dataset. The extracted content is suitable for feeding LLMs.

AI_Builder

Website Screenshot Generator

apify/screenshot-url

Create a screenshot of a website based on a specified URL. The screenshot is stored as the output in a key-value store. It can be used to monitor web changes regularly after setting up the scheduler.

Apify

2.5k

Pinecone Integration

apify/pinecone-integration

This integration transfers data from Apify Actors to a Pinecone and is a good starting point for a question-answering, search, or RAG use case.

Apify

OpenAI Vector Store Integration

jiri.spilka/openai-vector-store-integration

The Apify OpenAI Vector Store integration uploads data from Apify Actors to the OpenAI Vector Store linked to OpenAI Assistant.

Jiří Spilka

Qdrant Integration

apify/qdrant-integration

Transfer data from Apify Actors to a Qdrant vector database.

Apify

Pinecone GPT Chatbot

tri_angle/pinecone-gpt-chatbot

Pinecone GPT Chatbot combines OpenAI's GPT models with Pinecone's database to generate insightful responses. Its interactive chatbot interface presents precise and comprehensive answers to user queries. Benefit from semantic understanding, efficient workflows, and enriched knowledge integration!

Tri⟁angle

WCC Pinecone Integration

tri_angle/wcc-pinecone-integration

Crawl any website and store its content in your Pinecone vector database. Enhance the accuracy and reliability of your own AI Assistant with facts fetched from external sources or connect this integration to our Pinecone GPT Chatbot assistant available in Apify Store.

Tri⟁angle

Website Content Vector Retriever

hamza.alwan/website-content-vector-retriever

Hamza Alwan

Neo4j Graph Creator

orange_knife-owner/neo4j

Create a Knowledge Graph of any Website! Knowledge graphs give you a good understanding of how a website is structured and can be used for RAG applications with the new hot field of GraphRAG!