No credit card required

Website Content Crawler

apify/website-content-crawler

No credit card required

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗LangChain, LlamaIndex, and the wider LLM ecosystem.

Back to issues Create new issue

bug: Actor gets stuck on large sitemaps

Open

MavenAGI opened this issue

Based on the logs from this actor's run it looks like it was migrated from host to host without much else happening. It would be helpful to know what the issue was here and if there is a way for us to avoid it in the future.

We were also charged for quite a bit of usage without anything to show for it so perhaps we could get some of that credited back?

Jindřich Bär (jindrich.bar)

Hello and thank you for your interest in this Actor!

This seems like an internal error in the Actor. We are working on the fix (we already have it prepared, now we just need to test it properly). I'll let our support team know and they'll look into reimbursing your expenses promptly.

I'll also keep this issue open to track the underlying issue - and will keep you posted once this gets fixed.

Thank you - and sorry for the inconvenience.

Jindřich Bär (jindrich.bar)

Just so you know - our Support team just reimbursed you the lost credits as a credit offset.

The issue is still there (i.e. running the Actor now will likely result in the same outcome) - we'll keep you posted once that changes.

Cheers!

MavenAGI

Thanks for the prompt reply and wonderful support!

MavenAGI

We just hit this issue again on another run[1]. Is there something we can check for (programmatically) to determine if we're running into this bug? Most of our runs complete without a problem so the error case is definitely an outlier.

[1] https://console.apify.com/organization/5WhuE8XiPsnLiYsmv/actors/aYG0l9s7dbB7j3gbS/runs/vMRA3ElqRIaV3TQ4x#log

Jindřich Bär (jindrich.bar)

Hello again,

we're sorry to hear that you've run into problems with this Actor again.

You can set up monitoring alerts on the Alerts tab - here, you can e.g. have Apify send you an email if the run duration exceeds a given limit. To further mitigate the credit loss, you can also set the Actor timeout lower (see Run options > Timeout) - after checking your previous runs, I found out all of them (that weren't failing) finished under 41000 seconds (cca 11.5 hours). The current timeout limit of 604,800 seconds might therefore be too excessive and can be lowered.

Thank you again for bringing this up and sorry for the inconvenience.

Add comment

Developer

Apify

Actor metrics

2.8k monthly users
317 stars
100.0% runs succeeded
4 days response time
Created in Mar 2023
Modified 1 day ago

Categories

Developer tools

Business

AI Content Markdown Scraper

quaking_pail/ai-content-markdown-scraper

This Apify Actor, "Website Content Crawler with Markdown Extraction," is designed to perform a comprehensive crawl of specified websites, extract their text content, convert it into Markdown format, and store it in a structured dataset. The extracted content is suitable for feeding LLMs.

AI_Builder

Website Screenshot Generator

apify/screenshot-url

Create a screenshot of a website based on a specified URL. The screenshot is stored as the output in a key-value store. It can be used to monitor web changes regularly after setting up the scheduler.

Apify

2.5k

Pinecone Integration

apify/pinecone-integration

This integration transfers data from Apify Actors to a Pinecone and is a good starting point for a question-answering, search, or RAG use case.

Apify

OpenAI Vector Store Integration

jiri.spilka/openai-vector-store-integration

The Apify OpenAI Vector Store integration uploads data from Apify Actors to the OpenAI Vector Store linked to OpenAI Assistant.

Jiří Spilka

Qdrant Integration

apify/qdrant-integration

Transfer data from Apify Actors to a Qdrant vector database.

Apify

Pinecone GPT Chatbot

tri_angle/pinecone-gpt-chatbot

Pinecone GPT Chatbot combines OpenAI's GPT models with Pinecone's database to generate insightful responses. Its interactive chatbot interface presents precise and comprehensive answers to user queries. Benefit from semantic understanding, efficient workflows, and enriched knowledge integration!

Tri⟁angle

WCC Pinecone Integration

tri_angle/wcc-pinecone-integration

Crawl any website and store its content in your Pinecone vector database. Enhance the accuracy and reliability of your own AI Assistant with facts fetched from external sources or connect this integration to our Pinecone GPT Chatbot assistant available in Apify Store.

Tri⟁angle

Website Content Vector Retriever

hamza.alwan/website-content-vector-retriever

Hamza Alwan

Neo4j Graph Creator

orange_knife-owner/neo4j

Create a Knowledge Graph of any Website! Knowledge graphs give you a good understanding of how a website is structured and can be used for RAG applications with the new hot field of GraphRAG!