No credit card required

Website Content Crawler

apify/website-content-crawler

No credit card required

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗LangChain, LlamaIndex, and the wider LLM ecosystem.

165

147

Website Screenshot

Opened 8 days ago by athletic_trophy, last comment 4 days ago

I get lots of: ERROR AdaptiveCrawler: Request failed and reached maximum retries. Error: Attribute selector didn't terminate

Opened 13 days ago by upright_yoga, last comment 9 days ago

this is very good i will use it found my likes

Opened 14 days ago by communicative_computer, last comment 14 days ago

The actor replace any URL including (%) with (%25) which broke the actual link.

Opened 15 days ago by The25th, last comment 14 days ago

Invalid regular expression: /^https?://(www.)?kupbilecik.pl/imprezy/132726/Wroc%C5%82aw/Mamma+Mia++-+The+best+of+ABBA[/#?](?!.*[.](pdf|doc|docx|xls|xlsx)([?#].*)?$).*$/: Nothing to repeat

Opened 16 days ago by Yulia_Archipova, last comment 15 days ago

Content incorrectly removed from page

Opened 16 days ago by civic-roundtable, last comment 15 days ago

Scrape files that don't originate from start URLs

Opened 16 days ago by civic-roundtable, last comment 16 days ago

Timeout = 60, Crawler duration = forever (aborted at 36 minutes)

Opened 18 days ago by Lovgren, last comment 18 days ago

Does not remove cookies

Opened 23 days ago by tituskex, last comment 15 days ago

Full Runs' history

Opened 25 days ago by sestek, last comment 23 days ago

Not crawling all content

Opened a month ago by samuelsander, last comment 23 days ago

Actor sometimes grabs only one screenshot or repeated 'mismatched' screenshots in output

Opened a month ago by jjohnson-dev, last comment a month ago

Actor runs timing out after message about migration to another host

Opened a month ago by jjohnson-dev, last comment a month ago

Crawling does not work with some specific custom sitemaps

Opened a month ago by sai_sampath, last comment a month ago

Integrating with Datafetcher || Run Actor without "startUrls" ?

Opened a month ago by lofty_world, last comment 23 days ago

Actor stalled discovering sitemaps

Opened a month ago by MavenAGI, last comment a month ago

Basic Auth in Header not working

Opened a month ago by cirez_d, last comment a month ago

The Actor stopped working from us since it upgraded to 0.3.36

Opened 2 months ago by jorge_rauda, last comment 2 months ago

Crawler misses text content

Opened 2 months ago by innovum_admin, last comment 2 months ago

Crawling is taking lot of time because of some exception and credits are getting exhausted

Opened 2 months ago by atomicwork, last comment 2 months ago

Developer

Apify

Actor metrics

2.8k monthly users
317 stars
100.0% runs succeeded
4 days response time
Created in Mar 2023
Modified 1 day ago

Categories

Developer tools

Business

AI Content Markdown Scraper

quaking_pail/ai-content-markdown-scraper

This Apify Actor, "Website Content Crawler with Markdown Extraction," is designed to perform a comprehensive crawl of specified websites, extract their text content, convert it into Markdown format, and store it in a structured dataset. The extracted content is suitable for feeding LLMs.

AI_Builder

Website Screenshot Generator

apify/screenshot-url

Create a screenshot of a website based on a specified URL. The screenshot is stored as the output in a key-value store. It can be used to monitor web changes regularly after setting up the scheduler.

Apify

2.5k

Pinecone Integration

apify/pinecone-integration

This integration transfers data from Apify Actors to a Pinecone and is a good starting point for a question-answering, search, or RAG use case.

Apify

OpenAI Vector Store Integration

jiri.spilka/openai-vector-store-integration

The Apify OpenAI Vector Store integration uploads data from Apify Actors to the OpenAI Vector Store linked to OpenAI Assistant.

Jiří Spilka

Qdrant Integration

apify/qdrant-integration

Transfer data from Apify Actors to a Qdrant vector database.

Apify

Pinecone GPT Chatbot

tri_angle/pinecone-gpt-chatbot

Pinecone GPT Chatbot combines OpenAI's GPT models with Pinecone's database to generate insightful responses. Its interactive chatbot interface presents precise and comprehensive answers to user queries. Benefit from semantic understanding, efficient workflows, and enriched knowledge integration!

Tri⟁angle

WCC Pinecone Integration

tri_angle/wcc-pinecone-integration

Crawl any website and store its content in your Pinecone vector database. Enhance the accuracy and reliability of your own AI Assistant with facts fetched from external sources or connect this integration to our Pinecone GPT Chatbot assistant available in Apify Store.

Tri⟁angle

Website Content Vector Retriever

hamza.alwan/website-content-vector-retriever

Hamza Alwan

Neo4j Graph Creator

orange_knife-owner/neo4j

Create a Knowledge Graph of any Website! Knowledge graphs give you a good understanding of how a website is structured and can be used for RAG applications with the new hot field of GraphRAG!