Pricing

Pay per usage

Go to Store

Algolia Webcrawler

Try for free

Developed by

Jan Čurn

Crawls a website using one or more sitemaps and imports the data to Algolia search index. The text content is identified using simple CSS selectors.

0.0 (0)

Pricing

Pay per usage

Monthly users

Last modified

4 years ago

Other

Open source

Back to issues Create new issue

Module not found

Open

Toolbase opened this issue

2023-11-12T19:49:48.889Z Exiting the process with code 34 2023-11-12T19:49:48.890Z npm ERR! code ELIFECYCLE 2023-11-12T19:49:48.891Z npm ERR! errno 34 2023-11-12T19:49:48.891Z npm ERR! my-actor@0.0.1 start: node main.js 2023-11-12T19:49:48.892Z npm ERR! Exit status 34 2023-11-12T19:49:48.892Z npm ERR! 2023-11-12T19:49:48.893Z npm ERR! Failed at the my-actor@0.0.1 start script. 2023-11-12T19:49:48.893Z npm ERR! This is probably not a problem with npm. There is likely additional logging output above. 2023-11-12T19:49:48.894Z 2023-11-12T19:49:48.894Z npm ERR! A complete log of this run can be found in: 2023-11-12T19:49:48.895Z npm ERR! /root/.npm/_logs/2023-11-12T19_49_48_855Z-debug.log

Add comment

Pricing

Pricing model

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage.

Algolia Website Indexer

apify/algolia-website-indexer

The Indexer crawls recursively a website using the Puppeteer browser (headless Chrome) and indexes the selected pages to the Algolia index.

Apify

Cheerio Scraper

apify/cheerio-scraper

Crawls websites using raw HTTP requests, parses the HTML with the Cheerio library, and extracts data from the pages using a Node.js code. Supports both recursive crawling and lists of URLs. This actor is a high-performance alternative to apify/web-scraper for websites that do not require JavaScript.

Apify

7.2k

Website extract

mrahil/my-actor

It is website extractor

Mohammed Rahil

Website Content Extractor

fastidious_drawer/website-content-extractor

This extractor lets you extract content from any website with a single or multiple URLs. Use selectors to choose specific sections like the body and exclude elements like headers or navigation. It also extracts images and links, providing data in JSON and DataTable formats for easy processing.

fastidious_drawer

Website Content Crawler

apify/website-content-crawler

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.

Apify

45.5k

Get URLs from link

boring_code/get-urls-from-link

Extracts URLs from a sitemap or webpage with intuitive path matching. Use comma-separated patterns to include or exclude URL paths with smart matching: '/tags/' for exact paths, '/product' for paths starting with, or simple text for substring matches.

Audrius L.

Web Scraper

apify/web-scraper

Crawls arbitrary websites using a web browser and extracts structured data from web pages using a provided JavaScript function. The Actor supports both recursive crawling and lists of URLs, and automatically manages concurrency for maximum performance.

Apify

80.7k

Text Scraper (Free)

karamelo/text-scraper-free

Website Text Extractor. Extract Text from Webpages and Feed Your LLMs

karamelo

445

AI Website Content Markdown Scraper

quaking_pail/ai-website-content-markdown-scraper

This Apify Actor, "Website Content Crawler with Markdown Extraction," is designed to perform a comprehensive crawl of specified websites, extract their text content, convert it into Markdown format, and store it in a structured dataset. The extracted content is suitable for feeding LLMs.

AI_Builder

489

AI-Powered Web Content & Link Extractor

scrapercoder/ai-powered-web-content-link-extractor

Crawls websites to extract clean, structured content for AI/LLM use, ideal for training datasets, knowledge bases, and RAG systems. Json output includes: * text: Normalized page content * links: Extracted sub-URLs