Pay $2.00 for 1,000 results

Deep Website Content Crawler

6sigmag/deep-website-content-crawler

Pay $2.00 for 1,000 results

Scrape Failed Killer! A high-performance web scraper that rapidly extracts and analyzes content from multiple websites simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

Back to issues Create new issue

If one of websites hangs, the whole task fails

Open

Nikita Sviridenko (nikita-sviridenko) opened this issue

Could you please add a per-website timeout? And an error column so that we know which of them will fail?

This way other websites will be fetched successfully.

Nikita Sviridenko (nikita-sviridenko)

F.e. from this list 7 websites were imported and then it timed out after 5 min:

1{
2  "startUrls": [
3    "pearl-executivesearch.com",
4    "e-epitech.com",
5    "taylormaderecrutement.com",
6    "carrhure.com",
7    "focus-recrutement.fr",
8    "vauban-recrutement.fr",
9    "beager.com",
10    "job-tourisme.fr",
11    "eurojob-consulting.com",
12    "headhuntingfactory.com",
13    "pentabell.com",
14    "la-releve.com",
15    "haussmann-es.com",
16    "prismemploi.eu",
17    "briks.fr",
18    "88jobs.com",
19    "archibat.com",
20    "solantis.fr",
21    "selescope.com",
22    "ltd-international.com",
23    "sintel-recrutement.fr",
24    "macanders.eu",
25    "jobmania.fr",
26    "quadra-consultants.com",
27    "recruter.tn",
28    "grantalexander.com",
29    "fedlegalimports.com",
30    "asap.work",
31    "synapse-executive.com",
32    "dlsi-emploi.com",
33    "easypartner.fr",
34    "vendome-associes.com",
35    "now-consulting.fr",
36    "recrutop.com",
37    "myrecrutement.eu",
38    "skillwise.fr",
39    "ccld.com",
40    "alerys.fr",
41    "talysio.fr",
42    "gif-emploi.fr",
43    "gif-emploi.fr",
44    "tasterh.fr",
45    "axxis-interimetrecrutement.com",
46    "abg.asso.fr",
47    "lynkus.fr",
48    "morganmallet.agency",
49    "highdev.com",
50    "achil.io",
51    "fedafrica.com",
52    "aperlead.com"
53  ]
54}

Nikita Sviridenko (nikita-sviridenko)

It'll be also helpful to see in the logs which websites are already scraped.

Add comment

Developer

David Deng

Actor Metrics

52 monthly users
6 stars
>99% runs succeeded
33 days response time
Created in Oct 2024
Modified 2 months ago

Categories

Other

Lead generation

Website Content Crawler

apify/website-content-crawler

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.

Apify

30.6k

814

Amazon Reviews Scraper

junglee/amazon-reviews-scraper

Amazon scraper to extract reviews from Amazon products. Scrape and download detailed reviews without using the Amazon API, including rating score, review description, reactions and images. Download your data as HTML table, JSON, CSV, Excel, XML.

Junglee

3.4k

Amazon Product Scraper

junglee/Amazon-crawler

Use this Amazon scraper to collect data based on URL and country from the Amazon website. Extract product information without using the Amazon API, including reviews, prices, descriptions, and Amazon Standard Identification Numbers (ASINs). Download data in various structured formats.

Junglee

7.6k

100

Instagram Scraper

apify/instagram-scraper

Scrape and download Instagram posts, profiles, places, hashtags, photos, and comments. Get data from Instagram using one or more Instagram URLs or search queries. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Apify

65k

690

Google Maps Reviews Scraper

compass/Google-Maps-Reviews-Scraper

Extract all reviews of Google Maps places using place URLs. Get review text, published date, response from owner, review URL, and reviewer's details. Download scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Compass

117

Tripadvisor Scraper

maxcopell/tripadvisor

This unofficial Tripadvisor API is a data extraction tool able to get data on hotels, restaurants, things to do, vacation rentals, attractions, tours, and public trips. Get pricing, contact details, amenities, awards, ratings, and more. Download your data in Excel, JSON, CSV, and other formats.

Maximillian Copelli

6.6k

Amazon ASINs Scraper

junglee/amazon-asins-scraper

Gets you product data from Amazon. Unofficial API. Scrapes and downloads product information without using the Amazon API, including reviews, prices, descriptions.

Junglee

701

Amazon Scraper

junglee/free-amazon-product-scraper

Gets you product data from Amazon. Unofficial API. Scrapes and downloads product information without using the Amazon API, including reviews, prices, descriptions, and ASIN.

Junglee

4.2k

📩📍 Google Maps Email Extractor

lukaskrivka/google-maps-with-contact-details

Extract Google Maps contact details. Scrape websites of Google Maps places for contact details and get email addresses, website, location, address, zipcode, phone number, social media links. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Lukáš Křivka

10.1k

287

Facebook Posts Scraper

apify/facebook-posts-scraper

Extract data from hundreds of Facebook posts from one or multiple Facebook pages and profiles. Get post URL, post text, page or profile URL, timestamp, number of likes, shares, comments, and more. Download the data in JSON, CSV, and Excel and use it in apps, spreadsheets, and reports.

Apify

16.4k

231