Deep Website Content Crawler avatar
Deep Website Content Crawler

Pricing

$5.00 / 1,000 results

Go to Store
Deep Website Content Crawler

Deep Website Content Crawler

Developed by

David Deng

David Deng

Maintained by Community

Scrape Failed Killer! A high-performance web scraper that rapidly extracts and analyzes content from multiple websites simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

3.0 (1)

Pricing

$5.00 / 1,000 results

10

Total users

285

Monthly users

60

Runs succeeded

>99%

Last modified

14 days ago

nikita-sviridenko avatar

If one of websites hangs, the whole task fails

Open

Nikita Sviridenko (nikita-sviridenko) opened this issue
5 months ago

Could you please add a per-website timeout? And an error column so that we know which of them will fail?

This way other websites will be fetched successfully.

nikita-sviridenko avatar

F.e. from this list 7 websites were imported and then it timed out after 5 min:

{
"startUrls": [
"pearl-executivesearch.com",
"e-epitech.com",
"taylormaderecrutement.com",
"carrhure.com",
"focus-recrutement.fr",
"vauban-recrutement.fr",
"beager.com",
"job-tourisme.fr",
"eurojob-consulting.com",
"headhuntingfactory.com",
"pentabell.com",
"la-releve.com",
"haussmann-es.com",
"prismemploi.eu",
"briks.fr",
"88jobs.com",
"archibat.com",
"solantis.fr",
"selescope.com",
"ltd-international.com",
"sintel-recrutement.fr",
"macanders.eu",
"jobmania.fr",
"quadra-consultants.com",
"recruter.tn",
"grantalexander.com",
"fedlegalimports.com",
"asap.work",
"synapse-executive.com",
"dlsi-emploi.com",
"easypartner.fr",
"vendome-associes.com",
"now-consulting.fr",
"recrutop.com",
"myrecrutement.eu",
"skillwise.fr",
"ccld.com",
"alerys.fr",
"talysio.fr",
"gif-emploi.fr",
"gif-emploi.fr",
"tasterh.fr",
"axxis-interimetrecrutement.com",
"abg.asso.fr",
"lynkus.fr",
"morganmallet.agency",
"highdev.com",
"achil.io",
"fedafrica.com",
"aperlead.com"
]
}
nikita-sviridenko avatar

It'll be also helpful to see in the logs which websites are already scraped.