3 days trial then $10.00/month - No credit card required now

Hacker News Data Scraper

epctex/hackernews-scraper

3 days trial then $10.00/month - No credit card required now

Extract Y Combinator's Hacker News based on any search criteria. Crawl the front page, Show HN, Ask HN, news, job listings, and historical data. Get links, titles, comments, ratings, and more!

All issues Create new issue

We got blocked

Closed

bravura opened this issue

What is the best way to avoid this error:

2023-06-20T08:12:38.109Z ERROR BasicCrawler: Request failed and reached maximum retries. Error: We got blocked 2023-06-20T08:12:38.111Z at BasicCrawler.requestHandler (file:///home/myuser/src/main.js:49:19) 2023-06-20T08:12:38.113Z at process.processTicksAndRejections (node:internal/process/task_queues:95:5) 2023-06-20T08:12:38.116Z at async BasicCrawler._runRequestHandler (/home/myuser/node_modules/@crawlee/basic/internals/basic-crawler.js:572:9) 2023-06-20T08:12:38.118Z at async wrap (/home/myuser/node_modules/@apify/timeout/index.js:52:21) {"id":"7VnKzRXCQ8psNG9","url":"https://news.ycombinator.com/item?id=33178562","method":"GET","uniqueKey":"https://news.ycombinator.com/item?id=33178562"} 2

beremekdar

Hello,

Thank you very much for reaching out. Can you please share your Run ID so that we can investigate what went wrong there? Then we can immediately reach out to you about the solutions.

Best

bravura

See this issue: https://console.apify.com/actors/jv1G7XhUz1R34fV2L/issues/De5KmFdg84cg9GMKU/comments

Add comment

Developer

epctex

Actor metrics

3 monthly users
100.0% runs succeeded
0.0 days response time
Created in Mar 2021
Modified about 15 hours ago

Categories

News

Business

Other

Google Maps Scraper

compass/crawler-google-places

Extract data from hundreds of Google Maps locations and businesses. Get Google Maps data including reviews, images, contact info, opening hours, location, popular times, prices & more. Export scraped data, run the scraper via API, schedule and monitor runs, or integrate with other tools.

Compass

62.4k

Website Content Crawler

apify/website-content-crawler

Automatically crawl and extract text content from websites with documentation, knowledge bases, help centers, or blogs. This Actor is designed to provide data to feed, fine-tune, or train large language models such as ChatGPT or LLaMA.

Apify

12.9k

Twitter Scraper

quacker/twitter-scraper

Scrape tweets from any Twitter user profile. Top Twitter API alternative to scrape Twitter hashtags, threads, replies, followers, images, videos, statistics, and Twitter history. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Quacker

24.3k

GPT Scraper

drobnikj/gpt-scraper

Extract data from any website and feed it into GPT via the OpenAI API. Use ChatGPT to proofread content, analyze sentiment, summarize reviews, extract contact details, and much more.

Jakub Drobník

4.2k

AI Product Matcher

equidem/ai-product-matcher

Match products across multiple e-commerce websites. Use this AI product matching Actor whenever you need to find matching pairs of products from different online shops for dynamic pricing, competitor analysis or market research.

Matěj Sochor

308

Facebook Ads Scraper

apify/facebook-ads-scraper

Extract advertising data from one or multiple Facebook Pages. Get page details, reach estimates, publisher platforms, report count, number of impressions, ad IDs, timestamps, and more. Download Facebook ads data in JSON, CSV, and Excel and use it in apps, spreadsheets, and reports.

Apify

3.9k

Google Trends Scraper

emastra/google-trends-scraper

Scrape data from Google Trends by search terms or URLs. Specify locations, define time ranges, select categories to get interest by subregion and over time, related queries and topics, and more. Export scraped data, run the scraper via API, schedule and monitor runs, or integrate with other tools.

Emiliano Mastragostino

3.2k

📩📍 Google Maps Email Extractor

lukaskrivka/google-maps-with-contact-details

Extract Google Maps contact details. Scrape websites of Google Maps places for contact details and get email addresses, website, location, address, zipcode, phone number, social media links. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Lukáš Křivka

3.2k

Twitter URL Scraper

quacker/twitter-url-scraper

Copy any Twitter URL and extract Twitter usernames, profile photos, follower count, tweets, hashtags, favorite count, and more. Export scraped datasets, run the scraper via API, schedule and monitor runs or integrate with other tools.

Quacker

4.3k

Smart Article Extractor

lukaskrivka/article-extractor-smart

📰 Smart Article Extractor extracts articles from any scientific, academic, or news website with just one click. The extractor crawls the whole website and automatically distinguishes articles from other web pages. Download your data as HTML table, JSON, Excel, RSS feed, and more.

Lukáš Křivka

3.3k