No credit card required

Smart Article Extractor

lukaskrivka/article-extractor-smart

No credit card required

📰 Smart Article Extractor extracts articles from any scientific, academic, or news website with just one click. The extractor crawls the whole website and automatically distinguishes articles from other web pages. Download your data as HTML table, JSON, Excel, RSS feed, and more.

Do you want to learn more about this Actor?

Get a demo

Back to issues Create new issue

Timeout and Storage check

Open

hectorlca opened this issue

Could you please help understand how to pass timeout parameter using API? I never see it reflected, even when using the web interface. Additionally, I believe it occasionally fails to check the articles that are already stored.

i have used this actor for many months now, and in the recent weeks, some of my runs have been running indefinitely. I don-t know what happened, but I didn't have to monitor every day because the results were as expected. Now I've had some terrible experiences with the actor running infinitely, costing me loads of money.

Edit: these two runs have identical input:

https://console.apify.com/actors/runs/TrPjXGgDm5PHNiM48#output - 3,099 results. (Not desired)
https://console.apify.com/actors/runs/SJcFnVa96s3OFZy2w#output - 66 results. (Desired).

Ondrej Klinovský (ondrejklinovsky)

Hey,

when starting a new run through API, you can define the timeout with query parameter. For example, to start a run with timeout 60 seconds: POST https://api.apify.com/v2/acts/<ACTOR>/runs?timeout=60. Here's the docs.

The issue with the run is that the website changed the urls of the articles - they added ?amp query parameter. This caused that the actor scraped them again because their URL was different from those there were already stored. We'll need to figure out how to avoid situatoins like this. We cannot ignore the query parameters completely because they may define the article (e.g. ?articleId=edasdas), so this will require more thinking. Thank you for the report, we'll let you know when we have any updates on this. Let me know if you have any questions.

Add comment

Developer

Lukáš Křivka

Actor Metrics

261 monthly users
78 stars
>99% runs succeeded
2.3 days response time
Created in Nov 2019
Modified a month ago

Categories

News

Website Content Crawler

apify/website-content-crawler

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.

Apify

31.4k

837

Cheerio Scraper

apify/cheerio-scraper

Crawls websites using raw HTTP requests, parses the HTML with the Cheerio library, and extracts data from the pages using a Node.js code. Supports both recursive crawling and lists of URLs. This actor is a high-performance alternative to apify/web-scraper for websites that do not require JavaScript.

Apify

Web Scraper

apify/web-scraper

Crawls arbitrary websites using the Chrome browser and extracts data from pages using JavaScript code. The Actor supports both recursive crawling and lists of URLs and automatically manages concurrency for maximum performance. This is Apify's basic tool for web crawling and scraping.

Apify

73.5k

323

Tripadvisor Scraper

maxcopell/tripadvisor

This unofficial Tripadvisor API is a data extraction tool able to get data on hotels, restaurants, things to do, vacation rentals, attractions, tours, and public trips. Get pricing, contact details, amenities, awards, ratings, and more. Download your data in Excel, JSON, CSV, and other formats.

Maximillian Copelli

6.7k

Genius Scraper

epctex/genius-scraper

The Genius.com scraper extracts songs and lyrics from the platform. It fetches details from song, album, artist pages, supports tag-based retrieval, searches, and includes comments. Simplify accessing and analyzing music content from Genius.com.

epctex

LinkedIn Job Postings Scraper

eunit/linkedin-job-postings-scraper

Designed for both personal and professional use, simply enter your job search keywords and location to extract tailored job titles, companies names, locations, job URL and more. This scraper handles infinite scrolling and exports data to any format such CSV, JSON, XML and more. Try it today 🚀

Emmanuel Uchenna

Google Search Results Scraper

apify/google-search-scraper

Scrape Google Search Engine Results Pages (SERPs). Select the country or language and extract organic and paid results, AI overviews, ads, queries, People Also Ask, prices, reviews, like a Google SERP API. Export scraped data, run the scraper via API, schedule runs, or integrate with other tools.

Apify

52k

287

Google Trends Scraper

emastra/google-trends-scraper

Scrape data from Google Trends by search terms or URLs. Specify locations, define time ranges, select categories to get interest by subregion and over time, related queries and topics, and more. Export scraped data, run the scraper via API, schedule and monitor runs, or integrate with other tools.

Emiliano Mastragostino

3.9k

Twitter Tweets and Profiles Scraper

web.harvester/twitter-scraper

Easily search and extract tweets from profiles or directly using a URL with our tool. Scrape tweets and replies effortlessly. Download data in formats like JSON, CSV, XML, RSS, or HTML Table, perfect for integration with various applications, databases, and social media analytics tools.