No credit card required

Smart Article Extractor

lukaskrivka/article-extractor-smart

No credit card required

📰 Smart Article Extractor extracts articles from any scientific, academic, or news website with just one click. The extractor crawls the whole website and automatically distinguishes articles from other web pages. Download your data as HTML table, JSON, Excel, RSS feed, and more.

All issues Create new issue

pls fix doesnt work with fox news

Closed

cardinal_lobster opened this issue

https://newsinfo.inquirer.net/category/latest-stories it only scraped 8, pls fix now

Lukáš Křivka (lukaskrivka)

Hello,

Thanks for the report.

I see you are using Max articles per start URL set to 50 which will limit you further.
From the specific URL https://www.foxnews.com/category/us/crime, it doesn't find any more subcategories. So you will need to use more generic pseudo URL like https://www.foxnews.com[.*]
If you want only under /crime, it will need to keep scrolling down and clicking on the button "show more". This requires switching to the browser in the Browser options and adding selector to the button. See run and input as guidance: https://console.apify.com/view/runs/H4HiXiA5kXoYZUewS. Unfortunatel,y browsers are much slower but without them, you cannot click on "show more"

Add comment

Developer

Lukáš Křivka

Actor metrics

173 monthly users
74.9% runs succeeded
2.8 days response time
Created in Nov 2019
Modified about 2 months ago

Categories

News

Google Search Results Scraper

apify/google-search-scraper

Scrape Google Search Engine Results Pages (SERPs). Select the country or language and extract organic and paid results, ads, queries, People Also Ask, prices, reviews, like a Google SERP API. Export scraped data, run the scraper via API, schedule and monitor runs, or integrate with other tools.

Apify

43.5k

Website Content Crawler

apify/website-content-crawler

Automatically crawl and extract text content from websites with documentation, knowledge bases, help centers, or blogs. This Actor is designed to provide data to feed, fine-tune, or train large language models such as ChatGPT or LLaMA.

Apify

13k

Twitter Scraper

quacker/twitter-scraper

Scrape tweets from any Twitter user profile. Top Twitter API alternative to scrape Twitter hashtags, threads, replies, followers, images, videos, statistics, and Twitter history. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Quacker

24.4k

GPT Scraper

drobnikj/gpt-scraper

Extract data from any website and feed it into GPT via the OpenAI API. Use ChatGPT to proofread content, analyze sentiment, summarize reviews, extract contact details, and much more.

Jakub Drobník

4.3k

AI Product Matcher

equidem/ai-product-matcher

Match products across multiple e-commerce websites. Use this AI product matching Actor whenever you need to find matching pairs of products from different online shops for dynamic pricing, competitor analysis or market research.

Matěj Sochor

308

Google Trends Scraper

emastra/google-trends-scraper

Scrape data from Google Trends by search terms or URLs. Specify locations, define time ranges, select categories to get interest by subregion and over time, related queries and topics, and more. Export scraped data, run the scraper via API, schedule and monitor runs, or integrate with other tools.

Emiliano Mastragostino

3.2k

AI Web Agent

apify/ai-web-agent

Use natural language prompts to browse the web, click on elements, fill and submit forms, extract data, and take screenshots using the OpenAI API.

Apify

412

Twitter URL Scraper

quacker/twitter-url-scraper

Copy any Twitter URL and extract Twitter usernames, profile photos, follower count, tweets, hashtags, favorite count, and more. Export scraped datasets, run the scraper via API, schedule and monitor runs or integrate with other tools.

Quacker

4.3k

Extended GPT Scraper

drobnikj/extended-gpt-scraper

Extract data from any website and feed it into GPT via the OpenAI API. Use ChatGPT to proofread content, analyze sentiment, summarize reviews, extract contact details, and much more.

Jakub Drobník

855