Only startUrl required, for example this could be the main page: https://www.webdesignernews.com, some of the pagination pages: https://www.webdesignernews.com/page/2 or some specific section: https://www.webdesignernews.com/category/apps or https://www.webdesignernews.com/category/apps/page/3

wayToScrape can be old or new, by default old. If, for example, the starting link is on the third page (https://webdesignernews.com/page/3 for example), then scraper will scrape third page, fourth page, fifth and so on, until pagination or maxRequestsPerCrawl limit ends. But if the wayToScrape is new - third, second and first pages will be scraped. So you decide which way actor will work.

Local use

It is not necessary to use the apify.com service to use actor. Copy this repository and install node dependencies, for example, this way:

git clone git@github.com:Ganevru/actor-webdesignernews-scraper.git
cd actor-webdesignernews-scraper
npm install

Create in the root of project: apify_storage/key_value_stores/default/INPUT.json

In INPUT.json, write your settings, for example:

{
  "startUrl": "https://webdesignernews.com/page/3",
  "wayToScrape": "old",
  "maxRequestsPerCrawl": 3,
  "maxRequestRetries": 3,
  "maxConcurrency": 3,
  "liveView": true,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}

Start scrapping with command:

npm run local-start

Or, if you need a clean start:

npm run local-start-fresh

This will remove the results of all previous launches!

The result of the launch will be in apify_storage/datasets/default

It will be something like this:

{
  "title": "AMP Pages Fundamentally Changing How the Web Works",
  "vote": 47,
  "thumb": "https://webdesignernews.s3.amazonaws.com/post_images/2657750/vlcsnap-2019-04-17-00h58m00s057-728x407-1d95-140x112.jpg",
  "link": "https://www.webdesignernews.com/redirect/id/2657750",
  "source": "androidpolice.com",
  "date": "2 days ago",
  "requestUrl": "https://webdesignernews.com/page/3"
}

On this page

Web Designer News - Scraper
- Input settings
- Local use

Share Actor:

Feedly Scraper

mscraper/feedly-scraper

Feedly Scraper is a specialized web scraping tool designed to extract news from Feedly. The scraper exports the accumulated data to various formats like JSON, XML, CSV, or Excel

mscraper

Investing News Scraper

mscraper/investing-news-scraper

Investing News Scraper is a specialized web scraping tool designed to extract news from Investing. The Scraper exports the accumulated data to various formats like JSON, XML, CSV or EXCEL

mscraper

MyStocks News Scraper

mscraper/mystocks-news-scraper

MyStocks News Scraper is a specialized web scraping tool designed to extract news from MyStocks. The Scraper exports the accumulated data to various formats like JSON, XML, CSV or EXCEL

mscraper

Barchart News Scraper

mscraper/barchart-news-scraper

Berchart News Scraper is a specialized web scraping tool designed to extract news from Berchart. The Scraper exports the accumulated data to various formats like JSON, XML, CSV or EXCEL

mscraper

TipRanks News Scraper

mscraper/tipranks-news-scraper

TipRanks News Scraper is a specialized web scraping tool designed to extract news from TipRanks. The scraper exports the accumulated data to various formats like JSON, XML, CSV, or Excel.

mscraper

App Store Search Scraper

mscraper/app-store-search-scraper

App Store Search Scraper is a specialized web scraping tool designed to extract news from App Store. The Scraper exports the accumulated data to various formats like JSON, XML, CSV or EXCEL

mscraper

5.0

Stock Tracker News Scraper

mscraper/stock-tracker-news-scraper

Stock Tracker News Scraper is a specialized web scraping tool designed to extract news from Stock Tracker. The scraper exports the accumulated data to various formats like JSON, XML, CSV, or Excel.

mscraper

TradingView News Scraper

mscraper/tradingview-news-scraper

TradingView News Scraper is a specialized web scraping tool designed to extract news from the charting platform TradingView. The scraper exports the accumulated data to various formats like JSON, XML, CSV, or Excel.

mscraper

102

Stocks Plus News Scraper

mscraper/stocks-plus-news-scraper

Stock+ News Scraper is a specialized web scraping tool designed to extract news data from the Stock+. The scraper exports the collected news data to various formats like JSON, XML, CSV, or Excel.

mscraper

Investors Business Daily News Scraper

mscraper/investors-business-daily-news-scraper

Investors Business Daily News Scraper is a specialized web scraping tool designed to extract news from the IBD platform. The scraper exports the accumulated data to various formats like JSON, XML, CSV, or Excel.

mscraper

Tipranks Scraper

mscraper/tipranks-scraper

TipRanks News Scraper is a specialized web scraping tool designed to provide valuable real-time data on stock prices, market trends, and financial metrics from TipRanks. The scraper exports the accumulated data to various formats like JSON, XML, CSV, or Excel.

mscraper