No credit card required

Puppeteer Scraper

apify/puppeteer-scraper

No credit card required

Crawls websites with the headless Chrome and Puppeteer library using a provided server-side Node.js code. This crawler is an alternative to apify/web-scraper that gives you finer control over the process. Supports both recursive crawling and list of URLs. Supports login to website.

Do you want to learn more about this Actor?

Get a demo

Back to issues Create new issue

Ignore URLs with certain query strings

Open

linen_torch opened this issue

The site i want to scrape has it in multiple languages all denoted with the query '?=hl'. Can i get the crawler to ignore these?

Martin Adámek (adamek)

By ignoring it you mean you want to skip enqueuing such URLs if they were already processed? Are you sure the URLs are otherwise the same? If so, you could trim the query parameter manually and provide uniqueKey explicitly when adding new requests to the queue (which you would need to do manually, inside your page function, while disabling the automatic enqueueing, e.g. by setting the selector option to some gibberish).

Alternatively, you could skip the pages inside the request handler.

Add comment

Developer

Apify

Actor Metrics

358 monthly users
70 stars
>99% runs succeeded
30 days response time
Created in Apr 2019
Modified 6 months ago

Categories

Developer tools

Cloudflare Web Scraper

dtrungtin/cloudflare-web-scraper

Prevents Puppeteer from being detected as a bot in services like Cloudflare and allows you to pass captchas without any problems

Tin Duong

Redfin Fast Scraper

mantisus/redfin-fast-scraper

Redfin: Scrape fast, stay light! Skip bloated browser tools. My Redfin scraper extracts property data in a flash, no heavy lifting is needed. Scrape/monitor listings with ease, all without Puppeteer or Playwright. ⚡️

Maksym Bohomolov

Thefork Fast Scraper

mantisus/thefork-fast-scraper

Scrape TheFork.com quickly and easily! Skip bloated browser tools. This scraper extracts restaurant data in a flash, no heavy lifting is needed. Scrape and monitor data with ease, all without Puppeteer or Playwright. ⚡️

Maksym Bohomolov

Redfin Fast Scraper Per Results

mantisus/redfin-fast-scraper-per-results

Maksym Bohomolov

Zoopla.co.uk Fast Scraper

mantisus/zoopla-actor

Zoopla.co.uk: Scrape fast, stay light! Skip bloated browser tools. My Zoopla scraper extracts property data in a flash, no heavy lifting is needed. Scrape/monitor listings with ease, all without Puppeteer or Playwright. ⚡️

Maksym Bohomolov

Thefork Fast Scraper Per Result

mantisus/thefork-fast-scraper-per-result

Maksym Bohomolov

cambridge dictionary scraper

dev_bodex/cambridge-dictionary-scraper

This Cambridge Dictionary Scraper Apify Actor automates extracting word definitions, synonyms, examples, and translations from the Cambridge Dictionary. Built with Node.js and Puppeteer, it returns structured data, ideal for language processing, research, and educational use.

Festus Befgrp

Google Direction Scraper

dev_bodex/google-direction-scraper

This actor automates extracting route directions from Google Maps. Using Node.js and Puppeteer, it retrieves travel routes, distances, and estimated times for various transport modes, offering structured data for navigation, logistics, or travel planning applications.

Festus Befgrp

Google Translation Scraper

dev_bodex/google-translation-scraper

This Google Translation Scraper Actor automates extracting translations for any input text from Google Translate. Built with Node.js and Puppeteer, it efficiently retrieves translations in multiple languages, providing structured data for use in language apps, research, or educational projects.

Festus Befgrp

Simple Contact Info and Social Media Scraper

pajoe/simple-contact-info-and-social-media-scraper

This Apify actor is designed to crawl web pages and extract social media handles, emails, and phone numbers using Puppeteer. It can handle dynamic content and navigate through multiple pages, making it suitable for comprehensive data extraction tasks.