This actor takes a list of URLs and downloads HTML of each page.

{
  "sources": [{
    "requestsFromUrl": "http://example.com/my-url-list.txt"
  }]
}

Use the following form to configure this scraper. The URL list is required and all other fields are optional.

Maximum time the scraper will spend processing one page.

Page timeout

Choose to use no proxy, Apify Proxy, or provide custom proxy URLs.

Proxy configuration

Start URLs

The scraper will use a real Chrome browser instead of a Chromium masking as Chrome. Using this option may help with bypassing certain anti-scraping protections, but risks that the scraper will be unstable or not work at all.

Settings that help mask as a real user and prevent scraper detection.

Use Chrome

The scraper will apply various browser emulation techniques to match a real user as closely as possible. This feature works best in conjunction with the Use Chrome option and also carries the risk of making the scraper unstable.

Use Stealth

Input

url-list-download-html

URL List HTML Downloader

# Dockerfile contains instructions how to build a Docker image that
# will contain all the code and configuration needed to run your actor.
# For a full Dockerfile reference,
# see https://docs.docker.com/engine/reference/builder/

# First, specify the base Docker image. Apify provides the following
# base images for your convenience:
#  apify/actor-node-basic (Node.js 10 on Alpine Linux, small and fast)
#  apify/actor-node-chrome (Node.js 10 + Chrome on Debian)
#  apify/actor-node-chrome-xvfb (Node.js 10 + Chrome + Xvfb on Debian)
# For more information, see https://apify.com/docs/actor#base-images
# Note that you can use any other image from Docker Hub.
FROM apify/actor-node-chrome

# Second, copy just package.json since it should be the only file
# that affects NPM install in the next step
COPY package.json ./

# Install NPM packages, skip optional and development dependencies to
# keep the image small. Avoid logging too much and print the dependency
# tree for debugging
RUN npm --quiet set progress=false \
 && npm install --only=prod --no-optional \
 && echo "Installed NPM packages:" \
 && npm list \
 && echo "Node.js version:" \
 && node --version \
 && echo "NPM version:" \
 && npm --version

# Next, copy the remaining files and directories with the source code.
# Since we do this after NPM install, quick build will be really fast
# for most source file changes.
COPY . ./

# Optionally, specify how to launch the source code of your actor.
# By default, Apify's base Docker images define the CMD instruction
# that runs the source code using the command specified
# in the "scripts.start" section of the package.json file.
# In short, the instruction looks something like this:
# CMD npm start


Dockerfile

INPUT_SCHEMA.json

This actor simply scrapes the full HTML code for all given URLs.
In input you can define which proxies should be used.

Additionally, you can define a selector that will be awaited for each
of the URLs simply by adding `waitForSelector` field to `userData` property
of the request:

```javascript
{
    "waitForSelector": ".class-i-want-to-wait-for"
}
```

README.md

URL List HTML Downloader

Build new actors

Buy custom solution