Actor picture

Crawl Url List 1by1


Crawls given list of urls with one crawler execution per url.

Crawl Url List 1 by 1 act that takes a list of urls and starts given crawler for each of the urls.

Crawler is published at as mtrunkat/crawl-url-list-1by1.

You can start this act via POST request to following url with it's input as JSON payload:[YOUR_API_TOKEN]

Example input:

You can either send url of publicly hosted file containing your url list (one url per line):

    "urlListFile": "",
    "crawlerId": "ytXL3jaRKwrfWC9tz",
    "concurrency": 2

Or you can pass urls directly:

    "urlList": ["", ""],
    "crawlerId": "ytXL3jaRKwrfWC9tz",
    "concurrency": 2

Possible options:

Options crawlerId, cocurrency and one of urlListFile and urlList are required!

Option Type Description
urlListFile String Url of the texfile containing urls to be crawled with one url per line
urlList Array Array of urls to be crawled.
crawlerId String Crawler ID.
concurrency Number Concurrency of crawler executions.
crawlerSettings Object Overrides of crawler settings passed to startExecution call