Dataset Image Downloader & Uploader avatar

Dataset Image Downloader & Uploader

Try for free

No credit card required

Go to Store
Dataset Image Downloader & Uploader

Dataset Image Downloader & Uploader

lukaskrivka/images-download-upload
Try for free

No credit card required

Download image files from image URLs in your datasets and save them to a Zip file, Key-Value store, or directly your AWS S3 bucket.

Do you want to learn more about this Actor?

Get a demo

Dataset Id

datasetIdstringOptional

Id of the dataset where the data are located. Image URLs will be extracted from there.

Items

itemsarrayOptional

Array of items that includes the image URLs.

Path to image URLs

pathToImageUrlsstringOptional

Path from item object to an array or string where the URL(s) is/are located. Provide in "javascript style", e.g. "details[0].images

Filename function

fileNameFunctionstringOptional

Function that specifies how will be image filename created from its URL. If you keep this empty, it will be md5 hash of the URL.

Limit

limitintegerOptional

Max items to load from the dataset. Use with offset to paginate over the data (can reduce memory requirement of large loads).

Offset

offsetintegerOptional

How many items to skip from the dataset. Use with limit to paginate over the data (can reduce memory requirement of large loads)

Output to

outputToEnumOptional

Where to save the data from input after possibly transforming them during the download process.

Value options:

"no-output": string"key-value-store": string"dataset": string

Output dataset Name or ID

outputDatasetIdstringOptional

Name or ID of the dataset where the data will be saved. Only relevant if you want to output to dataset!

Key Value store input

storeInputstringOptional

If you want to input the data from key-value store instead of dataset. Notation: storeId-recordKey, e.g. - kWdGzuXuKfYkrntWw-OUTPUT

Upload to

uploadToEnumOptional

Where do you want to upload the image files

Value options:

"zip-file": string"key-value-store": string"s3": string"no-upload": string

Key-value store name

uploadStoreNamestringOptional

Key-value store name where the images will be upload. Empty field means it will be uploaded to the default key-value store

S3 Bucket

s3BucketstringOptional

Only relevant if you want to upload to S3! Name of the bucket where to upload.

S3 Access key id

s3AccessKeyIdstringOptional

Only relevant if you want to upload to S3! You can create these credentials for IAM user.

S3 Secret access key

s3SecretAccessKeystringOptional

Only relevant if you want to upload to S3! You can create these credentials for IAM user.

Check if key is already on S3

s3CheckIfAlreadyTherebooleanOptional

This option is useful if you don't want to rewrite the same image. GET requests are also cheaper than PUT requests

Pre-download function

preDownloadFunctionstringOptional

Function that specifies how will be the data transformed before downloading the image. The input and output of the function is the whole data array. You can skip downloading images of any item if you add skipItem: true field to it.

Post-download function

postDownloadFunctionstringOptional

Function that specifies how will be the data transformed before downloading the image. The input and output of the function is the whole data array. By default it adds either the file URL or errors array depending if the download was successfull.

Max retries

imageCheckMaxRetriesintegerOptional

How many times should actor retry if the file it tries to download fails to pass the tests. Setting this too high can lead to unecessary loops.

Default value of this property is 6

Image check type

imageCheckTypeEnumOptional

Type of the image check. If the image will not pass, the download will be retied with proxy and if that doesn't pass, the image is not uploaded.

Value options:

"none": string"content-type": string"image-size": string

Min size in KB

imageCheckMinSizeintegerOptional

Minimum size of the image to pass the image check test

Min width

imageCheckMinWidthintegerOptional

Minimim width of the image in pixels to pass the image check. Works only if the image check type is 'jimp'.

Min height

imageCheckMinHeightintegerOptional

Minimim height of the image in pixels to pass the image check. Works only if the image check type is 'jimp'.

Proxy configuration

proxyConfigurationobjectOptional

Select proxies to be used.

Max concurrency

maxConcurrencyintegerOptional

You can specify how many maximum parallel downloading/uploading requests will be running. Keep in mind that the limit is here to not overload the host server.

Default value of this property is 40

Download timeout in ms

downloadTimeoutintegerOptional

How long we will wait to download each image

Default value of this property is 15000

Batch Size

batchSizeintegerOptional

Number of items loaded from dataset in one batch.

Default value of this property is 10000

Convert webp to png

convertWebpToPngbooleanOptional

If checked, the actor will automatically convert all webp type images to standard png. This increases the size of the image.

State fields

stateFieldsarrayOptional

You can specify fields that you want in your state to make it more readable and use less memory. By default it uses all.

Run without download

noDownloadRunbooleanOptional

If checked, the actor will not download and upload the images. Usefull for checking duplicates or transformations.

Developer
Maintained by Apify

Actor Metrics

  • 35 monthly users

  • 13 stars

  • >99% runs succeeded

  • 2 days response time

  • Created in Nov 2018

  • Modified 22 days ago