This is looks what I need but I am not sure how to call this task on a schedule run dynamically so I can keep passing references to the datasets IDs as I run them and merge all this calls into a single dataset

thanks!

Míša Fialová (misa)

Hello! You may want to check out this blog post for guidance on using the scraper: https://blog.apify.com/how-to-scraped-data-deduplication/ It should provide valuable insights. If you encounter any difficulties during setup, don't hesitate to reopen the issue for further assistance.

Add comment

Batch Runner

useful-tools/batch-runner

Actor for starting several Runs from single Input. It also has capability to merge the Dataset items from the created batch.

Useful tools

5.0

PGVector Integration

apify/pgvector-integration

This integration transfers data from Apify Actors to a Postgres SQL database (with PGVector extension).

Apify

4.6

API / JSON scraper

pocesar/json-downloader

Scrape any API / JSON URLs directly to the dataset, and return them in CSV, XML, HTML, or Excel formats. Transform and filter the output. Enables you to follow pagination recursively from the payload without the need to visit the HTML page.

Paulo Cesar

530

Forward dataset as POST data

anchor/forward-dataset-webhook

This actor forwards the results of an Actor to an endpoint, instead of having to fetch the results manually. It will download the dataset and attach it to the body of a POST request you will specify. It acts as a new webhook. Simplify your Actor process !!!

Anchor

Contact Details Merge & Deduplicate

lukaskrivka/contact-details-merge-deduplicate

Merge and deduplicate all contacts extracted by Contact Details Scraper. Works with multiple datasets. One row per domain.

Lukáš Křivka

S3 Uploader

apify/s3-uploader

Upload data from an Apify dataset to an Amazon S3 bucket. Providing various filters and transformation options, this Actor allows precise control over data structure, formatting, and upload settings to ensure seamless integration into your data pipeline.

Apify

HTTP API - Flexible Client for Prototyping and Automation

xyzzy/http-api

Easily test, integrate, and automate APIs—configure method, headers, query params, body, timeout, and retries, no code or setup required. Ideal for rapid prototyping, endpoint monitoring, workflows, and sales & marketing automation.

xyzzy

OpenRouter - Unified LLM Interface for ChatGPT, Claude, Gemini

xyzzy/open-router

Use the OpenRouter platform to choose the best and most cost effective model for your prompts utilizing a standardized interface (including ChatGPT, Claude, Gemini, Llama, Mistral, and more). See instructions for creating an OpenRouter account and API key.

xyzzy

MongoDB Import

drobnikj/mongodb-import

Act import objects to specific mongodb collection. You can pass list of plain objects or Apify key-value store as input. For info see readme bellow.

Jakub Drobník

BeautifulSoup Scraper

apify/beautifulsoup-scraper

Crawls websites using raw HTTP requests. It parses the HTML with the BeautifulSoup library and extracts data from the pages using Python code. Supports both recursive crawling and lists of URLs. This Actor is a Python alternative to Cheerio Scraper.

Apify

826

4.4