Dataset Toolbox
No credit card required
Dataset Toolbox
No credit card required
Perform common actions on datasets - merge, unify, validate, transform, order fields etc.
Input
| Field | Type | Optional | Description |
|---|---|---|---|
| Actor IDs | Array | depends | Load latest default datasets |
| Dataset IDs | Array | depends | Load the specified datasets |
Access 3rd party datasets
Define secret environment variable to access other users' datasets: CUSTOM_SOURCE_APIFY_TOKEN
Features
- Dataset unification
- merge datasets in uniform output
- limit number of items extracted
- Latest dataset detection
- detect and use the latest datasets
- Output fields management
- filter out fields from source data
- output data in custom field order
- Dataset post-processing
- apply custom function to each item
- Output schema validation
- validate records against schema
- capture invalid items for recycling
Dataset unification
Use features described below to produce a single uniform dataset from datasets sharing a single common output schema and expected output structure
Latest dataset detection
Automatically detects and uses default datasets of the latest actor runs when:
- Actor ID(s) are specified AND
- Dataset ID(s) are not specified
Output fields management
Produces a download link for obtaining the resulting dataset with top level fields sorted and filtered based on the list of fields provided on input. This link is stored in default KV store:
DATASET_DOWNLOAD-CUSTOM_FIELD_ORDER-{selected file type}
Filter fields
Filter and pick only certain fields from source dataset(s)
Order fields
Apply custom order to top level fields in custom order instead of alphabetical (default)
Dataset post-processing
Apply custom javascript function to every item from source dataset(s) before saving to result dataset
Output schema validation
Validate schema
Validate every item against JSON schema specified on input and filter out invalid items before saving to result dataset
Reuse invalid items
Invalid items are captured in separate requestListSources saved in KV store.
Vasek Codey VlcekActor Metrics
3 monthly users
-
3 stars
Created in Mar 2020
Modified 2 years ago
