Pricing

Pay per usage

Go to Apify Store

Firestore Import

Try for free

Developed by

Jakub Drobník

Imports dataset items to Firestone DB.

0.0 (0)

Pricing

Pay per usage

Last modified

2 years ago

Developer tools

Integrations

Open source

.eslintrc

{
  "extends": "@apify"
}

.gitignore

apify_storage
node_modules

.npmignore

# This file tells Git which files shouldn't be added to source control

.idea
node_modules

Dockerfile

# Dockerfile contains instructions how to build a Docker image that will contain
# all the code and configuration needed to run your actor. For a full
# Dockerfile reference, see https://docs.docker.com/engine/reference/builder/

# First, specify the base Docker image. Apify provides the following base images
# for your convenience:
#  apify/actor-node-basic (Node.js 10 on Alpine Linux, small and fast image)
#  apify/actor-node-chrome (Node.js 10 + Chrome on Debian)
#  apify/actor-node-chrome-xvfb (Node.js 10 + Chrome + Xvfb on Debian)
# For more information, see https://apify.com/docs/actor#base-images
# Note that you can use any other image from Docker Hub.
FROM apify/actor-node-basic

# Second, copy just package.json and package-lock.json since they are the only files
# that affect NPM install in the next step
COPY package*.json ./

# Install NPM packages, skip optional and development dependencies to keep the
# image small. Avoid logging too much and print the dependency tree for debugging
RUN npm --quiet set progress=false \
 && npm install --only=prod --no-optional \
 && echo "Installed NPM packages:" \
 && npm list \
 && echo "Node.js version:" \
 && node --version \
 && echo "NPM version:" \
 && npm --version

# Next, copy the remaining files and directories with the source code.
# Since we do this after NPM install, quick build will be really fast
# for simple source file changes.
COPY . ./

# Specify how to run the source code
CMD npm start

INPUT_SCHEMA.json

{
  "title": "Firestore Import input",
  "description": "Imports dataset to Firestore DB",
  "type": "object",
  "schemaVersion": 1,
  "properties": {
    "datasetId": {
      "title": "Dataset",
      "type": "string",
      "description": "Dataset ID of dataset you want to import to Firestore",
      "editor": "textfield"
    },
    "apiKey": {
      "title": "Api key",
      "type": "string",
      "description": "Firestore API key",
      "editor": "textfield"
    },
    "authDomain": {
      "title": "Auth domain",
      "type": "string",
      "description": "Firestore authentication domain",
      "editor": "textfield"
    },
    "projectId": {
      "title": "Project ID",
      "type": "string",
      "description": "Firestore project ID",
      "editor": "textfield"
    },
    "collectionName": {
      "title": "Collection name",
      "type": "string",
      "description": "Firestore collection name",
      "editor": "textfield"
    }
  },
  "required": ["datasetId", "apiKey", "authDomain", "projectId", "collectionName"]
}

apify.json

{
	"name": "firestore-import",
	"version": "0.0",
	"buildTag": "latest",
	"env": null,
	"template": "hello_world"
}

main.js

1const Apify = require('apify');
2const firebase = require('firebase');
3
4Apify.main(async () => {
5    const input = await Apify.getInput();
6
7    const { data } = input;
8    let islegacyPhantomJSTask = false;
9    if (data && typeof data === 'string') {
10        // It runs from legacy phantomjs crawler task finished webhook
11        const legacyInput = JSON.parse(data);
12        Object.assign(input, legacyInput);
13        islegacyPhantomJSTask = true;
14    }
15
16    const { datasetId, apiKey, authDomain, projectId, collectionName } = input;
17
18    if (!datasetId) {
19        throw new Error('DatasetId is required on input.');
20    }
21
22    firebase.initializeApp({
23        apiKey,
24        authDomain,
25        projectId,
26    });
27
28    // Initialize Cloud Firestore through Firebase
29    const db = firebase.firestore();
30    console.log(`Start importing dataset ${datasetId} to firestore.`);
31    const dataset = await Apify.openDataset(datasetId, { forceCloud: true });
32    const datasetInfo = await dataset.getInfo();
33    // Import dataset from actor/task
34    const limit = 1000;
35    let counter = 0;
36    for (let offset = 0; offset < datasetInfo.itemCount; offset += limit) {
37        const pagination = await dataset.getData({
38            simplified: islegacyPhantomJSTask,
39            clean: !islegacyPhantomJSTask,
40            limit,
41            offset,
42        });
43        console.log(`Get dataset items offset: ${pagination.offset}`);
44        for (const item of pagination.items) {
45            try {
46                await db.collection(collectionName).add(item);
47                counter++;
48            } catch (err) {
49                console.log(`Cannot import item ${JSON.stringify(item)}: ${err.message}`);
50            }
51        }
52    }
53
54    console.log(`Imported ${counter} from dataset ${datasetId}.`);
55
56    console.log('Done!');
57});

package.json

{
	"name": "firestore-import",
	"version": "0.0.1",
	"description": "This is a boilerplate of an Apify actor.",
	"dependencies": {
		"apify": "^0.16.0",
		"firebase": "^7.2.0"
	},
	"devDependencies": {
		"@apify/eslint-config": "0.0.3",
		"eslint": "^6.5.1"
	},
	"scripts": {
		"start": "node main.js",
		"test": "echo \"Error: oops, the actor has no tests yet, sad!\" && exit 1"
	},
	"author": "It's not you it's me",
	"license": "ISC"
}

Firebase Firestore Import

danielwebr/firebase-firestore-import

This actor is designed to simplify the process of transferring data from an Apify dataset into a Firebase Firestore collection. Simply configure the actor with your Firebase credentials, Firestore collection ID, and dataset ID, and it will handle the rest, including optional data transformations.

Daniel Wébr

Firestore Import

apify/firestore-import

Seamlessly import data from Apify datasets into Firebase Firestore database. This integration allows full control over document IDs, conflict resolution (overwrite, merge, skip), and data transformation using a custom JavaScript function.

Apify

5.0

Google Dataset Items Translator

web.harvester/google-dataset-items-translator

Translate any dataset field(s) to any of the supported languages using the Google Translate website, it goes through all the items in the dataset and translates all of the selected fields

Web Harvester

Sort Dataset Items

lukaskrivka/sort-dataset-items

Add this actor as a webhook to your scraper to sort the dataset by index field

Lukáš Křivka

MongoDB Import

drobnikj/mongodb-import

Act import objects to specific mongodb collection. You can pass list of plain objects or Apify key-value store as input. For info see readme bellow.

Jakub Drobník

XMLs To Dataset

mtrunkat/xmls-to-dataset

Go to actor anytime you need to download XML files and store them in the dataset.

Marek Trunkát

105

Dataset(s) To Schema

zuzka/dataset-to-schema

Takes a Dataset ID(s) and outputs a JSON schema of the contents of the dataset into key value store.

Zuzka Pelechová

Error Messages Deduplication

petrpatek/get-debug-items-from-dataset

Filter items from the dataset with `#debug` fieldName and saves them to the dataset and deduplicates `errorMessages` so you don't have to go through all the errors.

Petr Pátek

Google Sheets Import & Export

lukaskrivka/google-sheets

Import data from datasets or JSON files to Google Sheets. Programmatically process data in Sheets. Easier and faster than the official Google Sheets API and perfect for importing data from scraping.

Lukáš Křivka

1.9K

5.0