My monitoring task report

This actor is a part of the Apify Monitoring Suite (apify/monitoring). See its readme for more information and how to use this.

This is an actor input schema for the monitoring master actor.

Ids of checker actors or tasks that needs

Checkers

Checkers input

Works with Subject type 'Actor', 'Task' 'Named dataset'. Starts dashboard actor. LINK TO PA

Dashboard

Checks for duplicates based on defined unique fields. Based on 'https://apify.com/lukaskrivka/duplications-checker'

Duplication checker

Sends report to mail inbox. LINK TO PA for input

Email

If passed the resources will be filtered by the frequency of the task. For example if you pass MONTHLY only resources modified last month will be collected.

The frequency of task

Last items to be checked. If 100 is passed, only last hundred runs/datasets will be checked. 

Max items to be checked

Name of your monitoring suite. It will be used in notifications and to identify related resources in the Apify dashboard.

This section of input allows you to select your targets. In other words, what you want to monitor. It can be actors, tasks or datasets and you can select them by providing a pattern that matches their names, or by individual IDs.

Monitoring suite name

Reporters

Works with Subject type 'Actor', 'Task' and 'Named dateaset', Gets information about the runs TODO: LINK TO PA

Resource stats collector

Run status validator

All monitoring runs results are saved in named dataset

Save history

Validates dataset items based on the Type check library

Schema validator checker

Sends report to custom slack channel. LINK TO PA for input

Slack

If for whatever reason the Target name pattern option does not suit you, targets can also be specified by providing their IDs, as found in your Apify dashboard.

Target IDs

Regular expressions that will be matched against selected actors / tasks or datasets under your Apify account. All matching targets will then be monitored by this monitoring suite. This is typically also the fastest way to select a single target. Just type its full name. Datasets are going to be automatically group by these patterns when dashboard statistics is counted for dataset target type.

Target name patterns

Only one type of target can be monitored by a single monitoring suite. If you want to watch more types, create more monitoring suites.

Type of target

Monitoring input

monitoring-runner

Monitoring Runner

MONITORING_CONFIG

.eslintrc

.gitignore

If true only new entities will be check based on the 'Subject type'. If 'Actor' is selected the checkers will run only for new runs. If 'Task' is selected only new task runs will be checked. If 'Named dataset' is selected, only new datasets will be checked 

Check only new

list of actor ID or task ID to be monitored

Monitoring subject ids

Regex that will match actors to be monitored

Subject regex

Subject type

INPUT_SCHEMA.json

# Apify project template

Before you start writing your first actor, please read [Apify Marketplace developers WIKI](https://gitlab.com/apify-public/wiki/wikis/home).


README.md

monitoring-run

apify.json

package-lock.json

monitoring-poc

package.json

src/main.js

// Checkers can cover more operations in order to optimize the large datasets.
const ERROR_TYPE = 'ERROR'; // more like test type
const STATISCTICS_TYPE = 'STATS_TYPE';

const duplicationCheckerResult = {
    name: 'duplication-checker',
    output: {
        checks: [
            {
                name: 'deduplication',
                level: 5, // levels scale to determine how bad the error is. @TODO: DEFINE SCALE
                result: {
                    type: ERROR_TYPE,
                    status: 'PASSED|FAILED',
                    expected: 0,
                    received: 100,
                    errorName: 'Duplication found',
                },
            },
            {
                name: 'missing-keys',
                level: 2, // levels scale to determine how bad the error is. @TODO: DEFINE SCALE
                result: {
                    type: ERROR_TYPE,
                    status: 'PASSED|FAILED',
                    errorsFound: [],
                    errorName: 'Missing keys found',
                },
            },
        ],
    },
};

const runStatusChecker = {
    name: 'run-status-checker',
    output: {
        checks: [
            {
                name: 'failed-runs',
                level: 3, // levels scale to determine how bad the error is. @TODO: DEFINE SCALE
                result: {
                    type: ERROR_TYPE,
                    status: 'PASSED|FAILED',
                    errorsFound: [],
                    errorName: 'Found %s failed runs',
                },
            },
            {
                name: 'timout-runs',
                level: 1, // levels scale to determine how bad the error is. @TODO: DEFINE SCALE
                result: {
                    type: ERROR_TYPE,
                    status: 'PASSED|FAILED',
                    errorsFound: [],
                    errorName: 'Found %s timouted runs',
                },
            },
        ],
    },
};

const datasetItemCountChecker = {
    name: 'dataset-item-count-checker',
    output: {
        checks: [
            {
                name: 'dataset-item-count',
                result: {
                    type: STATISCTICS_TYPE,
                    values: [{ name: 'Dataset item count', value: 150 }],
                },
            },
        ],
    },
};

const runStatusCountChecker = {
    name: 'run-status-count-checker',
    output: {
        checks: [
            {
                name: 'run-status-count',
                result: {
                    type: STATISCTICS_TYPE,
                    values: [{ name: 'Succceeded runs count', value: 100 }, { name: 'Failed runs count', value: 10 }],
                },
            },
        ],
    },
};
const runChecker = (checker) => {
    console.log(checker);
    return new Promise((resolve, reject) => {
        setTimeout(() => resolve(checker === 'DUPLICATION' ? duplicationCheckerResult : runStatusChecker), 100);
    });
};

const runCheckers = async (checkers) => {
    return await Promise.all(checkers.map(runChecker));
};
module.exports = {
    runCheckers,
};


Monitoring runner

Build new tools

Order custom tool