Run webook digest

Through webhooks installed in your tasks / actors, allows you to provide multiple HTTP endpoints, that receive a more complete JSON from the run, and allow to hit those endpoints using a proxy.

Enables you to do conditional webhook calls, when only certain conditions are met. You can set individual thresholds to get only some emails with some lines of Javascript code as you have access to most platform resources.

Sends you an email containing a digest with the run information whenever the run succeeds, timeouts, or fails.

Webhook

The installed webhook looks like this:

1{
2    "emails": ["email1@example.com", "email2@example.com"],
3    "endpoints": ["https://api.apify.com/v2/acts/username~another-actor/runs/?token=YourTokenHere"],
4    // ...
5    "resource": {{resource}}
6}

The email looks like this:

1Task my-task-107	ptew7Wypww36aZNUZ
2Status:	            TIMED-OUT
3Run:	            GBsPE3a78ZBH74Yq3
4Started At:	        2021-08-27T07:25:07.056Z
5Finished At:	    2021-08-27T07:25:18.236Z
6Duration:	        9.014 seconds | 0.15 minutes
7CU:	                0.0003129861111111111
8Dataset Count:	    0

By default, the HTTP endpoints will receive a JSON POST with:

1{
2    "type": "Task", // or "Actor"
3    "name": "my-actor",
4    "resource": {
5        "id": "hhj4yUPrncjTRsfas",
6        "actId": "S2xxbN3BVLXLmU2da",
7        "userId": "aurPRTH47KhmiaPNJ",
8        "startedAt": "2021-08-23T06:00:52.923Z",
9        "finishedAt": "2021-08-23T06:00:56.828Z",
10        "status": "SUCCEEDED",
11        "meta": {
12            "origin": "API",
13            "userAgent": "axios/0.21.1"
14        },
15        "stats": {
16            "inputBodyLen": 1216,
17            "restartCount": 0,
18            "durationMillis": 3759,
19            "resurrectCount": 0,
20            "runTimeSecs": 3.759,
21            "metamorph": 0,
22            "computeUnits": 0.00013052083333333335,
23            "memAvgBytes": 35479552,
24            "memMaxBytes": 35999744,
25            "memCurrentBytes": 35479552,
26            "cpuAvgUsage": 0,
27            "cpuMaxUsage": 0,
28            "cpuCurrentUsage": 0,
29            "netRxBytes": 530,
30            "netTxBytes": 150
31        },
32        "options": {
33            "build": "latest",
34            "timeoutSecs": 10,
35            "memoryMbytes": 128,
36            "diskMbytes": 256
37        },
38        "buildId": "dxryivn95ynb9",
39        "exitCode": 0,
40        "defaultKeyValueStoreId": "gnr59b7jh9d",
41        "defaultDatasetId": "485gwe485gv",
42        "defaultRequestQueueId": "4b5c0w845b",
43        "buildNumber": "0.0.6",
44        "containerUrl": "https://dryuvbdxpory.runs.apify.net"
45    },
46    "customData": {
47        "any": "custom information"
48    },
49    "datasetCount": 1000,
50    "run": {
51        "id": "rotne4amGv3YF",
52        "name": "496hvw94X5L7XAj",
53        "userId": "54vineirn4mBZmm",
54        "createdAt": "2019-12-12T07:34:14.202Z",
55        "modifiedAt": "2019-12-13T08:36:13.202Z",
56        "accessedAt": "2019-12-14T08:36:13.202Z",
57        "itemCount": 7,
58        "cleanItemCount": 5,
59        "actId": null,
60        "actRunId": null,
61        "fields": []
62    }
63}

Trigger condition

The email and endpoints will only be called if the triggerCondition parameter returns a truthy value.

1{
2    triggerCondition: async ({ Apify, dataset, requestQueue, keyValueStore, abort, data, input: { customData } }) => {
3        const { cleanItemCount } = await dataset.getInfo();
4
5        return cleanItemCount === 0 // execute the remote endpoint only in case the dataset yield nothing
6            || run.stats.computeUnits > 10 // or the compute units is over 10
7            || (await requestQueue.handledCount()) === 0 // or the requestQueue had an issue and processed 0 items
8    }
9}

You have full control of the data of your run here, you can do as many checks you need before sending out the request through the endpoints.

Custom HTTP endpoints and payloads

The custom HTTP endpoints webhook allows you to use proxies, something that the Apify platform don't provide.

This is mainly useful, using a proxy group like StaticUS3 with static IPs to do tunneling or IP whitelist!

You'll also be able to hit multiple endpoints with your data at once.

Apify Slack actor:

1{
2    endpoints: [
3        "https://api.apify.com/v2/acts/katerinahronik~slack-message/runs?token=YOUR_TOKEN"
4    ],
5    transformEndpoint: async ({ data, url }) => {
6        // you can differentiate by URL
7        if (url.includes('slack')) {
8            return {
9                token: "slack-token",
10                channel: "#your-channel",
11                text: `<https://my.apify.com/tasks/${data.resource.actorTaskId}|Task> finished with status ${data.resource.status}`
12            }
13        }
14    }
15}

MS Team:

1{
2    endpoints: [
3        "https://m341231.webhook.office.com/..."
4    ],
5    transformEndpoint: async ({ data, url }) => {
6        return {
7            "@type": "MessageCard",
8            "@context": "http://schema.org/extensions",
9            themeColor: "0076D7",
10            summary: "Larry Bryant created a new task",
11            sections: [{
12                activityTitle: "Larry Bryant created a new task",
13                activitySubtitle: "On Project Tango",
14                activityImage: "https://teamsnodesample.azurewebsites.net/static/img/image5.png",
15                facts: [{
16                    name: "Assigned to",
17                    value: "Unassigned"
18                }, {
19                    name: "Due date",
20                    value: "Mon May 01 2017 17:07:18 GMT-0700 (Pacific Daylight Time)"
21                }, {
22                    name: "Status",
23                    value: "Not started"
24                }],
25                markdown: true
26            }],
27        };
28    }
29}

Wordpress:

1{
2    endpoints: [
3        "https://your-wordpress-website.com/wp-json/wp/v2/posts"
4    ],
5    transformEndpoint: async ({ dataset, keyValueStore }) => {
6        // if your actor stores the data inside OUTPUT
7        const output = await keyValueStore.getValue('OUTPUT');
8        // otherwise access the dataset
9        const { items } = await dataset.getData({ desc: true, limit: 1 });
10
11        return {
12            // important, this will make the object to be treated as a plain
13            // request object instead of data, since we need to update
14            // the headers too with the tokens
15            __isRawRequest: true,
16            headers: {
17                'Authorization': 'Bearer your_token_here'
18            },
19            // Body can be a string or an object
20            body: {
21                date: new Date().toISOString(),
22                status: "publish",
23                title: "My blog post",
24                content: output.content,
25                tags: items[0].tags
26            }
27        }
28    }
29}

Google App Scripts:

1{
2    endpoints: [
3        "https://script.google.com/macros/s/###/exec"
4    ],
5    transformEndpoint: async ({ dataset }) => {
6        // will be sent as an array that you can access through e.postData.contents
7        return {
8            data: (await dataset.getData()).items,
9        };
10    }
11}

Remote form submission:

1{
2    endpoints: [
3        "https://your-remote-form-website.com/form"
4    ],
5    transformEndpoint: async ({ data, dataset }) => {
6        return {
7            // needs this to access the headers parameter
8            __isRawRequest: true,
9            headers: {
10                'Content-Type': 'application/x-www-form-urlencoded'
11            },
12            body: `cus=${data.run.stats.computeUnits}&count=${data.datasetCount}&startedAt=${data.run.startedAt}`
13        }
14    }
15}

Advanced usage:

1{
2    transformEndpoint: async () => {
3        return {
4            __isRawRequest: true,
5            gotScraping: {
6                // this can effectively call gotScraping directly
7                url: 'https://new URL',
8                retry: {
9                    limit: 10,
10                    statusCodes: [502,503,504]
11                }
12            }
13        }
14    }
15}

If you throw an Error inside transformEndpoint function, the payload won't be delivered.

License

Apache 2.0