API Response Watcher

A utility to check if some API endpoint's response has changed.

Purpose

This Actor allows to to test a list of endpoints to check if their response change over time. For example, you may want to test some API you are using to scrape the content of a website and be notified if some data has been added, changed, or removed to a specific endpoint.

How it works, and how to use it

The first time you test an endpoint, the Actor creates a JSON schema out of its response and stores it in the Key-value store. You can specify the name or the ID of the store you want to use for this, into "Next Key-value store", in the Actor's input. For more information about Key-value stores, see here.

The next time you want to test the same endpoint, you can move the Key-value store ID from "Next Key-value store" to "Previous Key-value store": in this way, the Actor will use the previously generated JSON schema to validate the new response. If the validation fails, it will output the differences.

Finally, the Actor will regenerate the JSON schema out of the new answer, merge it with the old schema and save it in the Key-value store pointed by "Next Key-value store". If the previous and next stores are the same, the old schema will be overwritten.

Input

Here is a sample input:

1{
2    "endpoints": [
3        "https://dummyjson.com/products",
4        "curl 'https://dummyjson.com/carts' -H 'Accept: application/json'"
5    ],
6    "prevKvsName": "abcde12345",
7    "nextKvsName": "schemas",
8    "noRequired": false,
9    "doMergeSchemas": true,
10    "reportEmail": "my.email@apify.com",
11    "reportSlackChannel": "#api-watcher",
12    "reportSlackToken": "*******"
13}

Some notes

• The Actor will test two endpoints: https://dummyjson.com/products and https://dummyjson.com/carts:
  • the first endpoint is a simple public endpoint's URL, which will be contacted through a GET request;
  • the second endpoint is actually a cURL command, which will be parsed. You can use the cURL syntax to describe more complex requests, for example ones using a method different from GET, custom headers, and custom payload.
• The Actor will validate their responses against some JSON schema from the Key-value store abcde12345, if found.
• The Actor will store the updated schemas in a named Key-value store called schemas. Each endpoint will have a record key in the Key-value store based on its URL.
• While generating and updating the JSON schema, the Actor will mark all the JSON properties as "required", because noRequired is set to false.
• The Actor will merge the old and new schemas before storing it into the Key-value store, because doMergeSchema is true.
• At the end of the Run, if some differences were found:
  • an email will be sent to my.email@apify.com, calling the Actor apify/send-mail, with a link to the Run default dataset, where those differences were stored;
  • a message will be written to #api-watcher, calling the Actor katerinahronik/slack-message and using the given token.

Further considerations

• The Actor is able to automatically distinguish between store names and IDs.
• If you don't specify a "Next Key-value store" in the input, the Actor won't perform validation.
• If you don't specify a "Previous Key-value store" in the input, the Actor will still store the generated schema into the default, unnamed storage. Unnamed storage has an expiration date: for more information, see here.

Output

Here is a sample output:

1{
2	"url": "https://some-api/data/1",
3	"data": {
4        "id": 1
5    },
6	"prevSchema": {
7		"type": "object",
8		"properties": {
9			"id": {
10                "type": "string"
11            }
12		}
13	},
14	"nextSchema": {
15		"type": "object",
16		"properties": {
17			"id": {
18                "type": "integer"
19            }
20		}
21	},
22	"mergedSchema": {
23		"type": "object",
24		"properties": {
25			"id": {
26                "type": [
27                    "integer",
28                    "string"
29                ]
30            }
31		}
32	},
33	"validationErrors": [
34		{
35			"instancePath": "/id",
36			"schemaPath": "#/properties/id/type",
37			"message": "must be string"
38		}
39	]
40}

The id in the response, which was previously a string, is now an integer. The old and new schemas were merged, because doMergeSchema in the input was true. The merged schema admits bot a string and an integer as id, so, if it will be used to validate the next Run, both types will pass the validation.

How to monitor some endpoints

You can leverage Apify's schedules. Just create a Task with the desired input and run it periodically: you can set it up to receive a notification when some changes are detected.

If you set the same previous and next Key-value stores, the reference schema will be updated every time, so that you will be notified just once when a change is detected.

Or, if you prefer, you can set two different values for the two stores, even leaving the next Key-value store blank, and the change will be detected every time, until you manually update the reference schema.