DataKitchen DataOps Documention

The DataKitchen platform sends alerts via email for various states of every order run. Users can configure the recipients of these notifications at both the kitchen and recipe levels.

Alert Emails

The platform sends email notifications when an order run moves into any of the following states.

Event Types that Generate Alerts

  • Start: An order run started
  • Warning: An order run met the threshold set for the number of node test warnings.
  • Time Limit Exceeded: An order run met the threshold set for the maximum process time for an order.
  • Success/Complete: An order run completed successfully
  • Failure/Error: An order run failed
Sample alert emailed to a user

Sample alert emailed to a user

Alert Integrations

In addition to email notifications, users can configure alerts to display in Slack channels or Microsoft Teams channels.

  • Slack: An app add-on supports email integration. The app generates custom email address, which you can enter in your DataKitchen alerts configurations. Any emails sent to that address are displayed in your specified Slack channel.
  • MS Teams: There are a number of ways to integrate email in your Teams channels. See Office Support and the MailClark app for options.

Alerts Tip

To address your alerts faster, consider segmenting the alerts you send to Slack or MS Teams by alert type (alert or error) and by release environment (dev or prod), so that DataKitchen alerts appear in different channels.

Viewing Order Run Details

The Order Run Details page contains all information related to the run. While the alert emails are point-in-time status reports, this page is the best resource for verifying or troubleshooting order runs.

View the current order run

  • From an alert email, click the linked Order Run ID to access the Order Run Details page for that order. See UI Order Run Detail for more information.
  • From the Orders page, click the linked Order Run ID.
  • From the command line, issue the orderrun-info command. See CLI Order Run Details for more information.

View order runs over time

  • From an alert email, click the linked Order ID to access the metrics for all order runs in the recipe variation in the kitchen.
  • From the Orders page, click the linked Recipe/Variation.

Alerts Configuration

Users can configure alerts at the kitchen level or at the recipe level or both. At these locations, users can enter alert recipients and select the event types for which to send notifications. For some alert types, users must set thresholds at the variation level.

As a best practice, users should set alerts for their kitchens because recipe alert settings get promoted during merges to parent kitchens. Recipe alerts should be used for temporary testing only.

When both kitchen and recipe alerts are configured, the system sends emails to both recipient lists but does not send duplicate emails. The two configurations are ANDed; one does not override the other.

Configure Kitchen Alerts

You can configure kitchen-level alerts in two areas of the web app.

  • During the Create Kitchen process in the Configure Alerts step
  • On the Configure Kitchen page in the Alerts tab

Both locations offer a simple interface to set alert recipients and the type of alerts they will receive. The system will send alerts to the listed recipients for the specified order run event types that occur within the kitchen.

  1. Access a kitchen alerts form.
    • While creating a new kitchen, use the Configure Alerts step.
    • From the Kitchens page or the green menu bar of the web app, click the wrench icon next to the kitchen name to open the Configure Kitchen page, then click the Alerts tab.
  2. Click the Add Recipient button.
  3. In the Email field, enter an email address for the new recipient, or select an address from the drop-down list of active users.
  4. Check one or more checkboxes to enable order run alerts by event type for the recipient.
    • Start: sends an alert at the start of an order run.
    • Warning: sends an alert when the number of test warnings meets a threshold set for the variation. The limit must be set for this alert to be sent.
      Note that node tests may be configured to take different actions on failures--stop, warning, or log.
    • Time limit exceeded: sends an alert if an order run meets a time threshold set for the variation. The limit must be set for this alert to be sent.
    • Success: sends an alert when an order run completes without errors.
    • Failure: sends an alert when an order run stops on an error.
  5. Add more recipients as needed.
  6. Click the Update button.

Configure Recipe Alerts

Users can configure alerts at the recipe level, which are useful for temporary, localized testing. These alerts are configured in a web app dialog or directly in the recipe's description.json file. Optionally, users may create variables for alert recipients then reference them while setting alerts.

Recipe Alerts Merge Up!

Any recipe-level alerts you set will be promoted to a parent kitchen during a kitchen merge. Ensure that the alert settings are appropriate or remove them before merging.

Creating alerts variables
You can set alerts variables in the Variables tab on the Variation page or in the variables.json file for the recipe.

  1. On the Variation page, click the Variables tab.
  2. Click the Add Variable button and enter a name, such as "alerts."
  3. Click in the Recipe Value field, and enter a JSON string to list email recipients for each of the four event types.
  4. Click the Update button.

The following examples show how you can set recipients in the web app tab or in the JSON file.

Users can set alert variables on the Variations page.

Users can set alert variables on the Variations page.

{
    "variable-list": {
        "alerts": {
            "start": [
                "[email protected]",
                "[email protected]"
            ],
            "success": [
                "[email protected]",
                "[email protected]"
            ],
            "failure": [
                "[email protected]",
                "[email protected]",
                "[email protected]"
            ]
        }
    }
}

Referencing variables during alerts setup
The Recipes page provides access to an Alerts dialog as an interface for the underlying description.json file.

  1. On the Recipes page, select a recipe from the list on the left.
  2. Using the Recipe Actions drop-down list at the top right of the page, select Edit Alerts.
  3. In the Edit Recipe Order Run Alerts dialog, click the Add Recipient button.
  4. In the Email field, select a pre-set variable or enter an email address for a new recipient.
    Alternatively, define the recipients as variables in the recipe's variables.json file and reference those variables here.
  5. Check one or more checkboxes to enable order run alerts by event type for the recipient.
    Warning and Time limit exceeded alerts will not be sent unless the thresholds are set in the relevant variation.
  6. Add more recipients as needed.
  7. Click the Add notes for changelog link, and enter a description of your changes. Then click the Update button.

You can switch to the File Editor to edit the recipe's description.json file manually using variables or address values.

{
    "description": "All tests to run in staging environment",
    "recipe-emails": {
        "email-delivery-service": "aws-email",
        "recipe-startup-email-recipients": "{{alerts.start}}",
      	"recipe-complete-email-recipients": "{{alerts.success}}",
      	"recipe-error-email-recipients": "{{alerts.failure}}",
        "recipe-over-duration-email-recipients": [ "[email protected]" ],
        "recipe-warning-email-recipients": [ "[email protected]", "[email protected]" ]
    }
}

Avoid Non-Portable Alerts

You can hard-code alert recipients in the description.json file, or you can define alert variables as kitchen overrides and reference the variables in every recipe. These methods are not optimal, however, because you have to copy the configurations across recipes or enforce a standard variable naming convention.

Configure Alert Thresholds per Variation

The alerts for test warnings and order runs exceeding a time limit require threshold settings at the variation level, regardless of where the alerts are configured--at the kitchen or recipe level.

Thresholds Required

Any time limit exceeded and warning alerts, configured for recipients in a kitchen or in a recipe, will not send unless the order's variation has thresholds configured for the time limit and/or number of test warnings to trigger those alerts.

  1. Access an alert thresholds form.
    • From the Recipes page with the relevant recipe selected, click the Variations tab then click the Actions menu for the relevant variation and select Alert Thresholds.
    • From the Variation page, click the Alert Thresholds tab.
  2. In the Time Limit section, select minutes or hours to define the unit of time, then enter the number of minutes or hours allowed for an order run to process.
  3. Select a radio button to instruct the system to either just send the alert or to stop the order run in addition to sending the alert when an order run exceeds the threshold.
  4. In the Test Warnings Threshold section, enter the number of warnings generated by node tests that will trigger an alert.
  5. Click the Add notes for changelog link, and enter a description of your changes. Then click the Update button.

Alternatively, you can switch to the File Editor and edit the variations.json file manually to add the thresholds.

{
    "variation-list": {
        "my_variation": {
            "graph-setting": "graph_to_run",
            "minutes-to-timeout": 60,
            "action-on-timeout": "warn" | "stop",
            "warnings-count-threshold": 10,
        }
    },
    ...
} 

Updated a day ago

Alerts


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.