DataKitchen DataOps Documention

Variations

Variations define a specific set of parameters to be used when compiling a Recipe at runtime.
These parameters include runtime settings, schedule settings, graph structure, infrastructure definitions, and source data. These configurations are defined in the variations.json file. Here, variable values defined in variables.json can be overridden via variation overrides.

Below is an example variations.json file:

{
  "mesos-setting-list": {
    "run-now":{
      "schedule":"now",
      "epsilon":"PT30M"
    },
    "daily-run":{
      "schedule":"R/2018-03-08T20:00:00.000Z/PT2H",
      "epsilon":"P1D"
    }
  },
  "override-setting-list": {
    "overrides-1":{
      "ordersfile":"orders-all.csv"
    },
    "overrides-2":{
    	"ordersfile":"orders-daily.csv"
    }
  },
  "variation-list": {
    "manual-run": {
      "mesos-setting":"run-now",
      "override-setting":"overrides-1"
    },
    "scheduled-run": {
    	"mesos-setting":"daily-run",
      "override-setting":"overrides-2"
    }
  }
}

Naming Conventions

Recipe Variations should be named according to DataKitchen's object-naming best practices.

Naming Variations

DataKitchen object names support alpha-numeric and underscore characters and are case-sensitive. Variable names cannot start with numbers.

Variation Definitions

Each variation has at least the following settings included in the variation-list:

  • A schedule setting (AKA mesos setting)
  • An override set

Since these settings can be leveraged by multiple variations, they are declared on their own sections and then referenced by each variation.

Runtime & Scheduling Settings

Here is a list of all available runtime and scheduling settings as defined via mesos-setting-list in the variations.json file:

Setting
Description

schedule

An ISO 8601 scheduling expression, or "now" for running on the moment the order run is created.

epsilon

If the scheduler misses the scheduled run time for any reason, it will still run the job if the time is within this interval. Epsilon must be formatted like an ISO 8601 Duration.

max-ram

The maximum RAM memory the order run should consume, if passes this setting the order run is automatically killed.
The value is expressed in MB.
The default value is 1024 MB.

max-disk

The maximum amount of disk space the order run should consume.
If passes this value the order run is automatically killed.
The value is expressed in MB.
The default value is 2048 MB.

scheduleTimeZone

The time zone to associate to the schedule time.
The default value is "America/New_York"

Additional Order Scheduling Details

You may find additional details regarding the syntax for Chronos scheduling syntax here.

Variable Overrides

The values of the variables declared in variables.json can be overridden by variation overrides in variations.json's override-setting-list. The declaration of these overrides follows the same syntax used for declaring variables in variables.json. The override setting that should serve as the default setting should be placed first in the list.

Updated 14 days ago


Next Up:

Ingredients

Variations


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.