DataKitchen DataOps Documention

Ingredient Nodes

Nodes that run recipe variations that have been declared as ingredients.

Recipe variations that have been declared for reuse, to be embedded within other recipes variation graphs, are called Ingredients.

Description.json

Ingredients are called with other recipes via ingredient nodes, which are denoted as DKNode_Ingredient in the node's description.json type field.

{
    "type": "DKNode_Ingredient",
    "description": "[YOUR DESCRIPTION HERE]"
}

Notebook.json

Ingredient nodes specify the Ingredient that they call via their notebook.json configuration file. This is also where the user would specify whether to reprocess the Ingredient or to use its formerly-processed output.

{
  "dk-cloud-ip": "{{https://cloud.datakitchen.io}}",
  "dk-cloud-password": "{{<PASSWORD>}}",
  "dk-cloud-port": "{{443}}",
  "dk-cloud-username": "{{<USERNAME>}}",
  "DKDOC": "this is node definition for a node that calls a recipe ingredient",
  "ingredient-name": "",
  "ingredient-recipe-name": "",
  "ingredient-description": "This is an example of an Ingredient node's notebook.json configuration.",
  "required-ingredient-variables": [
      "{{pre-defined-variable-format}}", 
      "$runtime-variable-format"
  ],
  "ingredient-required-orderrun-results": {
    "orderrun-poll-interval": 10,
    "orderrun-timeout": 60,
    "orderrun-kitchen": "master",
    "orderrun-allow-log-test-results": true,
    "orderrun-allow-warning-test-results": true,
    "orderrun-allow-failure-test-results": false
  }
}

Properties

property
description
input type

dk-cloud-ip

DK Cloud host name. Default value is https://cloud.datakitchen.io

String; host name or IP address. Minimum of 5, maximum of 30 characters.

dk-cloud-password

The user's DataKitchen password.

String; characters, upper and lower case, and numbers, with no spaces. Minimum of 8, maximum of 25 characters.

dk-cloud-port

TCP Port. The default value is 443.

Integer; numeric values between 1 and 65535.

dk-cloud-username

The user's DataKitchen username.

Alphanumeric; letters and numbers with no spaces.
Minimum of 5, maximum of 30 characters.

DKDOC

ingredient-description

ingredient-name

The name of the Ingredient called by this node. This name need not be equivalent to the name of the Recipe Variation from which it is declared.

Alphanumeric; letters and numbers with no spaces.
Minimum of 1, maximum of 256 characters.

ingredient-recipe-name

The name of the Recipe containing the Variation declared as an Ingredient, to be called by this node.

Alphanumeric; letters and numbers with no spaces.
Minimum of 1, maximum of 256 characters.

ingredient-required-orderrun-results

required-ingredient-variables

Variables that flow into an Ingredient node. May or may not be required, depending on configuration for the declared Ingredient.

orderrun-allow-failure-test-results

Outflowing variable. Determines whether test failures are exported to the parent OrderRun for the Recipe containing the Ingredient node.

Boolean; true or false.

orderrun-allow-log-test-results

Outflowing variable. Determines whether test logs are exported to the parent OrderRun for the Recipe containing the Ingredient node.

Boolean; true or false.

orderrun-allow-warning-test-results

Outflowing variable. Determines whether test warnings are exported to the parent OrderRun for the Recipe containing the Ingredient node.

Boolean; true or false.

orderrun-kitchen

Outflowing variable.

Alphanumeric; letters and numbers with no spaces.
Minimum of 1, maximum of 256 characters.

orderrun-poll-interval

Outflowing variable. The frequency, in seconds, at which the parent OrderRun should poll the child OrderRun, which runs the Ingredient, for its status.

Integer.

orderrun-timeout

Outflowing variable. The length of time, in seconds, after which the child OrderRun, running the Ingredient, will timeout.

Integer.

Examples

Example: Mapping US ZIP codes to Metropolitan Statistical Areas

{
    "type": "DKNode_Ingredient",
    "description": ""
}
{
    "dk-cloud-ip": "{{ dkcloudconfig.ip }}",
    "dk-cloud-password": "{{ dkcloudconfig.password }}",
    "dk-cloud-port": "{{ dkcloudconfig.port }}",
    "dk-cloud-username": "{{ dkcloudconfig.username }}",
    "required-ingredient-variables": [],
    "ingredient-name": "ingredient-put-us-zips",
    "ingredient-recipe-name": "ingredient-provider-recipe",
    "ingredient-description": "this ingredient either uses the existing table in redshift or re-builds the table by running a recipe",
    "ingredient-required-orderrun-results": {
        "orderrun-checks-timing": [
            2,
            4,
            8,
            16,
            32,
            64,
            128,
            256,
            512,
            1024,
            2048
        ],
        "orderrun-kitchen": "",
        "orderrun-allow-log-test-results": true,
        "orderrun-allow-warning-test-results": true,
        "orderrun-allow-failure-test-results": false
    }
}

Updated 2 months ago


Ingredient Nodes


Nodes that run recipe variations that have been declared as ingredients.

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.