DataKitchen DataOps Documention

Ingredients

Introduction

After recipe variations are declared as Ingredients, they may be incorporated into and reused by other recipes, or called when creating or deleting kitchens. A recipe with these variations is known as an ingredient provider.

When another process calls ingredients, the ingredients execute their functions as a kind of sub-process.

Declaring Ingredients

The first step in declaring an ingredient is to enable a recipe variation as an ingredient variation. You can access this setting in two ways.

Enabling a variation as an ingredient

  • From the Recipes page:
    1. Click the Actions menu for the relevant variation, and select Edit.
    2. In the Edit Variation dialog, check the Use Variation as Ingredient checkbox.
    3. Enter a name for the ingredient.
    4. Enter a change message, and click the Update button.
  • From the Variation page:
    1. From the default Graph tab view, click the Ingredient Settings tab.
    2. Check the Use Variation as Ingredient checkbox.
    3. Enter a name for the ingredient in the Ingredient Title field.
    4. You can add ingredient details on this tab if needed. See instructions below.
    5. Enter a change message, and click the Update button.

Runtime & Timing Settings

When declaring a recipe variation as an ingredient, make sure that the Variation's Mesos setting is set to run once ("schedule": "now"). Also, be sure to provide adequate RAM and disk space for the ingredient to run, if you wish to set an upper limit.

Adding ingredient details
You are required to provide a unique name or title for the new ingredient, but you can include other configurations.

  • Description
  • The name of an associated rollback ingredient
    If the ingredient variation runs during the kitchen creation process, a rollback ingredient can be used to reverse the action during the kitchen deletion process. The rollback ingredient must be another ingredient variation in the same recipe.
  • The variables that flow into the ingredient when it runs
  • The variables that flow out of an ingredient after it runs

Rollback Ingredients

An Ingredient may be built such that it performs a reversible action, like creating an AWS Redshift schema. This reversed action can be built into another Ingredient. These two Ingredients may be coupled together by defining the second Ingredient as the first's rollback ingredient.

Calling Ingredients

With an ingredient variation declared, you can call it during kitchen creation, kitchen deletion, or within a variation as an ingredient node.

From the Kitchen Wizard

The kitchen creation and deletion processes can call ingredients. This feature is most often used to create infrastructure to be associated with a kitchen and delete that infrastructure with kitchen teardown.

  • Ingredients for creating kitchens: Ingredient variations intended for use in the create kitchen process must exist in recipes in the parent kitchen so they may be copied and used while creating the child kitchen.
  • Ingredients for delete kitchens: Ingredient variations that will run during the delete kitchen process must exist in the kitchen being deleted.

For example, a data engineer may create a recipe variation to create an AWS Redshift schema, declare this recipe variation as an ingredient, and configure the kitchen wizard with the option to call this ingredient during the kitchen creation process. This allows downstream users to create (and later, delete) a Redshift schema where they can iterate on recipes without impacting production.

Kitchen Wizard Updates

Note that modifications to the default kitchen wizard may be made only through the command line interface.

From Nodes Within Recipes

Ingredients may be called by other recipes via the use of Ingredient Nodes.

Create and Configure an Ingredient Node

  1. Select your recipe, and click on the variation to open the graphical view.
  2. Click the Add Node button, and select the Create New Node radio button.
  3. Select the Ingredient box, enter a node name, and click the Create button.
  4. Connect the ingredient node to other nodes in the graph using edges as needed.
  5. Enter a change message, and click the Update button.
  6. Click on the node, then click the Edit Node link the in the sidebar.
  7. In the Node Editor, select the kitchen, the recipe, and the ingredient variation to run in this node.
    Note that only kitchens you are authorized to access appear in the Kitchen drop-down list.
  8. Review the ingredient summary information and graph.
    You can use the Edit Ingredient Variation link to make changes to the underlying ingredient, but be aware you may be changing the ingredient for other processes dependent on that ingredient.
  9. In the Configuration tab, change the Poll Interval if needed to instruct the ingredient node to check more or less frequently for the completion of ingredient processing. The default interval is 60 seconds.
  10. Modify the ingredient Timeout Duration if needed to control the time the ingredient node will allow for the ingredient to run. The default is 600 seconds.
  11. Enter values or select secrets or variables for any Inflowing Variables required by the ingredient variation. See Inflowing Variable Validations for more information.
    You can replace the default entries in this section, which are the active values of the variables in the ingredient variation.
  12. Review the Outflowing Variables configured in the ingredient variation if present.
  13. In the Tests tab, create data verification tests as needed. These tests will run after the ingredient is executed.
  14. Enter a change message, and click the Update button.

Ingredient Node Response to Ingredient Tests
The results of the tests run on the ingredient variation may affect the platform's processing of the ingredient node that calls that ingredient variation. The ingredient tests do not, however, affect the tests defined for the ingredient node itself.

  • If the ingredient order run has test failures or failures of any kind, the ingredient node reports an error status at the next 60-second polling interval and stops the main order run immediately.
  • If the ingredient order run has test warnings (and no failures), the ingredient node reports a success status and continues the main order run as usual.
  • If the ingredient order run has test logs (and no failures), the ingredient node reports a success status and continues the main order run as usual.

Ingredient Nodes Generate Separate Order Runs

When a recipe variation containing an ingredient node calls the underlying ingredient, that ingredient runs via a new order run of a wholly separate order.

In fact, when the system processes any ingredient node, it creates a temporary child kitchen of the ingredient variation's kitchen in which to process the separate order run.

Updated 2 days ago


Next Up:

Orders

Ingredients


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.