DataKitchen DataOps Documention

Ingredients

Introduction

After recipe variations are declared as Ingredients, they may be incorporated into and reused by other recipes, or called when creating or deleting kitchens. A recipe with these variations is known as an ingredient provider.

Ingredients may be configured such that when they are called, their code may either be rerun to generate fresh output, or a previous run's output may be reused.

Declaring Ingredients

The first step in declaring an ingredient is to enable a recipe variation as an ingredient variation. You can access this setting in two ways.

Enabling a variation as an ingredient

  • From the Recipes page:
    1. Click the Actions menu for the relevant variation, and select Edit.
    2. In the Edit Variation dialog, check the Use Variation as Ingredient checkbox.
    3. Enter a name for the ingredient.
    4. Enter a change message, and click the Update button.
  • From the Variation page:
    1. From the default Graph tab view, click the Ingredient Settings tab.
    2. Check the Use Variation as Ingredient checkbox.
    3. Enter a name for the ingredient in the Ingredient Title field.
    4. You can add ingredient details on this tab if needed. See instructions below.
    5. Enter a change message, and click the Update button.

Runtime & Timing Settings

When declaring a recipe variation as an ingredient, make sure that the Variation's Mesos setting is set to run once ("schedule": "now"). Also, be sure to provide adequate RAM and disk space for the ingredient to run, if you wish to set an upper limit.

Adding ingredient details
You are required to provide a unique name or title for the new ingredient, but you can include other configurations.

  • Description
  • The name of an associated rollback ingredient
    If the ingredient variation runs during the kitchen creation process, a rollback ingredient can be used to reverse the action during the kitchen deletion process. The rollback ingredient must be another ingredient variation in the same recipe.
  • The variables that flow into the ingredient when it runs
  • The variables that flow out of an ingredient after it runs

Rollback Ingredients

An Ingredient may be built such that it performs a reversible action, like creating an AWS Redshift schema. This reversed action can be built into another Ingredient. These two Ingredients may be coupled together by defining the second Ingredient as the first's rollback ingredient.

Calling Ingredients

With an ingredient variation declared, you can call it during kitchen creation, kitchen deletion, or within a variation as an ingredient node.

From the Kitchen Wizard

The kitchen creation and deletion processes can call Ingredients. This feature is most often used to create infrastructure to be associated with a kitchen and delete that infrastructure with kitchen teardown. This feature supports the concept of a kitchen serving as a segregated working environment for a user.

For example, a data engineer may create a recipe variation to create an AWS Redshift schema, declare this recipe variation as an ingredient, and configure the kitchen wizard with the option to call this ingredient during the kitchen creation process. This allows downstream users to create (and later, delete) a Redshift schema where they can iterate on recipes without impacting production.

Kitchen Wizard Updates

Note that modifications to the default kitchen wizard may be made only through the command line interface.

From Nodes Within Recipes

Ingredients may be called by other recipes via the use of Ingredient Nodes.

Ingredient Nodes Generate Separate Order Runs

When an order executes for a recipe variation, each run of that order is called an order run. If the recipe variation contains an ingredient node that calls an ingredient, this ingredient runs via a new order run of a wholly separate order. In fact, ingredient nodes automatically generate child kitchens where the ingredient's order run is processed.

Updated a day ago


Next Up:

Orders

Ingredients


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.