DataKitchen DataOps Documention

CLI Quickstart3

In this guide we'll cover an overview of DataOps orchestration via DKCloudCommand, DataKitchen's command line tool. Specific topics covered include:

  • Adding & Configuring a Container-based Node for GUI tools, specifically for Pentaho DI
  • Validating Local Changes Applied to a Recipe
  • Creating Recipe Variations to Provide Runtime Flexibility
  • Using Locally Configured Tool to View Local vs. Remote File Differences
  • Compiling Entire Recipes to Check the Proper Application of Parameterization
  • Viewing Lists of Recipe Variations
  • Deleting Remote Copies of Recipe Files
  • Deleting Entire Versions of Remote Recipes

Super-Fast CLI Quickstart3

Super-Fast CLI Quickstart3

Advanced users who need less detailed documentation to get up and running with DataKitchen's DataOps platform may jump right to the SUPER-FAST CLI QUICKSTART3 tab of this guide's Synopsis for a menu of abridged commands. This route provides a fully complete recipe for this guide. Alternatively, you may follow the steps below to build out the recipe starting from a copy of the recipe built in CLI Quickstart2.

Prepare the Environment

  1. Use the kitchen-list command to view the current lineup of kitchens?
~/Kitchens $ dk kitchen-list
~/Kitchens $ dk kl

Output: kitchen-list command

YYYY-MM-DD HH:MM:SS - Getting the list of kitchens
kitchen-list returned 3 kitchens
+---- master
    +---- Production
        +---- Dev_Sprint_1
  1. Use the recipe-list command to determine what recipes exist in the Dev_Sprint_1 kitchen.
~/Kitchens $ cd Dev_Sprint_1
~/Kitchens/Dev_Sprint_1 $ dk recipe-list
~/Kitchens $ cd Dev_Sprint_1
~/Kitchens/Dev_Sprint_1 $ dk rl
~/Kitchens $ dk rl -k Dev_Sprint_1

Output: recipe-list command

YYYY-MM-DD HH:MM:SS - Getting the list of Recipes for Kitchen 'Dev_Sprint_1'
list_recipe - elapsed: 0
DKCloudCommand.list_recipe returned 1 recipes
  Recipe_Template
  1. Use the kitchen-create command to generate a new child kitchen named Feature_Sprint_3 that you will used to further build out the Recipe_Template recipe.

Create a Uniquely-Named Kitchen

Append your name when creating the kitchen so that you have your own, segregated workspace. For example, if your name is Chef, you would create a kitchen named Feature_Sprint_3_Chef.

~/Kitchens/Dev_Sprint_1 $ cd ..
~/Kitchens $ dk kitchen-create --parent Dev_Sprint_1 Feature_Sprint_3
~/Kitchens/Dev_Sprint_1 $ cd ..
~/Kitchens $ dk kc -p Dev_Sprint_1 Feature_Sprint_3

Output: kitchen-create command

YYYY-MM-DD HH:MM:SS - Creating kitchen Feature_Sprint_3 from parent kitchen Dev_Sprint_1
DKCloudCommand.create_kitchen created Feature_Sprint_3
  1. Use the kitchen-get command to get a local copy of the new kitchen and the Recipe_Template recipe it has inherited from its parent kitchen.
~/Kitchens $ dk kitchen-get Feature_Sprint_3 --recipe Recipe_Template
~/Kitchens $ dk kg Feature_Sprint_3 -r Recipe_Template

Output: kitchen-get command

YYYY-MM-DD HH:MM:SS - Getting kitchen 'Feature_Sprint_3'
list_recipe - elapsed: 0
Got Kitchen 'Feature_Sprint_3'
DKCloudCommand.get_recipe has 6 sections
  Recipe_Template/placeholder-node1
  Recipe_Template
  Recipe_Template/resources
  Recipe_Template/load-profits-node
  Recipe_Template/placeholder-node2
  Recipe_Template/load-profits-node/actions

Add Pentaho DI Node

Understand the Tasks

In Quickstart2, you added an action node to the Recipe_Template recipe that leveraged raw SQL to copy data from an AWS S3 bucket to an AWS Redshift database. Next, you will create another action node that will complete data work similar to that performed by the load-profits-node.

Rather than leveraging raw SQL, this new node will encapsulate the data integration work that might be completed by a Data Engineer using Pentaho's Data Integration graphical interface tool. Recall that DataKitchen's DataOps Platform integrates well with all existing tools in your team's toolchain, meaning that you can implement DataOps at your organization without updating your existing resources.

The Data Engineer might perform the following steps using the Pentaho DI GUI.

A view from [Transmit](https://panic.com/transmit/) of the AWS S3 bucket containing source data in TXT files.

A view from Transmit of the AWS S3 bucket containing source data in TXT files.

Creating an AWS Redshift database schema using [Pentaho DI](http://www.pentaho.com/product/data-integration).  Note the use of parameterization, which allows for runtime flexibility.

Creating an AWS Redshift database schema using Pentaho DI. Note the use of parameterization, which allows for runtime flexibility.

Creating AWS Redshift tables and populating them with data copied from an AWS S3 bucket using [Pentaho DI](http://www.pentaho.com/product/data-integration).

Creating AWS Redshift tables and populating them with data copied from an AWS S3 bucket using Pentaho DI.

Using [Pentaho DI](http://www.pentaho.com/product/data-integration) to generate a row count from one of the newly created tables, which will serve as automated test data for the **pentaho-di-node**.  The row count is exported to a .txt file.

Using Pentaho DI to generate a row count from one of the newly created tables, which will serve as automated test data for the pentaho-di-node. The row count is exported to a .txt file.

In the DataKitchen platform, you will add a new node to the Recipe_Template recipe that is designed to create schemas and generate tables in AWS Redshift, populate the tables with data gathered from an AWS S3 bucket, and finally, procure data, via a key, to be leveraged by a node test.

Your work begins by declaring variables, creating a node directory, defining the node, then connecting it in the recipe graph. Then you will work on the node configuration.

Update variables.json I

In this exercise, you will update variables.json to include a number of new Docker Hub variables. Since you are building your new node to leverage a GUI tool rather than raw SQL, you will place the data operations completed by the new node with a Docker Container.

  1. Recall the list of secrets that exist in the vault associated with your customer account.
~/Kitchens $ dk secret-list
~/Kitchens $ dk sl

Output: secret-list command

YYYY-MM-DD HH:MM:SS - Getting the list of secrets
---- global secrets ----
  vault://aws/account_id
  vault://aws/role
  vault://dockerhub/email
  vault://dockerhub/namespace
  vault://dockerhub/password
  vault://dockerhub/username
  vault://redshift/database
  vault://redshift/hostname
  vault://redshift/password
  vault://redshift/port
  vault://redshift/username
  vault://s3_schema/bucket
  1. Update variables.json by replacing the contents of the file with the text below.
    Note the ability to group variables for a common tool within a Python dictionary.

Update Email and Schema Values

Just as you did in CLI Quickstart2, update the alerts and schema values after copying the text below into the variables.json file. Replace YOUR-EMAIL-HERE with your preferred email address, and replace YOUR-SCHEMA-NAME-HERE with your first initial and last name. Save the file.

{
    "name": "Recipe_Template QS3 Variables",
    "variable-list": {
        "awsaccountid": "#{vault://aws/account_id}",
        "awsrole": "#{vault://aws/role}",
        "dockerhub": {
              "email": "#{vault://dockerhub/email}",
              "namespace": "#{vault://dockerhub/namespace}",
              "password": "#{vault://dockerhub/password}",
              "username": "#{vault://dockerhub/username}"
        },
        "alerts": {
              "start": "YOUR-EMAIL-HERE", 
              "success": "YOUR-EMAIL-HERE", 
              "error": "YOUR-EMAIL-HERE"
        },
        "redshiftdatabase": "#{vault://redshift/database}",
        "redshifthostname": "#{vault://redshift/hostname}",
        "redshiftpassword": "#{vault://redshift/password}",
        "redshiftport": "#{vault://redshift/port}",
        "redshiftschema": "YOUR-SCHEMA-NAME-HERE",
        "redshiftusername": "#{vault://redshift/username}",
        "s3bucketname": "#{vault://s3_schema/bucket}",
        "s3bucketpath": "superstore/customer/sales-profitability"
      }
}
  1. Use the recipe-status command to verify your change.
~/Kitchens $ cd Feature_Sprint_3/Recipe_Template
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk recipe-status
~/Kitchens $ cd Feature_Sprint_3/Recipe_Template
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rs

Output: recipe-status command

YYYY-MM-DD HH:MM:SS - Getting the status of Recipe 'Recipe_Template' in Kitchen 'Feature_Sprint_3'
    versus directory '~/Kitchens/Feature_Sprint_3/Recipe_Template'
1 files are modified on local:
    variables.json

11 files are unchanged

For now, do not update individual changes to the remote version of the Recipe_Template recipe. Later, you will batch update both new and modified recipe files using the recipe-update command.

Add Node Description

Every recipe graph node requires its own sub-directory and description.json file.

  1. Create the node directory. Name this new node pentaho-di-node.
~/Kitchens/Feature_Sprint_3/Recipe_Template $ mkdir pentaho-di-node
~/Kitchens/Feature_Sprint_3/Recipe_Template $ cat > pentaho-di-node/description.json
  1. Paste the following text to your command-line prompt:
{
    "type": "DKNode_Action",
    "description": "Encapsulates work done by a Data Engineer using Pentaho DI to setup an AWS Redshift database schema, create tables, populate tables with data from an AWS S3 bucket, and generate a row count to be used for automated testing."
}
  1. Press Return/Enter followed by Ctrl+D to save the new file and return to the prompt.
A view of the pentaho-di-node's description.json file.

A view of the pentaho-di-node's description.json file.

  1. Check your work by running the recipe-status command again.
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk recipe-status
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rs

Output: recipe-status command

YYYY-MM-DD HH:MM:SS - Getting the status of Recipe 'Recipe_Template' in Kitchen 'Feature_Sprint_3'
    versus directory '~/Kitchens/Feature_Sprint_3/Recipe_Template'
1 files are modified on local:
    variables.json

1 files are local only:
    pentaho-di-node/description.json

1 directories are local only:
    pentaho-di-node

11 files are unchanged

Update variations.json

The recipe's variations.json file stores information about recipe variations, variation overrides, schedules and graph nodes.

You will now update the graph-setting-list in variations.json to define the new node's sequence and connections in the recipe graph. You can do this before or after configuring the node, but the node's directory and description must be in place first.

  1. View the current contents of the variations.json file.
    It contains the load-profits-node created in Quickstart 2, along with the default placeholder nodes.
~/Kitchens/Feature_Sprint_3/Recipe_Template $ cat variations.json

Output: concatenate command for the recipe's variations.json file

{
    "variation-list": {
        "variation1": {
            "graph-setting": "placeholder-graph"
        }
    }
, 
    "graph-setting-list": {
        "placeholder-graph": [
            [
                "placeholder-node1", 
                "placeholder-node2"            ], 
            [
                "placeholder-node2", 
                "load-profits-node"            ]        ]
    }
}
  1. Replace the existing content in variations.json with the following text.
    This content adds the new pentaho-di-node node to the recipe graph such that it is positioned as the final node in the series.
{
    "variation-list": {
        "variation1": {
            "graph-setting": "placeholder-graph"
        }
    }
, 
    "graph-setting-list": {
        "placeholder-graph": [
            [
                "placeholder-node1", 
                "placeholder-node2" 
           ] ,
           [
               "placeholder-node2",
               "load-profits-node"
           ] ,
           [
               "load-profits-node",
               "pentaho-di-node"
           ]
       ]
    }
}
  1. Run the recipe-status command to check your work.
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk recipe-status
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rs

Output: recipe-status command

YYYY-MM-DD HH:MM:SS - Getting the status of Recipe 'Recipe_Template' in Kitchen 'Feature_Sprint_3'
    versus directory '~/Kitchens/Feature_Sprint_3/Recipe_Template'
2 files are modified on local:
    variables.json
    variations.json

1 files are local only:
    pentaho-di-node/description.json

1 directories are local only:
    pentaho-di-node

10 files are unchanged

Configure Node

The pentaho-di-node is an action node that contains an /actions subdirectory. Because pentaho-di-node leverages a container, the configuration is somewhat different from the action node you created in Quickstart 2.

  1. Create an /actions sub-directory and then a JSON configuration file named DKDataSource_Container.json.
~/Kitchens/Feature_Sprint_3/Recipe_Template $ mkdir pentaho-di-node/actions
~/Kitchens/Feature_Sprint_3/Recipe_Template $ cat > pentaho-di-node/actions/DKDataSource_Container.json
  1. Copy the following text to your command-line prompt.
{
    "name": "DKDataSource_Container",
    "type": "DKDataSource_Container",
    "dockerhub-email": "{{dockerhub.email}}",
    "dockerhub-namespace": "{{dockerhub.namespace}}",
    "dockerhub-password": "{{dockerhub.password}}",
    "dockerhub-username": "{{dockerhub.username}}",
    "image-repo": "ac_pentahodi_container",
    "image-tag": "latest",
    "data-container-name": "data_ac_pentahodi",
    "inside-container-file-mount": "/dk/AnalyticContainer/PentahoDIContainer",
    "inside-container-file-directory": "docker-share",
    "container-input-configuration-file-path": "Recipe_Template/pentaho-di-node/actions/docker-share",
    "container-input-configuration-file-name": "config.json",
    "container-output-log-file": "ac_logger.log",
    "container-output-progress-file": "progress.json",
    "hide-dockerfile-location": "/dk/AnalyticContainer/PentahoDIContainer/Dockerfile",
    "keys": {
        "get-test-data": {
            "container-key": "get-test-data"
        }
    },
    "tests": {
        "get-test-data-count": {
            "test-variable": "get-test-data",
            "type": "test-contents-as-integer",
            "applies-to-keys": ["get-test-data"],
            "action": "stop-on-error",
            "keep-history": "True",
            "test-logic": {
                "test-variable": "get-test-data",
                "test-compare": "greater-than",
                "test-metric": "10000"
            }
        }
    }
}
  1. Press Enter/Return followed by Ctrl+D to save the new file and return to the prompt.
The configuration file for the container used by pentaho-di-node.  This container encapsulates the processing of files generated from Pentaho DI's GUI tool.

The configuration file for the container used by pentaho-di-node. This container encapsulates the processing of files generated from Pentaho DI's GUI tool.

  1. Review the code in the new configuration file.
    Note that the configuration for the node includes the definition of a key and a test.

The data work completed by this node (creating schemas and tables) is driven by the Pentaho Data Integration KTR files, which are located within a container. Still, the code for the automated testing functionality for this node is located outside of the container in the container configuration file above.

In short, Pentaho DI generates a row count from one of the tables it populates; it then generates a temporary file to house this value; and the node's configuration file tests this row count. In this case, if the test fails, the order will stop in place as the action field has been set to stop-on-error.

Implementing automated tests for each node in your recipe increases your team's confidence in the quality of the insights generated by your data analytic pipelines.

  1. Confirm your change using the recipe-status command.
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk recipe-status
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rs

Output: recipe-status command

YYYY-MM-DD HH:MM:SS - Getting the status of Recipe 'Recipe_Template' in Kitchen 'Feature_Sprint_3'
    versus directory '~/Kitchens/Feature_Sprint_3/Recipe_Template'
2 files are modified on local:
    variables.json
    variations.json

2 files are local only:
    pentaho-di-node/description.json
    pentaho-di-node/actions/DKDataSource_Container.json

2 directories are local only::
    pentaho-di-node
    pentaho-di-node/actions

10 files are unchanged
  1. Create an additional node sub-directory under /actions named /docker-share.
    This directory is common to all recipe graph nodes that leverage containers to encapsulate toolchain files. This is the recipe directory where Pentaho's KTR files will be located.
~/Kitchens/Feature_Sprint_3/Recipe_Template $ mkdir pentaho-di-node/actions/docker-share

Container Node Files

The files that perform data work for a given node are most often located in a recipe's /resources directory. However, for nodes that leverage containers, these files are instead located within the node itself, in the node's /docker-share directory. This is because container nodes encapsulate everything needed to complete the work for which they were designed.

  1. Create a config.json file to sit within pentaho-di-node/actions/docker-share.
    This file passes the required infrastructure credentials into the container.
~/Kitchens/Feature_Sprint_3/Recipe_Template $ cat > pentaho-di-node/actions/docker-share/config.json
  1. Copy the following text to your command-line prompt:
{
  "loglevel": "Maximum",
  "DKDOC-logelevel": ["Miminum", "Maximum"],
  "keys": {
    "create-schema": {
      "filename": "create-schema.ktr",
      "file-parameters": {
        "database": "{{redshiftdatabase}}",
        "hostname": "{{redshifthostname}}",
        "password": "{{redshiftpassword}}",
        "port": "{{redshiftport}}",
        "schema": "{{redshiftschema}}",
        "username": "{{redshiftusername}}"
      }
    },
    "load-data": {
      "filename": "create-and-load-superstore-data.ktr",
      "file-parameters": {
        "accountid": "{{awsaccountid}}",
        "role": "{{awsrole}}",
        "database": "{{redshiftdatabase}}",
        "hostname": "{{redshifthostname}}",
        "password": "{{redshiftpassword}}",
        "port": "{{redshiftport}}",
        "bucket": "{{s3bucketname}}",
        "schema": "{{redshiftschema}}",
        "username": "{{redshiftusername}}"
      }
    },
    "get-test-data": {
      "filename": "get-count-of-superstore-data.ktr",
      "file-parameters": {
        "database": "{{redshiftdatabase}}",
        "hostname": "{{redshifthostname}}",
        "password": "{{redshiftpassword}}",
        "port": "{{redshiftport}}",
        "schema": "{{redshiftschema}}",
        "username": "{{redshiftusername}}"
      },
      "integration-output-filename": "get-test-data.txt"
    }
  }
}
  1. Press Enter/Return then Ctrl+D to save the file and return to the prompt.
A view of pentaho-di-node's configuration file, config.json.

A view of pentaho-di-node's configuration file, config.json.

  1. Check your work by running the recipe-status command again.
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk recipe-status
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rs

Output: recipe-status command

YYYY-MM-DD HH:MM:SS - Getting the status of Recipe 'Recipe_Template' in Kitchen 'Feature_Sprint_3'
    versus directory '~/Kitchens/Feature_Sprint_3/Recipe_Template'
2 files are modified on local:
    variables.json
    variations.json

3 files are local only:
    pentaho-di-node/description.json
    pentaho-di-node/actions/DKDataSource_Container.json
    pentaho-di-node/actions/docker-share/config.json

3 directories are local only::
    pentaho-di-node
    pentaho-di-node/actions
    pentaho-di-node/actions/docker-share

10 files are unchanged

Add Toolchain Files

This example container node reflects the data integration work a Data Engineer might complete leveraging the Pentaho Data Integration GUI, specifically to create database schemas in AWS Redshift, populate tables, and return test data.

  1. Obtain the path to the public repository containing the KTR files for this exercise.
  2. Navigate to your local /docker-share sub-directory for the pentaho-di-node, and execute the following commands to download copies of the Pentaho KTR files to be used by this recipe.
    • create-schema.ktr
    • create-and-load-superstore-data.ktr
    • get-count-of-superstore-data.ktr
~/Kitchens/Feature_Sprint_3/Recipe_Template $ cd pentaho-di-node/actions/docker-share
~/Kitchens/Feature_Sprint_3/Recipe_Template/pentaho-di-node/actions/docker-share $ curl -L -O (link to raw public repo file1)
~/Kitchens/Feature_Sprint_3/Recipe_Template/pentaho-di-node/actions/docker-share $ curl -L -O (link to raw public repo file2)
~/Kitchens/Feature_Sprint_3/Recipe_Template/pentaho-di-node/actions/docker-share $ curl -L -O (link to raw public repo file3)
KTR files added to the structure of Recipe_Template in Feature_Kitchen_3.

KTR files added to the structure of Recipe_Template in Feature_Kitchen_3.

  1. Again, check the output of the recipe-status command.
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk recipe-status
~/Kitchens/Feature_Sprint_3/Recipe-Template $ dk rs

Output: recipe-status command

YYYY-MM-DD HH:MM:SS - Getting the status of Recipe 'Recipe_Template' in Kitchen 'Feature_Sprint_3'
    versus directory '~/Kitchens/Feature_Sprint_3/Recipe_Template'
2 files are modified on local:
    variables.json
    variations.json

6 files are local only:
    pentaho-di-node/description.json
    pentaho-di-node/actions/DKDataSource_Container.json
    pentaho-di-node/actions/docker-share/config.json
    pentaho-di-node/actions/docker-share/create-and-load-superstore-data.ktr
    pentaho-di-node/actions/docker-share/create-schema.ktr
    pentaho-di-node/actions/docker-share/get-count-of-superstore-data.ktr

3 directories are local only::
    pentaho-di-node
    pentaho-di-node/actions
    pentaho-di-node/actions/docker-share

10 files are unchanged

Recipe Validation I

  1. Before updating local recipes to remote, use the recipe-validate command to check for errors that would prevent a successful order run.
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk recipe-validate --variation Variation1
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rv -v Variation1

Output: recipe-validate command

YYYY-MM-DD HH:MM:SS - Validating recipe/variation Recipe_Template.Variation1 in Kitchen Feature_Sprint_3
Validating recipe with local changes applied
DKCloudCommand.recipe_validate succeeded
No recipe issues identified.

Recipe Update I

  1. Since no issues were identified by the recipe validation, push local changes in batch to the remote copy of Recipe_Template via the recipe-update command.
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk recipe-update --message "Add Pentaho DI node"
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk ru -m "Add Pentaho DI node"

Output: recipe-update command

YYYY-MM-DD HH:MM:SS - Updating all changed files in Recipe (Recipe_Template) in Kitchen(Feature_Sprint_3) with message (Add Pentaho DI node)
Update results:

New files:
    pentaho-di-node/description.json
    pentaho-di-node/actions/DKDataSource_Container.json
    pentaho-di-node/actions/docker-share/config.json
    pentaho-di-node/actions/docker-share/create-and-load-superstore-data.ktr
    pentaho-di-node/actions/docker-share/create-schema.ktr
    pentaho-di-node/actions/docker-share/get-count-of-superstore-data.ktr
Updated files:
    variables.json
    variations.json
Deleted files:
    None

Issues:

No issues found
  1. Confirm that the local and remote versions of the Recipe_Template recipe are now equivalent using the recipe-status command.
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk recipe-status
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rs

Output: recipe-status command

YYYY-MM-DD HH:MM:SS - Getting the status of Recipe 'Recipe_Template' in Kitchen 'Feature_Sprint_3'
    versus directory '~/Kitchens/Feature_Sprint_3/Recipe_Template'
18 files are unchanged

Test Order I

  1. Use the order-run command to run an order for the updated Recipe_Template recipe.
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk order-run Variation1 --yes
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk or Variation1 -y
~/Kitchens $ dk or -k Feature_Sprint_3 -r Recipe_Template Variation1 -y

Output: order-run command

YYYY-MM-DD HH:MM:SS - Create an Order:
    Kitchen: Feature_Sprint_3
    Recipe: Recipe_Template
    Variation: Variation1

Order ID is: DKRecipe#hw#Recipe_Template#Variation1#Feature_Sprint_3#1508524645286
  1. Review information on the last order run processed in the Feature_Sprint_3 kitchen using the orderrun-info command.

You can also check the Order Run Details page in the DataKitchen UI for order run results.

~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk orderrun-info
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk ori
~/Kitchens $ dk ori -k Feature_Sprint_3

Output: orderrun-info command

YYYY-MM-DD HH:MM:SS - Display Order-Run details from kitchen Feature_Sprint_3

ORDER RUN SUMMARY

Order ID:       DKRecipe#hw#Recipe_Template#Variation1#Feature_Sprint_3#1508524645286
Order Run ID:   ct:1508524645448:0:DKRecipe#hw#Recipe_Template#Variation1#Feature_Sprint_3#1508524645286:
Status:        COMPLETED_SERVING
Kitchen:      Feature_Sprint_3
Recipe:        Recipe_Template
Variation:    Variation1
Start time:    YYYY-MM-DD HH:MM:SS
Run duration:    0:00:32 (H:M:S)
Email confirmation of successful order run of Variation1.  Note the feedback that two tests passed.

Email confirmation of successful order run of Variation1. Note the feedback that two tests passed.

Create New Variation

At present, the Recipe_Template recipe has a single variation named Variation1, which you have referenced previously when running recipe order runs. Should you wish to executed a differently-shaped recipe graph, or to apply different variable values to your recipe, you need to create a new Recipe Variation. Leveraging variations allows you to parameterize your processing, thereby providing runtime flexibility for your recipes.

In the following steps, you will build a new recipe variation named Variation2 that will alter the way the recipe graph is processed.

Recall that when creating the Recipe_Template recipe, you started with two placeholder nodes, placeholder-node1 and placeholder-node2, which do no real data work and are solely used for demonstration purposes. Variation2 will omit these nodes from its defined recipe graph.

Update variations.json II

  1. Replace the existing content in variations.json with the following text.
    This update creates Variation2 and defines a set of recipe nodes and edges without the placeholder nodes. It also illustrates how a schedule setting (mesos-setting) can be configured.
{
    "variation-list": {
        "variation1": {
            "graph-setting": "placeholder-graph",
            "mesos-setting": "production-mesos"
        },
        "variation2": {
            "graph-setting": "no-placeholders",
            "mesos-setting": "production-mesos"
        }

    } , 
    "graph-setting-list": {
        "placeholder-graph": [
            [
                "placeholder-node1", 
                "placeholder-node2"            ], 
            [
                "placeholder-node2", 
                "load-profits-node"            ], 
            [
                "load-profits-node", 
                "pentaho-di-node"            ]
        ] ,
        "no-placeholders": [
            [
                "load-profits-node", 
                "pentaho-di-node"            ]
        ]
    } ,
    "mesos-setting-list": {
        "production-mesos": {
            "schedule": "now"
        }

}
  1. Confirm the local changes applied via the recipe-status command.
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk recipe-status
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rs

Output: recipe-status command

YYYY-MM-DD HH:MM:SS - Getting the status of Recipe 'Recipe_Template' in Kitchen 'Feature_Sprint_3'
    versus directory '~/Kitchens/Feature_Sprint_3/Recipe_Template'
1 files are modified on local:
    variations.json

17 files are unchanged

File Differential

You have taken the steps to build a new recipe variation named Variation2. Per the recipe-status response above, the changes you applied to Recipe_Template only exist in the local copy of the recipe.

You can view the differences in recipe files between their local and remote copies using the merge and diff tool you configured to work with DKCloudCommand.

  1. View the file differences via the file-diff command, which brings up a two-pane window highlighting the differences between the local and remote copies of the file specified (in this case variations.json).
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk file-diff variations.json
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk fdi variations.json
~/Kitchens/Feature_Sprint_3/Recipe_Template $ cd ../..
~/Kitchens $ dk fdi -k Feature_Sprint_3 -r Recipe_Template variations.json

Output: file-diff command

YYYY-MM-DD HH:MM:SS - File Diff for file variations.json, in Recipe (Recipe_Template) in Kitchen (Feature_Sprint_3)
Executing command: /usr/local/bin/charm diff ~/Kitchens/Feature_Sprint_3/Recipe_Template/variations.json  {USER_HOME}/.dk/diffs/Feature_Sprint_3/Recipe_Template/variations.json

File diff done.
File differential between the local (left) and remote (right) copies of variations.json viewed in PyCharm.

File differential between the local (left) and remote (right) copies of variations.json viewed in PyCharm.

Validate and Compile Recipe

  1. Validate the local changes you made to the Recipe_Template recipe by using the recipe-validate command.
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk recipe-validate --variation Variation2
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rv -v Variation2

Output: recipe-validate command

YYYY-MM-DD HH:MM:SS - Validating recipe/variation Recipe_Template.Variation2 in Kitchen Feature_Sprint_3
Validating recipe with local changes applied
DKCloudCommand.recipe_validate succeeded
No recipe issues identified.
  1. Use the recipe-compile command to compile all Recipe_Template's recipe files, per the Variation2 variation, to confirm your syntax is correct and parameterization has been implemented correctly.
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk recipe-compile --variation Variation2
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rc -v Variation2
~/Kitchens $ dk rc -k Feature_Sprint_3 -r Recipe_Template -v Variation2

Output: recipe-compile command

YYYY-MM-DD HH:MM:SS - Compiling Recipe:
Kitchen:        Feature_Sprint_3
Recipe:         Recipe_Template
Variation:        Variation2

DKCloudCommand.get_compiled_serving succeeded, compiled recipe stored in folder 'compiled-recipe'

As expected, the compiled recipe within the new, local recipe-compile directory omits the placeholder nodes as Variation2 no longer includes these in its defined recipe graph.

The recipe-compile command generates a new recipe-compile directory within the local copy of the recipe you are editing.

The recipe-compile command generates a new recipe-compile directory within the local copy of the recipe you are editing.

  1. Delete the local recipe-compile directory, so that you do not push its content to the remote version of the Recipe_Template recipe.
~/Kitchens/Feature_Sprint_3/Recipe_Template $ rm -r compiled-recipe

Delete Local Copy of Compiled Recipe

Be sure to delete the new, local recipe-compile directory below before proceeding with a subsequent recipe update.

Recipe Update II

Now that you have confirmed that Recipe_Template compiles properly, you can push your local changes to the remote version of the recipe.

  1. Use the recipe-update command to commit your changes.
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk recipe-update --message "Adding Variation2 to remote copy of Recipe_Template"
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk ru -m "Adding Variation2 to remote copy of Recipe_Template"

Output: recipe-update command

YYYY-MM-DD HH:MM:SS - Updating all changed files in Recipe (Recipe_Template) in Kitchen(Feature_Sprint_3) with message (Adding Variation2 to remote copy of Recipe_Template)
Update results:

New files:
    None
Updated files:
    variations.json
Deleted files:
    None

Issues:

No issues found

Test Order II

  1. Use the order-run command to run an order for the updated Recipe_Template recipe, this time leveraging Variation2.
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk order-run Variation2 --yes
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk or Variation2 -y
~/Kitchens $ dk or -k Feature_Sprint_3 -r Recipe_Template Variation2 -y

Output: order-run command

YYYY-MM-DD HH:MM:SS - Create an Order:
    Kitchen: Feature_Sprint_3
    Recipe: Recipe_Template
    Variation: Variation2

Order ID is: DKRecipe#hw#Recipe_Template#Variation2#Feature_Sprint_3#1508539853555
  1. Use the orderrun-info command to view details regarding the last order run in the Feature_Sprint_3 kitchen. Alternatively, check the Order Run Details page in the DataKitchen UI for your order run results.
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk orderrun-info
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk ori
~/Kitchens $ dk ori -k Feature_Sprint_3

Output: orderrun-info command

YYYY-MM-DD HH:MM:SS - Display Order-Run details from kitchen Feature_Sprint_3

ORDER RUN SUMMARY

Order ID:       DKRecipe#hw#Recipe_Template#Variation2#Feature_Sprint_3#1508539853555
Order Run ID:   ct:1508539853715:0:DKRecipe#hw#Recipe_Template#Variation2#Feature_Sprint_3#1508539853555:
Status:        COMPLETED_SERVING
Kitchen:    Feature_Sprint_3
Recipe:        Recipe_Template
Variation:    Variation2
Start time:    YYYY-MM-DD HH:MM:SS
Run duration:    0:00:24 (H:M:S)
A view of the notification email generated by the successful completion of the order run.

A view of the notification email generated by the successful completion of the order run.

Merge Parent to Child

  1. Start the parent-to-child kitchen merge process by using the kitchen-merge-preview command with the --clean_previous_run option.
~/Kitchens/Feature_Sprint_3/Recipe_Template $ cd ../../Dev_Sprint_1
~/Kitchens/Dev_Sprint_1 $ dk kitchen-merge-preview --target_kitchen Feature_Sprint_3 --clean_previous_run
~/Kitchens/Feature_Sprint_3/Recipe_Template $ cd ../../Dev_Sprint_1
~/Kitchens/Dev_Sprint_1 $ dk kmp -tk Feature_Sprint_3 -cpr
~/Kitchens $ dk kmp -sk Dev_Sprint_1 -tk Feature_Sprint_3 -cpr

Output: kitchen-merge-preview command

YYYY-MM-DD HH:MM:SS - Previewing merge Kitchen Dev_Sprint_1 into Kitchen Feature_Sprint_3
Merge Preview Results (only changed files are being displayed):
--------------------------------------------------------------

Nothing to merge.

Kitchen merge preview done.
  1. You can skip running the parent-to-child kitchen merge and viewing test results from an order run since the parent-to-child kitchen merge preview indicated that there were no changes to merge.

Merge Child to Parent

  1. Use the kitchen-merge-preview command with the --clean_previous_run option to preview the child-to-parent merge of Feature_Sprint_3 into the Dev_Sprint_1 Kitchen.
~/Kitchens/Dev_Sprint_1 $ cd ../Feature_Sprint_3
~/Kitchens/Feature_Sprint_3 $ dk kitchen-merge-preview --target_kitchen Dev_Sprint_1 --clean_previous_run
~/Kitchens/Dev_Sprint_1 $ cd ../Feature_Sprint_3
~/Kitchens/Feature_Sprint_3 $ dk kitchen-merge-preview -tk Dev_Sprint_1 -cpr
~/Kitchens $ dk kitchen-merge-preview -sk Feature_Sprint_3 -tk Dev_Sprint_1 -cpr

Output: kitchen-merge-preview command

YYYY-MM-DD HH:MM:SS - Previewing merge Kitchen Feature_Sprint_3 into Kitchen Dev_Sprint_1
Merge Preview Results (only changed files are being displayed):
--------------------------------------------------------------

      ok                Recipe_Template/pentaho-di-node/actions/docker-share/create-schema.ktr
      ok                Recipe_Template/pentaho-di-node/actions/docker-share/get-count-of-superstore-data.ktr
      ok                Recipe_Template/variations.json
      ok                Recipe_Template/pentaho-di-node/actions/docker-share/create-and-load-superstore-data.ktr
      ok                Recipe_Template/pentaho-di-node/actions/DKDataSource_Container.json
      ok                Recipe_Template/pentaho-di-node/actions/docker-share/config.json
      ok                Recipe_Template/pentaho-di-node/description.json
      ok                Recipe_Template/description.json
      ok                Recipe_Template/variables.json

Kitchen merge preview done.
  1. Since the child-to-parent kitchen merge preview indicated that all files to be merged are free of conflicts, you may proceed with the kitchen merge. Use the kitchen-merge command to merge the updates you have applied to the Recipe_Template recipe in the Feature_Sprint_3 child kitchen into its parent kitchen Dev_Sprint_1.
~/Kitchens/Feature_Sprint_3 $ dk kitchen-merge --target_kitchen Dev_Sprint_1 --yes
~/Kitchens/Feature_Sprint_3 $ dk km -tk Dev_Sprint_1 -y
~/Kitchens $ dk km -sk Feature_Sprint_3 -tk Dev_Sprint_1 -y

Output: kitchen-merge command

YYYY-MM-DD HH:MM:SS - Merging Kitchen Feature_Sprint_3 into Kitchen Dev_Sprint_1
looking for manually merged files in temporary directory {USER_HOME}/.dk/merges/Feature_Sprint_3_to_Dev_Sprint_1 
Calling Merge ...
Merge done. You can check your changes in target kitchen and delete the source kitchen.
Url: https://cloud.datakitchen.io:443/dk/index.html#/history/dk/dev_sprint_1/[MERGE_ID]

Updating Recipes in the local version of Target Kitchen Dev_Sprint_1 to receive merged changes applied to remote...
{USER_HOME}/.../dev_sprint_1 kitchen has been updated
  1. Use the recipe-variation-list command to confirm that Variation2 has been merged into the Dev_Sprint_1 kitchen.
~/Kitchens/Feature_Sprint_3 $ cd ../Dev_Sprint_1/Recipe_Template
~/Kitchens/Dev_Sprint_1/Recipe_Template $ dk recipe-variation-list
~/Kitchens/Feature_Sprint_3 $ cd ../Dev_Sprint_1/Recipe_Template
~/Kitchens/Dev_Sprint_1/Recipe_Template $ dk rvl
~/Kitchens $ dk rvl -k Dev_Sprint_1 -r Recipe_Template

Output: recipe-variation-list command

Getting variations from local ...
YYYY-MM-DD HH:MM:SS - Listing variations for recipe Recipe_Template in Kitchen Dev_Sprint_1
Variations:
        Variation1
        Variation2
  1. Run a test order using the order-run command to confirm that the merged Recipe_Template recipe runs as expected in the Dev_Sprint_1 kitchen.
~/Kitchens/Dev_Sprint_1/Recipe_Template $ dk order-run Variation2 --yes
~/Kitchens/Dev_Sprint_1/Recipe_Template $ dk or Variation2 -y
~/Kitchens $ dk or -k Dev_Sprint_1 -r Recipe_Template Variation2 -y

Output: order-run command

YYYY-MM-DD HH:MM:SS - Create an Order:
    Kitchen: Dev_Sprint_1
    Recipe: Recipe_Template
    Variation: Variation2

Order ID is: DKRecipe#hw#Recipe_Template#Variation2#Dev_Sprint_1#1508768898876
  1. Use the orderrun-info command with the - -summary and - -test options to confirm that the order run in the Dev_Sprint_1 kitchen processed successfully.
    Alternatively, check the Order Run Details page in the DataKitchen UI for your order run results.
~/Kitchens/Dev_Sprint_1/Recipe_Template $ dk orderrun-info --summary --test
~/Kitchens/Dev_Sprint_1/Recipe_Template $ dk ori -s -q
~/Kitchens $ dk ori -k Dev_Sprint_1 -s -q

Output: orderrun-info command

YYYY-MM-DD HH:MM:SS - Display Order-Run details from kitchen Dev_Sprint_1

ORDER RUN SUMMARY

Order ID:       DKRecipe#hw#Recipe_Template#Variation2#Dev_Sprint_1#1508768898876
Order Run ID:   ct:1508768899024:0:DKRecipe#hw#Recipe_Template#Variation2#Dev_Sprint_1#1508768898876:
Status:         COMPLETED_SERVING
Kitchen:        Dev_Sprint_1
Recipe:         Recipe_Template
Variation:      Variation2
Start time:     YYYY-MM-DD HH:MM:SS
Run duration:   0:00:22 (H:M:S)

TEST RESULTS

Tests: Failed
        No Tests Failed

Tests: Warning
        No Tests Gave Warning Messages

Tests: Log
        No Tests Gave Log Messages

Tests: Passed
   Step (load-profits-node)
      1. test-load-data-key (1589 greater-than-equal-to 1589)
   Step (pentaho-di-node)
      1. get-test-data-count (51289 greater-than 10000)
A notification email confirming the successful processing of the order run of Variation2 of the Recipe_Template recipe.

A notification email confirming the successful processing of the order run of Variation2 of the Recipe_Template recipe.

Delete Recipe Files

Now that you have successfully merged your updated Recipe_Template recipe into the Dev_Sprint_1 parent kitchen, you may teardown the Feature_Sprint_3 child kitchen and its associated recipe copies.

First, however, you can practice deleting the remote and local copies of individual recipe files using the file-delete command.

  1. Create a throwaway recipe in the Feature_Sprint_3 kitchen, named TempRecipe.
~/Kitchens/Dev_Sprint_1/Recipe_Template $ cd ../../Feature_Sprint_3
~/Kitchens/Feature_Sprint_3 $ dk recipe-create TempRecipe
~/Kitchens/Dev_Sprint_1/Recipe_Template $ cd ../../Feature_Sprint_3
~/Kitchens/Feature_Sprint_3 $ dk re TempRecipe

Output: recipe-create command

YYYY-MM-DD HH:MM:SS - Creating Recipe TempRecipe for Kitchen 'Feature_Sprint_3'
list_recipe - elapsed: 1
DKCloudCommand.recipe_create created recipe TempRecipe
  1. Get a local copy of TempRecipe.
~/Kitchens/Feature_Sprint_3 $ dk recipe-get TempRecipe
~/Kitchens/Feature_Sprint_3 $ dk rg TempRecipe

Output: recipe-get command

YYYY-MM-DD HH:MM:SS - Getting the latest version of Recipe 'TempRecipe' in Kitchen 'Feature_Sprint_3'
DKCloudCommand.get_recipe has 4 sections
  TestRecipe/resources
  TestRecipe
  TestRecipe/placeholder-node2
  TestRecipe/placeholder-node1
  1. Delete both the local and remote copies of the variables.json file for TempRecipe.
~/Kitchens/Feature_Sprint_3 $ cd TempRecipe
~/Kitchens/Feature_Sprint_3/TempRecipe $ dk file-delete --message "Practice deleting a recipe file" variables.json
~/Kitchens/Feature_Sprint_3 $ cd TempRecipe
~/Kitchens/Feature_Sprint_3/TempRecipe $ dk fd -m "Practice deleting a recipe file" variables.json
~/Kitchens/Feature_Sprint_3 $ cd ..
~/Kitchens $ dk fd -k Feature_Sprint_3 -r TempRecipe -m "Practice deleting a recipe file" variables.json
YYYY-MM-DD HH:MM:SS - Deleting ((u'variables.json',)) in Recipe (TempRecipe) in kitchen(Feature_Sprint_3) with message (Practice deleting a recipe file)
DKCloudCommand.delete_file for variables.json succeed on local delete
DKCloudCommand.delete_file for variables.json succeed on remote delete

Delete Entire Recipes

You can also delete only the remote copy of an entire recipe.

  1. Use the recipe-delete command to delete the remote TempRecipe recipe.
~/Kitchens/Feature_Sprint_3/TempRecipe $ cd ..
~/Kitchens/Feature_Sprint_3 $ dk recipe-delete TempRecipe --yes
~/Kitchens/Feature_Sprint_3/TempRecipe $ cd ..
~/Kitchens/Feature_Sprint_3 $ dk rd TempRecipe -y
~/Kitchens $ dk rd -k Feature_Sprint_3 TempRecipe -y

Output: recipe-delete command

This command will delete the remote copy of recipe 'TempRecipe' for kitchen 'Feature_Sprint_3'.
YYYY-MM-DD HH:MM:SS - Deleting Recipe TempRecipe for Kitchen 'Feature_Sprint_3'
recipe_delete - elapsed: 0
DKCloudCommand.recipe_delete deleted recipe TempRecipe

Teardown Child Kitchen

Your last step is to teardown the remote copy of the Feature_Sprint_3 child kitchen and its recipes.

  1. Use the kitchen-list command to view a list of kitchens that exist under your company's account.
~/Kitchens/Feature_Sprint_3 $ cd ..
~/Kitchens $ dk kitchen-list
~/Kitchens/Feature_Sprint_3 $ cd ..
~/Kitchens $ dk kl

Output: kitchen-list command

YYYY-MM-DD HH:MM:SS - Getting the list of kitchens
kitchen-list returned 3 kitchens
+---- master
    +---- Production
            +---- Dev_Sprint_1
                   +---- Feature_Sprint_3
  1. Since you have already merged its contents to the Dev_Sprint_1 kitchen, you can use the kitchen-delete command to teardown the Feature_Sprint_3 kitchen.
    Note that you are deleting the remote copy of the Feature_Sprint_3 kitchen; you should retain a local copy of the kitchen and its recipe for continued reference.
~/Kitchens $ dk kitchen-delete Feature_Sprint_3 --yes
~/Kitchens $ dk kd Feature_Sprint_3 -y

Output: kitchen-delete command

No child kitchens found.
YYYY-MM-DD HH:MM:SS - Deleting remote copy of kitchen Feature_Sprint_3. Local files will not change.

Kitchen Feature_Sprint_3 has been deleted

Wrapping Up

To recap, this CLI Quickstart3 provided an overview of DataOps orchestration via DKCloudCommand, DataKitchen's command line tool. Specific topics covered include:

  • Adding and configuring a container-based action node for GUI tools, specifically for Pentaho DI
  • Validating local changes applied to a recipe
  • Creating recipe variations to provide runtime flexibility
  • Using locally configured tool to view local vs. remote file differences
  • Compiling entire recipes to check the proper application of parameterization
  • Viewing lists of recipe variations
  • Deleting remote copies of recipe files
  • Deleting entire versions of remote recipes

You have learned how, using the existing tools in your data analytic pipeline, to implement DataOps in your organization via orchestration by the DataKitchen platform. By mirroring the solid DataOps foundation you have quickly built in these three CLI Quickstarts, your organization is well on its way to delivering to your stakeholders insights of substantially greater quality at a drastically increased rate of speed!

Synopsis

CHECK THE MENU
~/Kitchens $ dk kitchen-list
~/Kitchens $ cd Dev_Sprint_1
~/Kitchens/Dev_Sprint_1 $ dk recipe-list
~/Kitchens/Dev_Sprint_1 $ cd ..
~/Kitchens $ dk kitchen-create --parent Dev_Sprint_1 Feature_Sprint_3
~/Kitchens $ dk kitchen-get Feature_Sprint_3 --recipe Recipe_Template

ADD PENTAHO NODE - UPDATE VARIABLES.JSON
~/Kitchens $ dk secret-list
[REPLACE VARIABLES.JSON CONTENT WITH PROVIDED TEXT]
[UPDATE VARIABLES.JSON EMAIL & SCHEMA VALUES; CHECK JSON SYNTAX]
~/Kitchens $ cd Feature_Sprint_3/Recipe_Template
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk recipe-status

ADD PENTAHO NODE - ADD NODE DIRECTORY & DESCRIPTION.JSON
~/Kitchens/Feature_Sprint_3/Recipe_Template $ mkdir pentaho-di-node
~/Kitchens/Feature_Sprint_3/Recipe_Template $ cat > pentaho-di-node/description.json
[PASTE SUPPLIED TEXT AT COMMAND LINE, <ENTER>, <CTRL+D>]
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk recipe-status

ADD PENTAHO NODE - UPDATE VARIATIONS.JSON FOR GRAPH SETTINGS
~/Kitchens/Feature_Sprint_3/Recipe_Template $ cat variations.json
[REPLACE VARIATIONS.JSON CONTENT WITH PROVIDED TEXT]
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk recipe-status

ADD PENTAHO NODE - CONFIGURE NODE
~/Kitchens/Feature_Sprint_3/Recipe_Template $ mkdir pentaho-di-node/actions
~/Kitchens/Feature_Sprint_3/Recipe_Template $ cat > pentaho-di-node/actions/DKDataSource_Container.json
[PASTE SUPPLIED TEXT AT COMMAND LINE, <ENTER>, <CTRL+D>]
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk recipe-status
~/Kitchens/Feature_Sprint_3/Recipe_Template $ mkdir pentaho-di-node/actions/docker-share
/Kitchens/Feature_Sprint_3/Recipe_Template $ cat > pentaho-di-node/actions/docker-share/config.json
[PASTE SUPPLIED TEXT AT COMMAND LINE, <ENTER>, <CTRL+D>]
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk recipe-status

ADD PENTAHO NODE - ADD TOOLCHAIN FILES
[ADD THREE .ktr FILES TO /docker-share DIRECTORY WITH CURL COMMANDS]
~/Kitchens/Feature_Sprint_3/Recipe_Template $ cd pentaho-di-node/actions/docker-share
~/Kitchens/Feature_Sprint_3/Recipe_Template/pentaho-di-node/actions/docker-share $ curl -L -O (link to raw public repo file1)
~/Kitchens/Feature_Sprint_3/Recipe_Template/pentaho-di-node/actions/docker-share $ curl -L -O (link to raw public repo file2)
~/Kitchens/Feature_Sprint_3/Recipe_Template/pentaho-di-node/actions/docker-share $ curl -L -O (link to raw public repo file3)
~/Kitchens/Feature_Sprint_3/Recipe_Template/pentaho-di-node/actions/docker-share $ cd ../../..
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk recipe-status

ADD PENTAHO NODE - RECIPE VALIDATION I
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk recipe-validate --variation Variation1

ADD PENTAHO NODE - RECIPE UPDATE I
~Kitchens/Feature_Sprint_3/Recipe_Template $ dk recipe-update --message "Add Pentaho DI node"
~Kitchens/Feature_Sprint_3/Recipe_Template $ dk recipe-status

ADD PENTAHO NODE - TEST ORDER I
~Kitchens/Feature_Sprint_3/Recipe_Template $ dk order-run Variation1 --yes
~Kitchens/Feature_Sprint_3/Recipe_Template $ dk orderrun-info

CREATE NEW RECIPE VARIATION
[REPLACE VARIATIONS.JSON CONTENT WITH PROVIDED TEXT]
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk recipe-status

CREATE NEW RECIPE VARIATION - FILE DIFFERENTIAL
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk file-diff variations.json

CREATE NEW RECIPE VARIATION - RECIPE VALIDATION II
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk recipe-validate --variation Variation2

CREATE NEW RECIPE VARIATION - RECIPE COMPILE
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk recipe-compile --variation Variation2
~/Kitchens/Feature_Sprint_3/Recipe_Template $ rm -r compiled-recipe/

CREATE NEW VARIATION - RECIPE UPDATE II
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk recipe-update --message "Add Variation2 to remote copy of Recipe_Template"

CREATE NEW VARIATION - TEST ORDER II
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk order-run Variation2 --yes
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk orderrun-info

MERGE PARENT TO CHILD
~/Kitchens/Feature_Sprint_3/Recipe_Template $ cd ../../Dev_Sprint_1
~/Kitchens/Dev_Sprint_1 $ dk kitchen-merge-preview --target_kitchen Feature_Sprint_3 --clean_previous_run

MERGE CHILD TO PARENT
~/Kitchens/Dev_Sprint_1 $ cd ../Feature_Sprint_3
~/Kitchens/Feature_Sprint_3 $ dk kitchen-merge-preview --target_kitchen Dev_Sprint_1 --clean_previous_run
~/Kitchens/Feature_Sprint_3 $ dk kitchen-merge --target_kitchen Dev_Sprint_1 --yes

LIST RECIPE VARIATIONS
~/Kitchens/Feature_Sprint_3 $ cd ../Dev_Sprint_1/Recipe_Template
~/Kitchens/Dev_Sprint_1/Recipe_Template $ dk recipe-variation-list

PARENT TEST ORDER
~/Kitchens/Dev_Sprint_1/Recipe_Template $ dk order-run Variation2 --yes
~/Kitchens/Dev_Sprint_1/Recipe_Template $ dk orderrun-info --summary --test

DELETE RECIPE FILES
~/Kitchens/Dev_Sprint_1/Recipe_Template $ cd ../../Feature_Sprint_3
~/Kitchens/Feature_Sprint_3 $ dk recipe-create TempRecipe
~/Kitchens/Feature_Sprint_3 $ dk recipe-get TempRecipe
~/Kitchens/Feature_Sprint_3 $ cd TempRecipe
~/Kitchens/Feature_Sprint_3/TempRecipe $ dk file-delete --message "Practice deleting a recipe file" variables.json

DELETE ENTIRE RECIPE
~/Kitchens/Feature_Sprint_3/TempRecipe $ cd ..
~/Kitchens/Feature_Sprint_3 $ dk recipe-delete TempRecipe --yes

TEARDOWN CHILD KITCHEN
~/Kitchens/Feature_Sprint_3 $ cd ..
~/Kitchens $ dk kitchen-list
~/Kitchens $ dk kitchen-delete Feature_Sprint_3 --yes
CHECK THE MENU
~/Kitchens $ dk kl
~/Kitchens $ cd Dev_Sprint_1
~/Kitchens/Dev_Sprint_1 $ dk rl
~/Kitchens/Dev_Sprint_1 $ cd ..
~/Kitchens $ dk kc -p Dev_Sprint_1 Feature_Sprint_3
~/Kitchens $ dk kg Feature_Sprint_3 -r Recipe_Template

ADD PENTAHO NODE - UPDATE VARIABLES.JSON
~/Kitchens $ dk sl
[REPLACE VARIABLES.JSON CONTENT WITH PROVIDED TEXT]
[UPDATE VARIABLES.JSON EMAIL & SCHEMA VALUES; CHECK JSON SYNTAX]
~/Kitchens $ cd Feature_Sprint_3/Recipe_Template
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rs

ADD PENTAHO NODE - ADD NODE DIRECTORY & DESCRIPTION.JSON
~/Kitchens/Feature_Sprint_3/Recipe_Template $ mkdir pentaho-di-node
~/Kitchens/Feature_Sprint_3/Recipe_Template $ cat > pentaho-di-node/description.json
[PASTE SUPPLIED TEXT AT COMMAND LINE, <ENTER>, <CTRL+D>]
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rs

ADD PENTAHO NODE - UPDATE VARIATIONS.JSON FOR GRAPH SETTINGS
~/Kitchens/Feature_Sprint_3/Recipe_Template $ cat variations.json
[REPLACE VARIATIONS.JSON CONTENT WITH PROVIDED TEXT]
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rs

ADD PENTAHO NODE - CONFIGURE NODE
~/Kitchens/Feature_Sprint_3/Recipe_Template $ mkdir pentaho-di-node/actions
~/Kitchens/Feature_Sprint_3/Recipe_Template $ cat > pentaho-di-node/actions/DKDataSource_Container.json
[PASTE SUPPLIED TEXT AT COMMAND LINE, <ENTER>, <CTRL+D>]
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rs
~/Kitchens/Feature_Sprint_3/Recipe_Template $ mkdir pentaho-di-node/actions/docker-share
/Kitchens/Feature_Sprint_3/Recipe_Template $ cat > pentaho-di-node/actions/docker-share/config.json
[PASTE SUPPLIED TEXT AT COMMAND LINE, <ENTER>, <CTRL+D>]
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rs

ADD PENTAHO NODE - ADD TOOLCHAIN FILES
[ADD FOUR .ktr FILES TO /docker-share DIRECTORY WITH CURL COMMANDS]
~/Kitchens/Feature_Sprint_3/Recipe_Template $ cd pentaho-di-node/actions/docker-share
~/Kitchens/Feature_Sprint_3/Recipe_Template/pentaho-di-node/actions/docker-share $ curl -L -O (link to raw public repo file1)
~/Kitchens/Feature_Sprint_3/Recipe_Template/pentaho-di-node/actions/docker-share $ curl -L -O (link to raw public repo file2)
~/Kitchens/Feature_Sprint_3/Recipe_Template/pentaho-di-node/actions/docker-share $ curl -L -O (link to raw public repo file3)
~/Kitchens/Feature_Sprint_3/Recipe_Template/pentaho-di-node/actions/docker-share $ cd ../../..
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rs

ADD PENTAHO NODE - UPDATE GRAPH.JSON
[REPLACE GRAPH.JSON CONTENT WITH PROVIDED TEXT]
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rs

ADD PENTAHO NODE - RECIPE VALIDATION I
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rv -v Variation1

ADD PENTAHO NODE - RECIPE UPDATE I
~Kitchens/Feature_Sprint_3/Recipe_Template $ dk ru -m "Change message"
~Kitchens/Feature_Sprint_3/Recipe_Template $ dk rs

ADD PENTAHO NODE - TEST ORDER I
~Kitchens/Feature_Sprint_3/Recipe_Template $ dk or Variation1 -y
~Kitchens/Feature_Sprint_3/Recipe_Template $ dk ori

CREATE NEW RECIPE VARIATION
[REPLACE VARIATIONS.JSON CONTENT WITH PROVIDED TEXT]
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rs

CREATE NEW RECIPE VARIATION - FILE DIFFERENTIAL
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk fdi variations.json

CREATE NEW RECIPE VARIATION - RECIPE VALIDATION II
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rv -v Variation2

CREATE NEW RECIPE VARIATION - RECIPE COMPILE
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rc -v Variation2
~/Kitchens/Feature_Sprint_3/Recipe_Template $ rm -r compiled-recipe/

CREATE NEW VARIATION - RECIPE UPDATE II
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk ru -m "Change message"

CREATE NEW VARIATION - TEST ORDER II
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk or Variation2 -y
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk ori

MERGE PARENT TO CHILD
~/Kitchens/Feature_Sprint_3/Recipe_Template $ cd ../../Dev_Sprint_1
~/Kitchens/Dev_Sprint_1 $ dk kmp -tk Feature_Sprint_3 -cpr

MERGE CHILD TO PARENT
~/Kitchens/Dev_Sprint_1 $ cd ../Feature_Sprint_3
~/Kitchens/Feature_Sprint_3 $ dk kmp -tk Dev_Sprint_1 -cpr
~/Kitchens/Feature_Sprint_3 $ dk km -tk Dev_Sprint_1 -y

LIST RECIPE VARIATIONS
~/Kitchens/Feature_Sprint_3 $ cd ../Dev_Sprint_1/Recipe_Template
~/Kitchens/Dev_Sprint_1/Recipe_Template $ dk rvl

PARENT TEST ORDER
~/Kitchens/Dev_Sprint_1/Recipe_Template $ dk or Variation2 -y
~/Kitchens/Dev_Sprint_1/Recipe_Template $ dk ori -s -q

DELETE RECIPE FILES
~/Kitchens/Dev_Sprint_1/Recipe_Template $ cd ../../Feature_Sprint_3
~/Kitchens/Feature_Sprint_3 $ dk re TempRecipe
~/Kitchens/Feature_Sprint_3 $ dk rg TempRecipe
~/Kitchens/Feature_Sprint_3 $ cd TempRecipe
~/Kitchens/Feature_Sprint_3/TempRecipe $ dk fd -m "Change message" variables.json

DELETE ENTIRE RECIPE
~/Kitchens/Feature_Sprint_3/TempRecipe $ cd ..
~/Kitchens/Feature_Sprint_3 $ dk rd TempRecipe -y

TEARDOWN CHILD KITCHEN
~/Kitchens/Feature_Sprint_3 $ cd ..
~/Kitchens $ dk kl
~/Kitchens $ dk kid Feature_Sprint_3 -y
CHECK THE MENU
~/Kitchens $ dk kl
~/Kitchens $ dk rl -k Dev_Sprint_1
~/Kitchens $ dk kc -p Dev_Sprint_1 Feature_Sprint_3
~/Kitchens $ dk kg Feature_Sprint_3 -r Recipe_Template

ADD PENTAHO NODE - UPDATE VARIABLES.JSON
~/Kitchens $ dk sl
[REPLACE VARIABLES.JSON CONTENT WITH PROVIDED TEXT]
[UPDATE VARIABLES.JSON EMAIL & SCHEMA VALUES; CHECK JSON SYNTAX]
~/Kitchens $ cd Feature_Sprint_3/Recipe_Template
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rs

ADD PENTAHO NODE - ADD NODE DIRECTORY & DESCRIPTION.JSON
~/Kitchens/Feature_Sprint_3/Recipe_Template $ mkdir pentaho-di-node
~/Kitchens/Feature_Sprint_3/Recipe_Template $ cat > pentaho-di-node/description.json
[PASTE SUPPLIED TEXT AT COMMAND LINE, <ENTER>, <CTRL+D>]
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rs

ADD PENTAHO NODE - UPDATE VARIATIONS.JSON FOR GRAPH SETTINGS
~/Kitchens/Feature_Sprint_3/Recipe_Template $ cat variations.json
[REPLACE VARIATIONS.JSON CONTENT WITH PROVIDED TEXT]
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rs

ADD PENTAHO NODE - CONFIGURE NODE
~/Kitchens/Feature_Sprint_3/Recipe_Template $ mkdir pentaho-di-node/actions
~/Kitchens/Feature_Sprint_3/Recipe_Template $ cat > pentaho-di-node/actions/DKDataSource_Container.json
[PASTE SUPPLIED TEXT AT COMMAND LINE, <ENTER>, <CTRL+D>]
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rs
~/Kitchens/Feature_Sprint_3/Recipe_Template $ mkdir pentaho-di-node/actions/docker-share
/Kitchens/Feature_Sprint_3/Recipe_Template $ cat > pentaho-di-node/actions/docker-share/config.json
[PASTE SUPPLIED TEXT AT COMMAND LINE, <ENTER>, <CTRL+D>]
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rs

ADD PENTAHO NODE - ADD PENTAHO DI KTR FILES
[ADD FOUR .ktr FILES TO /docker-share DIRECTORY WITH CURL COMMANDS]
~/Kitchens/Feature_Sprint_3/Recipe_Template $ cd pentaho-di-node/actions/docker-share
~/Kitchens/Feature_Sprint_3/Recipe_Template/pentaho-di-node/actions/docker-share $ curl -L -O (link to raw public repo file1)
~/Kitchens/Feature_Sprint_3/Recipe_Template/pentaho-di-node/actions/docker-share $ curl -L -O (link to raw public repo file2)
~/Kitchens/Feature_Sprint_3/Recipe_Template/pentaho-di-node/actions/docker-share $ curl -L -O (link to raw public repo file3)
~/Kitchens/Feature_Sprint_3/Recipe_Template/pentaho-di-node/actions/docker-share $ cd ../../..
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rs

ADD PENTAHO NODE - UPDATE GRAPH.JSON
[REPLACE GRAPH.JSON CONTENT WITH PROVIDED TEXT]
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rs

ADD PENTAHO NODE - RECIPE VALIDATION I
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rv -v Variation1

ADD PENTAHO NODE - RECIPE UPDATE I
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk ru -m "Change message"
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rs

ADD PENTAHO NODE - TEST ORDER I
~/Kitchens/Feature_Sprint_3/Recipe_Template $ cd ../..
~/Kitchens $ dk or -k Feature_Sprint_3 -r Recipe_Template Variation1 -y
~/Kitchens $ dk ori -k Feature_Sprint_3

CREATE NEW RECIPE VARIATION
[REPLACE VARIATIONS.JSON CONTENT WITH PROVIDED TEXT]
~/Kitchens $ cd Feature_Sprint_3/Recipe_Template
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rs

CREATE NEW RECIPE VARIATION - FILE DIFFERENTIAL
~/Kitchens/Feature_Sprint_3/Recipe_Template $ cd ../..
~/Kitchens $ dk fdi -k Feature_Sprint_3 -r Recipe_Template variations.json

CREATE NEW RECIPE VARIATION - RECIPE VALIDATION II
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rv -v Variation2

CREATE NEW VARIATION - RECIPE COMPILE
~/Kitchens/Feature_Sprint_3/Recipe_Template $ cd ../..
~/Kitchens $ dk rc -k Feature_Sprint_3 -r Recipe_Template -v Variation2
~/Kitchens $ rm -r /Feature_Sprint_3/Recipe_Template/compiled-recipe/

CREATE NEW VARIATION - RECIPE UPDATE II
~/Kitchens $ cd Feature_Sprint_3/Recipe_Template
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk ru -m "Change message"

CREATE NEW VARIATION - TEST ORDER II
~/Kitchens/Feature_Sprint_3/Recipe_Template $ cd ../..
~/Kitchens $ dk or -k Feature_Sprint_3 -r Recipe_Template Variation2 -y
~/Kitchens $ dk ori -k Feature_Sprint_3

MERGE PARENT TO CHILD & RUN TEST ORDER
~/Kitchens $ dk kmp -sk Dev_Sprint_1 -tk Feature_Sprint_3 -cpr

MERGE CHILD TO PARENT & RUN TEST ORDER
~/Kitchens $ dk kmp -sk Feature_Sprint_3 -tk Dev_Sprint_1 -cpr
~/Kitchens $ dk km -sk Feature_Sprint_3 -tk Dev_Sprint_1 -y

LIST RECIPE VARIATIONS
~/Kitchens $ dk rvl -k Dev_Sprint_1 -r Recipe_Template

PARENT TEST ORDER
~/Kitchens $ dk or -k Dev_Sprint_1 -r Recipe_Template Variation2 -y
~/Kitchens $ dk ori -k Dev_Sprint_1 -s -q

DELETE RECIPE FILES
~/Kitchens $ dk re -k Feature_Sprint_3 TempRecipe
~/Kitchens $ dk fd -k Feature_Sprint_3 -r TempRecipe -m "Change message" variables.json

DELETE ENTIRE RECIPE
~/Kitchens $ dk rd -k Feature_Sprint_3 TempRecipe -y

TEARDOWN CHILD KITCHEN
~/Kitchens $ dk kl
~/Kitchens $ dk kd Feature_Sprint_3 -y
CHECK THE MENU
~/Kitchens $ dk kl
~/Kitchens $ dk rl -k Dev_Sprint_1
~/Kitchens $ dk kc -p Dev_Sprint_1 Feature_Sprint_3
~/Kitchens $ dk kg Feature_Sprint_3 -r Recipe_Template

DELETE QS2 RECIPE & CREATE QS3 RECIPE FROM TEMPLATE
[SKIPS STEPS TO BUILD QS3 RECIPE_TEMPLATE VIA QS2 RECIPE_TEMPLATE]
~/Kitchens $ cd Feature_Sprint_3
~/Kitchens/Feature_Sprint_3 $ dk rd Recipe_Template -y
~/Kitchens/Feature_Sprint_3 $ rm -r Recipe_Template
~/Kitchens/Feature_Sprint_3 $ dk re -tm qs3 Recipe_Template
~/Kitchens/Feature_Sprint_3 $ dk rg Recipe_Template

MAKE LOCAL RECIPE CHANGES AND UPDATE TO REMOTE
~/Kitchens/Feature_Sprint_2 $ cd Recipe_Template
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rs
[UPDATE EMAIL VALUE IN VARIABLES.JSON]
[UPDATE REDSHIFT SCHEMA VALUE IN VARIABLES.JSON - e.g. "jdoe"]
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk fdi variables.json
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rv -v Variation1
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk ru -m "Push local changes to remote"
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rs

VIEW RECIPE VARIATIONS
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rvl
~/Kitchens/Feature_Sprint_3/Recipe_Template $ cat variations.json

RECIPE COMPILE
~/Kitchens/Feature_Sprint_3/Recipe_Template $ dk rc -v Variation2
~/Kitchens/Feature_Sprint_3/Recipe_Template $ rm -r compiled-recipe/

TEST ORDER
~/Kitchens/Feature_Sprint_3/Recipe_Template $ cd ../..
~/Kitchens $ dk or -k Feature_Sprint_3 -r Recipe_Template Variation2 -y
~/Kitchens $ dk ori -k Feature_Sprint_3

MERGE PARENT TO CHILD
~/Kitchens $ dk kmp -sk Dev_Sprint_1 -tk Feature_Sprint_3 -cpr

MERGE CHILD TO PARENT
~/Kitchens $ dk kmp -sk Feature_Sprint_3 -tk Dev_Sprint_1 -cpr
~/Kitchens $ dk km -sk Feature_Sprint_3 -tk Dev_Sprint_1 -y

PARENT TEST ORDER
~/Kitchens $ dk or -k Dev_Sprint_1 -r Recipe_Template Variation2 -y
~/Kitchens $ dk ori -k Dev_Sprint_1 -s -q

DELETE RECIPE FILES
~/Kitchens $ dk re -k Feature_Sprint_3 TempRecipe
~/Kitchens $ dk fd -k Feature_Sprint_3 -r TempRecipe -m "Change message" variables.json

DELETE ENTIRE RECIPE
~/Kitchens $ dk rd -k Feature_Sprint_3 TempRecipe -y

TEARDOWN CHILD KITCHEN
~/Kitchens $ dk kl
~/Kitchens $ dk kd Feature_Sprint_3 -y

Updated 7 days ago

CLI Quickstart3


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.