DataKitchen DataOps Documention

CLI Quickstart2

This CLI Quickstart2 covers the use of DKCloudCommand, DataKitchen's command line tool. You will learn about the following topics.

  • The Vault and secure storage of secret credentials
  • Configuring recipe graph nodes
  • Viewing a file's change history
  • Compiling files to review the application of parameter values
  • Cooking an order run of a recipe variation with expanded options
  • Implementing an automated tests
  • Understanding test types and reviewing their feedback
  • Batch updating local recipe file changes to remote
  • Listing the orders that exist in a kitchen

Super-Fast Quickstart2

Super-Fast Quickstart2

Advanced users who need less detailed documentation to get up and running with DataKitchen's DataOps platform may jump right to the SUPER-FAST CLI QUICKSTART2 tab of this guide's Synopsis for a menu of abridged commands. This route provides a fully complete Recipe for this guide. Alternatively, you may follow the steps below to build out the Recipe starting from a copy of the Recipe generated in CLI Quickstart1.

Set The Table

What is our current lineup of Kitchens?

~/Kitchens $ dk kitchen-list
~/Kitchens $ dk kl
YYYY-MM-DD HH:MM:SS - Getting the list of kitchens
kitchen-list returned 3 kitchens
+---- master
    +---- Production
            +---- Dev_Sprint_1

What Recipes exist in the Dev_Sprint_1 Kitchen?

~/Kitchens $ cd Dev_Sprint_1
~/Kitchens/Dev_Sprint_1 $ dk recipe-list
~/Kitchens $ cd Dev_Sprint_1
~/Kitchens/Dev_Sprint_1 $ dk rl
~/Kitchens $ dk rl -k Dev_Sprint_1
YYYY-MM-DD HH:MM:SS - Getting the list of Recipes for Kitchen 'Dev_Sprint_1'
list_recipe - elapsed: 0
DKCloudCommand.list_recipe returned 1 recipes
  Recipe-Template

Create Child Kitchen

Let's now build upon our Recipe-Template Recipe so that it's processing will perform data work that generates an output, which we can test. Using the kitchen-create command, create another Child Kitchen named Feature_Sprint_2 with Dev_Sprint_1 as its parent. Feature_Sprint_2 will inherit a copy of the Recipe-Template Recipe, which is the copy where we will do our work for this section of this guide.

Create a Uniquely-Named Kitchen

Append your name when creating the Kitchen so that you have your own, segregated workspace. For example, if your name is Chef, you would create a Kitchen named Feature_Sprint_2_Chef.

~/Kitchens/Dev_Sprint_1 $ cd ..
~/Kitchens $ dk kitchen-create --parent Dev_Sprint_1 Feature_Sprint_2
~/Kitchens/Dev_Sprint_1 $ cd ..
~/Kitchens $ dk kc -p Dev_Sprint_1 Feature_Sprint_2
YYYY-MM-DD HH:MM:SS - Creating kitchen Feature_Sprint_2 from parent kitchen Dev_Sprint_1
DKCloudCommand.create_kitchen created Feature_Sprint_2

Display the Kitchens available to you via the kitchen-list command.

~/Kitchens $ dk kitchen-list
~/Kitchens $ dk kl
YYYY-MM-DD HH:MM:SS - Getting the list of kitchens
kitchen-list returned 4 kitchens
+---- master
    +---- Production
            +---- Dev_Sprint_1
                    +---- Feature_Sprint_2

Get Kitchen & Recipes

Get a local copy of a Kitchen and all of its Recipes in a single recipe-get command using the --all option.

~/Kitchens $ dk kitchen-get Feature_Sprint_2 --all
~/Kitchens $ dk kg Feature_Sprint_2 -a
YYYY-MM-DD HH:MM:SS - Getting kitchen 'Feature_Sprint_2'
list_recipe - elapsed: 0
Got Kitchen 'Feature_Sprint_2'
DKCloudCommand.get_recipe has 5 sections
  Recipe-Template/resources/email-templates
  Recipe-Template
  Recipe-Template/placeholder-node1
  Recipe-Template/resources
  Recipe-Template/placeholder-node2

The Vault

The Vault stores, securely, sensitive infrastructure and toolchain credentials. These credentials are called Secrets, which are stored remotely. Note that the Vault is maintained at the customer account level, and thus the Secrets it contains can be leveraged by any Recipe in any Kitchen under a customer account. While these Secrets can be leveraged by users in data analytics pipelines Vault Secrets cannot be viewed without special permissions.

We'll now review how to interact with the Vault via DKCloudCommand to help us later understand how to build Vault references into our Recipes.

List Secrets

Use the secret-list command to view a list of the Secrets that are stored in the Vault.

Check Your Vault Secrets

Note that the list of Vault Secrets below match Vaults that have been set up for prospective customer sandboxes. If you are a new user for an existing customer, or if your company manages their own custom Vault configurations, the Vault paths you will need to reference as you build your Quickstart Recipes will likely differ from what is shown below. You can view the specific Vault paths available to reference in your variables.json and variations.json configuration files via the Secrets UI or using DKCC's secret-list command.

~/Kitchens $ dk secret-list --recursive
~/Kitchens $ dk sl -rc
YYYY-MM-DD HH:MM:SS - Getting the list of secrets
secret_list - elapsed: 0
secret-list returned 14 secrets
        vault://aws/account_id
        vault://aws/role
        vault://dockerhub/email
        vault://dockerhub/namespace
        vault://dockerhub/password
        vault://dockerhub/username
        vault://redshift/database
        vault://redshift/hostname
        vault://redshift/password
        vault://redshift/port
        vault://redshift/username
        vault://s3_schema/bucket
        vault://s3_schema/s3-access-key
        vault://s3_schema/s3-secret-key

Write Secrets

Write a Secret to the Vault using the secret-write command.

~/Kitchens $ dk secret-write service-name/test-credential=test-value --yes
~/Kitchens $ dk sw service-name/test-credential=test-value -y
YYYY-MM-DD HH:MM:SS - Writing secret
secret_write - elapsed: 0
Secret written.

Search Secrets

Search whether a specific Secret exists in the Vault via the secret-exists command.

~/Kitchens $ dk secret-exists service-name/test-credential
~/Kitchens $ dk se service-name/test-credential
YYYY-MM-DD HH:MM:SS Checking secret
secret_exists - elapsed: 0
True

Delete Secrets

Delete a Secret from the Vault via the secret-delete command.

~/Kitchens $ dk secret-delete service-name/test-credential --yes
~/Kitchens $ dk sd service-name/test-credential -y
YYYY-MM-DD HH:MM:SS - Deleting secret
secret_write - elapsed: 0
Secret deleted.

Add a Recipe Graph Node

Previously, we ran an Order to cook a Recipe Variation composed of nodes that had no interaction with data. These nodes were of type Synchronize, as they only serve as placeholders. Recipe graphs require at least two nodes in order for an edge to exist between them. Synchronize nodes are thus helpful by acting as placeholders when building new Recipes or cooking test Orders for a single node in a Recipe.

Connecting to an AWS S3 Bucket

Detailed documentation is available regarding how to setup a connection to an AWS S3 Bucket.

Next, we'll add a new node to our Recipe graph to begin to build a Recipe that does real data work. Specifically, we'll create from scratch a node of Action type that we will configure to create a new schema and a new table in an AWS Redshift database and copy example data from an AWS S3 bucket into said table. Additionally, we will then execute a SQL query against the table we created and test the returned result in order to give us confidence in the quality of the node's output.

A view of the AWS S3 Bucket where our data begins its journey through our **Recipe-Template** Recipe's data analytic pipeline.  This view is specifically from [Transmit](https://panic.com/transmit/), our recommended SFTP and S3 application.

A view of the AWS S3 Bucket where our data begins its journey through our Recipe-Template Recipe's data analytic pipeline. This view is specifically from Transmit, our recommended SFTP and S3 application.

Update Variables.json

View the existing content of the Recipe's variables.json configuration file. Notice that the contents of this file originated in CLI Quickstart1. The email string-replace-value will be set to the value you inputted previously in the Feature_Sprint_1 Kitchen. We will be adding to this file a number of new variables that will be leveraged by the updated version of the Recipe-Template Recipe we are building.

~/Kitchens $ cd Feature_Sprint_2/Recipe-Template
~/Kitchens/Feature_Sprint_2/Recipe-Template $ cat variables.json
{
  "name": "Recipe-Template QS1 Variables",
  "variable-list": {
    "email":  "[YOUR EMAIL HERE FROM QUICKSTART1]"
  }
}

Recall the Secrets stored in the Vault that are relevant to AWS S3 and Redshift services:

~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk secret-list --recursive
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk sl -rc
YYYY-MM-DD HH:MM:SS - Getting the list of secrets
secret_list - elapsed: 0
secret-list returned 14 secrets
        vault://aws/account_id
        vault://aws/role
        vault://dockerhub/email
        vault://dockerhub/namespace
        vault://dockerhub/password
        vault://dockerhub/username
        vault://redshift/database
        vault://redshift/hostname
        vault://redshift/password
        vault://redshift/port
        vault://redshift/username
        vault://s3_schema/bucket
        vault://s3_schema/s3-access-key
        vault://s3_schema/s3-secret-key

We want to add these Secrets to our Recipe's variables.json configuration file so that they can be leveraged when we run an Order for our Recipe. Because Secret credentials are inherently sensitive we create variables that reference their Vault locations rather than hard-coding their values.

AWS S3 Credentials

Note that while the load-profits-node we're building does gather its source data from an AWS S3 bucket, the node does not require use of the s3-access-key and s3-secret-key credentials to do its work. Thus, the updated variables.json file for this Quickstart2 guide will not contain references to these Secrets' Vault locations. We'll leverage these credentials in the next section of the guide, Quickstart3.

Replace all existing content in your Recipe's variables.json file with the text below. Be sure to replace the email value listed. Be careful to maintain proper .json syntax.

Update Variable Values

Email

Remember to update the email value after copying the text below into the variables.json file. Replace [YOUR EMAIL HERE] with your preferred destination email address and save the file.

Redshift Schema

Also update the redshiftschema value after copying the text below into variables.json file so that you have your own unique database schema. We recommend you follow the syntax of using your first initial and last name for the syntax name. For example, a user named John Doe would replace the redshiftschema value from [YOUR SCHEMA NAME HERE] to jdoe.

{
  "name": "Recipe-Template QS2 Variables",
  "variable-list": {
    "awsaccountid": "#{vault://aws/account_id}",
    "awsrole": "#{vault://aws/role}",
    "email": "[YOUR EMAIL HERE]",
    "redshiftdatabase": "#{vault://redshift/database}",
    "redshifthostname": "#{vault://redshift/hostname}",
    "redshiftpassword": "#{vault://redshift/password}",
    "redshiftport": "#{vault://redshift/port}",
    "redshiftschema": "[YOUR SCHEMA NAME HERE]",
    "redshiftusername": "#{vault://redshift/username}",
    "s3bucketname": "#{vault://s3_schema/bucket}",
    "s3bucketpath": "superstore/customer/sales-profitability"
  }
}
Comparison of **Recipe-Template**'s **variables.json** file between that used in Quickstart2 versus Quickstart1.

Comparison of Recipe-Template's variables.json file between that used in Quickstart2 versus Quickstart1.

Use the recipe-status command to confirm the change to variables.json has been applied to the local version of the Recipe-Template Recipe.

~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk recipe-status
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk rs
YYYY-MM-DD HH:MM:SS - Getting the status of Recipe 'Recipe-Template' in Kitchen 'Feature_Sprint_2'
    versus directory '~/Kitchens/Feature_Sprint_2/Recipe-Template'
1 files are modified on local:
    variables.json

9 files are unchanged

Update local changes to variables.json to remote via the file-update command.

~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk file-update --message "Update variables.json with QS2 variables" variables.json
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk fu -m "Update variables.json with QS2 variables" variables.json
~/Kitchens/Feature_Sprint_2/Recipe-Template $ cd ../..
~/Kitchens $ dk fu -k Feature_Sprint_2 -r Recipe-Template -m "Update variables.json with QS2 variables" variables.json
YYYY-MM-DD HH:MM:SS - Updating File(s) ((u'variables.json',)) in Recipe (Recipe-Template) in Kitchen(Feature_Sprint_2) with message (Update variables.json with QS2 variables)
DKCloudCommand.update_file for variables.json succeeded

Rather than update to remote each individual local Recipe file change piecemeal, instead we will now hold until we complete editing the Recipe-Template Recipe and instead later use the recipe-update command to batch update Recipe files.

File History

View the change history for a specific file via the file-history command.

~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk file-history variables.json
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk fh variables.json
YYYY-MM-DD HH:MM:SS - Retrieving file history
DKCloudCommand.file_history succeeded:
Author:    [DK USERNAME] <[DK USERNAME]>
Date:    YYYY-MM-DDTHH:MM:SSZ
Message:Update variables.json with QS2 variables

Author:    [DK USERNAME] <[DK USERNAME]>
Date:    YYYY-MM-DDTHH:MM:SSZ
Message:Push email value update within variables.json

Author:    [DK USERNAME] <[DK USERNAME]>
Date:    YYYY-MM-DDTHH:MM:SSZ
Message:New recipe Recipe-Template

Update Variations.json

For now, we will continue with a single recipe variation, Variation1, which we used previously in CLI Quickstart1. Later, in CLI Quickstart3, we will learn how to build additional Recipe Variations to provide runtime flexibility for Recipes.

Add Description.json

Every node in a recipe graph is defined by its own sub-directory within the Recipe, as seen by our two existing nodes placeholder-node1 and placeholder-node2. Create a new sub-directory called load-profits-node containing a description.json file. This will be an Action Node as defined by type in its description.json file.

Recipe versus Node Descriptions

Recipes contain recipe-level description.json files as well as node-level description.json files.

~/Kitchens/Feature_Sprint_2/Recipe-Template $ mkdir load-profits-node
~/Kitchens/Feature_Sprint_2/Recipe-Template $ cat > load-profits-node/description.json

After executing the above command, paste the following text at your command line:

{
   "type" : "DKNode_Action",
   "description": "Create AWS Redshift schema and table, copy to table example data from AWS S3, execute SQL query against table, and test returned results"
}

Type Enter then Ctrl+D to return to the command line prompt. You've now created the shell of a new Recipe graph node.

A view of the **load-profits-node**'s **description.json** file.

A view of the load-profits-node's description.json file.

Calling recipe-status confirms the addition of the new node structure.

~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk recipe-status
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk rs
YYYY-MM-DD HH:MM:SS - Getting the status of Recipe 'Recipe-Template' in Kitchen 'Feature_Sprint_2'
    versus directory '~/Kitchens/Feature_Sprint_2/Recipe-Template'
1 files are local only:
    load-profits-node/description.json

1 directories are local only:
    load-profits-node

10 files are unchanged

Add Node Configuration

Next, configure the load-profits-node by creating a configuration file located within an /actions subdirectory, which will orchestrate the data work to be completed by the node. Name this file load-data.json. Note that this file will use jinja templating to call the variable values from variables.json. As discussed, the variables set to sensitive credentials reference Vault Secrets.

~/Kitchens/Feature_Sprint_2/Recipe-Template $ mkdir load-profits-node/actions
~/Kitchens/Feature_Sprint_2/Recipe-Template $ cat > load-profits-node/actions/load-data.json

After executing the above command, paste the following text at your command line:

{
    "name": "load-profits-node",
    "type": "DKDataSource_PostgreSQL",
    "username": "{{redshiftusername}}",
    "password": "{{redshiftpassword}}",
    "hostname": "{{redshifthostname}}",
    "database": "{{redshiftdatabase}}",
    "port": "{{redshiftport}}",
    "keys": {
        "load_data": {
            "target-field": "sql",
            "resource-file": "s3-to-redshift.sql",
            "query-type": "execute_scalar"
        }
    }
}

Type Enter then Ctrl+D to return to the command line prompt.

A view of **load-data.json**, **load-profits-node**'s /actions configuration file.  Note the single **key** and numerous references to variables in **variables.json**.

A view of load-data.json, load-profits-node's /actions configuration file. Note the single key and numerous references to variables in variables.json.

~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk recipe-status
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk rs
YYYY-MM-DD HH:MM:SS - Getting the status of Recipe 'Recipe-Template' in Kitchen 'Feature_Sprint_2'
    versus directory '~/Kitchens/Feature_Sprint_2/Recipe-Template'
2 files are local only:
    load-profits-node/actions/load-data.json
    load-profits-node/description.json

2 directories are local only:
    load-profits-node
    load-profits-node/actions

10 files are unchanged

Additionally, this load-data.json configuration file for this node also contains a Key. Keys are the steps that do work with a Recipe graph node and are processed in the order the are defined. The single key load-data in this example is configured to execute a scalar (query-type) .sql (target-field) query as defined by s3-to-redshift.sql (resource-file), which we will create below and place in Recipe-Template's /resources directory.

Add Resource File(s)

The load-profits-node will leverage a .sql file named s3-to-redshift.sql within the /resources directory that will do the work of copying data from a .txt file located in an AWS S3 bucket into an AWS Redshift database table within a schema it has created. Note how this file also calls variable references from variables.json.

A view of **s3-to-redshift.sql**, a **/resource** file called to do work by a **key** in **load-profits-node**'s **load-data.json** configuration file.  Note the file's use of parameterization, which calls values from **variables.json**.

A view of s3-to-redshift.sql, a /resource file called to do work by a key in load-profits-node's load-data.json configuration file. Note the file's use of parameterization, which calls values from variables.json.

~/Kitchens/Feature_Sprint_2/Recipe-Template $ cat > resources/s3-to-redshift.sql

Next, paste the following text at your command line:

CREATE schema IF NOT EXISTS {{redshiftschema}} authorization {{redshiftusername}};

SET search_path TO {{redshiftschema}};

DROP TABLE if exists DEMO_SUPERSTORE_SALES_PROFITS;
CREATE TABLE DEMO_SUPERSTORE_SALES_PROFITS
(
  ID       VARCHAR(50),
  Sales    NUMERIC(10,2),
  Profit   NUMERIC(10,2)
);


copy DEMO_SUPERSTORE_SALES_PROFITS
from 's3://{{s3bucketname}}/{{s3bucketpath}}'
credentials 'aws_iam_role=arn:aws:iam::{{awsaccountid}}:role/{{awsrole}}'
ignoreheader 1
TRIMBLANKS delimiter '|';

select count(*) from DEMO_SUPERSTORE_SALES_PROFITS;

Type Enter then Ctrl+D to return to the command line prompt.

AWS S3 Paths

The from statement in the s3-to-redshift.sql file is pointing a a specific AWS S3 path that is composed of the S3 Bucket Name and the S3 Bucket Path, which are defined in variables.json. When we compile this file below we will see the following output. Sensitive Vault Secrets are compiled to their Vault paths whereas hard-coded values defined in variables.json are explicitly compiled.

s3://#{vault://s3_schema/bucket}/superstore/customer/sales-profitability

The .txt file containing our profits source data is located at this path.

A view of the AWS S3 bucket filepath from which **/resource** file **s3-to-redshift.sql** copies data to be inserted into a new AWS Redshift table.

A view of the AWS S3 bucket filepath from which /resource file s3-to-redshift.sql copies data to be inserted into a new AWS Redshift table.

~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk recipe-status
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk rs
YYYY-MM-DD HH:MM:SS - Getting the status of Recipe 'Recipe-Template' in Kitchen 'Feature_Sprint_2'
    versus directory '~/Kitchens/Feature_Sprint_2/Recipe-Template'
3 files are local only:
    load-profits-node/description.json
    load-profits-node/actions/load-data.json
    resources/s3-to-redshift.sql

2 directories are local only:
    load-profits-node
    load-profits-node/actions

10 files are unchanged

Compile Files

You may compile files to confirm variable calls have been configured correctly by using the file-compile command.

Security Assured

Note that when compiled, files with sensitive credentials compile to Vault paths rather than surfacing the sensitive Secrets directly. DataKitchen's DataOps orchestration provides all of your team members the agility to spin up and spin down infrastructure independently, based on the boundaries set by your technical staff.

~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk file-compile --variation Variation1 --file resources/s3-to-redshift.sql
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk fc -v Variation1 -f resources/s3-to-redshift.sql

Compiled Database Schema Name

Note that the value of the compiled database schema name will be specific to the value you set for the redshiftschema variable in the variables.json file for the Recipe.

YYYY-MM-DD HH:MM:SS - Get the Compiled File of Recipe Recipe-Template.Variation1 in Kitchen Feature_Sprint_2
DKCloudCommand.get_compiled_file succeeded:
CREATE schema IF NOT EXISTS jdoe authorization #{vault://redshift/username};

SET search_path TO jdoe;

DROP TABLE if exists DEMO_SUPERSTORE_SALES_PROFITS CASCADE;
CREATE TABLE DEMO_SUPERSTORE_SALES_PROFITS
(
  ID       VARCHAR(50),
  Sales    NUMERIC(10,2),
  Profit   NUMERIC(10,2)
);


copy DEMO_SUPERSTORE_SALES_PROFITS
from 's3://#{vault://s3_schema/bucket}/superstore/customer/sales-profitability'
credentials 'aws_iam_role=arn:aws:iam::#{vault://aws/account_id}:role/#{vault://aws/role}'
ignoreheader 1
TRIMBLANKS delimiter '|';

select count(*) from DEMO_SUPERSTORE_SALES_PROFITS;

Update Graph.json

We have added relevant Vault Secret references to variables.json, created a Recipe graph subdirectory, configuration file (including key), and an .sql /resources file. The final step in building a new Recipe graph node is to update graph.json to include the new node and edge we've added to the Recipe graph. Add the new load-profits-node to the Recipe graph such that it is positioned as the final node in the series.

~/Kitchens/Feature_Sprint_2/Recipe-Template $ cat graph.json
{
  "nodes": [
    ["placeholder-node1", {}],
    ["placeholder-node2", {}]
  ],
  "edges": [
    ["placeholder-node1", "placeholder-node2", {"weight": 0}]
  ]
}
**Recipe-Template**'s **graph.json** file *before* and *after* the addition of **load-profits-node**.

Recipe-Template's graph.json file before and after the addition of load-profits-node.

Replace the existing content in graph.json with the following text:

{
  "nodes": [
    ["placeholder-node1", {}],
    ["placeholder-node2", {}],
    ["load-profits-node", {}]
  ],
  "edges": [
    ["placeholder-node1", "placeholder-node2", {"weight": 0}],
    ["placeholder-node2", "load-profits-node", {"weight": 0}]
  ]
}
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk recipe-status
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk rs
YYYY-MM-DD HH:MM:SS - Getting the status of Recipe 'Recipe-Template' in Kitchen 'Feature_Sprint_2'
    versus directory '~/Kitchens/Feature_Sprint_2/Recipe-Template'
1 files are modified on local:
    graph.json

3 files are local only:
    load-profits-node/description.json
    load-profits-node/actions/load-data.json
    resources/s3-to-redshift.sql

2 directories are local only:
    load-profits-node
    load-profits-node/actions

9 files are unchanged

Recipe Update

Note that while we've accumulated a number of local changes to the Recipe-Template Recipe, we've yet to push these changes to the remote version of Recipe-Template. Rather than push to remote each of these new or modified local files piecemeal, we instead push all of the updates at once via the recipe-update command.

~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk recipe-update --message "Updating Recipe for addition of load-profits-node"
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk ru -m "Updating Recipe for addition of load-profits-node"
YYYY-MM-DD HH:MM:SS - Updating all changed files in Recipe (Recipe-Template) in Kitchen(Feature_Sprint_2) with message (Updating Recipe for addition of load-profits-node)
Update results:

New files:
    load-profits-node/actions/load-data.json
    load-profits-node/description.json
    resources/s3-to-redshift.sql
Updated files:
    graph.json
Deleted files:
    None

Issues:

No issues found

To be thorough, we can confirm that the local and remote versions of the Recipe-Template Recipe are now equivalent via the recipe-status command.

~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk recipe-status
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk rs
YYYY-MM-DD HH:MM:SS - Getting the status of Recipe 'Recipe-Template' in Kitchen 'Feature_Sprint_2'
    versus directory '~/Kitchens/Feature_Sprint_2/Recipe-Template'
13 files are unchanged

Run Test Order

Run an Order of the Recipe-Template Recipe per Variation1 using the order-run command. Next, input the orderrun-info command to gather information regarding the Order-Run.

~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk order-run Variation1 --yes
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk or Variation1 -y
~/Kitchens/Feature_Sprint_2/Recipe-Template $ cd ../..
~/Kitchens $ dk or -k Feature_Sprint_2 -r Recipe-Template Variation1 -y
YYYY-MM-DD HH:MM:SS - Create an Order:
    Kitchen: Feature_Sprint_2
    Recipe: Recipe-Template
    Variation: Variation1

Order ID is: DKRecipe#hw#Recipe-Template#Variation1#Feature_Sprint_2#1501261928072
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk orderrun-info --summary
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk ori -s
~/Kitchens $ dk ori -k Feature_Sprint_2 -s
YYYY-MM-DD HH:MM:SS - Display Order-Run details from kitchen Feature_Sprint_2

ORDER RUN SUMMARY

Order ID:       DKRecipe#hw#Recipe-Template#Variation1#Feature_Sprint_2#1507751549836
Order Run ID:   ct:1507751549986:0:DKRecipe#hw#Recipe-Template#Variation1#Feature_Sprint_2#1507751549836:
Status:        COMPLETED_SERVING
Kitchen:    Feature_Sprint_2
Recipe:        Recipe-Template
Variation:    Variation1
Start time:    YYYY-MM-DD HH:MM:SS
Run duration:    0:00:4 (H:M:S)
A view from [DataKitchen's DataOps web app](https://cloud.datakitchen.io) confirming the order run completed successfully.  The graph visualization shows successful processing of all three nodes.

A view from DataKitchen's DataOps web app confirming the order run completed successfully. The graph visualization shows successful processing of all three nodes.

How can we confirm that the data work performed by load-profits-node is as expected? In other words, how can we gain confidence that it has delivered quality output? One option is to connect to the AWS Redshift database directly and look for any changes from before and after we cooked the Order-Run for the Recipe-Template Recipe.

Connecting to AWS Redshift Databases

Thorough documentation is available for connecting to AWS Redshift. Many database query tools are available for this task, including SQL Workbench, which is open source. DataKitchen users without their own AWS account will not be able to connect to the Redshift database directly as the relevant AWS credentials are obscured and hidden as Vault Secrets.

The AWS Redshift database view associated with this guide will not contain a *quickckstart2 schema, as defined via variables.json, until after we cook the Order depicted above. After the OrderRun completes the database shows the *quickstart2 schema with the existence of a populated table named demo_superstore_sales_profits. This is the table generated by Recipe-Template's load-profits-node, specifically by the /resources/s3-to-redshift.sql file.

A view of the AWS Redshift table **demo_superstore_sales_profits** under the **quickstart2** schema.  This is the data that was loaded from the AWS S3 bucket.  This view is specifically from [SQL Workbench](), our recommended open-source database query tool.

A view of the AWS Redshift table demo_superstore_sales_profits under the quickstart2 schema. This is the data that was loaded from the AWS S3 bucket. This view is specifically from SQL Workbench, our recommended open-source database query tool.

This confirms that our Recipe-Template Recipe has processed as expected. Still, this is not an efficient process to confirm our Recipe has created the AWS Redshift table as expected . What is more, how can we gain confidence in the quality of the data that has been populated to the AWS Redshift table? There has to be a better way.

Thankfully, there is a better way -- implementing automated tests for each node in your Recipe graph via DataKitchen's DataOps platform.

Implement Node Test

Tests are configured at the node-level, specific to each Key defined by a Recipe node's configuration file. In this example Recipe-Template Recipe, /actions/load-data.json is the configuration file where we will implement a test. Note the file as it exists before an automated test is implemented.

~/Kitchens/Feature_Sprint_2/Recipe-Template $ cat load-profits-data/actions/load-data.json
{
    "name": "load-profits-node",
    "type": "DKDataSource_PostgreSQL",
    "username": "{{redshiftusername}}",
    "password": "{{redshiftpassword}}",
    "hostname": "{{redshifthostname}}",
    "database": "{{redshiftdatabase}}",
    "port": "{{redshiftport}}",
    "keys": {
        "load_data": {
            "target-field": "sql",
            "resource-file": "s3-to-redshift.sql",
            "query-type": "execute_scalar"
        }
    }
}
**Recipe-Template**'s **load-profits-node**'s configuration file, **/actions/load-data.json**, before an automated test has been implemented.

Recipe-Template's load-profits-node's configuration file, /actions/load-data.json, before an automated test has been implemented.

Recall the /resources/s3-to-redshift.sql file we previously created. Specifically, note the final line of the query, which generates a count of all row in the table the file has created. This return of this row count is the final output of the query. We will leverage this row count to implement a quality assurance test for our node.

~/Kitchens/Feature_Sprint_2/Recipe-Template $ cat resources/s3-to-redshift.sql
CREATE schema IF NOT EXISTS {{redshiftschema}} authorization {{redshiftusername}};

SET search_path TO {{redshiftschema}};

DROP TABLE if exists DEMO_SUPERSTORE_SALES_PROFITS CASCADE;
CREATE TABLE DEMO_SUPERSTORE_SALES_PROFITS
(
  ID       VARCHAR(50),
  Sales    NUMERIC(10,2),
  Profit   NUMERIC(10,2)
);


copy DEMO_SUPERSTORE_SALES_PROFITS
from 's3://{{s3bucketname}}/{{s3bucketpath}}'
credentials 'aws_iam_role=arn:aws:iam::{{awsaccountid}}:role/{{awsrole}}'
ignoreheader 1
TRIMBLANKS delimiter '|';

select count(*) from DEMO_SUPERSTORE_SALES_PROFITS;
The final statement in **Recipe-Template**'s **/resources** file **s3-to-redshift.sql** generates a row count that we'll use for our test.

The final statement in Recipe-Template's /resources file s3-to-redshift.sql generates a row count that we'll use for our test.

Update Contents for load-data.json

Replace the contents of your Recipe-Template's load-profits-node/actions/load-data.json file with the text below. The test-load-profits-node test we are implementing will warn us if the row count generated by the /resources/s3-to-redshift.sql is less than 1589, the row count we noted previously using SQL Workbench. This is a simple test that confirms that the data we have copied from the AWS S3 Bucket has been successfully loaded into an AWS Redshift database.

{
    "name": "load-profits-node",
    "type": "DKDataSource_PostgreSQL",
    "username": "{{redshiftusername}}",
    "password": "{{redshiftpassword}}",
    "hostname": "{{redshifthostname}}",
    "database": "{{redshiftdatabase}}",
    "port": "{{redshiftport}}",
    "keys": {
        "load_data": {
            "target-field": "sql",
            "resource-file": "s3-to-redshift.sql",
            "query-type": "execute_scalar"
        }
    },
    "tests": {
        "test-load-data-key": {
            "test-variable": "load_data",
            "type": "test-contents-as-integer",
            "applies-to-keys": ["load_data"],
            "action": "warning",
            "keep-history": "True",
            "test-logic": {
                "test-variable": "load_data",
                "test-compare": "greater-than-equal-to",
                "test-metric": "1589"
            }
        }
    }
}

See the file comparison of the /actions/load-data.json configuration file for the load-profits-node before and after applying a key test. The OLD file is included here for clarity.

**load-data.json** file before and after implementation of an automated test based on the output of the SQL query contained in **/resources/s3-to-redshift.sql**.

load-data.json file before and after implementation of an automated test based on the output of the SQL query contained in /resources/s3-to-redshift.sql.

Automated Tests

This example implements a simple row count automated test, which surprisingly goes a long way towards instilling confidence in your data analytic pipeline. The automated tests you implement can be vastly more complex and granular, specific to the data you work with and the structure of your Recipes.

Confirm the local change via the recipe-status command.

~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk recipe-status
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk rs
YYYY-MM-DD HH:MM:SS - Getting the status of Recipe 'Recipe-Template' in Kitchen 'Feature_Sprint_2'
    versus directory '~/Kitchens/Feature_Sprint_2/Recipe-Template'
1 files are modified on remote:
    load-profits-node/actions/load-data.json

12 files are unchanged

Push Local Change to Remote

Use the file-update command to push the local change to /actions/load-data.json to remote.

~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk file-update --message "Add test to load-profits-node" load-profits-node/actions/load-data.json
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk fu -m "Add test to load-profits-node" load-profits-node/actions/load-data.json
~/Kitchens/Feature_Sprint_2/Recipe-Template $ cd ../..
~/Kitchens $ dk fu -k Feature_Sprint_2 -r Recipe-Template -m "Add test to load-profits-node" load-profits-node/actions/load-data.json
YYYY-MM-DD HH:MM:SS - Updating File(s):
Kitchen:    Feature_Sprint_2
Recipe:        Recipe-Template
Message:  Add test to load-profits-node
DKCloudCommand.update_file for load-profits-node/actions/load-data.json succeeded

Run a Test Order

~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk order-run Variation1 --yes
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk or Variation1 -y
~/Kitchens $ dk or -k Feature_Sprint_2 -r Recipe-Template Variation1 -y
YYYY-MM-DD HH:MM:SS - Create an Order:
    Kitchen: Feature_Sprint_2
    Recipe: Recipe-Template
    Variation: Variation1

Order ID is: DKRecipe#hw#Recipe-Template#Variation1#Feature_Sprint_2#1507752884353

View Order-Run Information

Use the orderrun-info command with the --summary and --test options to gain confidence that the Recipe-Template Recipe processed as expected -- no need to connect to AWS Redshift and check the row count to confirm.

~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk orderrun-info --summary --test
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk ori -s -q
~/Kitchens $ dk ori -k Feature_Sprint_2 -s -q
YYYY-MM-DD HH:MM:SS - Display Order-Run details from kitchen Feature_Sprint_2

ORDER RUN SUMMARY

Order ID:       DKRecipe#hw#Recipe-Template#Variation1#Feature_Sprint_2#1507752884353
Order Run ID:   ct:1507752884508:0:DKRecipe#hw#Recipe-Template#Variation1#Feature_Sprint_2#1507752884353:
Status:        COMPLETED_SERVING
Kitchen:    Feature_Sprint_2
Recipe:        Recipe-Template
Variation:    Variation1
Start time:    YYYY-MM-DD HH:MM:SS
Run duration:    0:00:08 (H:M:S)

TEST RESULTS

Tests: Failed
    No Tests Failed

Tests: Warning
    No Tests Gave Warning Messages

Tests: Log
    No Tests Gave Log Messages

Tests: Passed
   Step (load-profits-node)
      1. test-load-data-key (1589 greater-than-equal-to 1589)
Email confirming the Order-Run for **Recipe-Template**'s **Variation1** completed successfully.  Note the inclusion of feedback for the automated test we implemented.

Email confirming the Order-Run for Recipe-Template's Variation1 completed successfully. Note the inclusion of feedback for the automated test we implemented.

List Orders

Display a list of Orders have taken place in a Kitchen via the order-list command. Recall that we have run two Orders in the Feature_Sprint_2 Kitchen; one before we implemented an automated test and one after implementation.

~/Kitchens/Feature_Sprint_2/Recipe-Template $ cd ..
~/Kitchens/Feature_Sprint_2 $ dk order-list
~/Kitchens/Feature_Sprint_2/Recipe-Template $ cd ..
~/Kitchens/Feature_Sprint_2 $ dk ol
~/Kitchens $ dk ol -k Feature_Sprint_2
YYYY-MM-DD HH:MM:SS - Get Order information for Kitchen Feature_Sprint_2

ORDER SUMMARY (order ID: DKRecipe#hw#Recipe-Template#Variation1#Feature_Sprint_2#1507752884353)
Kitchen:        Feature_Sprint_2
Recipe:         Recipe-Template
Variation:      Variation1
Schedule:       now
Status:         COMPLETED_ORDER

  1.  ORDER RUN (OrderRun ID: ct:1507752884508:0:DKRecipe#hw#Recipe-Template#Variation1#Feature_Sprint_2#1507752884353:)
        OrderRun Status OrderRun Completed
        Start time:    YYYY-MM-DD HH:MM:SS
    End time:    YYYY-MM-DD HH:MM:SS
        Duration:       0:00:08 (H:M:S)

ORDER SUMMARY (order ID: DKRecipe#hw#Recipe-Template#Variation1#Feature_Sprint_2#1507751549836)
Kitchen:    Feature_Sprint_2
Recipe:        Recipe-Template
Variation:    Variation1
Schedule:    now
Status:        COMPLETED_ORDER

  1.  ORDER RUN    (OrderRun ID: ct:1507751549986:0:DKRecipe#hw#Recipe-Template#Variation1#Feature_Sprint_2#1507751549836:))
    OrderRun Status OrderRun Completed
    Start time:    YYYY-MM-DD HH:MM:SS
    End time:    YYYY-MM-DD HH:MM:SS
    Duration:    0:00:04 (H:M:S)

Merge Parent to Child

Next, let's begin the process of merging our work on the Recipe-Template Recipe within the Feature_Sprint_2 Kitchen to its Parent Kitchen, Dev_Sprint_1. As always, first merge from Parent to Child to account for any changes that have occurred in the Parent since you branched to the Child Kitchen.

Kitchen-Merge-Preview

Before you complete the actual merge use the kitchen-merge-preview command to preview the forthcoming merge. Remember to use the --clean_previous_run option to clear any temporary local merge files that may exist as a result of a previously-aborted Kitchen merge.

As expected, no changes have occurred in the Parent Kitchen since the creation of the Child.

~/Kitchens/Feature_Sprint_2 $ cd ../Dev_Sprint_1
~/Kitchens/Dev_Sprint_1 $ dk kitchen-merge-preview --target_kitchen Feature_Sprint_2 --clean_previous_run
~/Kitchens/Feature_Sprint_2 $ cd ../Dev_Sprint_1
~/Kitchens/Dev_Sprint_1 $ dk kmp -tk Feature_Sprint_2 -cpr
~/Kitchens $ dk kmp -sk Dev_Sprint_1 -tk Feature_Sprint_2 -cpr
YYYY-MM-DD HH:MM:SS - Previewing merge Kitchen Dev_Sprint_1 into Kitchen Feature_Sprint_2
Merge Preview Results (only changed files are being displayed):
--------------------------------------------------------------

Nothing to merge.

Kitchen merge preview done.

Kitchen-Merge

Since the Parent-to-Child Kitchen merge preview indicated that there was nothing to merge we need not complete a Parent-to-Child Kitchen merge.

Child Test Order

Because no changes were merge from the Parent Kitchen, Dev_Sprint_1 to the Child Kitchen, Feature_Sprint_2 there is no need to run an Order in Feature_Sprint_2 with its associated tests.

Merge Child to Parent

Next, merge the Child Kitchen, Feature_Sprint_2 into its Parent Kitchen, Dev_Sprint.

Kitchen-Merge-Preview

Begin the Child-toParent merge process by using the kitchen-merge-preview command to preview the merge from Feature_Sprint_2 to Dev_Sprint_1. Clear any temporary local merge files from a previously-aborted Kitchen merge using the --clean_previous_run option.

~/Kitchens/Dev_Sprint_1 $ cd ../Feature_Sprint_2
~/Kitchens/Feature_Sprint_2 $ dk kitchen-merge-preview --target_kitchen Dev_Sprint_1 --clean_previous_run
~/Kitchens/Dev_Sprint_1 $ cd ../Feature_Sprint_2
~/Kitchens/Feature_Sprint_2 $ dk kkmp -tk Dev_Sprint_1 -cpr
~/Kitchens $ dk kkmp -sk Feature_Sprint_2 -tk Dev_Sprint_1 -cpr
YYYY-MM-DD HH:MM:SS - Previewing merge Kitchen Feature_Sprint_2 into Kitchen Dev_Sprint_1
Merge Preview Results (only changed files are being displayed):
--------------------------------------------------------------

      ok                Recipe-Template/load-profits-node/description.json
      ok                Recipe-Template/resources/s3-to-redshift.sql
      ok                Recipe-Template/load-profits-node/actions/load-data.json
      ok                Recipe-Template/graph.json
      ok                Recipe-Template/variables.json

Kitchen merge preview done.

Kitchen-Merge

Note that the command response summarizes all of the change we have built into the Recipe-Template Recipe with the Feature_Sprint_2 Kitchen. These changes will now be merged into the remote copy of the Recipe-Template Recipe in the Dev_Sprint Kitchen by use of teh kitchen-merge command.

~/Kitchens/Feature_Sprint_2 $ dk kitchen-merge --target_kitchen Dev_Sprint_1 --yes
~/Kitchens/Feature_Sprint_2 $ dk km -tk Dev_Sprint_1 -y
~/Kitchens $ dk km -sk Feature_Sprint_2 -tk Dev_Sprint_1 -y
YYYY-MM-DD HH:MM:SS - Merging Kitchen Feature_Sprint_2 into Kitchen Dev_Sprint_1
looking for manually merged files in temporary directory {USER_HOME}.dk/merges/Feature_Sprint_2_to_Dev_Sprint_1 
Calling Merge ...
 load-data.json             30         ++++++++++++++++++++++++++++++        
 graph.json                  6         ++++--                                
 variables.json             20         +++++++++++++++-----                  
 description.json            4         ++++                                  
 s3-to-redshift.sql         20         ++++++++++++++++++++                  
5 files changed, 73 insertions(+), 7 deletions(-)

Parent Test Order

~/Kitchens/Feature_Sprint_2 $ cd ../Dev_Sprint_1
~/Kitchens/Dev_Sprint_1 $ dk order-run --recipe Recipe-Template Variation1 --yes
~/Kitchens/Feature_Sprint_2 $ cd ../Dev_Sprint_1
~/Kitchens/Dev_Sprint_1 $ dk or -r Recipe-Template Variation1 -y
~/Kitchens $ dk or -k Dev_Sprint_1 -r Recipe-Template Variation1 -y
YYYY-MM-DD HH:MM:SS - Create an Order:
    Kitchen: Dev_Sprint_1
    Recipe: Recipe-Template
    Variation: Variation1

Order ID is: DKRecipe#hw#Recipe-Template#Variation1#Dev_Sprint_1#1501274634868

View information about the Order was ran in the Dev_Sprint_1 Kitchen via the orderrun-info command.

~/Kitchens/Dev_Sprint_1 $ dk orderrun-info --summary --tests
~/Kitchens/Dev_Sprint_1 $ dk ori -s -q
~/Kitchens $ dk ori -k Dev_Sprint_1 -s -q
YYYY-MM-DD HH:MM:SS - Display Order-Run details from kitchen Dev_Sprint_1

ORDER RUN SUMMARY

Order ID:    DKRecipe#hw#Recipe-Template#Variation1#Dev_Sprint_1#1501274634868
Order Run ID:    ct:1501274635037:0:DKRecipe#hw#Recipe-Template#Variation1#Dev_Sprint_1#1501274634868:
Status:        COMPLETED_SERVING
Kitchen:    Dev_Sprint_1
Recipe:        Recipe-Template
Variation:    Variation1
Start time:    YYYY-MM-DD HH:MM:SS
Run duration:    0:00:04 (H:M:S)

TEST RESULTS

Tests: Failed
    No Tests Failed

Tests: Warning
    No Tests Gave Warning Messages

Tests: Log
    No Tests Gave Log Messages

Tests: Passed
   Step (load-profits-node)
      1. test-load-data-key (1589 greater-than-equal-to 1589)

Teardown Child Kitchen

Since all development work has been merged into the Dev_Sprint_1 Parent Kitchen we may delete the Feature_Sprint_2 Kitchen via the kitchen-delete command. This will teardown Feature_Sprint_2's computing resources and infrastructure.

~/Kitchens/Dev_Sprint_1 $ cd ..
~/Kitchens $ dk kitchen-delete Feature_Sprint_2 --yes
~/Kitchens/Dev_Sprint_1 $ cd ..
~/Kitchens $ dk kd Feature_Sprint_2 -y
No child kitchens found.
YYYY-MM-DD HH:MM:SS - Deleting remote copy of kitchen Feature_Sprint_2. Local files will not change.
><
Kitchen Feature_Sprint_2 has been deleted

Wrapping Up

To recap, this CLI Quickstart2 provided an overview of DataOps orchestration via DKCloudCommand, DataKitchen's command line tool. Specific topics covered include:

  • The Vault and its Secure Storage of Secret Credentials
  • Configuring Recipe Graph Nodes
  • Viewing a File's Change History
  • Compiling Files to Review the Application of Parameter Values
  • Cooking an Order-Run of a Recipe Variation with Expanded Options
  • Implementing an Automated Tests
  • Understanding Test Types and Reviewing Their Feedback
  • Batch Updating Local Recipe File Changes to Remote
  • Listing the Orders that Exist in a Kitchen

We've now learned to build Recipe nodes from scratch that demonstrate some of the real heavy-lifting required for a robust data analytic pipeline. What's more, we've reviewed how to implement automated tests in a Recipe Graph node whose feedback give you confidence in the quality of the data flowing through your data analytic pipeline, and consequently, the insights you deliver to your stakeholders. In the next section of this guide we'll dive into building additional node types for the existing tools in your toolchain, creating Recipe Variations, and implementing parameterization for runtime flexibility of your Order-Runs.

Synopsis

SET THE TABLE
~/Kitchens $ dk kitchen-list
~/Kitchens $ cd Dev_Sprint_1
~/Kitchens/Dev_Sprint_1 $ dk recipe-list

CREATE CHILD KITCHEN
~/Kitchens/Dev_Sprint_1 $ cd ..
~/Kitchens $ dk kitchen-create --parent Dev_Sprint_1 Feature_Sprint_2
~/Kitchens $ dk kitchen-list

GET KITCHEN & RECIPES
~/Kitchens $ dk kitchen-get Feature_Sprint_2 --all

VAULT SECRETS
~/Kitchens $ dk secret-list --recursive
~/Kitchens $ dk secret-write service-name/test-credential=test-value --yes
~/Kitchens $ dk secret-exists service-name/test-credential
~/Kitchens $ dk secret-delete service-name/test-credential --yes

ADD NODE: UPDATE VARIABLES
~/Kitchens $ cd Feature_Sprint_2/Recipe-Template
~/Kitchens/Feature_Sprint_2/Recipe-Template $ cat variables.json
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk secret-list --recursive
[REPLACE VARIABLES.JSON CONTENT WITH SUPPLIED TEXT & UPDATE EMAIL FIELD]
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk recipe-status
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk file-update --message "Change message" variables.json

FILE HISTORY
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk file-history variables.json

ADD NODE: CREATE NODE DIRECTORY & DESCRIPTION
~/Kitchens/Feature_Sprint_2/Recipe-Template $ mkdir load-profits-node
~/Kitchens/Feature_Sprint_2/Recipe-Template $ cat > load-profits-node/description.json
[PASTE SUPPLIED TEXT AT COMMAND LINE, <ENTER>, <CTRL+D>]
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk recipe-status

ADD NODE: CREATE NODE CONFIGURATION
~/Kitchens/Feature_Sprint_2/Recipe-Template $ mkdir load-profits-node/actions
~/Kitchens/Feature_Sprint_2/Recipe-Template $ cat > load-profits-node/actions/load-data.json
[PASTE SUPPLIED TEXT AT COMMAND LINE, <ENTER>, <CTRL+D>]
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk recipe-status

ADD NODE: CREATE SQL RESOURCE FILE
~/Kitchens/Feature_Sprint_2/Recipe-Template $ cat > resources/s3-to-redshift.sql
[PASTE SUPPLIED TEXT AT COMMAND LINE, <ENTER>, <CTRL+D>]
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk recipe-status

COMPILE FILE
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk file-compile --variation Variation1 --file resources/s3-to-redshift.sql

ADD NODE: UPDATE RECIPE GRAPH
~/Kitchens/Feature_Sprint_2/Recipe-Template $ cat graph.json
[REPLACE GRAPH.JSON CONTENT WITH SUPPLIED TEXT]
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk recipe-status

RECIPE UPDATE
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk recipe-update --message "Change message"
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk recipe-status

RUN TEST ORDER
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk order-run Variation1 --yes
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk orderrun-info --summary

IMPLEMENT NODE TEST & RUN ORDER
~/Kitchens/Feature_Sprint_2/Recipe-Template $ cat load-profits-data/actions/load-data.json
~/Kitchens/Feature_Sprint_2/Recipe-Template $ cat resources/s3-to-redshift.sql
[REPLACE LOAD-DATA.JSON CONTENT WITH SUPPLIED TEXT]
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk recipe-status
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk file-update --message "Change message" load-profits-data/actions/load-data.json
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk order-run Variation1 --yes
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk orderrun-info --summary --test
[CONFIRM ORDER-RUN & TESTING FEEDBACK VIA EMAIL ALERTS]

LIST ORDERS
~/Kitchens/Feature_Sprint_2/Recipe-Template $ cd ..
~/Kitchens/Feature_Sprint_2 $ dk order-list

MERGE: PARENT TO CHILD
~/Kitchens/Feature_Sprint_2 $ cd ../Dev_Sprint_1
~/Kitchens/Dev_Sprint_1 $ dk kitchen-merge-preview --target_kitchen Feature_Sprint_2 --clean_previous_run

MERGE: CHILD TO PARENT
~/Kitchens/Dev_Sprint_1 $ cd ../Feature_Sprint_2
~/Kitchens/Feature_Sprint_2 $ dk kitchen-merge-preview --target_kitchen Dev_Sprint_1 --clean_previous_run
~/Kitchens/Feature_Sprint_2 $ dk kitchen-merge --target_kitchen Dev_Sprint_1 --yes

PARENT TEST ORDER
~/Kitchens/Feature_Sprint_2 $ cd ../Dev_Sprint_1
~/Kitchens/Dev_Sprint_1 $ dk order-run --recipe Recipe-Template Variation1 --yes
~/Kitchens/Dev_Sprint_1 $ dk orderrun-info --summary --test

TEARDOWN CHILD KITCHEN
~/Kitchens/Dev_Sprint_1 $ cd ..
~/Kitchens $ dk kitchen-delete Feature_Sprint_2 --yes
SET THE TABLE
~/Kitchens $ dk kl
~/Kitchens $ cd Dev_Sprint_1
~/Kitchens/Dev_Sprint_1 $ dk rl

CREATE CHILD KITCHEN
~/Kitchens/Dev_Sprint_1 $ cd ..
~/Kitchens $ dk kc -p Dev_Sprint_1 Feature_Sprint_2
~/Kitchens $ dk kl

GET KITCHEN & RECIPES
~/Kitchens $ dk kg Feature_Sprint_2 -a

VAULT SECRETS
~/Kitchens $ dk sl -rc
~/Kitchens $ dk sw service-name/test-credential=test-value -y
~/Kitchens $ dk se service-name/test-credential
~/Kitchens $ dk sd service-name/test-credential -y

ADD NODE: UPDATE VARIABLES
~/Kitchens $ cd Feature_Sprint_2/Recipe-Template
~/Kitchens/Feature_Sprint_2/Recipe-Template $ cat variables.json
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk sl -rc
[REPLACE VARIABLES.JSON CONTENT WITH SUPPLIED TEXT & UPDATE EMAIL FIELD]
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk rs
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk fu -m "Change message" variables.json

FILE HISTORY
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk fh variables.json

ADD NODE: CREATE NODE DIRECTORY & DESCRIPTION
~/Kitchens/Feature_Sprint_2/Recipe-Template $ mkdir load-profits-node
~/Kitchens/Feature_Sprint_2/Recipe-Template $ cat > load-profits-node/description.json
[PASTE SUPPLIED TEXT AT COMMAND LINE, <ENTER>, <CTRL+D>]
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk rs

ADD NODE: CREATE NODE CONFIGURATION
~/Kitchens/Feature_Sprint_2/Recipe-Template $ mkdir load-profits-node/actions
~/Kitchens/Feature_Sprint_2/Recipe-Template $ cat > load-profits-node/actions/load-data.json
[PASTE SUPPLIED TEXT AT COMMAND LINE, <ENTER>, <CTRL+D>]
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk rs

ADD NODE: CREATE SQL RESOURCE FILE
~/Kitchens/Feature_Sprint_2/Recipe-Template $ cat > resources/s3-to-redshift.sql
[PASTE SUPPLIED TEXT AT COMMAND LINE, <ENTER>, <CTRL+D>]
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk rs

COMPILE FILE
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk fc -v Variation1 -f resources/s3-to-redshift.sql

ADD NODE: UPDATE RECIPE GRAPH
~/Kitchens/Feature_Sprint_2/Recipe-Template $ cat graph.json
[REPLACE GRAPH.JSON CONTENT WITH SUPPLIED TEXT]
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk rs

RECIPE UPDATE
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk ru -m "Change message"
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk rs

RUN TEST ORDER
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk or Variation1 -y
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk ori -s

IMPLEMENT NODE TEST & RUN ORDER
~/Kitchens/Feature_Sprint_2/Recipe-Template $ cat load-profits-node/actions/load-data.json
~/Kitchens/Feature_Sprint_2/Recipe-Template $ cat resources/s3-to-redshift.sql
[REPLACE LOAD-DATA.JSON CONTENT WITH SUPPLIED TEXT]
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk rs
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk fu -m "Change message" load-profits-data/actions/load-data.json
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk or Variation1 -y
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk ori -s -q
[CONFIRM ORDER-RUN & TESTING FEEDBACK VIA EMAIL ALERTS]

LIST ORDERS
~/Kitchens/Feature_Sprint_2/Recipe-Template $ cd ..
~/Kitchens/Feature_Sprint_2 $ dk ol

MERGE: PARENT TO CHILD
~/Kitchens/Feature_Sprint_2 $ cd ../Dev_Sprint_1
~/Kitchens/Dev_Sprint_1 $ dk kmp -tk Feature_Sprint_2 -cpr

MERGE: CHILD TO PARENT
~/Kitchens/Dev_Sprint_1 $ cd ../Feature_Sprint_2
~/Kitchens/Feature_Sprint_2 $ dk km -tk Dev_Sprint_1 -cpr
~/Kitchens/Feature_Sprint_2 $ dk km -tk Dev_Sprint_1 -y

PARENT TEST ORDER
~/Kitchens/Feature_Sprint_2 $ cd ../Dev_Sprint_1
~/Kitchens/Dev_Sprint_1 $ dk or -r Recipe-Template Variation1 -y
~/Kitchens/Dev_Sprint_1 $ dk ori -s -q

TEARDOWN CHILD KITCHEN
~/Kitchens/Dev_Sprint_1 $ cd ..
~/Kitchens $ dk kd Feature_Sprint_2 -y
SET THE TABLE
~/Kitchens $ dk kl
~/Kitchens $ dk rl -k Dev_Sprint_1

CREATE CHILD KITCHEN
~/Kitchens $ dk kc -p Dev_Sprint_1 Feature_Sprint_2
~/Kitchens $ dk kl

GET KITCHEN & RECIPES
~/Kitchens $ dk kg Feature_Sprint_2 -a

VAULT SECRETS
~/Kitchens $ dk sl -rc
~/Kitchens $ dk sw service-name/test-credential=test-value -y
~/Kitchens $ dk se service-name/test-credential
~/Kitchens $ dk sd service-name/test-credential -y

ADD NODE: UPDATE VARIABLES
~/Kitchens $ cat Feature_Sprint_2/Recipe-Template/variables.json
~/Kitchens $ dk sl -rc
[REPLACE VARIABLES.JSON CONTENT WITH SUPPLIED TEXT & UPDATE EMAIL FIELD]
~/Kitchens $ cd Feature_Sprint_2/Recipe-Template
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk rs
~/Kitchens/Feature_Sprint_2/Recipe-Template $ cd ../..
~/Kitchens $ dk file-update -k Feature_Sprint_2 -r Recipe-Template -m "Change message" variables.json
~/Kitchens $ cd Feature_Sprint_2/Recipe-Template

FILE HISTORY
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk fh Feature_Sprint_2/Recipe-Template/variables.json

ADD NODE: CREATE NODE DIRECTORY & DESCRIPTION
~/Kitchens/Feature_Sprint_2/Recipe-Template $ mkdir load-profits-node
~/Kitchens/Feature_Sprint_2/Recipe-Template $ cat > load-profits-node/description.json
[PASTE SUPPLIED TEXT AT COMMAND LINE, <ENTER>, <CTRL+D>]
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk rs

ADD NODE: CREATE NODE CONFIGURATION
~/Kitchens/Feature_Sprint_2/Recipe-Template $ $ mkdir load-profits-node/actions
~/Kitchens/Feature_Sprint_2/Recipe-Template $ cat > load-profits-node/actions/load-data.json
[PASTE SUPPLIED TEXT AT COMMAND LINE, <ENTER>, <CTRL+D>]
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk rs 

ADD NODE: CREATE SQL RESOURCE FILE
~/Kitchens/Feature_Sprint_2/Recipe-Template $ cat > resources/s3-to-redshift.sql
[PASTE SUPPLIED TEXT AT COMMAND LINE, <ENTER>, <CTRL+D>]
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk rs

COMPILE FILE
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk fc -v Variation1 -f resources/s3-to-redshift.sql

ADD NODE: UPDATE RECIPE GRAPH
~/Kitchens/Feature_Sprint_2/Recipe-Template $ cat graph.json
[REPLACE GRAPH.JSON CONTENT WITH SUPPLIED TEXT]
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk rs

RECIPE UPDATE
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk ru -m "Change message"
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk rs

RUN TEST ORDER
~/Kitchens/Feature_Sprint_2/Recipe-Template $ cd ../..
~/Kitchens $ dk or -k Feature_Sprint_2 -r Recipe-Template Variation1 -y
~/Kitchens $ dk ori -k Feature_Sprint_2 -s

IMPLEMENT NODE TEST & RUN ORDER
~/Kitchens $ cat Feature_Sprint_2/Recipe-Template/load-profits-node/actions/load-data.json
~/Kitchens $ cat Feature_Sprint_2/Recipe-Template/resources/s3-to-redshift.sql
[REPLACE LOAD-DATA.JSON CONTENT WITH SUPPLIED TEXT]
~/Kitchens $ cd Feature_Sprint_2/Recipe-Template
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk rs
~/Kitchens/Feature_Sprint_2/Recipe-Template $ cd ../..
~/Kitchens $ dk fu -k Feature_Sprint_2 -r Recipe-Template -m "Change message" load-profits-data/actions/load-data.json
~/Kitchens $ dk or -k Feature_Sprint_2 -r Recipe-Template Variation1 -y
~/Kitchens $ dk ori -k Feature_Sprint_2 -s -q
[CONFIRM ORDER-RUN & TESTING FEEDBACK VIA EMAIL ALERTS]

LIST ORDERS
~/Kitchens $ dk ol -k Feature_Sprint_2

MERGE: PARENT TO CHILD
~/Kitchens $ dk kmp -sk Dev_Sprint_1 -tk Feature_Sprint_2 -cpr

MERGE: CHILD TO PARENT
~/Kitchens $ dk kmp -sk Feature_Sprint_2 -tk Dev_Sprint_1 -cpr
~/Kitchens $ dk km -sk Feature_Sprint_2 -tk Dev_Sprint_1 -y

PARENT TEST ORDER
~/Kitchens $ dk or -k Dev_Sprint_1 -r Recipe-Template Variation1 -y
~/Kitchens $ dk ori -k Dev_Sprint_1 -s -q

TEARDOWN CHILD KITCHEN
~/Kitchens $ dk kd Feature_Sprint_2 -y
SET THE TABLE
~/Kitchens $ dk kl
~/Kitchens $ dk rl -k Dev_Sprint_1

CREATE CHILD KITCHEN
~/Kitchens $ dk kc -p Dev_Sprint_1 Feature_Sprint_2
~/Kitchens $ dk kl

GET KITCHEN & RECIPES
~/Kitchens $ dk kg Feature_Sprint_2 -a

FILE HISTORY
~/Kitchens $ cd Feature_Sprint_2/Recipe-Template
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk fh variables.json

DELETE QS1 RECIPE & CREATE QS2 RECIPE FROM TEMPLATE
[SKIPS STEPS TO BUILD FROM SCRATCH QS2 RECIPE-TEMPLATE VIA QS1 RECIPE-TEMPLATE]
~/Kitchens/Feature_Sprint_2/Recipe-Template $ cd ..
~/Kitchens/Feature_Sprint_2 $ dk rd Recipe-Template -y
~/Kitchens/Feature_Sprint_2 $ rm -r Recipe-Template
~/Kitchens/Feature_Sprint_2 $ dk re -tm qs2 Recipe-Template
~/Kitchens/Feature_Sprint_2 $ dk rg Recipe-Template

MAKE LOCAL RECIPE CHANGES & UPDATE TO REMOTE
~/Kitchens/Feature_Sprint_2 $ cd Recipe-Template
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk rs
[UPDATE EMAIL VALUE IN VARIABLES.JSON]
[UPDATE REDSHIFT SCHEMA VALUE IN VARIABLES.JSON, e.g. "jdoe"]
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk ru -m "Push local changes to remote"
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk rs

VAULT SECRETS
~/Kitchens/Feature_Sprint_2 $ dk sl -rc

COMPILE FILE
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk fc -v Variation1 -f resources/s3-to-redshift.sql

IMPLEMENT NODE TEST & RUN ORDER
[REPLACE LOAD-DATA.JSON CONTENT WITH SUPPLIED TEXT]
[https://datakitchen.readme.iodocs/quickstart2#section-updated-contents-for-load-data-json]
~/Kitchens/Feature_Sprint_2/Recipe-Template $ dk rs
~/Kitchens/Feature_Sprint_2/Recipe-Template $ cd ../..
~/Kitchens $ dk fu -k Feature_Sprint_2 -r Recipe-Template -m "Implement automated test" action-node/actions/load-data.json
~/Kitchens $ dk or -k Feature_Sprint_2 -r Recipe-Template Variation1 -y
~/Kitchens $ dk ori -k Feature_Sprint_2 -s -q
[CONFIRM ORDER-RUN & TESTING FEEDBACK VIA EMAIL ALERTS]

LIST ORDERS
~/Kitchens $ dk ol -k Feature_Sprint_2

MERGE: PARENT TO CHILD
~/Kitchens $ dk kmp -sk Dev_Sprint_1 -tk Feature_Sprint_2 -cpr

MERGE: CHILD TO PARENT
~/Kitchens $ dk kmp -sk Feature_Sprint_2 -tk Dev_Sprint_1 -cpr
~/Kitchens $ dk km -sk Feature_Sprint_2 -tk Dev_Sprint_1 -y

PARENT TEST ORDER
~/Kitchens $ dk or -k Dev_Sprint_1 -r Recipe-Template Variation1 -y
~/Kitchens $ dk ori -k Dev_Sprint_1 -s

TEAR DOWN CHILD KITCHEN
~/Kitchens $ dk kd Feature_Sprint_2 -y

Updated about a month ago


CLI Quickstart2


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.