DataKitchen DataOps Documention

CLI Quickstart2

This CLI Quickstart2 covers the use of DKCloudCommand, DataKitchen's command line tool. You will learn about the following topics.

  • The Vault and secure storage of secret credentials
  • Configuring recipe graph nodes
  • Viewing a file's change history
  • Compiling files to review the application of parameter values
  • Cooking an order run of a recipe variation with expanded options
  • Implementing an automated tests
  • Understanding test types and reviewing their feedback
  • Batch updating local recipe file changes to remote
  • Listing the orders that exist in a kitchen

Super-Fast Quickstart2

Super-Fast Quickstart2

Advanced users who need less detailed documentation, can jump right to the SUPER-FAST CLI QUICKSTART2 tab of this guide's Synopsis for a menu of abridged commands. This route provides a fully complete recipe for this guide.

Review Your Environment

  1. Determine your current kitchen hierarchy.
~/Kitchens $ dk kitchen-list
~/Kitchens $ dk kl

Output: kitchen-list command

YYYY-MM-DD HH:MM:SS - Getting the list of kitchens
kitchen-list returned 3 kitchens
+---- master
    +---- Production
            +---- Dev_Sprint_1
  1. Determine what recipes exist in the Dev_Sprint_1 kitchen.
~/Kitchens $ cd Dev_Sprint_1
~/Kitchens/Dev_Sprint_1 $ dk recipe-list
~/Kitchens $ cd Dev_Sprint_1
~/Kitchens/Dev_Sprint_1 $ dk rl
~/Kitchens $ dk rl -k Dev_Sprint_1

Output: recipe-list command

YYYY-MM-DD HH:MM:SS - Getting the list of Recipes for Kitchen 'Dev_Sprint_1'
list_recipe - elapsed: 0
DKCloudCommand.list_recipe returned 1 recipes
  Recipe_Template

Create Child Kitchen

You will build on your Recipe_Template recipe so that when processed it will perform data work that generates an output. You can test this output.

  1. Use the kitchen-create command to create another child kitchen named Feature_Sprint_2 with Dev_Sprint_1 as its parent.
    Feature_Sprint_2 will inherit a copy of the Recipe_Template recipe, which is the copy where you will do your work.

Create a Uniquely-Named Kitchen

Append your name when creating the kitchen so that you have your own, segregated workspace. For example, if your name is Chef, you would create a kitchen named Feature_Sprint_2_Chef.

~/Kitchens/Dev_Sprint_1 $ cd ..
~/Kitchens $ dk kitchen-create --parent Dev_Sprint_1 Feature_Sprint_2
~/Kitchens/Dev_Sprint_1 $ cd ..
~/Kitchens $ dk kc -p Dev_Sprint_1 Feature_Sprint_2

Output: kitchen-create command

YYYY-MM-DD HH:MM:SS - Creating kitchen Feature_Sprint_2 from parent kitchen Dev_Sprint_1
DKCloudCommand.create_kitchen created Feature_Sprint_2
  1. Use the kitchen-list command to display the kitchens available to you.
~/Kitchens $ dk kitchen-list
~/Kitchens $ dk kl

Output: kitchen-list command

YYYY-MM-DD HH:MM:SS - Getting the list of kitchens
kitchen-list returned 4 kitchens
+---- master
    +---- Production
            +---- Dev_Sprint_1
                    +---- Feature_Sprint_2

Get Kitchen & Recipes

  1. Use the kitchen-get command with the --all option to get a local copy of a kitchen and all of its recipes.
~/Kitchens $ dk kitchen-get Feature_Sprint_2 --all
~/Kitchens $ dk kg Feature_Sprint_2 -a

Output: kitchen-get command

YYYY-MM-DD HH:MM:SS - Getting kitchen 'Feature_Sprint_2'
list_recipe - elapsed: 0
Got Kitchen 'Feature_Sprint_2'
DKCloudCommand.get_recipe has 5 sections
  Recipe_Template/resources/email-templates
  Recipe_Template
  Recipe_Template/placeholder-node1
  Recipe_Template/resources
  Recipe_Template/placeholder-node2

The Vault

The Vault secures sensitive infrastructure and toolchain credentials and stores them remotely. These credentials are called Secrets.

Note that the vault is maintained at the customer account level, so the secrets it contains can be leveraged by any recipe in any kitchen under a customer account. While these secrets can be leveraged by users in data analytics pipelines, vault secrets cannot be viewed without special permissions.

Now you will learn how to interact with the vault via DKCloudCommand to help build vault references into your recipes.

Manage Secrets

  1. Use the secret-list command to view a list of the secrets that are stored in the vault.

Check Your Vault Secrets

Note that the list of vault secrets below match vaults that have been set up for prospective customer sandboxes. If you are a new user for an existing customer, or if your company manages their own custom vault configurations, the vault paths you will need to reference as you build your quickstart recipes will likely differ from what is shown below.

You can view the specific vault paths available to reference in your variables.json and variations.json configuration files via the Secrets page in the web app or by using the DKCloudCommand's secret-list command.

~/Kitchens $ dk secret-list
~/Kitchens $ dk sl

Output: secret-list command

YYYY-MM-DD HH:MM:SS - Getting the list of secrets

---- global secrets ----
        vault://aws/account_id
        vault://aws/role
        vault://dockerhub/email
        vault://dockerhub/namespace
        vault://dockerhub/password
        vault://dockerhub/username
        vault://redshift/database
        vault://redshift/hostname
        vault://redshift/password
        vault://redshift/port
        vault://redshift/username
        vault://s3_schema/bucket
        vault://s3_schema/s3-access-key
        vault://s3_schema/s3-secret-key
  1. Use the secret-write command to write a secret to the vault.
~/Kitchens $ dk secret-write service-name/test-credential=test-value --yes
~/Kitchens $ dk sw service-name/test-credential=test-value -y

Output: secret-write command

YYYY-MM-DD HH:MM:SS - Writing secret
Secret written.
  1. Use the secret-exists command to determine if a specific secret exists in the vault.
~/Kitchens $ dk secret-exists service-name/test-credential
~/Kitchens $ dk se service-name/test-credential

Output: secret-exists command

YYYY-MM-DD HH:MM:SS Checking secret in global vault
True (found in global vault)
  1. Use the secret-delete command to delete a secret from the vault.
~/Kitchens $ dk secret-delete service-name/test-credential --yes
~/Kitchens $ dk sd service-name/test-credential -y

Output: secret-delete command

YYYY-MM-DD HH:MM:SS - Deleting secret
Secret deleted.

Add a Recipe Graph Node

In Quickstart1, you ran an order to cook a recipe variation composed of synchronize nodes that had no interaction with data. Recipe graphs require at least two nodes in order for an edge to exist between them, so synchronize nodes are helpful as placeholders for new recipes and test orders for a single node in a recipe.

Next, you will add a new node to your recipe graph to begin to build a recipe that does real data work.

Specifically, you will create an action node and configure it to create a new schema and table in an AWS Redshift database and copy example data from an AWS S3 bucket into the table.

Connecting to an AWS S3 Bucket

Detailed documentation is available regarding how to setup a connection to an AWS S3 Bucket.

Additionally, you will execute a SQL query against the table and test the returned result in order to give confidence in the quality of the node's output.

A view of the AWS S3 Bucket where the data begins its journey through the Recipe_Template recipe's data analytic pipeline.  This view is specifically from [Transmit](https://panic.com/transmit/), a recommended SFTP and S3 application.

A view of the AWS S3 Bucket where the data begins its journey through the Recipe_Template recipe's data analytic pipeline. This view is specifically from Transmit, a recommended SFTP and S3 application.

Update Variables.json

  1. View the existing content of the recipe's variables.json file.
    The contents of this file originated in CLI Quickstart1 in your Feature_Sprint_1 kitchen.

You will be adding to this file a number of new variables that will be leveraged by the updated version of the Recipe_Template recipe you are building.

~/Kitchens $ cd Feature_Sprint_2/Recipe_Template
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cat variables.json

Output: concatenate command for the variables.json file

{
    "variable-list": {
        "alerts": {
          "start": "your-email-from-quickstart1",
          "success": "your-email-from-quickstart1",
          "error": "your-email-from-quickstart1"
        }
    }
}
  1. Recall the secrets stored in the vault that are relevant to AWS S3 and Redshift services.
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk secret-list
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk sl

Output: secret-list command

YYYY-MM-DD HH:MM:SS - Getting the list of secrets
secret_list - elapsed: 0
secret-list returned 14 secrets
        vault://aws/account_id
        vault://aws/role
        vault://dockerhub/email
        vault://dockerhub/namespace
        vault://dockerhub/password
        vault://dockerhub/username
        vault://redshift/database
        vault://redshift/hostname
        vault://redshift/password
        vault://redshift/port
        vault://redshift/username
        vault://s3_schema/bucket
        vault://s3_schema/s3-access-key
        vault://s3_schema/s3-secret-key

You want to add these secrets to your recipe's variables.json configuration file so that they can be leveraged when you run an order for your recipe. Because secret credentials are inherently sensitive, we create variables that reference their vault locations rather than hard-coding their values.

  1. Replace all existing content in your recipe's variables.json file with the text below. Be sure to replace the email value listed. Be careful to maintain proper JSON syntax.

Update Variable Values

Email
Remember to replace the YOUR-EMAIL-HERE value with your preferred email address after copying the text below into the variables.json file, and save the file.

Redshift Schema
Update the redshiftschema value after copying the text below into variables.json file so that you have your own unique database schema. Consider using your first initial and last name for the schema name. For example, a user named John Doe would replace YOUR-SCHEMA-NAME-HERE with jdoe.

{
  "variable-list": {
        "alerts": {
          "start": "YOUR-EMAIL-HERE",
          "success": "YOUR-EMAIL-HERE",
          "error": "YOUR-EMAIL-HERE"
        },
    "awsaccountid": "#{vault://aws/account_id}",
    "awsrole": "#{vault://aws/role}",
    "redshiftdatabase": "#{vault://redshift/database}",
    "redshifthostname": "#{vault://redshift/hostname}",
    "redshiftpassword": "#{vault://redshift/password}",
    "redshiftport": "#{vault://redshift/port}",
    "redshiftschema": "YOUR-SCHEMA-NAME-HERE",
    "redshiftusername": "#{vault://redshift/username}",
    "s3bucketname": "#{vault://s3_schema/bucket}",
    "s3bucketpath": "superstore/customer/sales-profitability"
  }
}

AWS S3 Credentials

Note that while the load-profits-node you are building in this quickstart does gather its source data from an AWS S3 bucket, the node does not require use of the s3-access-key and s3-secret-key credentials to do its work. Thus, the updated variables.json file for this Quickstart2 will not contain references to these secrets' vault locations. You will leverage these credentials in Quickstart3.

  1. Use the recipe-status command to confirm the change to variables.json has been applied to the local version of the Recipe_Template recipe.
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk recipe-status
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk rs

Output: recipe-status command

YYYY-MM-DD HH:MM:SS - Getting the status of Recipe 'Recipe_Template' in Kitchen 'Feature_Sprint_2'
    versus directory '~/Kitchens/Feature_Sprint_2/Recipe_Template'
1 files are modified on local:
    variables.json

9 files are unchanged
  1. Update local changes to variables.json to remote via the file-update command.
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk file-update --message "Update variables.json with QS2 variables" variables.json
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk fu -m "Update variables.json with QS2 variables" variables.json
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cd ../..
~/Kitchens $ dk fu -k Feature_Sprint_2 -r Recipe_Template -m "Update variables.json with QS2 variables" variables.json

Output: file-update command

YYYY-MM-DD HH:MM:SS - Updating File(s) (('variables.json',)) in Recipe (Recipe_Template) in Kitchen(Feature_Sprint_2) with message (Update variables.json with QS2 variables)
DKCloudCommand.update_file for variables.json succeeded

You will hold off updating additional local changes to the remote recipe files until you complete editing the Recipe_Template recipe. You will use the recipe-update command in a later step to batch update recipe files.

File History

  1. View the change history for a specific file via the file-history command.
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk file-history variables.json
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk fh variables.json

Output: file-history command

YYYY-MM-DD HH:MM:SS - Retrieving file history
DKCloudCommand.file_history succeeded:
Author:   [DK USERNAME] <[DK USERNAME]>
Date:     YYYY-MM-DDTHH:MM:SSZ
Message:  Update variables.json with QS2 variables
Url:      [LINK TO HISTORY ENTRY]

Author:   [DK USERNAME] <[DK USERNAME]>
Date:     YYYY-MM-DDTHH:MM:SSZ
Message:  Push email value update within variables.json
Url:      [LINK TO HISTORY ENTRY]

Author:   [DK USERNAME] <[DK USERNAME]>
Date:     YYYY-MM-DDTHH:MM:SSZ
Message:  New recipe Recipe_Template
Url:      [LINK TO HISTORY ENTRY]

Add Description.json

Every node in a recipe graph is defined by its own sub-directory within the recipe.

  1. Create a new sub-directory called load-profits-node containing a description.json file. This will be an Action Node as defined by type in its description.json file.

Recipe versus Node Descriptions

Recipes contain recipe-level description.json files as well as node-level description.json files.

~/Kitchens/Feature_Sprint_2/Recipe_Template $ mkdir load-profits-node
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cat > load-profits-node/description.json
  1. After executing the above command, paste the following text at your command line:
{
   "type" : "DKNode_Action",
   "description": "Create AWS Redshift schema and table, copy to table example data from AWS S3, execute SQL query against table, and test returned results"
}
  1. Press Enter/Return then Ctrl+D to return to the command line prompt. You have now created the shell of a new recipe graph node.
A view of the **load-profits-node**'s **description.json** file.

A view of the load-profits-node's description.json file.

  1. Use the recipe-status command to confirm the addition of the new node structure.
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk recipe-status
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk rs

Output: recipe-status command

YYYY-MM-DD HH:MM:SS - Getting the status of Recipe 'Recipe_Template' in Kitchen 'Feature_Sprint_2'
    versus directory '~/Kitchens/Feature_Sprint_2/Recipe_Template'
1 files are local only:
    load-profits-node/description.json

1 directories are local only:
    load-profits-node

10 files are unchanged

Add Node Configuration

  1. Configure the load-profits-node by creating a configuration file located within an /actions subdirectory, which will orchestrate the data work to be completed by the node.
  2. Name this file load-data.json.
    Note that this file will use jinja templating to call the variable values from variables.json. The variables set to sensitive credentials reference vault secrets.
~/Kitchens/Feature_Sprint_2/Recipe_Template $ mkdir load-profits-node/actions
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cat > load-profits-node/actions/load-data.json
  1. Paste the following text at your command line.
{
    "name": "load-profits-node",
    "type": "DKDataSource_PostgreSQL",
    "username": "{{redshiftusername}}",
    "password": "{{redshiftpassword}}",
    "hostname": "{{redshifthostname}}",
    "database": "{{redshiftdatabase}}",
    "port": "{{redshiftport}}",
    "keys": {
        "load_data": {
            "target-field": "sql",
            "resource-file": "s3-to-redshift.sql",
            "query-type": "execute_scalar"
        }
    }
}
  1. Press Enter/Return then Ctrl+D to return to the command line prompt.
A view of **load-data.json**, **load-profits-node**'s /actions configuration file.  Note the single **key** and numerous references to variables in **variables.json**.

A view of load-data.json, load-profits-node's /actions configuration file. Note the single key and numerous references to variables in variables.json.

  1. Run the recipe-status command to check your work.
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk recipe-status
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk rs

Output: recipe-status command

YYYY-MM-DD HH:MM:SS - Getting the status of Recipe 'Recipe_Template' in Kitchen 'Feature_Sprint_2'
    versus directory '~/Kitchens/Feature_Sprint_2/Recipe_Template'
2 files are local only:
    load-profits-node/actions/load-data.json
    load-profits-node/description.json

2 directories are local only:
    load-profits-node
    load-profits-node/actions

10 files are unchanged

The load-data.json configuration file for this node also contains a Key. Keys are the steps that do work with a recipe graph node and are processed in the order the are defined. The single key load-data in this example is configured to execute a scalar (query-type) SQL (target-field) query as defined by s3-to-redshift.sql (resource-file), which you will create below and place in Recipe_Template's /resources directory.

Add Resource File(s)

The load-profits-node will leverage a SQL file named s3-to-redshift.sql within the /resources directory that will do the work of copying data from a .txt file located in an AWS S3 bucket into an AWS Redshift database table within a schema it has created.

Note how this file also calls variable references from variables.json.

A view of **s3-to-redshift.sql**, a **/resource** file called to do work by a **key** in **load-profits-node**'s **load-data.json** configuration file.  Note the file's use of parameterization, which calls values from **variables.json**.

A view of s3-to-redshift.sql, a /resource file called to do work by a key in load-profits-node's load-data.json configuration file. Note the file's use of parameterization, which calls values from variables.json.

  1. Run a concatenate command for the s3-to-redshift.sql file
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cat > resources/s3-to-redshift.sql
  1. Paste the following text at your command line.
CREATE schema IF NOT EXISTS {{redshiftschema}} authorization {{redshiftusername}};

SET search_path TO {{redshiftschema}};

DROP TABLE if exists DEMO_SUPERSTORE_SALES_PROFITS;
CREATE TABLE DEMO_SUPERSTORE_SALES_PROFITS
(
  ID       VARCHAR(50),
  Sales    NUMERIC(10,2),
  Profit   NUMERIC(10,2)
);


copy DEMO_SUPERSTORE_SALES_PROFITS
from 's3://{{s3bucketname}}/{{s3bucketpath}}'
credentials 'aws_iam_role=arn:aws:iam::{{awsaccountid}}:role/{{awsrole}}'
ignoreheader 1
TRIMBLANKS delimiter '|';

select count(*) from DEMO_SUPERSTORE_SALES_PROFITS;
  1. Press Enter/Return then Ctrl+D to return to the command line prompt.

AWS S3 Paths

The from statement in the s3-to-redshift.sql file is pointing to a specific AWS S3 path that is composed of the S3 Bucket Name and the S3 Bucket Path, defined in variables.json. The .txt file containing the profits source data is located at this path.

When you compile this file below, you will see the following output.

s3://#{vault://s3_schema/bucket}/superstore/customer/sales-profitability

Sensitive vault secrets are compiled to their vault paths whereas hard-coded values defined in variables.json are explicitly compiled.

A view of the AWS S3 bucket filepath from which **/resource** file **s3-to-redshift.sql** copies data to be inserted into a new AWS Redshift table.

A view of the AWS S3 bucket filepath from which /resource file s3-to-redshift.sql copies data to be inserted into a new AWS Redshift table.

  1. Run the recipe-status command to check your work.
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk recipe-status
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk rs

Output: recipe-status command

YYYY-MM-DD HH:MM:SS - Getting the status of Recipe 'Recipe_Template' in Kitchen 'Feature_Sprint_2'
    versus directory '~/Kitchens/Feature_Sprint_2/Recipe_Template'
3 files are local only:
    load-profits-node/description.json
    load-profits-node/actions/load-data.json
    resources/s3-to-redshift.sql

2 directories are local only:
    load-profits-node
    load-profits-node/actions

10 files are unchanged

Compile Files

  1. You can compile files to confirm variable calls have been configured correctly by using the file-compile command.

Security Assured

Note that when compiled, files with sensitive credentials compile to vault paths rather than surfacing the sensitive secrets directly. DataKitchen's DataOps orchestration provides all of your team members the agility to spin up and spin down infrastructure independently, based on the boundaries set by your technical staff.

~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk file-compile --variation Variation1 --file resources/s3-to-redshift.sql
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk fc -v Variation1 -f resources/s3-to-redshift.sql

Output: file-compile command

YYYY-MM-DD HH:MM:SS - Get the Compiled File of Recipe Recipe_Template.Variation1 in Kitchen Feature_Sprint_2
DKCloudCommand.get_compiled_file succeeded:
CREATE schema IF NOT EXISTS jdoe authorization #{vault://redshift/username};

SET search_path TO jdoe;

DROP TABLE if exists DEMO_SUPERSTORE_SALES_PROFITS CASCADE;
CREATE TABLE DEMO_SUPERSTORE_SALES_PROFITS
(
  ID       VARCHAR(50),
  Sales    NUMERIC(10,2),
  Profit   NUMERIC(10,2)
);


copy DEMO_SUPERSTORE_SALES_PROFITS
from 's3://#{vault://s3_schema/bucket}/superstore/customer/sales-profitability'
credentials 'aws_iam_role=arn:aws:iam::#{vault://aws/account_id}:role/#{vault://aws/role}'
ignoreheader 1
TRIMBLANKS delimiter '|';

select count(*) from DEMO_SUPERSTORE_SALES_PROFITS;

Compiled Database Schema Name

Note that the value of the compiled database schema name will be specific to the value you set for the redshiftschema variable in the variables.json file for the recipe.

Update Variations.json

The recipe's variations.json file stores information about recipe variations, variation overrides, schedules and graph nodes.

For now, you will continue with a single recipe variation, Variation1, used in CLI Quickstart1. In CLI Quickstart3, you will build additional recipe variations to provide runtime flexibility for recipes.

At this time, you will update the graph-setting-list in variations.json to define the new graph node and edge for the node configuration you have added to the recipe. This is the final step in building a new node after you successfully added relevant vault secret references to variables.json and created a recipe graph subdirectory, configuration file, and SQL resources file.

  1. View the current contents of the variations.json file.
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cat variations.json

Output: concatenate command for a recipe's variations.json file

{
    "variation-list": {
        "variation1": {
            "graph-setting": "placeholder-graph"
        }
    }
, 
    "graph-setting-list": {
        "placeholder-graph": [
            [
                "placeholder-node1", 
                "placeholder-node2"            ]        ]
    }
}
  1. Replace the existing content in variations.json with the following text.
    This content adds the new load-profits-node to the recipe graph such that it is positioned as the final node in the series.
{
    "variation-list": {
        "variation1": {
            "graph-setting": "placeholder-graph"
        }
    }
, 
    "graph-setting-list": {
        "placeholder-graph": [
            [
                "placeholder-node1", 
                "placeholder-node2" 
           ] ,
           [
               "placeholder-node2",
               "load-profits-node"
           ]
       ]
    }
}
  1. Run the recipe-status command to check your work.
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk recipe-status
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk rs

Output: recipe-status command

YYYY-MM-DD HH:MM:SS - Getting the status of Recipe 'Recipe_Template' in Kitchen 'Feature_Sprint_2'
    versus directory '~/Kitchens/Feature_Sprint_2/Recipe_Template'
1 files are modified on local:
    variations.json

3 files are local only:
    load-profits-node/description.json
    load-profits-node/actions/load-data.json
    resources/s3-to-redshift.sql

2 directories are local only:
    load-profits-node
    load-profits-node/actions

9 files are unchanged

Recipe Update

While you have accumulated a number of local changes to the Recipe_Template recipe, you have not yet pushed these changes to the remote version of Recipe_Template.

Rather than push each of these new or modified local files piecemeal, you will push all of the updates at once.

  1. Use the recipe-update command to batch update the remote recipe.
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk recipe-update --message "Updating Recipe for addition of load-profits-node"
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk ru -m "Updating Recipe for addition of load-profits-node"

Output: recipe-update command

YYYY-MM-DD HH:MM:SS - Updating all changed files in Recipe (Recipe_Template) in Kitchen(Feature_Sprint_2) with message (Updating Recipe for addition of load-profits-node)
Update results:

New files:
    load-profits-node/actions/load-data.json
    load-profits-node/description.json
    resources/s3-to-redshift.sql
Updated files:
    variations.json
Deleted files:
    None

Issues:

No issues found
  1. Use the recipe-status command to confirm that the local and remote versions of the Recipe_Template recipe are now equivalent.
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk recipe-status
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk rs

Output: recipe-status command

YYYY-MM-DD HH:MM:SS - Getting the status of Recipe 'Recipe_Template' in Kitchen 'Feature_Sprint_2'
    versus directory '~/Kitchens/Feature_Sprint_2/Recipe_Template'
13 files are unchanged

Run Test Order

  1. Run an order of the Recipe_Template recipe per Variation1 using the order-run command.
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk order-run Variation1 --yes
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk or Variation1 -y
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cd ../..
~/Kitchens $ dk or -k Feature_Sprint_2 -r Recipe_Template Variation1 -y

Output: order-run command

YYYY-MM-DD HH:MM:SS - Create an Order:
    Kitchen: Feature_Sprint_2
    Recipe: Recipe_Template
    Variation: Variation1

CREATE ORDER https://cloud.datakitchen.io:443/v2/order/create/Feature_Sprint_2/Recipe_Template/Variation1
Order ID is: d84513d2-e647-11ea-bb0c-ba2e780fb1a4
  1. Use the orderrun-info command with the --summary option to gather information regarding the order run.
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk orderrun-info --summary
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk ori -s
~/Kitchens $ dk ori -k Feature_Sprint_2 -s

Output: orderrun-info command

YYYY-MM-DD HH:MM:SS - Display Order-Run details from kitchen Feature_Sprint_2

ORDER RUN SUMMARY

Order ID:         d84513d2-e647-11ea-bb0c-ba2e780fb1a4
Order Run ID:     db199af6-e647-11ea-99a9-9a5de8cae983
Status:           COMPLETED_SERVING
Kitchen:          Feature_Sprint_2
Recipe:           Recipe_Template
Variation:        Variation1
Start time:       YYYY-MM-DD HH:MM:SS
Run duration:     0:00:4 (H:M:S)
A view from [DataKitchen's DataOps web app](https://cloud.datakitchen.io) confirming the order run completed successfully.  The graph visualization shows successful processing of all three nodes.

A view from DataKitchen's DataOps web app confirming the order run completed successfully. The graph visualization shows successful processing of all three nodes.

Implement Node Test

How can you confirm that the data work performed by load-profits-node is as expected? In other words, how can you gain confidence that it has delivered quality output?

Confirm Data in Database

One option is to connect to the AWS Redshift database directly and look for any changes from before and after you ran the order run for the Recipe_Template recipe.

Connecting to AWS Redshift Databases

Thorough documentation is available for connecting to AWS Redshift. Many database query tools are available for this task, including SQL Workbench, which is open source.

DataKitchen users without their own AWS account will not be able to connect to the Redshift database directly as the relevant AWS credentials are obscured and hidden as vault secrets.

The AWS Redshift database view associated with this guide will not contain a quickstart2 schema, as defined via variables.json, until after you run the order depicted above. After the order run completes, the database shows the quickstart2 schema with a populated table named demo_superstore_sales_profits. This is the table generated by Recipe_Template's load-profits-node, specifically by the /resources/s3-to-redshift.sql file.

A view of the AWS Redshift table **demo_superstore_sales_profits** under the **quickstart2** schema.  This is the data that was loaded from the AWS S3 bucket.  This view is from SQL Workbench, an open-source database query tool.

A view of the AWS Redshift table demo_superstore_sales_profits under the quickstart2 schema. This is the data that was loaded from the AWS S3 bucket. This view is from SQL Workbench, an open-source database query tool.

This confirms that the Recipe_Template recipe has processed as expected. Still, this is not an efficient process to confirm your recipe has created the AWS Redshift table as expected. What is more, it does not necessarily give you confidence in the quality of the data that has been populated to the AWS Redshift table.

Thankfully, there is a better way -- implementing automated tests for each node in your recipe graph via DataKitchen's DataOps platform.

Add an Automated Test

Tests are configured at the node-level, specific to each key defined by a recipe node's configuration file. In the Recipe_Template recipe, /actions/load-data.json is the configuration file where you will implement a test.

  1. View the file as it exists before an automated test is implemented.
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cat load-profits-data/actions/load-data.json

Output: concatenate command for a recipe's load-data.json file

{
    "name": "load-profits-node",
    "type": "DKDataSource_PostgreSQL",
    "username": "{{redshiftusername}}",
    "password": "{{redshiftpassword}}",
    "hostname": "{{redshifthostname}}",
    "database": "{{redshiftdatabase}}",
    "port": "{{redshiftport}}",
    "keys": {
        "load_data": {
            "target-field": "sql",
            "resource-file": "s3-to-redshift.sql",
            "query-type": "execute_scalar"
        }
    }
}
  1. Recall the /resources/s3-to-redshift.sql file you created.
    Specifically, note the final line of the query, which generates a count of all row in the table the file has created. This return of this row count is the final output of the query. You will leverage this row count to implement a quality assurance test for our node.
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cat resources/s3-to-redshift.sql

Output: concatenate command for a recipe's s3-to-redshift.sql file

CREATE schema IF NOT EXISTS {{redshiftschema}} authorization {{redshiftusername}};

SET search_path TO {{redshiftschema}};

DROP TABLE if exists DEMO_SUPERSTORE_SALES_PROFITS CASCADE;
CREATE TABLE DEMO_SUPERSTORE_SALES_PROFITS
(
  ID       VARCHAR(50),
  Sales    NUMERIC(10,2),
  Profit   NUMERIC(10,2)
);


copy DEMO_SUPERSTORE_SALES_PROFITS
from 's3://{{s3bucketname}}/{{s3bucketpath}}'
credentials 'aws_iam_role=arn:aws:iam::{{awsaccountid}}:role/{{awsrole}}'
ignoreheader 1
TRIMBLANKS delimiter '|';

select count(*) from DEMO_SUPERSTORE_SALES_PROFITS;
  1. Replace the contents of your Recipe_Template's load-profits-node/actions/load-data.json file with the text below.

The test-load-profits-node test you are implementing will warn you if the row count generated by the /resources/s3-to-redshift.sql is less than 1589, the row count we noted previously using SQL Workbench.

This is a simple test that confirms that the data you copied from the AWS S3 Bucket has been successfully loaded into an AWS Redshift database.

{
    "name": "load-profits-node",
    "type": "DKDataSource_PostgreSQL",
    "username": "{{redshiftusername}}",
    "password": "{{redshiftpassword}}",
    "hostname": "{{redshifthostname}}",
    "database": "{{redshiftdatabase}}",
    "port": "{{redshiftport}}",
    "keys": {
        "load_data": {
            "target-field": "sql",
            "resource-file": "s3-to-redshift.sql",
            "query-type": "execute_scalar"
        }
    },
    "tests": {
        "test-load-data-key": {
            "test-variable": "load_data",
            "type": "test-contents-as-integer",
            "applies-to-keys": ["load_data"],
            "action": "warning",
            "keep-history": "True",
            "test-logic": {
                "test-variable": "load_data",
                "test-compare": "greater-than-equal-to",
                "test-metric": "1589"
            }
        }
    }
}
**load-data.json** file before and after implementation of an automated test based on the output of the SQL query contained in **/resources/s3-to-redshift.sql**.

load-data.json file before and after implementation of an automated test based on the output of the SQL query contained in /resources/s3-to-redshift.sql.

Automated Tests

This example implements a simple row count automated test, which surprisingly goes a long way towards instilling confidence in your data analytic pipeline. The automated tests you implement can be vastly more complex and granular, specific to the data you work with and the structure of your recipes.

  1. Use the recipe-status command to confirm the local change.
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk recipe-status
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk rs

Output: recipe-status command

YYYY-MM-DD HH:MM:SS - Getting the status of Recipe 'Recipe_Template' in Kitchen 'Feature_Sprint_2'
    versus directory '~/Kitchens/Feature_Sprint_2/Recipe_Template'
1 files are modified on local:
    load-profits-node/actions/load-data.json

12 files are unchanged
  1. Use the file-update command to push the local change to /actions/load-data.json to remote.
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk file-update --message "Add test to load-profits-node" load-profits-node/actions/load-data.json
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk fu -m "Add test to load-profits-node" load-profits-node/actions/load-data.json
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cd ../..
~/Kitchens $ dk fu -k Feature_Sprint_2 -r Recipe_Template -m "Add test to load-profits-node" load-profits-node/actions/load-data.json

Output: file-update command

YYYY-MM-DD HH:MM:SS - Updating File(s) (('load-profits-node/actions/load-data.json',)) in Recipe (Recipe_Template) in Kitchen(Feature_Sprint_2) with message (Add test to load_profits_node)
DKCloudCommand.update_file for load-profits-node/actions/load-data.json succeeded

Run a Test Order

  1. Run another order of your recipe with the test in place to confirm the work.
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk order-run Variation1 --yes
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk or Variation1 -y
~/Kitchens $ dk or -k Feature_Sprint_2 -r Recipe_Template Variation1 -y

Output: order-run command

YYYY-MM-DD HH:MM:SS - Create an Order:
    Kitchen: Feature_Sprint_2
    Recipe: Recipe_Template
    Variation: Variation1

CREATE ORDER https://cloud.datakitchen.io:443/v2/order/create/Feature_Sprint_2/Recipe_Template/Variation1
Order ID is: 4cd4a90e-e64a-11ea-b70e-621639189e36
  1. Use the orderrun-info command with the --summary and --test options to gain confidence that the Recipe_Template recipe processed as expected—no need to connect to AWS Redshift and check the row count to confirm.
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk orderrun-info --summary --test
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk ori -s -q
~/Kitchens $ dk ori -k Feature_Sprint_2 -s -q

Output: orderrun-info command

YYYY-MM-DD HH:MM:SS - Display Order-Run details from kitchen Feature_Sprint_2

ORDER RUN SUMMARY

Order ID:    4cd4a90e-e64a-11ea-b70e-621639189e36
Order Run ID:    4fa8f446-e64a-11ea-af19-46700b0bc3f0
Status:        COMPLETED_SERVING
Kitchen:    Feature_Sprint_2
Recipe:        Recipe_Template
Variation:    Variation1
Start time:    YYYY-MM-DD HH:MM:SS
Run duration:    0:00:08 (H:M:S)

TEST RESULTS

Tests: Failed
    No Tests Failed

Tests: Warning
    No Tests Gave Warning Messages

Tests: Log
    No Tests Gave Log Messages

Tests: Passed
   Step (load-profits-node)
      1. test-load-data-key (1589 greater-than-equal-to 1589)
Email confirming the order run for **Recipe_Template**'s **Variation1** completed successfully.  Note the inclusion of feedback for the automated test you implemented.

Email confirming the order run for Recipe_Template's Variation1 completed successfully. Note the inclusion of feedback for the automated test you implemented.

List Orders

  1. Display a list of orders that have taken place in a kitchen via the order-list command.
    Recall that you have run two orders in the Feature_Sprint_2 kitchen; one before you implemented an automated test and one after implementation.
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cd ..
~/Kitchens/Feature_Sprint_2 $ dk order-list
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cd ..
~/Kitchens/Feature_Sprint_2 $ dk ol
~/Kitchens $ dk ol -k Feature_Sprint_2

Output: order-list command

YYYY-MM-DD HH:MM:SS - Get Order information for Kitchen Feature_Sprint_2

ORDER SUMMARY (order ID: 4cd4a90e-e64a-11ea-b70e-621639189e36)
Kitchen:        Feature_Sprint_2
Recipe:         Recipe_Template
Variation:      Variation1
Schedule:       now
Status:         COMPLETED_ORDER

  1.  ORDER RUN (OrderRun ID: 4fa8f446-e64a-11ea-af19-46700b0bc3f0)
        OrderRun Status OrderRun Completed
        Start time:    YYYY-MM-DD HH:MM:SS
    End time:    YYYY-MM-DD HH:MM:SS
        Duration:       0:00:08 (H:M:S)

ORDER SUMMARY (order ID: d84513d2-e647-11ea-bb0c-ba2e780fb1a4)
Kitchen:    Feature_Sprint_2
Recipe:        Recipe_Template
Variation:    Variation1
Schedule:    now
Status:        COMPLETED_ORDER

  1.  ORDER RUN    (OrderRun ID: db199af6-e647-11ea-99a9-9a5de8cae983)
    OrderRun Status OrderRun Completed
    Start time:    YYYY-MM-DD HH:MM:SS
    End time:    YYYY-MM-DD HH:MM:SS
    Duration:    0:00:04 (H:M:S)

Merge Parent to Child

Begin the process of merging your work on the Recipe_Template recipe within the Feature_Sprint_2 kitchen to its parent kitchen, Dev_Sprint_1. As always, first merge from parent to child to account for any changes that have occurred in the parent since you branched to the child kitchen.

Kitchen-Merge-Preview

  1. Use the kitchen-merge-preview command to preview the merge.
    Remember to use the --clean_previous_run option to clear any temporary local merge files that may exist as a result of a previously-aborted kitchen merge.
~/Kitchens/Feature_Sprint_2 $ cd ../Dev_Sprint_1
~/Kitchens/Dev_Sprint_1 $ dk kitchen-merge-preview --target_kitchen Feature_Sprint_2 --clean_previous_run
~/Kitchens/Feature_Sprint_2 $ cd ../Dev_Sprint_1
~/Kitchens/Dev_Sprint_1 $ dk kmp -tk Feature_Sprint_2 -cpr
~/Kitchens $ dk kmp -sk Dev_Sprint_1 -tk Feature_Sprint_2 -cpr

Output: kitchen-merge-preview command

YYYY-MM-DD HH:MM:SS - Previewing merge Kitchen Dev_Sprint_1 into Kitchen Feature_Sprint_2
--- Merge Preview Results (only changed files are being displayed): ---
Nothing to merge.

Kitchen merge preview done.

As expected, no changes have occurred in the parent kitchen since the creation of the child.

Merge and Test

Since the parent-to-child kitchen merge preview indicated that there was nothing to merge, you do not need to complete a parent-to-child kitchen merge. And, you do not need to run an order in Feature_Sprint_2 with its associated tests.

Merge Child to Parent

Merge the child kitchen, Feature_Sprint_2, into its parent kitchen, Dev_Sprint_1.

Preview and Merge Kitchens

  1. Use the kitchen-merge-preview command to preview the merge from Feature_Sprint_2 to Dev_Sprint_1.
    Clear any temporary local merge files from a previously-aborted kitchen merge using the --clean_previous_run option.
~/Kitchens/Dev_Sprint_1 $ cd ../Feature_Sprint_2
~/Kitchens/Feature_Sprint_2 $ dk kitchen-merge-preview --target_kitchen Dev_Sprint_1 --clean_previous_run
~/Kitchens/Dev_Sprint_1 $ cd ../Feature_Sprint_2
~/Kitchens/Feature_Sprint_2 $ dk kkmp -tk Dev_Sprint_1 -cpr
~/Kitchens $ dk kkmp -sk Feature_Sprint_2 -tk Dev_Sprint_1 -cpr

Output: kitchen-merge-preview command

YYYY-MM-DD HH:MM:SS - Previewing merge Kitchen Feature_Sprint_2 into Kitchen Dev_Sprint_1
--- Merge Preview Results (only changed files are being displayed): ---
      ok        Recipe_Template/load_profits_node/description.json
      ok        Recipe_Template/load_profits_node/actions/load_data.json
      ok        Recipe_Template/resources/s3_to_redshift.sql
      ok        Recipe_Template/variables.json
      ok        Recipe_Template/variations.json
-----------------------------------------------------------------------

Kitchen merge preview done.

Note that the command response summarizes all of the changes you have built into the Recipe_Template recipe with the Feature_Sprint_2 kitchen.

  1. Use the kitchen-merge command to merge these changes into the remote copy of the Recipe_Template recipe in the Dev_Sprint_1 kitchen.
~/Kitchens/Feature_Sprint_2 $ dk kitchen-merge --target_kitchen Dev_Sprint_1 --yes
~/Kitchens/Feature_Sprint_2 $ dk km -tk Dev_Sprint_1 -y
~/Kitchens $ dk km -sk Feature_Sprint_2 -tk Dev_Sprint_1 -y

Output: kitchen-merge command

YYYY-MM-DD HH:MM:SS - Merging Kitchen Feature_Sprint_2 into Kitchen Dev_Sprint_1
looking for manually merged files in temporary directory {USER_HOME}.dk/merges/Feature_Sprint_2_to_Dev_Sprint_1 
Calling Merge ...
Merge done. You can check your changes in target kitchen and delete the source kitchen.

Updating Recipes in the local version of Target Kitchen dev_sprint_1 to receive merged changes applied to remote...
/{USER_HOME}/Kitchens/.../dev_sprint_1 kitchen has been updated
  1. Run a test order on the parent kitchen following the merge using the order-run command.
~/Kitchens/Feature_Sprint_2 $ cd ../Dev_Sprint_1
~/Kitchens/Dev_Sprint_1 $ dk order-run --recipe Recipe_Template Variation1 --yes
~/Kitchens/Feature_Sprint_2 $ cd ../Dev_Sprint_1
~/Kitchens/Dev_Sprint_1 $ dk or -r Recipe_Template Variation1 -y
~/Kitchens $ dk or -k Dev_Sprint_1 -r Recipe_Template Variation1 -y

Output: order-run command

YYYY-MM-DD HH:MM:SS - Create an Order:
    Kitchen: Dev_Sprint_1
    Recipe: Recipe_Template
    Variation: Variation1

CREATE ORDER https://cloud.datakitchen.io:443/v2/order/create/Dev_Sprint_1/Recipe_Template/Variation1
Order ID is: 68666d08-e64d-11ea-983f-ba2e780fb1a4
  1. View information about the order you ran in the Dev_Sprint_1 kitchen via the orderrun-info command.
~/Kitchens/Dev_Sprint_1 $ dk orderrun-info --summary --test
~/Kitchens/Dev_Sprint_1 $ dk ori -s -q
~/Kitchens $ dk ori -k Dev_Sprint_1 -s -q

Output: orderrun-info command

YYYY-MM-DD HH:MM:SS - Display Order-Run details from kitchen Dev_Sprint_1

ORDER RUN SUMMARY

Order ID:          68666d08-e64d-11ea-983f-ba2e780fb1a4
Order Run ID:      6b3d9c40-e64d-11ea-b128-3a1b66adf6af
Status:            COMPLETED_SERVING
Kitchen:           Dev_Sprint_1
Recipe:            Recipe_Template
Variation:         Variation1
Start time:        YYYY-MM-DD HH:MM:SS
Run duration:      0:00:04 (H:M:S)

TEST RESULTS

Tests: Failed
    No Tests Failed

Tests: Warning
    No Tests Gave Warning Messages

Tests: Log
    No Tests Gave Log Messages

Tests: Passed
   Step (load-profits-node)
      1. test-load-data-key (1589 greater-than-equal-to 1589)

Teardown Child Kitchen

Since all development work has been merged into the Dev_Sprint_1 parent kitchen, you can delete the Feature_Sprint_2 kitchen.

  1. Use the kitchen-delete command to teardown Feature_Sprint_2's computing resources and infrastructure.
~/Kitchens/Dev_Sprint_1 $ cd ..
~/Kitchens $ dk kitchen-delete Feature_Sprint_2 --yes
~/Kitchens/Dev_Sprint_1 $ cd ..
~/Kitchens $ dk kd Feature_Sprint_2 -y

Output: kitchen-delete command

No child kitchens found.
YYYY-MM-DD HH:MM:SS - Deleting remote copy of kitchen Feature_Sprint_2. Local files will not change.
Kitchen Feature_Sprint_2 has been deleted

Wrapping Up

To recap, this CLI Quickstart2 provided an overview of DataOps orchestration via DKCloudCommand, DataKitchen's command line tool. Specific topics covered include:

  • The Vault and its Secure Storage of Secret Credentials
  • Configuring Recipe Graph Nodes
  • Viewing a File's Change History
  • Compiling Files to Review the Application of Parameter Values
  • Cooking an Order-Run of a Recipe Variation with Expanded Options
  • Implementing an Automated Tests
  • Understanding Test Types and Reviewing Their Feedback
  • Batch Updating Local Recipe File Changes to Remote
  • Listing the Orders that Exist in a Kitchen

You have learned to build recipe nodes that demonstrate some of the real heavy-lifting required for a robust data analytic pipeline. What's more, you have reviewed how to implement automated tests in a recipe graph node whose feedback give you confidence in the quality of the data flowing through your data analytic pipeline, and consequently, the insights you deliver to your stakeholders.

In the next quickstart, you will dive into building additional node types for the existing tools in your toolchain, creating recipe variations, and implementing parameterization for runtime flexibility of your order runs.

Synopsis

REVIEW YOUR ENVIRONMENT
~/Kitchens $ dk kitchen-list
~/Kitchens $ cd Dev_Sprint_1
~/Kitchens/Dev_Sprint_1 $ dk recipe-list

CREATE CHILD KITCHEN
~/Kitchens/Dev_Sprint_1 $ cd ..
~/Kitchens $ dk kitchen-create --parent Dev_Sprint_1 Feature_Sprint_2
~/Kitchens $ dk kitchen-list

GET KITCHEN & RECIPES
~/Kitchens $ dk kitchen-get Feature_Sprint_2 --all

VAULT SECRETS
~/Kitchens $ dk secret-list --recursive
~/Kitchens $ dk secret-write service-name/test-credential=test-value --yes
~/Kitchens $ dk secret-exists service-name/test-credential
~/Kitchens $ dk secret-delete service-name/test-credential --yes

ADD NODE: UPDATE VARIABLES
~/Kitchens $ cd Feature_Sprint_2/Recipe_Template
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cat variables.json
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk secret-list
[REPLACE VARIABLES.JSON CONTENT WITH SUPPLIED TEXT & UPDATE ALERTS OBJECT]
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk recipe-status
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk file-update --message "Change message" variables.json

FILE HISTORY
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk file-history variables.json

ADD NODE: CREATE NODE DIRECTORY & DESCRIPTION
~/Kitchens/Feature_Sprint_2/Recipe_Template $ mkdir load-profits-node
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cat > load-profits-node/description.json
[PASTE SUPPLIED TEXT AT COMMAND LINE, <ENTER>, <CTRL+D>]
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk recipe-status

ADD NODE: CREATE NODE CONFIGURATION
~/Kitchens/Feature_Sprint_2/Recipe_Template $ mkdir load-profits-node/actions
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cat > load-profits-node/actions/load-data.json
[PASTE SUPPLIED TEXT AT COMMAND LINE, <ENTER>, <CTRL+D>]
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk recipe-status

ADD NODE: CREATE SQL RESOURCE FILE
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cat > resources/s3-to-redshift.sql
[PASTE SUPPLIED TEXT AT COMMAND LINE, <ENTER>, <CTRL+D>]
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk recipe-status

COMPILE FILE
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk file-compile --variation Variation1 --file resources/s3-to-redshift.sql

ADD NODE: UPDATE RECIPE GRAPH
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cat variations.json
[REPLACE VARIATIONS.JSON CONTENT WITH SUPPLIED TEXT]
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk recipe-status

RECIPE UPDATE
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk recipe-update --message "Change message"
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk recipe-status

RUN TEST ORDER
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk order-run Variation1 --yes
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk orderrun-info --summary

IMPLEMENT NODE TEST & RUN ORDER
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cat load-profits-data/actions/load-data.json
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cat resources/s3-to-redshift.sql
[REPLACE LOAD-DATA.JSON CONTENT WITH SUPPLIED TEXT]
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk recipe-status
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk file-update --message "Change message" load-profits-data/actions/load-data.json
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk order-run Variation1 --yes
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk orderrun-info --summary --test
[CONFIRM ORDER-RUN & TESTING FEEDBACK VIA EMAIL ALERTS]

LIST ORDERS
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cd ..
~/Kitchens/Feature_Sprint_2 $ dk order-list

MERGE: PARENT TO CHILD
~/Kitchens/Feature_Sprint_2 $ cd ../Dev_Sprint_1
~/Kitchens/Dev_Sprint_1 $ dk kitchen-merge-preview --target_kitchen Feature_Sprint_2 --clean_previous_run

MERGE: CHILD TO PARENT
~/Kitchens/Dev_Sprint_1 $ cd ../Feature_Sprint_2
~/Kitchens/Feature_Sprint_2 $ dk kitchen-merge-preview --target_kitchen Dev_Sprint_1 --clean_previous_run
~/Kitchens/Feature_Sprint_2 $ dk kitchen-merge --target_kitchen Dev_Sprint_1 --yes

PARENT TEST ORDER
~/Kitchens/Feature_Sprint_2 $ cd ../Dev_Sprint_1
~/Kitchens/Dev_Sprint_1 $ dk order-run --recipe Recipe_Template Variation1 --yes
~/Kitchens/Dev_Sprint_1 $ dk orderrun-info --summary --test

TEARDOWN CHILD KITCHEN
~/Kitchens/Dev_Sprint_1 $ cd ..
~/Kitchens $ dk kitchen-delete Feature_Sprint_2 --yes
REVIEW YOUR ENVIRONMENT
~/Kitchens $ dk kl
~/Kitchens $ cd Dev_Sprint_1
~/Kitchens/Dev_Sprint_1 $ dk rl

CREATE CHILD KITCHEN
~/Kitchens/Dev_Sprint_1 $ cd ..
~/Kitchens $ dk kc -p Dev_Sprint_1 Feature_Sprint_2
~/Kitchens $ dk kl

GET KITCHEN & RECIPES
~/Kitchens $ dk kg Feature_Sprint_2 -a

VAULT SECRETS
~/Kitchens $ dk sl -rc
~/Kitchens $ dk sw service-name/test-credential=test-value -y
~/Kitchens $ dk se service-name/test-credential
~/Kitchens $ dk sd service-name/test-credential -y

ADD NODE: UPDATE VARIABLES
~/Kitchens $ cd Feature_Sprint_2/Recipe_Template
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cat variables.json
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk sl
[REPLACE VARIABLES.JSON CONTENT WITH SUPPLIED TEXT & UPDATE ALERTS OBJECT]
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk rs
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk fu -m "Change message" variables.json

FILE HISTORY
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk fh variables.json

ADD NODE: CREATE NODE DIRECTORY & DESCRIPTION
~/Kitchens/Feature_Sprint_2/Recipe_Template $ mkdir load-profits-node
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cat > load-profits-node/description.json
[PASTE SUPPLIED TEXT AT COMMAND LINE, <ENTER>, <CTRL+D>]
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk rs

ADD NODE: CREATE NODE CONFIGURATION
~/Kitchens/Feature_Sprint_2/Recipe_Template $ mkdir load-profits-node/actions
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cat > load-profits-node/actions/load-data.json
[PASTE SUPPLIED TEXT AT COMMAND LINE, <ENTER>, <CTRL+D>]
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk rs

ADD NODE: CREATE SQL RESOURCE FILE
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cat > resources/s3-to-redshift.sql
[PASTE SUPPLIED TEXT AT COMMAND LINE, <ENTER>, <CTRL+D>]
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk rs

COMPILE FILE
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk fc -v Variation1 -f resources/s3-to-redshift.sql

ADD NODE: UPDATE RECIPE GRAPH
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cat variations.json
[REPLACE VARIATIONS.JSON CONTENT WITH SUPPLIED TEXT]
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk rs

RECIPE UPDATE
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk ru -m "Change message"
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk rs

RUN TEST ORDER
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk or Variation1 -y
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk ori -s

IMPLEMENT NODE TEST & RUN ORDER
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cat load-profits-node/actions/load-data.json
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cat resources/s3-to-redshift.sql
[REPLACE LOAD-DATA.JSON CONTENT WITH SUPPLIED TEXT]
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk rs
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk fu -m "Change message" load-profits-data/actions/load-data.json
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk or Variation1 -y
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk ori -s -q
[CONFIRM ORDER-RUN & TESTING FEEDBACK VIA EMAIL ALERTS]

LIST ORDERS
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cd ..
~/Kitchens/Feature_Sprint_2 $ dk ol

MERGE: PARENT TO CHILD
~/Kitchens/Feature_Sprint_2 $ cd ../Dev_Sprint_1
~/Kitchens/Dev_Sprint_1 $ dk kmp -tk Feature_Sprint_2 -cpr

MERGE: CHILD TO PARENT
~/Kitchens/Dev_Sprint_1 $ cd ../Feature_Sprint_2
~/Kitchens/Feature_Sprint_2 $ dk km -tk Dev_Sprint_1 -cpr
~/Kitchens/Feature_Sprint_2 $ dk km -tk Dev_Sprint_1 -y

PARENT TEST ORDER
~/Kitchens/Feature_Sprint_2 $ cd ../Dev_Sprint_1
~/Kitchens/Dev_Sprint_1 $ dk or -r Recipe-Template Variation1 -y
~/Kitchens/Dev_Sprint_1 $ dk ori -s -q

TEARDOWN CHILD KITCHEN
~/Kitchens/Dev_Sprint_1 $ cd ..
~/Kitchens $ dk kd Feature_Sprint_2 -y
REVIEW YOUR ENVIRONMENT
~/Kitchens $ dk kl
~/Kitchens $ dk rl -k Dev_Sprint_1

CREATE CHILD KITCHEN
~/Kitchens $ dk kc -p Dev_Sprint_1 Feature_Sprint_2
~/Kitchens $ dk kl

GET KITCHEN & RECIPES
~/Kitchens $ dk kg Feature_Sprint_2 -a

VAULT SECRETS
~/Kitchens $ dk sl -rc
~/Kitchens $ dk sw service-name/test-credential=test-value -y
~/Kitchens $ dk se service-name/test-credential
~/Kitchens $ dk sd service-name/test-credential -y

ADD NODE: UPDATE VARIABLES
~/Kitchens $ cat Feature_Sprint_2/Recipe_Template/variables.json
~/Kitchens $ dk sl
[REPLACE VARIABLES.JSON CONTENT WITH SUPPLIED TEXT & UPDATE ALERTS OBJECT]
~/Kitchens $ cd Feature_Sprint_2/Recipe_Template
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk rs
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cd ../..
~/Kitchens $ dk file-update -k Feature_Sprint_2 -r Recipe_Template -m "Change message" variables.json
~/Kitchens $ cd Feature_Sprint_2/Recipe_Template

FILE HISTORY
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk fh Feature_Sprint_2/Recipe-Template/variables.json

ADD NODE: CREATE NODE DIRECTORY & DESCRIPTION
~/Kitchens/Feature_Sprint_2/Recipe_Template $ mkdir load-profits-node
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cat > load-profits-node/description.json
[PASTE SUPPLIED TEXT AT COMMAND LINE, <ENTER>, <CTRL+D>]
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk rs

ADD NODE: CREATE NODE CONFIGURATION
~/Kitchens/Feature_Sprint_2/Recipe_Template $ $ mkdir load-profits-node/actions
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cat > load-profits-node/actions/load-data.json
[PASTE SUPPLIED TEXT AT COMMAND LINE, <ENTER>, <CTRL+D>]
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk rs 

ADD NODE: CREATE SQL RESOURCE FILE
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cat > resources/s3-to-redshift.sql
[PASTE SUPPLIED TEXT AT COMMAND LINE, <ENTER>, <CTRL+D>]
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk rs

COMPILE FILE
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk fc -v Variation1 -f resources/s3-to-redshift.sql

ADD NODE: UPDATE RECIPE GRAPH
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cat variations.json
[REPLACE VARIATIONS.JSON CONTENT WITH SUPPLIED TEXT]
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk rs

RECIPE UPDATE
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk ru -m "Change message"
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk rs

RUN TEST ORDER
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cd ../..
~/Kitchens $ dk or -k Feature_Sprint_2 -r Recipe_Template Variation1 -y
~/Kitchens $ dk ori -k Feature_Sprint_2 -s

IMPLEMENT NODE TEST & RUN ORDER
~/Kitchens $ cat Feature_Sprint_2/Recipe_Template/load-profits-node/actions/load-data.json
~/Kitchens $ cat Feature_Sprint_2/Recipe_Template/resources/s3-to-redshift.sql
[REPLACE LOAD-DATA.JSON CONTENT WITH SUPPLIED TEXT]
~/Kitchens $ cd Feature_Sprint_2/Recipe_Template
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk rs
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cd ../..
~/Kitchens $ dk fu -k Feature_Sprint_2 -r Recipe_Template -m "Change message" load-profits-data/actions/load-data.json
~/Kitchens $ dk or -k Feature_Sprint_2 -r Recipe_Template Variation1 -y
~/Kitchens $ dk ori -k Feature_Sprint_2 -s -q
[CONFIRM ORDER-RUN & TESTING FEEDBACK VIA EMAIL ALERTS]

LIST ORDERS
~/Kitchens $ dk ol -k Feature_Sprint_2

MERGE: PARENT TO CHILD
~/Kitchens $ dk kmp -sk Dev_Sprint_1 -tk Feature_Sprint_2 -cpr

MERGE: CHILD TO PARENT
~/Kitchens $ dk kmp -sk Feature_Sprint_2 -tk Dev_Sprint_1 -cpr
~/Kitchens $ dk km -sk Feature_Sprint_2 -tk Dev_Sprint_1 -y

PARENT TEST ORDER
~/Kitchens $ dk or -k Dev_Sprint_1 -r Recipe-Template Variation1 -y
~/Kitchens $ dk ori -k Dev_Sprint_1 -s -q

TEARDOWN CHILD KITCHEN
~/Kitchens $ dk kd Feature_Sprint_2 -y
#NOTE: These steps have not been verified!!!
REVIEW YOUR ENVIRONMENT
~/Kitchens $ dk kl
~/Kitchens $ dk rl -k Dev_Sprint_1

CREATE CHILD KITCHEN
~/Kitchens $ dk kc -p Dev_Sprint_1 Feature_Sprint_2
~/Kitchens $ dk kl

GET KITCHEN & RECIPES
~/Kitchens $ dk kg Feature_Sprint_2 -a

FILE HISTORY
~/Kitchens $ cd Feature_Sprint_2/Recipe_Template
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk fh variables.json

DELETE QS1 RECIPE & CREATE QS2 RECIPE FROM TEMPLATE
[SKIPS STEPS TO BUILD FROM SCRATCH QS2 RECIPE_TEMPLATE VIA QS1 RECIPE_TEMPLATE]
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cd ..
~/Kitchens/Feature_Sprint_2 $ dk rd Recipe_Template -y
~/Kitchens/Feature_Sprint_2 $ rm -r Recipe_Template
~/Kitchens/Feature_Sprint_2 $ dk re -tm qs2 Recipe_Template
~/Kitchens/Feature_Sprint_2 $ dk rg Recipe_Template

MAKE LOCAL RECIPE CHANGES & UPDATE TO REMOTE
~/Kitchens/Feature_Sprint_2 $ cd Recipe_Template
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk rs
[UPDATE ALERTS OBJECT IN VARIABLES.JSON]
[UPDATE REDSHIFT SCHEMA VALUE IN VARIABLES.JSON, e.g. "jdoe"]
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk ru -m "Push local changes to remote"
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk rs

VAULT SECRETS
~/Kitchens/Feature_Sprint_2 $ dk sl

COMPILE FILE
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk fc -v Variation1 -f resources/s3-to-redshift.sql

IMPLEMENT NODE TEST & RUN ORDER
[REPLACE LOAD-DATA.JSON CONTENT WITH SUPPLIED TEXT]
[https://datakitchen.readme.iodocs/quickstart2#section-updated-contents-for-load-data-json]
~/Kitchens/Feature_Sprint_2/Recipe_Template $ dk rs
~/Kitchens/Feature_Sprint_2/Recipe_Template $ cd ../..
~/Kitchens $ dk fu -k Feature_Sprint_2 -r Recipe_Template -m "Implement automated test" action-node/actions/load-data.json
~/Kitchens $ dk or -k Feature_Sprint_2 -r Recipe_Template Variation1 -y
~/Kitchens $ dk ori -k Feature_Sprint_2 -s -q
[CONFIRM ORDER-RUN & TESTING FEEDBACK VIA EMAIL ALERTS]

LIST ORDERS
~/Kitchens $ dk ol -k Feature_Sprint_2

MERGE: PARENT TO CHILD
~/Kitchens $ dk kmp -sk Dev_Sprint_1 -tk Feature_Sprint_2 -cpr

MERGE: CHILD TO PARENT
~/Kitchens $ dk kmp -sk Feature_Sprint_2 -tk Dev_Sprint_1 -cpr
~/Kitchens $ dk km -sk Feature_Sprint_2 -tk Dev_Sprint_1 -y

PARENT TEST ORDER
~/Kitchens $ dk or -k Dev_Sprint_1 -r Recipe-Template Variation1 -y
~/Kitchens $ dk ori -k Dev_Sprint_1 -s

TEAR DOWN CHILD KITCHEN
~/Kitchens $ dk kd Feature_Sprint_2 -y

Updated 9 days ago


CLI Quickstart2


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.