DataKitchen DataOps Documention

Install & Configure

Install

User Credentials

Your account credentials were provided to you via email. If you experience any issues logging in please contact DataKitchen by emailing productsupport@datakitchen.io.

Install with pip3

pip3 install DKCloudCommand

MacOS-Specific Install Step

If you are running the DKCloudCommand (DKCC) on MacOS, you may start to encounter this error.

Error: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: 
self signed certificate in certificate chain (_ssl.c:1076)>

This error is related to a MacOS-specific change that was made in Python 3.6. See https://www.python.org/downloads/release/python-360/ for more information.

To prevent this error, you must run the following command. Paste the following line into the Mac Terminal app (adjust the Python version as needed), and press Enter/Return. This code will run a pip command that validates and updates certificates.

/Applications/Python\ 3.6/Install\ Certificates.command

Upgrade

Confirm the version that is currently installed.

dk --version

You will receive a warning message should you input a command using a deprecated version of DKCC. To upgrade to the most recent version:

pip3 install DKCloudCommand --upgrade

MacOS User Step

Run the command detailed above in MacOS-Specific Install Step after you complete your DKCloudCommand upgrade.

Configure

Setup Local Merge and File Diff Tool

Before configuring DKCloudCommand itself, set up a local tool that will be used to visualize merge conflicts and differences between recipe files.

Version Control

File Diff operations compare recipes local vs remote file copies within the same kitchen. File Merge operations compare remote recipe files across kitchens being merged.

Configuring Pycharm
The following steps help you to configure Pycharm Community Edition, though DKCloudCommand works with any tool that supports merging and viewing files differences via the command line.

  1. Ensure that your copy of Pycharm CE has a command line launcher in place. Find Tools in the menu bar and select Create Command-line Launcher.

Pycharm Command-Line Laucher

The option to create a command-line launcher is only available for Pycharm running on Linux and Mac operating systems. On Windows, it is recommended to add the path to the PyCharm executable to the Path environment variable. Alternatively, Windows users can configure a different tool like Meld.

  1. When prompted, save the launcher script with the default name and local path provided. This full path will later be used when configuring DKCloudCommand.

Note that you can call the full path of Pycharm CE's launcher script via the following shell command.

which charm
/usr/local/bin/charm

Pycharm Updates

After updating to a new version of Pycharm it may be necessary to recreate the Command Line Launcher via the steps above.

  1. Run commands to configure a local file diff and merge tool.
    The following command provides the configuration syntax required for Pycharm file diff operations and merge operations..
charm merge --help
Usage:
  /usr/local/bin/charm -h | -? | --help
  /usr/local/bin/charm [project_dir]
  /usr/local/bin/charm [-l|--line line] [project_dir|--temp-project] file[:line]
  /usr/local/bin/charm diff <left> <right>
  /usr/local/bin/charm merge <local> <remote> [base] <merged>
  1. Use this proper syntax for Pycharm to produce a three-pane merge window showing file versions as follows.
    Source Kitchen File Version ||| Base File Version ||| Target Kitchen File Version

Pycharm Diff and Merge View Configuration Syntax

Pycharm CE requires the use of merge and diff keywords as part of its configuration with DKCloudCommand.

Pycharm File Diff Config:

/usr/local/bin/charm diff {{local}} {{remote}}

Pycharm Merge Config:

/usr/local/bin/charm merge {{left}} {{right}} {{base}}

Meld

Meld Diff and Merge View Configuration Syntax

Meld does not require the use of merge and diff keywords as part of its configuration with DKCC.

Meld File Diff Config:

meld {{local}} {{remote}}

Meld Merge Config:

meld {{left}} {{base}} {{right}}

PC Setup of Local Merge and File Diff Tools

Pycharm

  1. Download the free Community Edition of Pycharm

Typical Installation Path

c:\Program Files\JetBrains\PyCharm Community Edition 2018.1.2\bin\

  1. Ensure that Pycharm is included in your path environment variable:

Restart Console

Remember to restart your console after editing your path environment variable in order for the changes to take effect.

c:\Program Files\JetBrains\PyCharm Community Edition 2018.1.2\bin\
  1. Run a file diff call for Pycharm.
c:\Users\User1\example_files>pycharm diff file1.txt file2.txt
  1. Run a file merge call for Pycharm.
c:\Users\User1\example_files>pycharm merge file1.txt file2.txt file3.txt
  1. Use the following example to complete the configuration for DKCloudCommand using Pycharm on PCs.
Username:                      <USERNAME>
Password:                      <PASSWORD>
Cloud IP:                      https://cloud.datakitchen.io
Cloud Port:                    443
Merge Tool:                    pycharm merge {{left}} {{right}} {{base}}
Diff Tool:                     pycharm diff {{local}} {{remote}

Meld

  1. Download the open source tool Meld for Windows.

Typical Installation Path

The file Meld-3.16.2-win32.msi is typically installed at the following path:

c:\Meld\

  1. Ensure that Meld is included in your path environment variable:
%SystemRoot%\system32;%SystemRoot%;c:\Meld\
  1. Run a file diff call for Meld on PCs.
c:\Users\User1\example_files>meld file1.txt file2.txt
  1. Run a file merge call for Meld on PCs.
c:\Users\Alejandro\example_files>meld file1.txt file2.txt file3.txt
  1. Use the following example to complete the configuration for DKCloudCommand using Meld on PCs.
Username:                       <USERNAME>
Password:                       <PASSWORD>
Cloud IP:                       https://cloud.datakitchen.io
Cloud Port:                     443
Merge Tool:                     meld {{left}} {{base}} {{right}}
Diff Tool:                      meld {{local}} {{remote}}

Configure DKCloudCommand

DKCloudCommand is configured using the config command. Upon initial configuration, you will be prompted to set up the general for DKCloudCommand and a default user context. Configuration for multiple users is supported, though only a default configuration context is required.

When prompted, enter the username and password (hidden) provided to you by DataKitchen. The default cloud address and port are provided. By default, DKCC responses will also include an announcement of the context currently in use. This is most helpful if you have configured multiple contexts on the same local machine.

To configure Pycharm CE with DKCC, for example, follow the syntax shown in the response below:

dk config
Enter username:  <USERNAME>
Enter password: <PASSWORD>
DK Cloud Address (default https://cloud.datakitchen.io):  <ENTER>
DK Cloud Port (default 443): <ENTER>

DK Cloud Merge Tool Template (default None):  /usr/local/bin/charm merge {{left}} {{right}} {{base}}
DK Cloud File Diff Tool Template (default None):  /usr/local/bin/charm diff {{local}} {{remote}}
Check current working path against existing contexts?[yes/No] (default No):No
Hide current context legend on each command response?[yes/No] (default No):No

Aside from the Kitchen and Recipe files located in your /Kitchens working directories, all local configuration and temporary files associated with DKCloudCommand are located in the {USER_HOME}/.dk directory. The contents of this directory will appear as follows:

Hidden Files & Folders

You may need to turn on the ability to view hidden files and folders on your local machine in order to view the contents of {USER_HOME}/.dk. Consult documentation specific to your operating system for instructions.

  • .context
  • .latest_version
  • general_config.json
  • default
    • /.dk-credentials
    • /config.json
    • /diffs
    • /merges
  • another_context
    • /.dk-credentials
    • /config.json
    • /diffs
    • /merges

The .latest_version file records the current version of DKCC and is used to prompt you when upgrades are available. The .context keep track of the user context under which you are presently working, should you have configured multiple contexts. The general-config.json file contains the configuration for your local diff and merge tool, which is applicable for all configured users. This file also contains your settings for working path warnings. The configuration for your /default user context contains a .dk-credentials file, which contains the web token for the configured user. The default/config.json file contains a username, password, hostname, and port information for that specific user context. Here also are /diffs and /merges subdirectories that store local, temporary files once you have executed diff and merge operations from the command line. Additionally configured user contexts follow a similar directory and file structure.

View User Information
Once you have configured your default user, view the user information with the user-info command. Note that your role is configured by your account administrator.

dk user-info
Name:                   <Email>
Email:                  <Email>
Customer Name:          <Customer>
Support Email:          [email protected]
Role:                   <Role>

View Full Configuration Details
View the details of your user's configuration with the config-list command.

dk config-list
Current configuration is ...
Customer Name:          <Customer>
Context:                default
Config Location:        /{USER_HOME}/.dk/default/config.json
General Config Location:/{USER_HOME}/.dk/general-config.json
Username:               <USERNAME>
Password:               <PASSWORD>
Cloud IP:               https://cloud.datakitchen.io
Cloud Port:             443
Merge Tool:             /usr/local/bin/charm merge {{left}} {{right}} {{base}}
Diff Tool:              /usr/local/bin/charm diff {{local}} {{remote}}
Check Working Path:     False
Hide Context Legend:      False

Create & Configure Additional User Contexts

DKCloudCommand supports the configuration of multiple users, both within the same account and across customer accounts. Each of these configurations is called a context. The only required context is the default context. The general configuration, specifically the configuration for your local diff and merge tool and the check-working-path setting, is shared across all contexts.

Announcement of Current Context

Note that DKCC responses will only occur if multiple contexts have been configured AND this feature is enabled in general-config.json.

View a List of Configured Contexts

dk context-list
Current context is: default
Available contexts are ...

default
another_context
third_context

Create & Configure a New, Additional Context
A new context is created via the context-switch command, whereby an as-of-yet created context is referenced.

dk context-switch another_context
Current context is: default
Context does not exist. Are you sure you want to create context another_context? [yes/No]yes
Switching to context another_context ...

Enter username: <USERNAME>
Enter password: <PASSWORD>
DK Cloud Address (default https://cloud.datakitchen.io): <ENTER>
DK Cloud Port (default 443): <ENTER>
dk context-list
Current context is: default
Available contexts are ...

default
another_context

Default Context

Note that moving forward in the documentation we assume the use of the default context.

Switch to an Existing Context

dk context-switch another_context
Current context is: default
Switching to context another_context ...
Context switch done.
Use dk user-info and dk config-list to get context details.

Delete a Context
You cannot delete the default context nor a context for which you are presently configured. Switch away from an existing context before deleting it.

~$ dk context-delete sample_context
Current context is: default



Credential information will be lost.
Are you sure you want to delete context sample_context? [yes/No]yes
Deleting context sample_context  ...

Done!

Setup Local Workspace

Kitchens, recipes, and recipe variations exist both in the cloud, accessible via the DataKitchen API (webapp or CLI), and optionally, local to a user's machine. For the sake of clarity in this guide we create a /Kitchens directory to house our DataKitchen analytics assets.

(DataKitchen) ~ $ mkdir Kitchens
(DataKitchen) ~ $ cd Kitchens
(DataKitchen) ~/Kitchens $ 

Virtual Environments

Moving forward, we'll omit references to virtual environments when demonstrating shell commands.

Directory Structure

Kitchens and recipes fall into a hierarchical directory structure.

Check Working Path

If you have configured multiple user contexts, and wish to make use of the general configuration Check Working Path option, add another directory level to your kitchens working directories. When this naming convention is followed properly you will receive a warning if you are operating in a kitchen working directory that does not match your currently selected user context. Name these levels such that they match the names of your configured contexts.

Configured User Contexts:

Customer1_User1
Customer1_User2
Customer2

Matching Kitchen Working Directory Naming Conventions:

~/Customer1_User1/Kitchens/
~/Customer1_User2/Kitchens/
~/Customer2/Kitchens/

or

~/Kitchens/Customer1_User1/
~/Kitchens/Customer1_User2/
~/Kitchens/Customer2/

Updated 9 days ago


Install & Configure


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.