DataKitchen DataOps Documention

Vault & Secrets

Introduction

Vaults securely store sensitive infrastructure and toolchain credentials as Secrets, which are encrypted and stored remotely. As we will see below, Vaults can be configured in a number of different ways to match your organization's operational and security needs. Vaults are ideal for associating Kitchens with environments like Production, Staging, and Development.

A view of the Global and Kitchen-level Vault interface.

A view of the Global and Kitchen-level Vault interface.

In the example above, the Global-Vault has a custom, private connection, while the Kitchen-Vault private connection has been inherited from the Parent Kitchen.

In the example above, the Global-Vault has a custom, private connection, while the Kitchen-Vault private connection has been inherited from the Parent Kitchen.

Global-Vault

The Global-Vault contains Secrets that are available for use by Recipes in any Kitchen. It is enabled by default. The Global-Vault can be used alone, disabled in favor of Kitchen-Vaults, or alternatively, used in conjunction with Kitchen-Vaults. All users can, from the DataKitchen platform, view Secrets (but not their underlying values), as well as edit, create or delete Secrets contained in the Global-Vault, so long as it is not configured as a private, custom Vault.

Disabling the Global-Vault

UI

To disable the Global-Vault in favor of solely using Kitchen-Vaults, select the None Vault option for the Global-Vault on the Secrets page within any Kitchen.

Disabling the Global Vault.

Disabling the Global Vault.

Command Line

Use the vault-info command with the --global option to see information about the Global-Vault.

~ $ dk vault-info --global



Current context is: default
YYYY-MM-DD HH:MM:SS - Getting the vault info

---- global config ----
 prefix                 vault_prefix
 private                False
 service                custom
 token                  ******
 url                    http://12.345.678.9:1234

Disable the Global-Vault using the vault-config command with the --disable option.

~ $ dk vault-config --disable



Current context is: default
You are configuring the global vault!. Are you sure you want to proceed? [yes/No]yes
YYYY-MM-DD HH:MM:SS - Setting the vault info
Done.

Reenabling the Global Vault

UI

Reenable the Global-Vault on the Secrets page in the opposite manner to how it is shown being disabled above.

Command Line

~ $ dk vault-config --url http://12.345.678.9:1234 --token sample-token



Current context is: default
You are configuring the global vault!. Are you sure you want to proceed? [yes/No]yes
YYYY-MM-DD HH:MM:SS - Setting the vault info
Done.

Kitchen-Vaults

Kitchen-Vault connection settings are defined at the Kitchen-level. When OrderRuns process, they first look to Kitchen-Vaults for Secret values. If a Secret value is not found in a Kitchen-level Vault, the OrderRuns next looks to the Global Vault for the Secret value. If a Secret is found in the Kitchen-Vault the Global-Vault's corresponding value will be ignored.

The Redshift Password Secret value from the Kitchen-Vault takes precedent over the value in the Global-Vault.

The Redshift Password Secret value from the Kitchen-Vault takes precedent over the value in the Global-Vault.

The Redshift Password Secret value from the Global-Vault is ignored.

The Redshift Password Secret value from the Global-Vault is ignored.

Each Kitchen may be connected to a maximum of one Kitchen-Vault at any given time. Management of Kitchen-Vault connection settings is limited to those users included in the connected Kitchens' Staffs.

If a user is not part of a Kitchen's staff they cannot enter the Kitchen to view or manage its Kitchen-Vault settings.

If a user is not part of a Kitchen's staff they cannot enter the Kitchen to view or manage its Kitchen-Vault settings.

Kitchen Vault Connection Inheritance

Kitchen-Vault connection settings, as long as they are of the custom variety, may be configured to be inheritable by Child Kitchens to the Parent Kitchen where they are defined. Default Kitchen-Vault connection settings may not be inherited.

Inheritance of Kitchen-Vault connection settings applies across generations of Child Kitchens. Edits to inheritable, custom Kitchen-Vault connection settings can only be applied from within the Kitchen where the connection was initially configured. In other words, if a Child Kitchen inherits a Vault connection from its Parent Kitchen, said Vault connection may only be edited from within the Parent Kitchen.

If a user acts to delete a Kitchen that has Child Kitchens with an inherited Vault connection, the Parent Kitchen deletion will be prevented with a warning to first delete relevant Child Kitchens.

Deleting a Kitchen is prevented, and a warning presented, if that Kitchen has a Kitchen-Vault custom connection configured that has been inherited by Child Kitchens.

Deleting a Kitchen is prevented, and a warning presented, if that Kitchen has a Kitchen-Vault custom connection configured that has been inherited by Child Kitchens.

If a user changes a Kitchen-Vault's connection setting to non-inheritance, and Child Kitchens exist that previously inherited this Kitchen-Vault connection, the Child Kitchens will lose their inherited connection. A warning for this case is provided to the user, as seen below. Should the user reimplement inheritability for the Kitchen-Vault, existing Child Kitchens that had previously inherited the connection settings will re-inherit the connection automatically.

Creating Kitchen-Vault Connections

Via Kitchen Wizard

You will be prompted to configure a Vault Service connection during a step in the Kitchen creation process.

The connection settings for a custom Kitchen-Vault.

The connection settings for a custom Kitchen-Vault.

Via Secrets Page

Vault connection settings may be edited at any time after they are created.

Configuring a Kitchen-Vault connection from the Secrets page.  Note that a custom connection provides more options, but requires more configuration.

Configuring a Kitchen-Vault connection from the Secrets page. Note that a custom connection provides more options, but requires more configuration.

Default vs. Custom Vaults

Both Global and Kitchen-Vaults may be configured as either Default or Custom Vault types.

Default Vault

By default, DataKitchen customers have access to a secure Vault hosted by DataKitchen. Default Vaults do not require the configuration of connection settings.

No connection setting configuration is required for Default Vault settings.  Vaults of this nature are securely hosted by DataKitchen.

No connection setting configuration is required for Default Vault settings. Vaults of this nature are securely hosted by DataKitchen.

Custom Vault

Alternatively, customers may choose to opt for their own, externally-hosted Vault services. Custom Vaults possess a number of configuration options.

A custom Vault's connection settings.

A custom Vault's connection settings.

Private

Setting a Vault service as private disables the viewing, editing, creation, and deletion of Secrets via the UI and command line. Secret values may still be leveraged by Recipes in private Vault service configurations as long as the Vault service is accessible on the environment where OrderRuns are executed (the Mesos Agent).

Vault Service URL

This field is required. This URL must be accessible by DataKitchen if the custom Vault configuration is NOT private. Set to a blank value for the Global-Vault if you wish to disable this Vault's connection. An example Vault Service URL is http://34.234.109.5:8200.

Vault Service Token

If this token is set to periodically expire, you must either renew the token or update its value at the configuration screen shown above. The token value is obscured from the user once it has been saved. Set to a blank value for the Global-Vault if you wish to disable this Vault's connection.

Vault Path Prefix

This is an optional value denoting a path prefix, allowing you to isolate DataKitchen-related Secrets from other Secrets in your Vault(s). A valid example entry for this field is "datakitchen". Vault paths follow the structure secret/[some path prefix]/[path]. Here, you need only denote the [some path prefix] value when defining the path prefix. The path prefix need not be included in the Secret path. For example, the secret path vault://dockerhub/password is valid whether or not a path prefix is configured.

Managing Secrets

Secret Management is Disabled for Private, Custom Vault Services

Note that if you have selected to configure a Vault service as private you will be unable to create, delete, edit, or view Secrets via DataKitchen's UI or command line tool.

Viewing Secrets

Global-Vault Secrets and Kitchen-Vault Secrets are listed on the Secrets page of the DataKitchen UI. If a Vault has a custom connection set to private, that Vault's Secrets will not be visible via the DataKitchen application, though the Secret values may still be called by OrderRuns in process.

Viewing Secrets in the UI for both Global and Kitchen-level Vaults.

Viewing Secrets in the UI for both Global and Kitchen-level Vaults.

Creating Secrets

Secret values may either be saved as string or files containing keys.

String Values

Vault entries for a common service can be grouped under a common path. For example, the following values can be inputted to create Secrets related to Docker Hub:

  • dockerhub/email
  • dockerhub/namespace
  • dockerhub/password
  • dockerhub/username
Adding a Vault Secret with a string value.

Adding a Vault Secret with a string value.

Key Files

For example, you may upload a .pem file:

Adding a Vault Secret with a value set equal to a file.

Adding a Vault Secret with a value set equal to a file.

Editing Secrets

Viewing Vault Secret Values

Note that once a Secret value is saved you cannot view its underlying value in the clear. However, you may still overwrite the saved value.

Editing a pre-existing Secret in the UI.  The user cannot view the existing Secret value in the clear, only the new value that is being inputted.

Editing a pre-existing Secret in the UI. The user cannot view the existing Secret value in the clear, only the new value that is being inputted.

Deleting Secrets

Deleting a Vault Secret via the UI.

Deleting a Vault Secret via the UI.

Using Secrets

Vault Secret Syntax

When referencing Secrets in Recipes or Kitchens use the {#} syntax. It is a best practice to construct Vault paths with solely lowercase characters. It is also recommended to group Secrets under subpaths as is appropriate. For example, all DockerHub related Secrets may be grouped together under a /dockerhub subpath.

#{vault://my_secret_example}

#{vault://dockerhub/username}

#{vault://dockerhub/password}

Populating Secret Paths

Secret value placeholders may be inserted into Recipes in various file editing screens across the DataKitchen UI, where the lock icon is found.

Secrets Are Not Populated in Dropdowns for Custom, Private Vault Configurations

In these cases, users will need to manually enter Vault paths when defining variable values, defining override values, or inserting Vault paths directly into Recipe node and resource files.

Quickly insert Vault Secret Paths into file forms.

Quickly insert Vault Secret Paths into file forms.

Best Practice for Use of Vault Secrets and Parameter Overrides

While Vault paths may be inserted directly into Recipe configuration and resource files, it is a best practice to use Vault paths once within Recipes (often at the Kitchen-level) by defining them as variables, and then using these variables wherever the relevant Vault paths are needed. This facilitates more streamlined management of Recipe code over the long term.

Variables.json

Inserting Vault paths into variables.json.

Inserting Vault paths into variables.json.

Variations.json Overrides

Quick insert of Vault Secret paths as Variation Overrides.

Quick insert of Vault Secret paths as Variation Overrides.

Resource Files

Inserting Vault paths into resource files via the Recipe Configure page route.

Inserting Vault paths into resource files via the Recipe Configure page route.

Node Files

Inserting Vault paths into configuration files via the Edit Variation page route.

Inserting Vault paths into configuration files via the Edit Variation page route.

Cooking Orders

Users will be prevented from cooking new Orders when the Recipe-Variation associated with the Order references Secrets absent from all connected Vaults.

Users are blocked from submitting Orders via the UI if the Recipe-Variation to be run contains references to Secrets not contained in either the Global-Vault or Kitchen-Vault, should they be configured.

Users are blocked from submitting Orders via the UI if the Recipe-Variation to be run contains references to Secrets not contained in either the Global-Vault or Kitchen-Vault, should they be configured.

Resolving Missing Secrets Warnings

If you are blocked from creating Orders due to missing Secret warnings, check that your Recipe-Variation and Kitchen settings only reference Secrets contained in connected Vaults.

To resolve, remove unused/deprecated Secrets from your Recipe/Kitchen or add the missing Secrets to one of the Vaults connected to the Kitchen.

Updated 8 months ago


Vault & Secrets


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.