DataKitchen DataOps Documention

DataKitchens

Kitchens are virtual workspaces where teams build, manage, and run pipelines that provision infrastructure and do data work. Think of them much like factories that contain assembly lines called Recipes. What follows is a description of the valuable characteristics of DataKitchens.

DataKitchen Object Naming Conventions

DataKitchen objects are alphanumeric, case-sensitive, and support underscores.

Permissioned

Each Kitchen has a designated Kitchen-Staff that limits access to manage its Infrastructure, edit its Recipes, and run its Orders.

Kitchens are staffed by teams consisting of folks with different skillsets using different tools working together to rapidly deliver high-quality insights.  Levels of technical skill vary across the team.

Kitchens are staffed by teams consisting of folks with different skillsets using different tools working together to rapidly deliver high-quality insights. Levels of technical skill vary across the team.

Segregated Infrastructure

Each Kitchen can be configured such that its Recipes run on and connect with Infrastructure associated with specific environments. This is accomplished by defining Infrastructure connections as part of the Kitchen workspace definition.

In this way, one can have Production, Hot Fix, User Acceptance Testing, Proof-of-Concept, Dev, and Feature Sprint environments that are isolated from one another.

A menu of DataKitchen I/O connector Node components known as [Data Sources and Data Sinks](https://datakitchen.readme.io/docs/data-sources-data-sinks).

A menu of DataKitchen I/O connector Node components known as Data Sources and Data Sinks.

A subset of the tools supported via [Container-type Nodes](https://datakitchen.readme.io/docs/container-nodes).

A subset of the tools supported via Container-type Nodes.

Secured

Every DataKitchen may leverage a Global Vault and/or optionally leverage a Kitchen-Vault. Vaults contain encrypted entries called Secrets that are most often data infrastructure connection credentials. Users may only manage and leverage those Secrets available to a Kitchen if they are a member of that kitchen's Staff. This mechanism provides freedom of user experimentation on one hand whilst also supporting isolated environments on the other.

Versioning

DataKitchen levereages v0.11.3 or higher of HashiCorp's Vault.

DataKitchen leverages [HashiCorp's Vault](https://www.vaultproject.io/) for encrypted secret credential storage.

DataKitchen leverages HashiCorp's Vault for encrypted secret credential storage.

Support Quality Output

DataKitchen's DataOps orchestration provides a framework to apply automated testing across the full span of a Recipe's assembly line. Tests can be configured to log results, warn upon failure, or stop an Order in place upon failure. The latter assures that quality issues never make it to the end of the production line and out the door to your customers.

Not only do tests help guarantee quality Recipe output, but they also expedite root-cause-analysis for Operations teams who adjudicate failed Orders for Recipes they did not personally build.

Not only do tests help guarantee quality Recipe output, but they also expedite root-cause-analysis for Operations teams who adjudicate failed Orders for Recipes they did not personally build.

Versioned

Recipe contents (code, configuration, and resource files) are versioned via git. This is accomplished by underpinning each Kitchen with its own unique git branch.

Version control centrally records the intellectual property represented by Recipes, avoiding haphazard, distributed storage of pipeline code and configuration that is fragile and prone to downtime and quality issues. With DataKitchen, users may review a detailed history of changes that have occurred in a Kitchen via each Kitchen's History.

Version control also supports team collaboration by assisting with the resolution of file merge conflicts. With DataKitchen, all users are guided to follow version control best practices, even if they do not understand git.

In the example above, the two parallel tracks represent git branches, with the longer serving as the parent branch (P) and the shorter the child branch (C).  Per the diagram, C is created, changes to its files are applied (+++, ++--), then C and its changes are merged up to P alongside changes that were made in the meantime in P (---).  Later, more changes are applied to C  (+,+) and these new changes are later merged up to P.  The changes made directly to branch P (---) are never merged down to C in this simple example, nor are any fie conflicts encountered.

In the example above, the two parallel tracks represent git branches, with the longer serving as the parent branch (P) and the shorter the child branch (C). Per the diagram, C is created, changes to its files are applied (+++, ++--), then C and its changes are merged up to P alongside changes that were made in the meantime in P (---). Later, more changes are applied to C (+,+) and these new changes are later merged up to P. The changes made directly to branch P (---) are never merged down to C in this simple example, nor are any fie conflicts encountered.

Rapid Creation/Teardown

Kitchens are hierarchical in nature as their underlying git branches hold hierarchical relationships. For every Child Kitchen there is a specific Parent Kitchen to which it is anchored. Child Kitchens may be quickly created, even by non-technical users, via the Kitchen Wizard, generating an exact copy of the Parent Kitchen's Recipes. Similarly, the Kitchen Wizard can quickly tear down any Kitchen but the master Kitchen.

Kitchens associated with release environments (PROD, UAT, POC, DEV etc.) and user-specific Kitchens (eg user1_dev) may exist indefinitely. Kitchens associated with hotfixes, feature development, or experimentation are often ephemeral.

In the example above, Hotfix and DEVELOP are children to MASTER and FEATURE is child to DEVELOP.  Hotfix and FEATURE are torn down after their changes are merged.  MASTER is tied to the Production environment, Hotifx to the Hotfix environment, with DEVELOP and FEATURE both tied to the Development environment.

In the example above, Hotfix and DEVELOP are children to MASTER and FEATURE is child to DEVELOP. Hotfix and FEATURE are torn down after their changes are merged. MASTER is tied to the Production environment, Hotifx to the Hotfix environment, with DEVELOP and FEATURE both tied to the Development environment.

Support Deployments

The Recipe content contained in Kitchens is under version control contained in that Kitchen's git branch. Kitchen Recipe content can thus be merged across Kitchens by merging their underlying git branches. Merges upwards in the Kitchen hierarchy are known as deployments.

Deployments via DataKitchen are fast and simple, even for non-technical users, who may use the Kitchen Wizard. Deployments are protected by the Kitchen Staff as users may only merge to Kitchens where they hold Kitchen Staff rights.

Because infrastructure connections are defined as part of the Kitchen workspace definition outside of Recipe version control, deployed Recipe contents are portable across environments, and connect to whichever infrastructure instances their Kitchen dictates.

Guided Kitchen merges provide supporting file diffs and file conflict resolution as needed in just a handful of clicks.

Guided Kitchen merges provide supporting file diffs and file conflict resolution as needed in just a handful of clicks.

The Value of DataOps

DataKitchen helps organizations follow DataOps best practices so that they can sustainably meet their clients' high expectations for rapid delivery of high-quality data insights.

Feeling overwhelmed?  DataKitchen can help you excel at DataOps.

Feeling overwhelmed? DataKitchen can help you excel at DataOps.

Updated 8 months ago


DataKitchens


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.