DataKitchen DataOps Documention

Agents & Resources

Agents provide resources for order runs. When users schedule orders, they can allocate disk space and memory for running those orders from the resources supplied by agents. The scheduler resides in a Kubernetes Owner/Scheduler, and order runs execute within Kubernetes components for secure, scalable, dependable management.

Agents

A DataKitchen Agent allocates resources and runs recipe orders. An agent is actually a Kubernetes worker node within a customer's Kubernetes Cluster, and it houses the DataKitchen recipe runner. This architecture ensures that each customer has a dedicated execution engine with its own agents.

High-level architecture with connections to a customer's Kubernetes components.

High-level architecture with connections to a customer's Kubernetes components.

Agent Resources

Each agent can run as many orders as there are available resources. One benefit of using Kubernetes services is that agent resources can be scaled by increasing or reducing the number of pods, or sets of running containers, in an agent. The flexibility of Kubernetes allows for customers to add as many agents (Kubernetes worker nodes) as they want, while the customer's Kubernetes Owner/Scheduler handles pod scaling for optimal order execution.

When users set resource allocation while configuring order schedules, they are reserving memory and disk space for processing specific orders. If multiple orders run at the same time, an order that has insufficient resources stays in a pending state until resources become available in the agent. If the agent runs out of memory or disk space completely, orders will end in errors.

Agent Status

The Agent Status feature, found in the top menu bar in the web app, displays the total current available resources for all of a customer's agents. If a customer uses agent groups to divide resources among kitchens, the actual resources available for an order will be less than what the agent status reports.

That same information is available when users click to run a variation or node. The Run Variation and Run Node dialogs have a section for Agent Status, so that users can assess the resources for processing their orders. If an agent is offline or being checked for its status, the Run button is not active and the order cannot initiate.

Order Scheduling

The cron expression configured for order runs in a variation becomes layers of Kubernetes jobs that dispatch each other. Basically, a Kubernetes cron job creates jobs on a repeating schedule, and those jobs create one or more pods in which to execute the work.

Agent Pools & Agent Groups

Customers can host and maintain their own pool of agents, a designation that DataKitchen implementation engineers configure to constrain all kitchens to the private pools. In this case, customers can define Agent Groups to segregate their agents by environment.

DataKitchen also offers a default agent pool from which customers can allocate resources. At the time of any order run, the system provides the best available agent to use.

Agent Groups

Customers may place constraints on resource usage by specifying agent groups that subdivide their agent pools. Agent groups enable users to reserve or restrict the resources for specific kitchens.

For example, in a pool of three agents, a DataKitchen customer may want a production kitchen to use agent1, a staging or UAT kitchen to use agent2, and all remaining kitchens to share agent3. This method ensures that production deployments will always have sufficient resources.

Specify an agent group for kitchen order runs
When your organization has chosen to segregate its agent pool with named agent groups, users can designate a valid agent group for each kitchen.

  1. On the Kitchens page, find the kitchen and click the associated wrench icon to configure the kitchen.
  2. Click the Agents tab.
  3. Enter a valid name in the Agent Group field.
    Note that this field is disabled where there are no customer-specific agents.
  4. Click the Update button.

Updated about a month ago

Agents & Resources


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.