Concepts

Kopilot is a LangChain/LangGraph multi-agent application wrapped in the operational plumbing a cluster tool needs: a safety-checked executor, a REST API, a Kubernetes operator, and an audit log.

The shape of the system

A request arrives — from the CLI, the REST API, Slack, a webhook, the Kubernetes event watcher, or an AITask resource.
The supervisor agent reads the request and delegates to one or more skill sub-agents.
Each sub-agent reasons and acts in a ReAct loop using four generic tools.
Every tool call passes through the executor middleware, which enforces safety rules before anything touches the cluster.
The supervisor synthesizes the results into one answer, optionally passing it through a reflection review first.

Supervisor

The supervisor is itself an agent. Each loaded skill is wrapped as a delegate_to_<skill> tool that the supervisor can call with a detailed instruction; it can call several sub-agents for a request that spans domains. As a fallback for simple queries, the supervisor also has direct access to the same four executor tools the skills use.

Skills as sub-agents

A skill is a YAML file: a name, a description, and a system prompt with optional reference documentation. At startup each enabled skill is compiled into its own ReAct agent. All skills share the same four tools:

Tool	What it does
`run_kubectl`	Runs a full kubectl command string against the cluster.
`run_helm`	Runs a full Helm command string.
`run_shell`	Runs a general diagnostic shell command (curl, dig, …).
`read_resource`	Reads a file (restricted to configured directories), a URL, or a ConfigMap via `configmap:<name>:<namespace>`.

Because the tools are generic, the difference between skills is knowledge and focus, not capability — and the safety layer applies identically no matter which skill is acting. See Skills for the built-in set and custom skills.

Executor middleware

Every command goes through the same pipeline, in order:

Hard blocklist. Patterns like rm -rf /, dd of=/dev/…, mkfs, fork bombs, and shutdown/reboot are refused unconditionally.
Rate limit. At most 50 tool calls per 5 minutes per task.
Risk assessment. The command is classified as low, medium, high, or critical. Destructive kubectl/helm verbs, --prune, and obfuscated invocations (command substitution, backticks, eval, xargs kubectl) are flagged. Destructive commands on protected namespaces are refused outright; other destructive commands require approval. See Security & safety.
Approval gate. An approval-required command is not executed; a pending approval request is registered and the agent reports its id. The command runs only after a human approves it, and only within the 10-minute approval window.
Bounded execution. Subprocesses run with a 90-second timeout; at most 2 MB is read from a process and at most 12,000 characters are returned to the model. Read-only commands are retried once after a timeout; mutating commands are never auto-retried, because a timed-out kubectl apply may already have taken effect server-side.

Every decision — executed, blocked, gated, approved, denied — is written to the structured audit log (JSON via structlog).

Reflection

When a task is submitted with reflect: true, a second LLM pass reviews the answer for completeness, accuracy, safety, and actionability. If the review finds it lacking, the task is retried once with the feedback appended to the prompt. Reflection is off by default.

Operator and CRDs

A Kopf-based operator watches three CRDs in the kubedevaiops.io/v1alpha1 API group:

AITask — a task as a Kubernetes resource; create one and the agent picks it up.
AISkill — skill definitions as cluster resources (the CRD is installed; loading skills from it is planned, not implemented yet).
AIPolicy — policy definitions for the agent.

apiVersion: kubedevaiops.io/v1alpha1
kind: AITask
metadata:
  name: health-check
  namespace: kubedevaiops
spec:
  prompt: "Perform a full cluster health check and report any issues"
  priority: normal
  channel: operator

Event watcher

With kopilot serve, a watcher observes Warning events across all namespaces. When the same resource/reason combination occurs 3 times within 5 minutes, it triggers an automatic investigation through the supervisor — capped at 3 concurrent investigations. Investigations follow the same safety rules as any other task; nothing destructive happens without approval.

Memory

Conversation state uses LangGraph's in-memory checkpointer: it is process-local and lost on restart, by design. Durable, multi-replica persistence would require wiring a different LangGraph checkpointer (for example Postgres) into get_checkpointer(). The same restart-loses-state caveat applies to the approval queue and task history.