Skip to main content
Control Plane is an open-source CI/CD and observability platform built specifically for AI agents. It gives teams the deployment pipelines, tracing, evaluation, and dataset management that agent development demands, without the vendor lock-in of hosted platforms. Think of Control Plane as the open-source alternative to LangSmith: self-hostable, GitOps-native, and designed to slot into the engineering workflows your team already uses.

​
What Control Plane does

CapabilityDescription
Pipeline CI/CDAutomated build, test, and deploy pipelines for agent projects
ObservabilityDistributed tracing for every LLM call, tool invocation, and agent step
EvaluationRun eval suites on every commit; catches regressions before they reach production
Dataset managementVersion-control your prompts, test cases, and golden sets alongside your code
GitHub Actions integrationFirst-class Actions workflows; triggers evals on PRs and blocks merges on failure
Self-hostableRun entirely on your infrastructure; no data leaves your environment

​
Why Control Plane

Agent development breaks standard CI/CD. A unit test can pass while the agent produces worse answers. A deploy can succeed while latency doubles. Control Plane adds the agent-specific checks (LLM evaluation, trace comparison, cost budgets) as first-class pipeline steps. 
git push → CI pipeline → eval suite → trace diff → deploy
Every PR gets an automated quality gate. Every deploy generates a trace baseline. Every rollback is one command.

​
Key features

GitOps-native: Your langship.yaml lives in the repo. Pipelines, eval configs, and deployment targets are all version-controlled. Trace-first observability: Every agent run produces an OpenTelemetry-compatible trace with LLM call details (model, tokens, latency, cost), tool calls, memory reads, and final outputs. Eval-in-CI: Define evaluators (LLM-as-judge, exact match, semantic similarity, custom Python) and run them on every push. Block merges when quality drops. Dataset versioning: Store prompt templates, test cases, and expected outputs in Control Plane’s dataset store. Pin evals to specific dataset versions so results are reproducible. Multi-environment deployments: Promote agent versions from dev → staging → production with environment-specific configs and automatic rollback on eval failure.

​
Architecture overview

Control Plane has three components:
  1. Control Plane Server: Receives traces, stores datasets, runs evals, manages deployments. Self-hosted via Docker or Kubernetes.
  2. Control Plane CLI: langship command for local development, running evals, and triggering deploys.
  3. Control Plane SDK: Lightweight instrumentation library (pip install langship) that adds trace collection to any LangChain, LlamaIndex, or raw OpenAI agent.

​
Where to go next

  • Quickstart: running Control Plane Server and tracing your first agent in 10 minutes
  • How It Works: pipeline architecture and core data model
  • Deployment: Docker Compose, Kubernetes, and scaling
  • GitHub Actions: integrating Control Plane evals into your PR workflow