Skip to content
Atlas by Comet Lab
Infrastructure layer

Observability and evals, compared

Observability tools record what an AI system actually did, call by call; evaluation tools measure whether it did it well. A system in production needs the first, and any change to prompts or retrieval needs the second. A testing tool does not replace a tracing platform: the two pair.

Learn the idea Testing AI before trusting it: how to run an eval on your own work on Horizon

Open this layer in the explorer

At a glance

Every tool in this layer, with where it fits and the facts that usually decide it.

Tool Where it fits Licence Self-hostable Hosted API Maturity
Langfuse LLM observability A strong default whenever an AI system reaches production and you need to see what it actually did, call by call. Open source Yes Yes Established
Braintrust Evaluation platform A fit for product teams who want disciplined, data-driven evaluation as the centre of their iteration loop. Proprietary No Yes Growing
DeepEval Evaluation framework A fit when you want LLM quality checks written as tests and run automatically in CI. Open source Yes No Growing
Helicone LLM observability A fit when you want fast, low-effort visibility into model calls, costs, and latency through a simple proxy. Open source Yes Yes Established
LangSmith LLM observability A natural fit when a project is built on LangChain or LangGraph and you want tracing and evals from the same ecosystem. Proprietary No Yes Established
Lunary LLM observability A fit when you want observability and versioned prompt management together, in a self-hostable tool. Open source Yes Yes Growing
OpenLLMetry Observability standard A fit when you want LLM tracing as standard OpenTelemetry data, feeding an observability backend you already use. Open source Yes No Growing
Opik LLM observability A fit when you want an open-source, self-hostable platform spanning tracing, evals, and production monitoring. Open source Yes Yes Growing
Phoenix LLM observability A fit when you want open, OpenTelemetry-based tracing and evaluation with a focus on debugging agent behaviour. Source available Yes No Growing
promptfoo Evaluation framework A fit when you want config-driven evaluation across prompts and models, plus security red-teaming, in one CLI tool. Open source Yes No Established
Ragas RAG evaluation Essential when retrieval quality matters and you need to measure changes to a RAG pipeline rather than judge them by eye. Open source Yes No Growing

How to choose

Start from your situation. Where a line says “our”, it comes from Comet Lab’s own builds.

  1. A system is going to production

    Our default: self-hostable, so the trace stays with you. Wire it in before launch, not after the first incident.

  2. Built on LangChain or LangGraph

    Tracing and evals from the same ecosystem. When traces must be self-hosted, we point to Langfuse instead.

  3. Tracing and evals in one self-hostable tool

    Opik spans tracing, evals and production monitoring; Phoenix is built on OpenTelemetry, with a focus on debugging agents.

  4. Quick visibility into calls, cost and latency

    A simple proxy logs every call. Less suited to deep, nested agent traces.

  5. Tests that gate every change

    promptfoo runs config-driven checks and red-teaming in CI; DeepEval writes quality checks as tests; Ragas measures how faithful a RAG pipeline's answers are.

  6. Traces as standard OpenTelemetry data

    Feeds an observability backend you already run. It is instrumentation, not a destination.