A conversation log can look completely clean and still represent a failure. The agent gave a confident answer. The user moved on. Nobody flagged it. But somewhere in the chain, a tool returned stale data, a handoff was silently dropped, or the final output was technically correct and practically useless.
Evaluation, not the build, is the hard part
A recurring theme at VB Transform this year was that most teams building AI agents are optimising the wrong stage. The build gets the attention. Evaluation gets a checkbox.
That ordering works fine for a simple chatbot. It breaks down for a multi-step agent, because a multi-agent system does not fail the way a single model call fails. A bad output from one model call is obvious. A silently wrong tool call three steps into a chain, corrected downstream by coincidence rather than design, is not.
Why pass or fail is not enough
Most teams treat evaluation as a single question: does the output look right? Real evaluation asks something harder: whether the system behaved correctly across every step, every tool call, and every branch it did not take. An agent can land on the right final answer while making three wrong decisions along the way, and pass or fail metrics will never catch that.
In our own LangGraph pipelines, we've had to build internal review layers specifically because pass or fail metrics miss this. Tracing the reasoning matters more than reading the response, which is why we track metrics like tool-call correctness, step-level task completion, and trajectory match against a known-good path, rather than scoring only the final answer. A workflow can look like it works for weeks before a real edge case exposes exactly where it was never actually checking what it claimed to check.
Practically, that means treating evaluation as two separate layers, not one. Offline evaluation runs before anything ships: a regression suite of golden traces checked in CI on every change, so a prompt tweak or a new tool doesn't quietly break a downstream step. We lean on DeepEval for this, since its pytest-style tests let us assert on tool correctness, task completion, and RAG faithfulness as part of the normal build pipeline, and on Promptfoo for matrix-testing prompts and models against regressions, jailbreaks, and PII leakage before a change goes out. Online evaluation is the second layer, and it's the one most teams skip: scoring live traffic continuously, not just at test time. LangSmith's evaluators are a natural fit here given the LangGraph pipelines we already run, because they trace every span of a run and can grade production traffic against the same rubrics used offline, which is what lets us catch drift instead of finding it three weeks later. Depending on the workload, we've also pulled in Ragas for RAG-specific scoring, and tools like Langfuse as well.
What this means before you ship
If your evaluation process for a production agent is still spot-checking a few outputs, you are flying blind, and the gap will not show up in a demo. It shows up three weeks into production, on the one input nobody thought to test, right when the cost of being wrong is highest.
Confidence in the demo is not confidence in the system. Building that confidence properly means instrumenting every step an agent takes, not just its final answer, with both a CI-gated regression suite and a live scoring layer watching production, before that agent is anywhere near a client-facing workflow.