AI agent evaluation tied to vendor methods, scalability‑grounding gap
At VB Transform 2026, three AI industry executives converged on the same finding: a single agent trace can score perfectly and still signal a broken product. The structural problem they surfaced, that automated judging is scalable but ungrounded while human review is grounded but not scalable, is what Conviva's Hui Zhang called a poison-pill tradeoff. Harrison Chase of LangChain, Zhang, and CoreWeave's Emmanuel Turlay each proposed a methodology that maps to the product their company sells, which is the shape of the conversation: three vendor-shaped answers to one structurally unsolved problem.
Zhang illustrated the gap with a retail example, though Conviva, where he is CTO and co-founder, sells the contrastive analysis methodology behind it. A shopper asks an agent for running shoes ahead of a half marathon, qualifies through the agent's questions, and completes the purchase inside the conversation. Scored individually, that interaction looks fine. The signal Zhang's team found only appears in cohort comparison: the clarification ratio was three times higher than baseline for that shoe category, and shoppers finished the purchase outside the conversation five times more often than baseline. Neither number shows up in the single trace. Both point to a category-specific debugging target, and both depend on a baseline that is itself a methodology choice. The source does not establish how the cohort baselines hold across product categories, or whether the same three-times and five-times multipliers surface in non-retail workloads.
Chase reframed evaluation criteria as a living product specification. According to Chase, evaluation sets now function as the document that defines what an agent should and shouldn't do, what he called 'evals are like the new PRD.' That language serves LangChain's product surface: a framework company benefits when every agent project treats evaluation as an ongoing build artifact, since that framing pulls the eval surface into the framework itself. Chase's separate warning against eval paralysis, his observation that 'the best teams launch and then iterate,' is more useful as a practical rule of thumb. It does not, however, address the failure mode the panel opened with: shipping a broken interaction that scores well. Turlay, the third speaker, supported the same instinct from a different angle: his team's effort to reach 100% test coverage still left bugs in production, which is the same gap a per-trace evaluation suite cannot close.
The judge-model discussion is where CoreWeave's role becomes clearest. Turlay's rule, start with the strongest model to confirm a task is solvable, then work down to cheaper variants, is a sizing argument that an inference provider can monetize. He also describes sampling a fraction of traffic once a pattern is proven rather than scoring every interaction, which shifts cost from judgment volume to traffic routing. Both choices sit in the layer where CoreWeave's infrastructure operates. LangChain, which sells orchestration rather than inference, optimized the same cost problem in a different layer. According to Chase, the company fine-tuned a Qwen model from Alibaba's open-source family to detect perceived error, the moment a user believes the agent made a mistake. Chase reports the result performed comparably to Claude Sonnet for a 10 to 100x cost reduction depending on serving configuration. That number is vendor-reported and unattributed to independent benchmarks, and the source does not specify which perceived-error categories the fine-tuned Qwen model covers.
On the simpler end of the same cost curve, Chase pointed to Claude Code's guardrails as a counterweight to the judge-model reflex. According to Chase, the guardrails he saw there were mostly regexes, not small language models. The observation is useful: not every evaluation problem needs a model. The source reports it as a passing example rather than a benchmark, and the workload coverage of regex-based guardrails is not specified. The contrast between regexes and small judges is part of the same pricing argument the panel was building, but the cost-and-coverage tradeoff between the two approaches is not quantified in the source.
The human-in-the-loop question sat underneath the methodology debate. Turlay drew on prior work at a self-driving car company, where his team compressed data intake and retraining into a two-week cycle. Even then, he said, someone had to sign off on the model shipped to the car. Turlay extends the same accountability logic to legal, finance, and healthcare, where responsibility still has to land on a person before agents can act on their own. Zhang agreed that humans remain the guardians of corner cases. Chase went further, arguing that humans in the loop matter not just for safety but for trust, memory, and learning. None of the three speakers describe an automated replacement for that accountability layer, and none describe a methodology that closes the scalable-versus-grounded gap they named at the outset.
The three methodologies are converging on the same gap rather than closing it. Each speaker proposed a methodology that maps to the product their company sells: Conviva's contrastive analysis, LangChain's eval-as-PRD, CoreWeave's judge-model sizing. The implication is that agent evaluation is consolidating around vendor-shaped methodologies that couple each customer's tooling stack to that vendor's data shape, whether that data is Conviva's cohort baselines, LangChain's eval artifacts, or CoreWeave's inference routing. The operational consequence: a team that picks one methodology commits to a vendor's data model before it has an independent ground-truth baseline to verify any of the three approaches. The panel did not test whether that coupling improves evaluation quality or just relocates the cost of ground truth into a vendor-controlled layer.