Amazon AI agents: 85% pilot, 5% production gap is a measurement problem
Amazon's AGI autonomy lead Bryan Silverthorn argued at VB Transform 2026 on Tuesday that the 85% pilot-to-5% production gap in enterprise AI agent deployment reflects a measurement problem, not a capability shortfall. The argument is correct in direction but lighter on mechanism than it appears, and the gap it claims to explain is wider than the framework he offers can close.
The headline number comes from Cisco: 85% of enterprises pilot AI agents, 5% ship them to production. Silverthorn, who joined Amazon through its acquisition of Adept AI and now leads multimodal agent training, proposed a four-dimension decomposition to structure the reliability problem. The dimensions are consistency, robustness, predictability, and safety, and he credited the framework to research from Princeton. As vocabulary for talking about why an agent fails, the taxonomy is useful. As an answer to why the 85% ceiling persists, it is more descriptive than diagnostic.
The case study Silverthorn used shows why. A customer deployed an agent for software QA involving serial number extraction from screens. The system worked flawlessly for two months, then began intermittently reading wrong numbers. The cause, per Silverthorn, was a vision encoder that behaved differently depending on where the serial number appeared on screen, triggered by a software change imperceptible to humans. The story cleanly illustrates consistency failing, but the failure mode is not generic. It is a specific encoder behavior tied to a specific visual-position trigger. The lesson Silverthorn drew, that teams need to identify their dimensions of variability and match measurement rigor to the stakes, is sound. It also means the framework does not generalize, because each deployment needs its own variability map and the framework does not tell teams how to build one.
The cultural reframe Silverthorn offered, calling Amazon's lab agents "interns," points at a real operational philosophy. The team treats agents the way a manager treats junior staff: ask what could go wrong, add backups and undo capabilities, decide what risk is acceptable. "You can ask the intern, 'What might you do wrong here? How might you mitigate your negative outcomes?'" Silverthorn said. Treating agents as fallible workers who need supervision is closer to how production agent systems actually behave today than treating them as software that should not fail. The cost is throughput. Amazon's lab accepts that agents occasionally run the wrong experiment in exchange for research velocity, including one agent running experiments around the clock on its own high-level research plan. Enterprises shipping agents to production at scale face a different tradeoff, since they cannot absorb the same error rate, and the source does not specify how teams should calibrate acceptable failure rates to their own domain.
VentureBeat's own research, presented before the session, complicates the picture further. Half of surveyed companies shipped agents that passed internal evals but failed real customers. Enterprises overwhelmingly track uptime while ignoring accuracy, and most default to the model makers' own evaluations with little else. If internal evals are passing while production fails, the variance dimension Silverthorn names is being tested against the wrong baseline. The eval pass rate says nothing about whether the eval surfaces the failure modes the deployment will hit. The measurement framework, in other words, is still measuring the wrong thing even when teams believe they have adopted it.
The limits of the framework are visible in Silverthorn's own remarks. Self-improving AI, he said, remains a loaded term, with autonomous self-improvement distant. LLM-as-judge techniques are one of several strategies for aligning agent capability with acceptable risk, not a substitute for the measurement work he is recommending. A commercial trucking customer uses browser automation to stitch warranty claims across fragmented systems, but Silverthorn stressed that no future agent will rely on computer use alone, since it will work alongside MCP, APIs, and other tools to complete end-to-end workflows. The honest concession is that the technology stack is layered and the failure modes are too. Reducing the agent reliability problem to four dimensions is convenient, but the production failure surface is wider than any four-bucket taxonomy can capture in advance.
The Cisco 85% number has the rhetorical force of a benchmark but the weakness of a survey aggregate. The 5% who ship to production are not solving a different reliability problem from the 85% who do not. They are running a different measurement process against their own deployment-specific failure modes. The framework Silverthorn presented is a useful shared vocabulary for talking about that gap, not a method for closing it, and closing it depends on each team building the instrumentation for their own system rather than borrowing a four-dimension checklist.