Credit card payment of $67.53 for a self-pickup order with a $66.85 discount, completed on 2023-12-26.
A new study evaluating 67 frontier models from 21 providers has measured something the multi-model orchestration approach has long assumed away: a shared failure mode that the standard diversity metric cannot detect. The paper's authors call it the co-failure ceiling, and on the MATH-500 benchmark the actual rate of all-models-wrong prompts came in at 5.2 percent, against a pairwise-correlation-based prediction of 2.3 percent. That 2.25x gap is the structural reason multi-model routing often pays its orchestration overhead but never collects a diversity dividend. According to researcher Josef Chen, the driver is a 'common-mode atom' of queries that the entire market fails on together, and no pairwise statistic can see it.
The study separates two failure metrics that practitioners routinely conflate. Pairwise error correlation describes how often two models in a pool disagree, and a low correlation number is what most teams treat as evidence that a routing layer will work. The co-failure rate describes something different: the share of prompts where every model in the pool fails simultaneously. Mathematically, no router, cascade, or voting scheme can ever exceed the co-failure rate, because once every model is wrong there is nothing left to route, escalate, or vote on. The new study's central empirical claim is that the second number is significantly larger than the first number suggests, and the gap is consistent.
The researchers tested 67 frontier models from 21 providers, including GPT-5.5, Claude Opus 4.8, and Gemini 3.1 Pro, on the open-ended MATH-500 math benchmark. Standard correlation-based statistical models predicted that the entire pool would fail together on only 2.3 percent of the questions. The observed co-failure rate was 5.2 percent, more than double the prediction. On the GPQA graduate-level science benchmark, converting multiple-choice questions to free-response format expanded the all-wrong tail to 12.7 percent, a reminder that the ceiling is not a fixed property of the pool but a function of task structure as well.
The architectural implications are concrete. The three multi-model patterns most teams use are routers, cascades, and Mixture-of-Agents (MoA) ensembles. Each carries a shadow price of added latency, multi-provider operations, and governance surface, which the source frames as paid up front on the assumption that a diversity dividend arrives later. The paper's experiments suggest that for many configurations the dividend does not arrive. When the pool is balanced, diverse ensembles at low pairwise correlation can outperform Self-MoA configurations, but only when the models sit within a matched quality band. When the pool is unbalanced, naive majority voting across diverse but unequal models posted a negative mean gain of minus 10 points on the study's hard mix, as Josef Chen noted in the source. Diverse-but-weaker models outvote the strongest one.
A constraint in the study runs directly against the deployment intuition most teams hold. Multi-model setups help least exactly on the open-ended, hard-to-verify tasks where the perceived benefit of ensembles is largest. Free-form generation, long-form reasoning, and the kind of complex prompts that are usually cited to justify a router in the first place are precisely the tasks where the co-failure ceiling is highest. The paper introduces a useful taxonomy: ceiling-bound environments, where the all-wrong rate is high because the underlying capability is missing, and realizability-bound environments, where at least one model in the pool usually knows the answer but the models agree so subtly that a router cannot pick the right one without an oracle.
The engineering lever the source endorses is not a more sophisticated ensemble design but a more rigorous pre-deployment test. The Clopper-Pearson bound, a classical statistical bound on binomial proportions, can be run on the eval logs teams already produce. If a team tests five models on 50 sample queries and finds two cases where the whole pool fails, naive arithmetic would put the production co-failure rate near 4 percent. The bound, applied to that small sample, says the true rate could be as high as 12 percent. That difference is the budget for whether a router is worth building at all. The source describes the integration as a counting job over existing eval logs, run in CI, re-triggered when the model pool or workload changes.
Two evidence boundaries are worth naming. The study only tested verifiable benchmarks where answers can be checked objectively; it does not cover subjective tasks like marketing copy, and the authors explicitly flag whether the findings hold outside verifiable benchmarks as an open question. The 5.2 percent and 12.7 percent figures are also from specific benchmark conditions and may not transfer to private enterprise workloads without the team running its own Clopper-Pearson calculation.
For tasks with objectively checkable answers, the study's data suggests a single best-performing model usually beats an ensemble, because the orchestration overhead is real while the diversity dividend on checkable problems is small. The lever the source argues buyers actually hold is failure-mode heterogeneity and market churn, not model count. The same co-failure bound that exposes the problem also provides a free pre-deployment test, but only for the verifiable task types the benchmark covers. Whether the ceiling behaves the same way on subjective, unverifiable enterprise workloads is the part the published data does not yet address.