AI Enthusiast Log

Sign in Subscribe

Topic

benchmark

A collection of 5 issues

Credit card payment of $67.53 for a self-pickup order with a $66.85 discount, completed on 2023-12-26.

A new study evaluating 67 frontier models from 21 providers has measured something the multi-model orchestration approach has long assumed away: a shared failure mode that the standard diversity metric cannot detect. The paper's authors call it the co-failure ceiling, and on the MATH-500 benchmark

A new benchmark finds policy gradients beat game-theory algorithms

A benchmark presented at ICLR by MIT researchers and collaborators finds that policy gradient methods, a general-purpose family of reinforcement learning tools that date to the early 1990s, can outperform specialized game-theory algorithms in two-player zero-sum imperfect-information games. The result, the source reports, is less

Kimi K2.7-Code's token savings are real; benchmark claims aren't

Moonshot AI released Kimi K2.7-Code as an open-source drop-in upgrade to its K2 coding family, claiming a 30% reduction in thinking tokens and double-digit gains on three of its own benchmarks. The token savings are immediately usable for teams running agentic workloads; the capability claims

ALE: GPT-5.5 edges Claude Fable 5 by 2 points; most tasks still fail

The new Agents' Last Exam (ALE) benchmark from UC Berkeley's Center for Responsible, Decentralized Intelligence (RDI) crowns OpenAI's gpt-5-5 with a 24.0% pass rate, with Anthropic's just-released claude-fable-5 sitting in third place at 22.0%. The article

Harness-1 edges GPT-5.4 on search recall by externalizing state, not scale

Harness-1, a 20-billion-parameter open-source search agent, scored 73% on average recall across eight search benchmarks, edging GPT-5.4's 70.9%. The 2.1-point margin is narrower than the source's "massive leap" framing suggests, and the comparison carries a