原文整理页

Artificial Analysis 更新了编程智能体指数,采用 DeepSWE 基准测试替代了易受污染的 SWE-Bench Pro,Claude Fable 5 登顶

来源作者:Peter Steinberger 🦞 (@steipete)原始来源:https://x.com/steipete/status/2065560834462670884

中文导读

Artificial Analysis 更新了编程智能体指数,采用 DeepSWE 基准测试替代了易受污染的 SWE-Bench Pro,Claude Fable 5 登顶。

正文 Markdown

We've updated the Artificial Analysis Coding Agent Index, replacing SWE-Bench Pro with Datacurve's DeepSWE benchmark - the swap lifts Codex with GPT-5.5 (xhigh) above Claude Code with Opus 4.8 (max), while the newly released Claude Fable 5 (max) in Claude Code debuts at the top DeepSWE, built by @datacurve, writes its tasks from scratch rather than adapting them from public GitHub issues or pull requests, so no model has seen the solutions during training. That matters because SWE-Bench Pro, the benchmark it replaces in our Coding Agent Index, had grown gameable, with some models recovering the fix from the repository's commit history instead of solving the task. The swap reorders the index: Codex with GPT-5.5 (xhigh) rises from 65 to 76, overtaking Claude Code with Opus 4.8 (max) at 73. Claude Code with Fable 5 (max), which enters directly on the refreshed index, leads at 77. SWE-Bench Pro had been flattering some combinations and penalizing others. More below.