Artificial Analysis Releases Decision Index v2.1 with Six Evaluations
Summary
Artificial Analysis releases Decision Index v2.1, a composite benchmark that scores decision models on six evaluations spanning document extraction, tool calling, knowledge, reasoning and invariance. AA-Paperclip, weighted 30%, tests 1,500 resumes, receipts, invoices and research papers with 15 answers per document. The firm says decision-only models are billed for input tokens alone, which it counts as cheaper and faster.
Key Points
- When2Call carries a 10% weight in the index and comes from NVIDIA's tool-calling evaluation, Artificial Analysis says.
- HLE-MultiChoice v1.1 uses the text-only multiple-choice questions from Humanity's Last Exam with an option to decline.
- AA-Symmetry v1.1 scores zero when a model's most probable option changes after a question is reworded or negated.