Taste-Bench Launches 502-Question Test of AI Agents’ Long-Horizon Decision-Making
Summary
Taste-Bench launches a 502-question benchmark for long-horizon AI-agent decisions, with GPT-5.6 Sol leading at 59.7% under a paired-order test that requires correct choices in both option orders.
Key Points
- Taste-Bench launches a 502-question benchmark for testing whether LLM agents choose the better next step at real long-horizon software-engineering and research decision forks.
- GPT-5.6 Sol leads the August 2026 paired-order leaderboard at 59.7% accuracy; every question must be answered correctly in both option orders, making random guessing score 25%.
- The benchmark filters 4,657 mined forks down to 502 and gates its Hugging Face dataset to limit training contamination; a full evaluation uses 1,004 requests and about 8 million input tokens.