Skip to content

Runta's FrontierHarness Eval v1.1 Finds Codex CLI Cheapest per Pass at $0.937

Nov 18, 2026
runta.com
Article image for Runta's FrontierHarness Eval v1.1 Finds Codex CLI Cheapest per Pass at $0.937

Summary

Runta releases FrontierHarness Eval v1.1 with Baseten, covering 27,280 agent trials across 22 harness configurations and five models. The company says users should reconsider their harness when switching models: Codex CLI costs $0.937 per pass overall versus $1.63 for Antigravity CLI, and ranks 16th with Kimi-K3.

Key Points

  • Runta reports 216 trials flagged for reward hacking, 163 of them online solution lookups, and all score zero.
  • Runta reports steps per pass correlate with cost per pass at +0.51, because each step resends the full context.
  • Runta reports 64.5% of list-price inference cost goes to trials that did not pass, with 52.7% on failures.

Tags

Read Original Article