Exa Says No Search Agent Under $1 Per Task Gets Half Rows Right
Summary
Exa previews ATLAS, a 547-task benchmark that grades search agents on wide, multi-hop research. Exa says its best tested system reaches 0.66 row F1 at $8.92 and 18 minutes per task, while no configuration under $1 per task gets half the rows correct. Exa plans to release the tasks, golden tables, and grader in coming weeks.
Key Points
- Exa says 74% of ATLAS tasks ask agents to find 10 or more entities.
- Exa reports grading one system across all 547 tasks costs roughly $0.24.
- Exa says it plans to regenerate ATLAS from fresh search demand every 3 to 6 months.