Skip to content

Stripe Says Agent Plugin Lifts Scores but Strict Pass Rate Falls to 29.4%

Nov 20, 2026
stripe.dev
Article image for Stripe Says Agent Plugin Lifts Scores but Strict Pass Rate Falls to 29.4%

Summary

Stripe releases an evaluation of 69 coding-agent scenarios for building Stripe integrations, run across 14 model and harness pairs. Stripe says its agent plugin lifted scores by 3 to 8 percentage points for every pair, yet a strict pass rate requiring all criteria fell to 29.4% from 78.1%.

Key Points

  • Stripe says web searches fell 96% per eval with the plugin, as models turned to its documentation tools.
  • Stripe reports outdated parameter references fall to 1.3% of API call parameters from 7% with the plugin.
  • Stripe publishes the Forge eval harness and plans to add cases and refresh results over time.

Tags

Read Original Article