track 05 · Technical research
τ-bench: studying tool-agent-user evaluation
A fork of Sierra's τ-bench, which scores language agents on dynamic conversations with a simulated user, domain tools and policy documents, and reports pass^k reliability.
StudyFork · studied, not originalAgent evaluationTool-callingBenchmarksPython
The question
Single-run accuracy hides how unreliable agents are. τ-bench's pass^k metric asks whether an agent succeeds on the same task k times in a row.
Caveat. Not original work: this is a fork kept to study and run the benchmark. Credit belongs to its authors.