Conversational-agent benchmark
τ³-bench
Following policy, using tools and retrieving knowledge through a conversation.
Key information
- Focus
- Customer-service agents in simulated business domains, including knowledge retrieval and voice.
- Measure
- Task success and repeat-run reliability; pass^k is not pass@k.
- Version / scope
- τ³-bench · tau2-bench v1.0.1 grading update
- Update pattern
- Task, grader and domain releases; submitted results can require regrading.
- Publisher update
- Not consistently disclosed
- Editorial verification
- 15 October 2026
In plain English
What makes it different?
Sierra's τ-bench family tests agents that talk with users while acting through tools under a business policy. The τ³ release adds knowledge-intensive and voice settings. A pleasant answer is not enough if the agent leaves the wrong state or breaks the task's rules.
Sources and attribution are listed below.
What it measures
Customer-service agents in simulated business domains, including knowledge retrieval and voice.
Strengths
- Combines conversation with verifiable actions.
- Repeated runs expose reliability gaps hidden by a single successful attempt.
Limitations
- A simulated customer and its prompts are not a representative sample of real support traffic.
- User simulator, retrieval pipeline and grader changes can alter the difficulty.
How the evaluation works
A simulated user and the agent interact in a domain with policies and tools. The evaluator checks whether the intended outcome was reached.
Repeated trials can measure consistent success. Text, full-duplex voice and knowledge-retrieval domains exercise different behaviours and belong on separate result views.
Configuration matters
Pin domain, task split, simulator, retrieval tools, model effort and voice mode. In banking_knowledge, results before v1.0.1 are not comparable with the corrected grader according to the maintainer.
Primary methods: Official repository and version warning ↗
Results at the publisher
Scores stay with the original source
We have not documented the permissions needed to reproduce this leaderboard here. You can still explore its methods and visit the publisher for results.
Versions & freshness
Verified 15 October 2026The official repository announces a July 2026 v1.0.1 banking grading fix and says affected submissions were regraded. The package remains named tau2; the name alone does not identify the benchmark generation.
Official leaderboard and current Sierra repository accessible. Task, grader and domain releases; submitted results can require regrading.
Our review date is not an evaluation date. Individual run dates are not consistently available; they remain unknown here. No automatic result collection is enabled.
Publication & reuse
The repository's MIT software licence is explicit. No separate scoped permission for reusing the hosted leaderboard and all submitted scores is documented.
Do not infer hosted-result or voice-asset rights from the package's MIT licence.
View the scope of the reuse decision
- Result fields
- No result fields cleared
- Public result display
- not enabled
- Storage
- not enabled
- Historical copies
- not enabled
- Updates
- not enabled
- Commercial reuse
- not enabled
- Derived ratings
- No scoped grant documented; not enabled.
- Attribution
- Credit the named publisher and link the primary sources. Attribution alone does not grant reproduction rights.
This decision does not clear mixed-source scientific recommendations. Publisher content, software, datasets and model licences are separate. No partnership is implied.
Primary sources & attribution
This guide is original scor.es commentary about Sierra Research's work. The links below support the identity, methods, versions and reuse review.
- Official τ-bench website ↗Checked 15 October 2026
- Official repository and version warning ↗Checked 15 October 2026
- Repository MIT licence ↗Checked 15 October 2026
Editorial edition 2026-10-15.1 · reuse review directory-rights-1. Re-check due 14 November 2026.