Soham Ray

Soham Ray

AI Research, Sierra

· Resume · GitHub · Google Scholar · LinkedIn · X

Hey, I'm Soham.

I'm a research engineer at Sierra with over seven years in ML research, specializing in task-oriented dialogue. I lead voice-agent benchmarking — from ASR evaluation (μ-Bench) to audio-native agents (τ-voice) — and earlier co-built τ²-Bench, our text-agent benchmark. All three are used by leading labs (OpenAI, Google, Anthropic, xAI, Moonshot, Deepgram, ...) to measure modelling progress in complex, real-world conditions. Most recently, I launched τ-i18n, which takes τ-voice to six languages, and τ-Entity, which tests whether voice agents get names, numbers, and addresses exactly right. I've been fortunate to learn from Dr. Victor Barres and Dr. Karthik Narasimhan (Princeton) along the way.

Prior to Sierra, I spent three and a half years at ASAPP under Dr. Kilian Weinberger (Cornell) and Dr. SRK Branavan, applying LLMs and NLP to customer service problems and shipping research into production. I hold a master's in Computer Science from Cornell.

I excel in environments with high agency, quick iteration, and fast adoption.

I'm married to the inspiring Aashna Banerjee, and we live in San Francisco. If you're ever in San Francisco and up for a boba or a game of squash, let me know.

Highlights

ICML 2026
Jul 2026

Presented the τ-benchmark suite at ICML 2026 (Seoul) — τ²-Bench as an Oral (top 0.7% of ~24,000 submissions), τ-Voice and τ-Knowledge as posters.

τ-voice
May 1, 2026

Released τ-voice, the first benchmark for full-duplex voice agents on real customer-service tasks.

μ-Bench
Apr 20, 2026

Released μ-Bench, an open multilingual transcription benchmark across 5 locales.

GPT-5 launch
Aug 7, 2025

GPT-5 launched by OpenAI, with τ²-Bench featured in the announcement; scored 96.7% on τ²-Bench telecom.

τ²-bench launch
Jun 10, 2025

Published τ²-bench, the first dual-control benchmark for collaborative agent tasks (arXiv).

News

★ highlight

Featured Talks

The next generation of voice AI with Google DeepMind · Google for Developers · September 2026

Google DeepMind

Talk on the τ-bench family at Google DeepMind · September 2026

Stanford AIMS Seminar: the τ-bench family and τ-voice — Introducing 'ticks' slide

Guest lecture on the τ-bench family & τ-voice at the Stanford AIMS Seminar · July 2026

Presenting τ-voice findings at OpenAI's Build Hour: GPT-Realtime-2 · May 2026

Code, Publications & Patents

  1. τ-bench — open-source benchmark for tool-agent-user interaction across retail, airline, telecom, banking, and voice domains; 2.0 adds 6 languages (τ-i18n) and entity capture (τ-Entity). GitHub · ★ 2.3k stars
  2. μ-Bench — open multilingual transcription benchmark across 5 locales, 4,270 human-annotated utterances. GitHub · ★ 9 stars
  3. τ-Multilingual: Benchmarking Voice Agents Across Languages. Soham Ray, Edgard dos Santos Paiva, Ruben Valenzuela, Karthik Narasimhan, Keshav Dhandhania, Victor Barres. Under review at ICASSP 2027. Released as τ-i18n.
  4. τ-Elicitation: Benchmarking Multi-Turn Entity Extraction in Voice Agents. Soham Ray, Victor Barres. Under review at ICASSP 2027. Released as τ-Entity. Cited by 1
  5. τ-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains. Soham Ray, Keshav Dhandhania, Victor Barres, Karthik Narasimhan. ICML 2026 (Poster). Cited by 38
  6. τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment. Victor Barres, Honghua Dong, Soham Ray, Xujie Si, Karthik Narasimhan. ICML 2026 (Oral). Cited by 552
  7. Sample-Efficient Diffusion for Text-to-Speech Synthesis. Justin Lovelace, Soham Ray, Kwangyoun Kim, Kilian Q. Weinberger, Felix Wu. Interspeech, 2024. Cited by 11
  8. Evaluating an Artificial Intelligence Agent in a Voice Conversation. Soham Ray, Keshav Dhandhania, Victor Barres, Karthik Narasimhan. US patent application 19/808,428, 2026 patent pending.
  9. Automated key-value extraction using natural language intents. Soham Ray, Volkan Cirik, Kilian Q. Weinberger, Satchuthananthavale R. K. Branavan, Paloma Sodhi. US Patent 12,591,603, 2026.
  10. SAGA: A system for conversational AI agents to learn from live agent supervision. Kilian Q. Weinberger, Soham Ray, Satchuthananthavale R. K. Branavan, Paloma Sodhi. US patent application, 2024 patent pending.
  11. Text-to-audio conversion with byte-encoding vectors. Justin R. Lovelace, Soham Ray, Felix Wu, Kilian Q. Weinberger, Kwangyoun Kim. US Patent 12,542,127, 2026.

Writing

τ-Entity: can voice agents get the details right?
τ-Entity: can voice agents get the details right? Sierra · October 22, 2026

200 voice tasks across ten entity types — names, phone numbers, addresses, emails. The best agent captures the value in 60% of calls, but gets it right under every condition on only 34% of tasks.

How we think about τ-bench
How we think about τ-bench X · June 29, 2026

A Tau Day essay, three versions in: a benchmark is a strong opinion about what matters, made measurable — τ's opinion is production realism in task-oriented dialogue.

Voice AI is only as good as what it hears
Voice AI is only as good as what it hears Sierra · May 18, 2026

Sierra's transcription stack dynamically routes across providers, supports 70+ languages, and ensembles outputs to cut utterance error rate by ~25%.

Education & Experience

  1. 2003 — 2013 School St Mary's School Pune, India
  2. 2013 — 2015 Higher Secondary Fergusson College Pune, India
  3. 2015 — 2019 B.E., Information Technology University of Pune Pune, India
  4. 2019 — 2020 MEng, Computer Science Cornell University Ithaca, NY
  5. 2020 — 2021 Data Scientist Afiniti Washington DC
  6. 2021 — 2025 Senior Research Engineer ASAPP New York
  7. 2025 — Now Senior Research Engineer Sierra San Francisco