Soham Ray

Hey, I'm Soham.

I'm a research engineer at Sierra with over seven years in ML research, specializing in task-oriented dialogue. I lead voice-agent benchmarking — from ASR evaluation (μ-Bench) to audio-native agents (τ-voice) — and earlier co-built τ²-Bench, our text-agent benchmark. All three are used by leading labs (OpenAI, Google, Anthropic, xAI, Moonshot, Deepgram, ...) to measure modelling progress in complex, real-world conditions. I've been fortunate to learn from Dr. Victor Barres and Dr. Karthik Narasimhan (Princeton) along the way.

Prior to Sierra, I spent three and a half years at ASAPP under Dr. Kilian Weinberger (Cornell) and Dr. SRK Branavan, applying LLMs and NLP to customer service problems and shipping research into production. I hold a master's in Computer Science from Cornell.

I excel in environments with high agency, quick iteration, and fast adoption.

I'm married to the inspiring Aashna Banerjee, and we live in San Francisco. If you're ever in San Francisco and up for a boba or a game of squash, let me know.

Highlights

Jul 2026

Presented the τ-benchmark suite at ICML 2026 (Seoul) — τ²-Bench as an Oral (top 0.7% of ~24,000 submissions), τ-Voice and τ-Knowledge as posters.

May 13, 2026

Featured in OpenAI's Build Hour: GPT-Realtime-2, presenting τ-voice findings.

May 1, 2026

Released τ-voice, the first benchmark for full-duplex voice agents on real customer-service tasks.

Apr 23, 2026

xAI announced Grok Voice Think Fast 1.0 — top spot on the τ-voice leaderboard (418M views).

Apr 20, 2026

Released μ-Bench, an open multilingual transcription benchmark across 5 locales.

Aug 7, 2025

GPT-5 launched by OpenAI, with τ²-Bench featured in the announcement; scored 96.7% on τ²-Bench telecom.

Jun 10, 2025

Published τ²-bench, the first dual-control benchmark for collaborative agent tasks (arXiv).

News

★ highlight

Featured Talk

Presenting τ-voice findings at OpenAI's Build Hour: GPT-Realtime-2 · May 2026

Code, Publications & Patents

1. τ²-bench — open-source benchmark for tool-agent-user interaction across retail, airline, telecom, and voice domains. GitHub · ★ 1.5k stars
2. μ-Bench — open multilingual transcription benchmark across 5 locales, 4,270 human-annotated utterances. GitHub · ★ 9 stars
3. τ-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains. Soham Ray, Keshav Dhandhania, Victor Barres, Karthik Narasimhan. ICML 2026 (Poster). Cited by 5
4. τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment. Victor Barres, Honghua Dong, Soham Ray, Xujie Si, Karthik Narasimhan. ICML 2026 (Oral). Cited by 310
5. Sample-Efficient Diffusion for Text-to-Speech Synthesis. Justin Lovelace, Soham Ray, Kwangyoun Kim, Kilian Q. Weinberger, Felix Wu. Interspeech, 2024. Cited by 11
6. Automated key-value extraction using natural language intents. Soham Ray, Volkan Cirik, Kilian Q. Weinberger, Satchuthananthavale R. K. Branavan, Paloma Sodhi. US Patent 12,591,603, 2026.
7. SAGA: A system for conversational AI agents to learn from live agent supervision. Kilian Q. Weinberger, Soham Ray, Satchuthananthavale R. K. Branavan, Paloma Sodhi. US patent application, 2024 patent pending.
8. Text-to-audio conversion with byte-encoding vectors. Justin R. Lovelace, Soham Ray, Felix Wu, Kilian Q. Weinberger, Kwangyoun Kim. US Patent 12,542,127, 2026.

Writing

How we think about τ-bench X · June 29, 2026

A Tau Day essay, three versions in: a benchmark is a strong opinion about what matters, made measurable — τ's opinion is production realism in task-oriented dialogue.

Voice AI is only as good as what it hears Sierra · May 18, 2026

Sierra's transcription stack dynamically routes across providers, supports 70+ languages, and ensembles outputs to cut utterance error rate by ~25%.

τ-voice: benchmarking real-time voice agents on real-world tasks Sierra · May 1, 2026

The first benchmark to combine grounded customer-service tasks with full-duplex voice under realistic conditions — diverse accents, noise, and turn-taking.

μ-Bench: an open multilingual transcription benchmark Sierra · April 20, 2026

Open multilingual transcription benchmark across 5 locales, with a new metric (UER) that isolates meaning-changing errors from surface-level ones.

τ³-Bench: advancing agent benchmarking to knowledge and voice Sierra · March 18, 2026

Expands τ-bench to two new frontiers — knowledge retrieval over 698 policy documents, and live full-duplex voice with realistic audio conditions.

Multilingual voice: building agents that speak to everyone Sierra · October 8, 2025

How Sierra builds voice agents across 34+ languages and dialects — combining human evaluation, automated benchmarking, and local language experts.

τ²-bench: benchmarking agents in collaborative real-world scenarios Sierra · June 10, 2025

A dual-control benchmark where users and agents share the action space. Top LLMs drop up to 25 points moving from solo to interactive mode.

Education & Experience

education work