SCB: SpeechConversationBench for Evaluating Multi-Turn Reasoning in Speech-to-Speech Model

VoiceBench evaluates voice assistants across content, speaker, and environmental variations [1], while Dynamic-SUPERB organizes diverse instruction-based speech tasks [6]. Multi-turn speech evaluation is also an established research direction. MTalk-Bench combines pairwise and rubric-based assessment across semantic, paralinguistic, and ambient-sound dimensions [4]. Audio MultiChallenge evaluates natural spoken interactions involving memory, instruction retention, self-coherence, and spoken repairs [5]. SpeechConversationBench complements these broader evaluations with a narrower comparison: numerical task completion under three presentations of corresponding problem information. It does not measure their full range of acoustic or interactional capabilities.

We adapt the Full/Concat/Sharded protocol of Laban et al. [7] to spoken mathematical tasks. LoCoMo instead evaluates memory over extended, multi-session conversations [9]; our setting concerns integrating the facts of one problem within an interaction.

Moshi models full-duplex speech-text dialogue [3], while MemGPT manages information across memory tiers [10]. Context-aware decoding strengthens adherence to spoken history [8]. These approaches motivate our comparison of five speech systems, including the context-managing cascade LEGO; they do not validate LEGO itself.

GPT-4o Realtime improves from 70.0% in Full to 76.6% in Sharded, even though it falls from 81.6% in Concat. A Full-only comparison would therefore conceal its disadvantage relative to the concatenated-shard baseline. More generally, the Concat-minus-Full changes are $+11.6$, $+2.2$, $+7.7$, and $-5.0$ pp for the four commercial systems, respectively. Both single-turn controls are therefore informative; degradation is not universal across baseline choices.

Gemini 2.5 Flash Preview Native Audio Dialog has the highest Full accuracy, 85.0%, but achieves 55.0% in Sharded. GPT-4o Realtime has lower Full accuracy yet the strongest Sharded result among the commercial systems. Selecting a system by single-turn accuracy can therefore favor a different system than selecting by multi-turn accuracy.

LEGO obtains 77.5% in all three conditions and the highest observed Sharded score, exceeding GPT-4o Realtime by 0.9 pp. Its zero aggregate change makes explicit context management worth investigating, but does not show that summarization caused the result. The systems differ in their components, and no memory ablation is available. Equal aggregate scores could also conceal problems that become correct and others that become incorrect. We therefore interpret LEGO as a promising system-level observation, not proof that a cascade is generally superior to native audio modeling.

The observed multi-turn gaps do not identify a failure mechanism. A wrong numerical answer can arise from misperception, loss or misuse of earlier information, arithmetic error, or answer rendering. Without intermediate transcripts and matched interventions, these explanations cannot be separated. The table establishes neither forgetting nor monotonic degradation with turn count.

This small mathematical evaluation does not measure open-domain or multilingual dialogue, interruptions, prosody, noise robustness, latency, speech naturalness, or memory-management cost. The aggregate record does not specify exact model snapshots, audio-generation settings, sampling parameters, per-condition denominators, repeat counts, or answer-extraction and scoring details. LEGO’s component identities and memory-update implementation are also unspecified. These omissions limit reproduction and statistical interpretation.

A matched text-only baseline is needed to quantify an audio-specific penalty. To test whether LEGO’s context mechanism is responsible for its observed stability, the informative comparison would hold recognition, reasoning, and synthesis components fixed while varying context summarization. Problem-level outcomes and repeated runs would then support paired comparisons and uncertainty estimates. Such evidence would distinguish improvements in remembering information from improvements in using it, a distinction also raised by work on context-aware decoding [8]. Broader spoken benchmarks remain necessary to assess whether gains in numerical correctness extend to natural interaction.

SpeechConversationBench compares spoken mathematical task completion across original, consolidated, and incrementally disclosed inputs. All four commercial systems lose accuracy from Concat to Sharded, whereas LEGO preserves its reported aggregate accuracy. The strongest supported lesson is methodological: retain both single-turn controls and report absolute task accuracy alongside the multi-turn change.