VoiceLongMemEval: Do Assistants Remember How You Sounded?

Each instance additively extends a LongMemEval-compatible record (Wu et al., 2025), preserving compatibility with existing tooling. Every user turn has five annotations: an emotion from 12 everyday labels spanning the valence–arousal plane (Russell, 1980) (neutral, happy, excited, content, sad, disappointed, anxious, frustrated, angry, embarrassed, bored, affectionate); a categorical prosody tuple covering rate, pitch, loudness, pauses, and emphasized words that must appear verbatim in the turn; voice events drawn from five reliably synthesized nonverbals (laughs, sighs, coughs, clears_throat, gasps); pragmatic flags for sarcasm and uncertainty; and a free-text delivery description. Descriptions must be acoustic-only: what a microphone captures, not an interpretation. A lexical gate rejects emotion names, inflections, and $\sim$60 interpretive glosses (relieved, wry, sarcastic, …); for example, “quick and light, laughs mid-sentence” passes, whereas “relieved” does not. Because descriptions enter the model’s text input, interpretive labels would reduce the task to string matching.

The layer has three renders: blind (transcript only, byte-identical to the original), descriptive (transcript plus acoustic stage directions; a structured tagged variant also ships), and audio (Section 5.6). The blind render is both the control and the adversary’s view in Section 3.3.

An LLM authored 4–12-turn evidence sessions ($\leq$26 evidence turns per instance) in 14 themed batches (two pilots, twelve 16-item batches). A protocol¹¹ 1 Themes (work, home, health, travel, money, community, …) partition topics and persona names; collision scans ensure that no lexical topic recurs across instances. enforced four validator-checked invariants: (i) lexical flatness (needles read as neutral logistics), (ii) affect against pragmatics (when possible, affect opposes the event’s prior), (iii) question neutrality (functional, valence-free questions), and (iv) annotation uniformity (every user turn is fully annotated, so annotation presence cannot reveal the needle).

For an instance with $k$ evidence sessions, a deterministic seeded assembler adds 40 topic-screened LongMemEval filler sessions with synthetic neutral annotations, plus $6+2(k{-}1)$ emotive, answer-free distractors from a disjoint 48-session pool; the distractor budget scales with $k$ to prevent emotive-density shortcuts. Needle positions are stratified (early/middle/late), and multi-evidence arcs are distributed over time. The corpus is released in oracle (evidence only, ${\leq}1$k tokens) and full ($\sim$100k tokens) regimes, mirroring LongMemEval${}_{\textsc{s}}$.

The main risk is lexical leakage: if an item is solvable from text alone, it is not measuring paralinguistic memory. We audit each taxonomy item three ways. G1 (blind-unsolvable): an adversary answers a blind render and an LLM judge applies a type-specific rubric with the lexical-only answer as an explicit trap (Zheng et al., 2023); any correct blind answer fails. G2 (aware-solvable): the same model answers a descriptive render; we report results by solver strength (not gating), since failures may reflect model limits. G3 (surface-clean): static checks for interpretive terms, stock phrases, and valence presuppositions.

We iterated with a 7B judge–adversary, then gated with Qwen2.5-72B-Instruct-AWQ, requiring two consecutive clean runs on a frozen file to reduce nondeterminism. The 72B blind adversary solved 8 items (7.5%) that passed the 7B gate. Post-mortems identified five leak mechanisms—pragmatic-prior leakage, outcome tells, gold-matches-prior, default-recovery priors, and A/B gifts—now a checklist; later batches had zero authoring-time leaks. We rerun the terminal gate on the assembled corpus, since date/order shifts can affect marginal verdicts. Final: 0 of 175 non-abstention taxonomy items are blind-solvable, G3 flags none, and the 72B aware-solve rate is 57.9%. Derived families (Section 3.4) are not separately blind-attacked; they rely on gated evidence sessions and a mechanical invariant check. Corpus probes add two controls: ranking sessions by emotive-annotation density finds the needle in 13.1% (top-1; random 3.3%), under the 15% budget, and a session-length probe scores 0.

We generate two-speaker evidence-session clips with Dia (1.6B) (Nari Labs, 2025). Because Dia has no emotion control, each clip is audio-prompted with a trimmed, peak-normalized RAVDESS reference (Livingstone and Russo, 2018) for the target (“needle”) emotion (fixed 12$\to$8 mapping). The reference transcript becomes the first [S1] line, then the dialogue alternates [S1]/[S2] over a typically six-turn, needle-centered window; when possible, we start on an assistant turn to preserve alternation after the reference. Sampling uses guidance_scale 3.0, temperature 1.8,²² 2 This is Dia’s native temperature. Lowering it for “stability” breaks voice cloning under classifier-free guidance and yields silence; pinned per-clip seeds provide reproducibility. top-$p$ 0.9, and top-$k$ 45. The manifest records all parameters, seeds, and references.

We initially used a Whisper-large-v3 speech-emotion-recognition (SER) gate, but moved it to an advisory check after it reached only $\sim$30% on acted RAVDESS and showed similar per-emotion patterns across TTS backends, consistent with cross-corpus SER bias (Schuller et al., 2010). Quality control is now a human listen-through; each annotator records pass/fail in append-only sidecar logs. The human annotator passed 91/104 clips (87.5%); failed clips will be regenerated. All audio is machine-generated (no real recordings), with timbre cloned from acted RAVDESS references.

Each test case places target evidence within $n_{d}=5$ randomly sampled distractor sessions, creating a context window of roughly 10k-15k tokens. Models process the context followed by the query to generate free-text responses, which an LLM judge evaluates against ground-truth answers using task-specific rubrics. We run the experiments for three random seeds, controlling distractor selection and arrangement, and report performance as mean accuracy $\pm$ standard deviation. We primarily compare two input formats: blind (plain transcripts without non-verbal metadata) and descriptive (transcripts enriched with natural-language stage directions detailing vocal delivery). Additional ablations in (Section 5.2) isolate individual metadata components.

We evaluated for three different question-set conditions: Nuanced 181Q, Original 202Q and Indirect 140Q. All pairwise comparisons use paired bootstrap resampling (Efron et al., 2000) (10,000 iterations) and McNemar’s test (McNemar, 1947) for matched-pair binary outcomes. We report $p$-values; all reported gaps are significant at $p<0.001$.

We presents our core findings in  Table 2 and show a positive and a statistically significant affect gap between blind and descriptive conditions. Across all the models, we observe a consistently positive affect gap, indicating that the effect generalizes to both proprietary and open-weight systems. Moreover, the magnitude of this gap increases with model capability, rising from $+0.089$ for Gemma 3-12B to $+0.383$ for Opus 4.8. Importantly, comparing performances between Llama 4 Maverick (${\sim}$400B MoE) and Llama 3.3-70B, show that the size of the model alone doesn’t inform about the model’s performance. Given that blind accuracy is uniformly low (0.09–0.18) across models, one might hypothesize that the affect gap is merely an artifact of overall capability. However, normalizing by headroom recovered, $\Delta/(1-\text{blind})$, preserves the same ranking between models. The uniformly low blind accuracy further supports the interpretation that the adversarial gates effectively remove items that can be solved via text alone.

A per-type analysis ( Table 3) shows that the affect gap is maximal for question types in which delivery most directly encodes the correct response, and minimal for types requiring integration across multiple sessions. The resulting type-level ordering is consistent across both models, suggesting that the associated difficulty hierarchy is inherent to the question types rather than contingent on model-specific behavior.

To understand which paralinguistic cues drive the affect gap, we evaluate Claude Opus 4.8 and GPT-5.5 on nine render conditions ( Table 4). Several findings emerge, consistent across both models: Models genuinely use metadata. The wrong-metadata condition (Opus: 0.228, GPT: 0.183) is barely above blind (0.193, 0.129), confirming that models do not simply benefit from the presence of metadata annotations; they read and use the content.

Emotion labels are the single most informative cue. Emotion-only (Opus: 0.614, GPT: 0.535) surpasses the full descriptive condition (0.589, 0.475) in both models, despite containing far less information. Explicit categorical labels are easier for models to integrate into reasoning than free-text acoustic descriptions.

Structured tags outperform natural language. The tagged condition (Opus: 0.757, GPT: 0.624) exceeds descriptive by +0.168 (Opus) and +0.149 (GPT), indicating that frontier models extract paralinguistic information more reliably from structured formats.

CoT helps but cannot compensate. Chain-of-thought prompting (Wei et al., 2022) without metadata (cot-blind: 0.302, 0.203) improves over blind but falls far short of any metadata-equipped condition. Adding CoT to descriptive input (cot-descriptive: 0.767, 0.668) yields the best overall accuracy.

Prosody and events provide partial signal. Events-only and prosody-only each exceed blind substantially, but neither alone approaches the performance of emotion labels. The ranking of conditions is identical across both models, suggesting the hierarchy of cue informativeness is model-independent.

We evaluate three question-set conditions, from explicit paralinguistic cues to fully natural phrasing in Table 5. The nuanced set, whose questions explicitly reference voice, tone, or delivery, produces the largest affect gap (+0.61 to +0.69). The indirect set (140 items, fully natural, open-ended) shows the smallest gap (+0.11 to +0.18) indicating that models struggle to connect natural questions to paralinguistic evidence. The final row previews the prompting result detailed in Section 5.4 showing that adding a retrieval-time hint nearly triples the indirect gap.

The indirect result poses a practical question: if models have paralinguistic metadata in context but fail to attend to it, can a simple prompt intervention close the gap? We test this by prepending a single instruction to the descriptive condition: “When answering, consider not just what was said but how it was said.” Table 6 shows that a retrieval-time hint substantially improves accuracy on natural questions across all eight models. Critically, the hint also lifts the blind condition: on indirect v1, hint-on-blind raises Opus from $0.169$ to $0.557{\pm 0.031}$ ($+0.388$) and GPT-5.5 from $0.143$ to $0.536{\pm 0.026}$ ($+0.393$), exceeding even unprompted descriptive ($0.305$, $0.285$). This reveals that prompting and annotation are partially interchangeable: prompting for affective reasoning recovers much of the signal that metadata provides.

To determine whether this reflects genuine reasoning or judge reward hacking, we run three controls: (1) Scrambled context: hint with wrong evidence sessions collapses to $0.043$, ruling out plausible made-up guessing. (2) Cross-judge: re-judging hint-on-blind outputs with GPT-5.5 yields $0.600$ (vs. $0.536$ with Opus 4.5), ruling out self-preference bias. (3) Wrong-metadata + hint: randomized annotations with the hint score $0.564$, comparable to blind+hint ($0.536$), confirming that the hint operates on conversational content rather than annotation content. The tightest estimate of metadata’s content contribution comes from comparing descriptive+hint ($0.631$) against wrong-metadata+hint ($0.564$): a clean $+0.067$, clean by annotation presence or prompt effects.

Crucially, the interchangeability is question-set-dependent. On the adversarially-gated Original 202Q (Opus, seed=42), the hint lifts blind only modestly ($0.188\rightarrow 0.267$, $+0.079$), and the affect gap grows under the hint (descriptive+hint $0.733$ minus blind+hint $0.267$ = $+0.465$, vs. $+0.376$ without hint). On the LLM-generated indirect v1 (3-seed means), the hint lifts blind dramatically ($0.169\rightarrow 0.557$, $+0.388$), and the gap narrows to $+0.074$ for Opus and $+0.026$ for GPT-5.5. This divergence reflects the adversarial gates: 202Q items were authored to resist text-only reasoning, making metadata genuinely irreplaceable; indirect v1 items, generated without such gates, are more amenable to general affective reasoning. The headline affect gap on the gated benchmark is robust to prompting. This result also serves as an empirical validation of the adversarial gates themselves: gated items (202Q) resist the strongest known prompting attack ($+0.079$ blind hint lift), while ungated items (indirect v1) do not ($+0.367$). The observed leak is confined to the LLM-generated question sets that were not subjected to the adversarial gates The core 202Q benchmark, which passed these gates, remains robust to the same prompting intervention.

Table 7 shows that increasing distractors from 3 to 10 mildly reduces descriptive accuracy across all model types, but blind accuracy remains flat. The affect gap persists at all scales for both frontier and open-weight models, confirming that the benchmark’s difficulty is not an artifact of haystack size.

To measure the paralinguistic-memory deficit of cascaded pipelines, we synthesize the 114 indirect v2 evidence sessions with Dia TTS (Nari Labs, 2025), conditioned on emotion-matched RAVDESS reference clips (Livingstone and Russo, 2018) across four speaker voices. We evaluate two audio-native models (Qwen2-Audio-7B (Chu et al., 2024), Qwen2.5-Omni-7B) under four conditions each, against a cascade (Whisper large-v3 $\rightarrow$ Opus 4.8 or GPT-5.5) on the same clips (Table 8). All conditions are regime-matched (evidence-only); text baselines use all 114 items, while audio and cascade rows use the 104 with valid TTS output.

Three findings emerge. First, audio-native models hear paralinguistic cues: Qwen2-Audio ($0.354$) and Qwen2.5-Omni ($0.412$) outperform the blind text baseline ($0.325$), and supplementary metadata lifts both further ($0.509$, $0.541$), approaching the descriptive upper bound ($0.675$). Second, the cascade loses this signal: Whisper $\rightarrow$ Opus 4.8 scores $0.254$, below both 7B audio-native models and even the blind baseline, despite a frontier-scale capability advantage. The loss has two sources: Whisper model strips all delivery cues (transcript analysis finds no bracketed voice events, fillers, or other paralinguistic markers in any of the 104 clips) and introduces content errors relative to the ground-truth transcript. Third, the hint compensates for cascade loss, raising Opus from $0.254$ to $0.515$ ($+0.261$) and GPT-5.5 from $0.468$ to $0.552$ ($+0.084$); per Section 5.4, this reflects general affective-reasoning gains rather than recovery of delivery cues, since comparable lifts appear on blind text.