TurnNat: Automatic Evaluation of Turn-Taking Naturalness in Dyadic Spoken Dialogue

Turn-taking prediction provides the modeling basis for TurnNat. Skantze [29] reviews turn-taking as a continuous interactional process involving shifts, holds, pauses, overlap, interruptions, and backchannels. Earlier computational work models local turn-taking decisions, such as whether a pause should lead to a hold or a speaker shift [26, 20, 14]. Skantze [28] further formulates turn-taking as continuous future speech-activity prediction. TurnGPT predicts turn shifts from linguistic context [7]. VAP predicts future two-speaker voice activity from speech and provides the probabilistic prediction target most directly related to our scorer [8]. Subsequent VAP-style work extends this target to multilingual, prompt-guided, real-time, and multimodal settings [11, 10, 12, 23]. Other recent models combine acoustic and language-model features for turn-taking and backchannel prediction [17, 30]. DualTurn learns dyadic conversational dynamics through dual-channel generative speech pretraining [21]. Taken together, these studies show that turn-taking prediction models can learn temporally structured expectations about two-speaker conversational behavior. Our work repurposes these models for evaluation, using future-activity likelihood as an automatic signal for turn-taking naturalness.

Prior evaluation of conversational timing often relies on human listening tests or user studies. Roddy and Harte [22] model dialogue response timing and show that perceived timing naturalness depends on dialogue context. Full-Duplex-Bench evaluates full-duplex spoken dialogue systems with behavior-specific diagnostics such as pause handling, backchanneling, smooth turn-taking, and interruption management [16]. Its follow-up benchmark focuses on overlap-heavy scenarios such as interruptions, backchannels, side conversations, and ambient speech [15]. Talking Turns trains a supervised event-decision judge and uses event-specific thresholds to evaluate whether a system speaks, continues, backchannels, interrupts, or yields at appropriate moments [2]. These benchmarks provide useful behavioral diagnostics, but they evaluate separate interactional behaviors or event decisions. TurnNat instead scores heterogeneous timing failures in a shared continuous space using the future two-speaker voice-activity likelihood.

We study automatic evaluation of turn-taking naturalness in two-channel spoken dialogue. Let $x=(x^{(1)},x^{(2)})$ denote a dialogue segment, where $x^{(1)}$ and $x^{(2)}$ correspond to the audio channels of speakers 1 and 2, respectively.

Let $f_{\theta}$ denote a turn-taking prediction model with parameters $\theta$. For each frame $t$, the model uses the available dialogue context $x_{\leq t}$ to estimate the likelihood of the observed future two-speaker voice-activity state $c_{t}$. TurnNat converts this likelihood into frame-level negative log-likelihood values and aggregates them into a dialogue-level turn-taking naturalness score $m_{\theta}(x)\in\mathbb{R}$, where higher values indicate more natural turn-taking. TurnNat does not require human judgments, perturbation labels, or manually annotated turn-taking events at inference time. Fig. 1 shows the overall TurnNat framework in detail.

TurnNat extracts TBUs from a cleaned two-speaker VAD sequence. We first identify speech islands for each speaker and merge adjacent islands from the same speaker when the gap is at most 1.0s, and the other speaker’s active ratio in the gap is at most 0.2. After merging, islands shorter than 200ms are discarded to remove spurious fragments while retaining brief feedback events.

For each retained utterance candidate, we define two TBUs: one associated with the utterance onset and one associated with the utterance offset. For a boundary time $\tau_{j}$, the corresponding TBU $u_{j}$ contains frames in the pre-boundary interval $[\tau_{j}-L,\tau_{j}]$, where $L=2\mathrm{s}$. For a dialogue segment $x$, we extract all TBUs $\mathcal{U}(x)=\{u_{j}\}_{j=1}^{J}$, where ${J}$ is the total amount. Let $\mathcal{T}(u_{j})$ denote the set of frames contained in unit $u_{j}$. We define the set of all TBU frames as follows:

$$\mathcal{T}_{\mathrm{TBU}}(x)=\bigcup_{u_{j}\in\mathcal{U}(x)}\mathcal{T}(u_{j}).$$

(1)

Although the scoring frames are selected before the boundary, each frame is later evaluated through a two-speaker voice-activity prediction over a future horizon. Thus, a TBU covers the local turn-taking region around an onset or offset boundary, including possible responses, gaps, overlaps, holds, or backchannels following the boundary.

denote the future voice-activity pattern after frame $t$. For speaker $s$ and future bin $k$, $b^{(s)}_{t,k}=1$ if more than $50\%$ of the frames in that bin are active according to the VAD, and $b^{(s)}_{t,k}=0$ otherwise. Each possible binary pattern $\mathbf{b}_{t}$ is indexed as one categorical future voice-activity state:

$$c_{t}=\mathrm{index}({\mathbf{b}_{t}}),\quad c_{t}\in\{1,\ldots,2^{2K}\}.$$

(3)

With $K=4$, this yields $2^{8}=256$ possible joint future voice-activity states.

Setting $\alpha=1$ gives the standard uniformly weighted objective, while $\alpha>1$ emphasizes frames inside TBUs.

As depicted in Fig. 1, TurnNat converts the future-activity likelihood into frame-level NLL during evaluation time:

$$\ell_{\theta}(t;x)=-\log\mathbf{p}_{\theta}(t;x)[c_{t}].$$

(7)

A larger $\ell_{\theta}(t;x)$ indicates that the observed future two-speaker activity is less typical under natural conversational dynamics.

For each TBU $u_{j}$, TurnNat computes the unit-level NLL:

$$s_{\theta}(u_{j})=\frac{1}{|\mathcal{T}(u_{j})|}\sum_{t\in\mathcal{T}(u_{j})}\ell_{\theta}(t;x).$$

(8)

Given the $J$ TBU scores $\{s_{\theta}(u_{j})\}_{j=1}^{J}$, TurnNat aggregates them using mean and tail terms:

$$\begin{gathered}\mathrm{MeanNLL}_{\theta}(x)=\frac{1}{J}\sum_{j=1}^{J}s_{\theta}(u_{j}),\\ \mathrm{TailNLL}_{\theta}(x)=\mathrm{AvgTop}_{\rho}(\{s_{\theta}(u_{j})\}_{j=1}^{J}),\end{gathered}$$

(9)

where $\mathrm{AvgTop}_{\rho}(\cdot)$ averages the top fraction of highest-NLL TBUs, so that strongly unnatural local turn-taking events are not diluted by the global mean. We use $\rho=0.25$ in all experiments. The final dialogue-level naturalness score is:

$$m_{\theta}(x)=-\left[\lambda\mathrm{MeanNLL}_{\theta}(x)+(1-\lambda)\mathrm{TailNLL}_{\theta}(x)\right],$$

(10)

where $\lambda=0.5$, giving equal weight to mean and tail evidence. The negative sign converts NLL-oriented atypicality into naturalness, so higher $m_{\theta}(x)$ indicates more natural turn-taking.

We use English two-channel dialogue recordings from the naturalistic portion of the Seamless Interaction dataset [1]. Following the released train, development, and test partitions, we select controlled subsets and focus on ordinary prompted dyadic conversations based on the Interpersonal Circumplex framework. We exclude task-oriented interaction types, such as collaborative storytelling, or charades, because their turn-taking patterns are more strongly constrained by the task format than by open-ended conversational dynamics.

Table I summarizes the resulting speaker-disjoint splits. The training split is used only to train future-activity prediction models, the development split is used for model selection, and the held-out test split is used as the source pool for constructing the paired perturbation benchmark.

We sample 20–25s dialogue clips from the held-out test split. Candidate regions are identified using the Silero voice activity detector [27], with transcript text used only for auditing and backchannel identification. Each crop is centered around a turn-taking event targeted by one of the perturbation types, and its start and end points are selected from regions where both speakers are silent to avoid truncating ongoing speech. After basic audio and VAD quality control, we randomly downsample 200 natural–perturbed pairs per perturbation type, yielding 1,000 pairs for automatic evaluation.

The perturbation magnitudes are chosen to be clearly outside typical human–human response timing while remaining plausible for listening evaluation. Prior work reports that many human responses begin within a few hundred milliseconds of the previous turn ending, for example, from about -200 to 400ms [24]. We therefore use larger timing shifts to create controlled degradations rather than borderline timing variation.

For hold and shift perturbations, we select clean candidate regions using the VAP shift/hold criterion [8]: a mutual-silence interval is bounded by a single active speaker in a 1s pre-offset region and a single active speaker in a 1s post-onset region. If the active speaker is before and after the silence differs, the region is treated as a speaker shift; if the speaker is the same, it is treated as a speaker hold. Backchannels are selected using the transcript-based criterion of Wang et al. [30]: isolated one- or two-word listener responses, such as “yeah” or “mmhmm”, during the other speaker’s turn.

The benchmark covers five localized timing and floor-management perturbations as follows:

• •

  Late response: We delay a responding utterance by 1.2–2.0s, creating an abnormally long response gap.

• •

  Early entry: We advance a responding utterance by 1.2–2.5s, creating premature overlap with the current speaker.

• •

  Hold instead of shift: We start from a clean speaker-shift region and remove the responding turn, so the original speaker appears to retain the floor instead of shifting it.

• •

  Shift instead of hold: We start from a clean speaker-hold region and insert a complete turn from the other speaker, so the dialogue appears to shift speakers where the original speaker would normally continue.

• •

  Excessive backchanneling: We insert two to three additional listener-feedback events into regions where the listener is originally silent, using naturally occurring backchannels from the same speaker and conversation.

Human judgments are collected to assess the perceptual validity of the perturbation benchmark. For validation, we sample a balanced subset of 150 natural–perturbed pairs, with 30 pairs from each perturbation type. Each pair is rated by three independent native English-speaking annotators.

In each trial, annotators listen to the two clips in randomized order, choose which clip has more natural turn-taking (A/B test), and rate each clip on a five-point scale of naturalness and artifact scales. The resulting judgments are analyzed using the human evaluation metrics described in Section V-A.

For human validation, we report two A/B preference statistics at different aggregation levels. Natural preference rate is computed over individual annotations, as the fraction of judgments choosing the original non-perturbed clip as having more natural turn-taking. Majority-natural pair rate is computed after majority voting within each pair, as the fraction of pairs where the natural clip wins the majority vote.

For five-point ratings, we report the mean naturalness-rating difference between natural and perturbed clips, where positive values favor natural clips. We also report mean artifact ratings and the perturbed-minus-natural artifact difference to check whether audible artifacts confound naturalness judgments.

We compare TurnNat scorers instantiated with VAP and DualTurn backbones. The comparison includes released checkpoints and fully fine-tuned variants, using each backbone’s native future-activity target or the shared 256-way categorical target used by TurnNat.

For comparison with the 256-way categorical TurnNat scorer, we evaluate two DualTurn scoring forms. The first uses DualTurn’s native eight Bernoulli future-activity targets, defined by 4 horizon bins per speaker at 0.24, 0.48, 0.96, and 2.00 s. The second replaces this output with the shared 256-way categorical head used by TurnNat.

For each natural–perturbed pair $(x_{i}^{\mathrm{nat}},x_{i}^{\mathrm{pert}})$, an automatic metric produces dialogue-level naturalness scores $m_{\theta}(x_{i}^{\mathrm{nat}})$ and $m_{\theta}(x_{i}^{\mathrm{pert}})$, where higher values indicate more natural turn-taking. We define the paired score difference as

$$\Delta m_{\theta,i}=m_{\theta}(x_{i}^{\mathrm{nat}})-m_{\theta}(x_{i}^{\mathrm{pert}}).$$

(11)

A positive $\Delta m_{\theta,i}$ indicates that the natural clip receives a higher naturalness score than its perturbed counterpart.

As shown in Table II, human judgments consistently favor natural turn-taking. Annotators prefer the natural clip in 70.0% (95% Wilson CI: 65.6–74.1) of individual pairwise judgments. After majority voting within each pair, the natural clip is preferred in 78.0% of pairs (95% Wilson CI: 70.7–83.9). Natural clips also receive higher five-point naturalness ratings by 0.678 points on average. A participant-level paired bootstrap confirms that the naturalness-rating difference is significantly larger than the artifact-rating difference by 0.389 points (95% CI: [0.138, 0.651]), suggesting that the observed preference pattern is not primarily driven by waveform artifacts.