Simulating realistic speech overlaps improves multi-talker ASR

The token-level serialized output training (t-SOT) was proposed to generate transcriptions of multi-talker overlapping speech in a streaming fashion [15]. In the t-SOT framework, the maximum number of active speakers at the same time is assumed to be $M$. In our work, we used the t-SOT with $M=2$ for its simplicity, thus explaining the t-SOT with this condition. With t-SOT, the transcriptions of multiple speakers are serialized into a single sequence of recognition tokens (e.g., words, subwords, etc.) by sorting them in chronological order. A special token $\langle\mathrm{cc}\rangle$, indicating a change of virtual output channels, is inserted between two adjacent words spoken by different speakers. A streaming end-to-end ASR model [20] is trained based on pairs of such serialized transcriptions and the corresponding audio samples. During inference, an output sequence including $\langle\mathrm{cc}\rangle$ is generated by the ASR model in a streaming fashion, which is then converted to separate transcriptions based on the recognition of the $\langle\mathrm{cc}\rangle$. The t-SOT model showed superior performance compared to prior multi-talker ASR models even with streaming processing. See [15] for more details.

Prior multi-talker ASR studies used a simple simulation technique to generate training data by mixing single-talker speech signals with random delays [7, 8, 9, 10, 11, 12, 13, 14, 15, 16]. In this work, as a baseline technique, we adopt the procedure used in [15] where multiple single-talker audio samples are mixed with random delay while limiting the maximum number of active speakers at the same time to up to two. Let $S_{0}$ denote a single-talker ASR training set. For each training data simulation, the number of audio signals $\tilde{K}$ to be mixed is first uniformly sampled from 1 to $K$, where $K$ is the maximum number of utterances to be mixed. Then, $\tilde{K}$ utterances are randomly sampled from $S_{0}$. Finally, 2nd to $\tilde{K}$-th utterances are iteratively mixed with the 1st utterance after prepending random silence signals. The duration of the silence prepended for $k$-th utterance is sampled from $\mathcal{U}(\mathrm{end2}(a_{k-1}),\mathrm{len}(a_{k-1}))$, where $\mathcal{U}(\alpha,\beta)$ is the uniform distribution function in the value range of $[\alpha,\beta)$, $a_{k-1}$ is the mixed audio after mixing $(k-1)$-th utterance, $\mathrm{len}(a)$ is a function to return the duration of audio $a$, and $\mathrm{end2}(a)$ is a function to return the end time of the penultimate utterance in $a$.

In this work, we propose to simulate multi-talker speech based on the statistics of speech overlaps in real conversation. The overall workflow of our proposed method is as follows. (1) Convert time- and speaker-annotated transcription of real conversation into a sequence of discrete tokens, with each representing the status of speaker overlap within a short time- or word-based unit. (2) Train SLM $\mathcal{M}$ using the discrete token sequence. (3) Simulate multi-talker speech based on single-talker ASR training data $S_{0}$ and the SLM $\mathcal{M}$. (4) Train multi-talker ASR by using the simulated multi-talker speech.

The core of our proposed method lies in converting the time- and speaker-annotated transcription into a discrete token sequence that represents the speaker overlapping pattern. In this paper, we propose two algorithms for such conversion, named time-based discretization and word-based discretization, which are explained in Section 3.2 and Section 3.3, respectively. Note that our algorithms use the notion of “virtual channel” proposed in t-SOT [15]. In the following explanation, we assume two virtual channels (i.e. $M=2$) for simplicity and consistency with the t-SOT. Our algorithm can be easily extended for $M>2$ by assuming more virtual channels.

In this method, the speech overlapping pattern is represented by a sequence of discrete tokens, where each token indicates the speech overlapping status of a $d$-sec time window. The algorithm is depicted in line 1–17, 26 of Algorithm 1. Suppose we have a time- and speaker-annotated transcription $\mathcal{T}$ of a small amount of real conversation data. We first sort $\mathcal{T}$ based on the end time of each word. We then assign each word a “virtual” channel index $c\in\{0,1\}$. Here, $c=0$ is always assigned to the first (i.e 0-th) word (line 4, 5). For words thereafter, the value of $c$ is changed when the adjacent words are spoken by different speakers, which is the sign of either speech overlap or speaker turn (lines 7–9). Once the list $\mathcal{C}$ of the channel indices is prepared, the transcription is converted to a sequence $\mathcal{X}$ of discrete tokens $x\in\{0,1,2,3\}$, which indicates the speech activity of each virtual channel for a short $d$-sec time window (lines 10–17). More specifically, $x=0$ (or $q={\tiny\begin{bmatrix}0\\ 0\end{bmatrix}}$) indicates no speech activity is observed during the $d$-sec time window. On the other hand, $x=1,2,3$ (or $q={\tiny\begin{bmatrix}1\\ 0\end{bmatrix},\begin{bmatrix}0\\ 1\end{bmatrix},\begin{bmatrix}1\\ 1\end{bmatrix}}$) indicates that the speech activity is observed in 0-th channel, 1-st channel, or both channels during the $d$-sec time window.

SLM $\mathcal{M}$ can be trained based on the derived sequence of discrete tokens, and then used to yield speech overlapping patterns for simulating more realistic multi-talker speech. The algorithm of generating a multi-talker speech sample is described in lines 1–12, 19, 20 of Algorithm 2 and depicted as follows. We first sample a discretized token sequence $\tilde{\mathcal{X}}$ from SLM $\mathcal{M}$ (line 3) based on the time-based discretization. We then apply $\rm Decode$ function that converts $\tilde{\mathcal{X}}$ into a sequence $\tilde{\mathcal{Q}}$, consisting of a list of binary pairs $q\in\{{\tiny\begin{bmatrix}0\\ 0\end{bmatrix},\begin{bmatrix}1\\ 0\end{bmatrix},\begin{bmatrix}0\\ 1\end{bmatrix},\begin{bmatrix}1\\ 1\end{bmatrix}}\}$, where 1 in $c$-th element of $q$ indicates the existence of speech in the $c$-th channel (line 4). Next, a target mixed audio $a$ is initialized with empty (line 5). We then call $\rm ConsectiveOne$ to extract the beginning index $i^{b}$ and ending index $i^{e}$ of the consecutive sequence of $1$ in either 0-th or 1-st channel of $\tilde{\mathcal{Q}}$, in chronological order of $i^{b}$ (line 7). For example, when $\tilde{\mathcal{Q}}=[{\tiny\begin{bmatrix}1\\ 0\end{bmatrix},\begin{bmatrix}1\\ 0\end{bmatrix},\begin{bmatrix}1\\ 1\end{bmatrix},\begin{bmatrix}1\\ 1\end{bmatrix},\begin{bmatrix}0\\ 1\end{bmatrix}}]$, $\rm ConsectiveOne$ returns $i^{b}=0,i^{e}=3$ for the first iteration, and returns $i^{b}=2,i^{e}=4$ for the second iteration. Based on $i^{b}$ and $i^{e}$, we sample a speech segment $u$ from source ASR training data $S_{0}$, with the duration within or nearest to the expected interval [$d^{\rm min}$, $d^{\rm max}$] (lines 9--11).¹¹ 1 To save computation, we created a pool of 10K utterances from $S_{0}$, and used it to sample $u$. When we created the pool, utterances in $S_{0}$ were short-segmented at the point of silence longer than 0.5 sec. We randomly sampled $u$ among samples whose duration is within [$d^{\rm min},d^{\rm max}$]. If there was no such sample, we sample an utterance having closest duration to $(d^{\rm min},d^{\rm max})/2$. A duration $\gamma$ of the silence prepended to $u$ is randomly determined such that $u$ is mixed to the audio $a$ with the expected position from $i^{b}$ (line 12, 19). These procedures are repeated to form the mixed audio $a$, which is returned as the training sample for multi-talker ASR (line 20).

The word-based discretization algorithm differs from the time-based one in a sense that the discretized unit representing the overlapping status is based on “word” rather than a $d$-sec time window. The algorithm of converting $\mathcal{T}$ into the discretized speaker overlapping pattern $\mathcal{X}$ is depicted in lines 1–9, 18–26 of Algorithm 1. The word-based discretization shares the same procedure with the time-based discretization for the first 9 steps, and then performs discretization for each word in $\mathcal{T}$, where $x$ represents the overlapping status of speech overlaps of each token (lines 18–25). Note that $x$ cannot be 0 (or $q$ cannot be $\tiny\begin{bmatrix}0\\ 0\end{bmatrix}$) in this algorithm. Fundamentally, word-based discretization is assumed to be applied to a short segment that does not include long-silence signals. This can be achieved by segmenting the transcription based on the existence of long silence.

The multi-speaker speech generation algorithm with word-based discretization is introduced in Algorithm 2. The first 7 steps are the same as the time-based discretization algorithm except that we create a buffer $D$ to keep the end time of each word of utterances being mixed (line 6). The buffer $D$ is necessary to determine the duration $\gamma$ of silence when we mix $u$ to $a$ with targeted overlaps (lines 16--18).²² 2 We assume $D$ will return 0 for an undefined index. In line 14, we first apply $\rm WordIndices$ function, which returns a list $\mathcal{I}$ of indices indicating where each word should be allocated. This function is necessary because around half of the consecutive sequence of $\tiny\begin{bmatrix}1\\ 1\end{bmatrix}$ represents the words in the other channel. For example, given $\tilde{\mathcal{Q}}=[{\tiny\begin{bmatrix}1\\ 0\end{bmatrix},\begin{bmatrix}1\\ 0\end{bmatrix},\begin{bmatrix}1\\ 1\end{bmatrix},\begin{bmatrix}1\\ 1\end{bmatrix},\begin{bmatrix}0\\ 1\end{bmatrix}}]$, $\rm ConsectiveOne$ first returns $i^{b}=0,i^{e}=3$, and $\rm WordIndices$ returns $[0,1,2]$. For the second iteration, $\rm ConsectiveOne$ returns $i^{b}=2,i^{e}=4$, and $\rm WordIndices$ returns $[3,4]$. We then sample $u$ whose number of words are closest to the length of $\mathcal{I}$ (line 15).³³ 3 We used a pool of 10K utterances from $S_{0}$, and randomly sampled $u$ among samples whose number of words is $\mathrm{len}(\mathcal{I})$. If there was no such sample, we sample an utterance having the closest word count to $\mathrm{len}(\mathcal{I})$. We mix $u$ to $a$ with a silence signal whose duration $\gamma$ is determined such that the word in $u$ is overlapping with $a$ as represented by $\mathcal{Q}$ (lines 16, 19).

Table 1 shows the dataset used in our experiments. As public meeting evaluation sets, we used the AMI meeting corpus [21] and the ICSI meeting corpus [22]. For the AMI corpus, we used the first channel of the microphone array signals, also known as the single distant microphone (SDM) signal. For the ICSI corpus, we used the SDM signal from the D2 microphone. Both corpora were pre-processed and split into training, development, and evaluation sets by using the Kaldi toolkit [23]. In addition to these public corpora, we also used 33 internal meeting recordings based on an SDM, denoted as MS^mtg. For all datasets, we applied causal logarithmic-loop-based automatic gain control to normalize the significant volume differences among different recordings. Evaluation was conducted based on the utterance-group segmentation [17], and speaker-agnostic WER was calculated by using the multiple dimension Levenshtein edit distance [24, 16].

For our proposed multi-talker audio simulation, the AMI training data with official word-level time stamps was used to train the SLM, where we trained N-gram with $N=30$ based on the NLTK toolkit [25]. For the source data ($S_{0}$) of the multi-talker audio simulation, we used 75 thousand (K) hours of 64 million (M) anonymized and transcribed single-talker English utterances, denoted as MS^asr [17]. MS^asr consists of audio signals from various domains, such as dictation and voice commands, and each audio was supposed to contain single-talker speech. However, we found it occasionally contained untranscribed background human speech, which broke the assumption for the multi-talker audio simulation. Therefore, we filtered out the audio sample that potentially contains more than two speaker audio. To detect such audio samples, we applied serialized output training-based multi-talker ASR pre-trained by MS^asr and fine-tuned by AMI [17] for all audio samples in MS^asr with a beam size of 1. With this procedure, transcriptions of more than one speaker were generated for 14M utterances out of 64M utterances, which were excluded in the multi-talker audio simulation. The effect of this data filtering is examined in Section 4.3.1.

As an instance of multi-talker ASR, we trained t-SOT based transformer transducer (TT) [26] with chunk-wise look-ahead [27], where the algorithmic latency of the model can be controlled based on the chunk size of the attention mask. We trained TT models with 18 and 36 layers of transformer encoders, which were denoted by TT18 and TT36, respectively. Each transformer block consisted of a 512-dim multi-head attention with 8 heads and a 2048-dim point-wise feed-forward layer. The prediction network consisted of two layers of 1024-dim long short-term memory. 4,000 word pieces plus blank and $\langle\mathrm{cc}\rangle$ tokens were used as the recognition units. We used 80-dim log mel-filterbank extracted for every 10 msec.

All models were first pre-trained by using the multi-talker simulation data based on MS^asr with the random simulation algorithm with $K=2$. We performed 425K training steps with 32 GPUs, with each GPU processing a mini-batch of 24K frames. A linear decay learning rate schedule with a peak learning rate of 1.5e-3 after 25K warm-up iterations were used. After the pre-training, the model was further fine-tuned by using the simulation data based on MS^asr and/or AMI & ICSI training data. We performed various fine-tuning configurations, which will be presented in the next section.