From Monolingual to Multilingual: Evaluating Mamba for ASR in South African Languages

For our experiments and analysis, we use ASR data from three sources, which are:

Swivuriso: [10] is a large speech corpus for South African languages that covers seven languages, including isiNdebele (nbl), Sesotho (sot), Setswana (tsn), Xitsonga (tso), Tshivenda (ven), isiXhosa (xho), and isiZulu (zul). It contains scripted and unscripted speech across domains such as agriculture, healthcare, and general conversation, with over 90 speakers per language. The transcriptions include diacritics, punctuations, special symbols such as [cs] to denote code-switching, [pause] for pause, and [?].

For this study, we define short utterances as speech segments between 0 and 30 seconds, reflecting typical ASR training conditions. Segments longer than this are considered long utterances. We sample 50 hours of short utterances per language to construct a balanced dataset comprising scripted and unscripted speech from the train set. Short utterances from the dev set are used for validation, while the dev_test set is split into short and long subsets for evaluation. Table I summarizes the statistics of the data set. In the test split, short utterances outnumber long utterances by at least a factor of two in most languages, except isiZulu.

NCHLT Speech corpus: [16, 17] is another large speech corpus that covers all 11 official South African languages. It contains more than 100 hours of scripted speech and more than 200 speakers per language. We used only the test splits of the seven languages shared with Swivuriso.

FLEURS: [20] is a multilingual speech corpus that covers 102 languages, created by recording text translated by humans into the respective languages. It includes only four South African languages, of which only isiXhosa and isiZulu were used in this work. We used test splits for both languages.

All audio files were downsampled to 16 kHz. The transcriptions were preprocessed by normalizing them using Unicode Normalization Form C (NFC), removing all punctuation except a few. The retained punctuation includes ?!-́,.;%=+*# and numeric digits. We also removed special tokens we noticed in the data set, including [pause], [um], [cs], and [?], and lowercase all transcriptions.

For our experiments, we use encoder-only ASR models based on Mamba or Conformer architectures, both trained with Connectionist Temporal Classification (CTC) [28], and use character-level modeling. We chose CTC due to its simplicity, computational efficiency, and previous use in related work [12]. To ensure a fair comparison, both architectures share the same configuration, including 18 encoder layers, a hidden size of 512, and a feed-forward dimension of 2048, resulting in comparable model scales (114M parameters for Conformer and 123M parameters for Mamba).

The Conformer follows the standard architecture [4], consisting of relative positional self-attention [29], convolution modules, and feed-forward networks. In contrast, Mamba entirely replaces self-attention with a state-space sequence modeling mechanism. For the Mamba-based system, we adopt ConMamba [30], an architecture designed for speech processing that integrates a bidirectional Mamba module (BiMamba; $d_{\text{state}}=16$, expand=2, $d_{\text{conv}}=4$), a feed-forward network and a convolutional module to efficiently capture local and long-range dependencies.

Our experiments consider both monolingual and multilingual training regimes. In the monolingual setting, we train a separate model for each language using language-specific data and evaluate it in the corresponding language. For the multilingual setting, we evaluate five training strategies by combining monolingual data.

1. 1.

   Multilingual-Implicit (MI): A single model trained on the pooled data from all languages without any explicit language information.

2. 2.

   Multilingual-Implicit Family (MIF): For each language family, one model trained on pooled data from all languages in that family, without using any explicit language-identifying information.

3. 3.

   Multilingual Language Embedding (MLE): Similar to (1), but with the introduction of a learnable language embedding matrix $E^{(\ell)}\in\mathbb{R}^{n_{\text{lang}}\times 32}$. For each input sequence that belongs to language $\ell$, the corresponding embedding is added to the acoustic representations produced by the CNN downsampling module:

   $$\mathbf{h}^{\prime}_{t}=\mathbf{h}_{t}+\mathbf{e}_{\ell},\quad\forall t,$$

   (1)

   where $\mathbf{h}_{t}\in\mathbb{R}^{32}$ is the frame-level acoustic representation and $\mathbf{e}_{\ell}\in\mathbb{R}^{32}$ is the embedding corresponding to language $\ell$. The resulting representations are then fed into the Mamba module. Unlike [31], we do not concatenate language embeddings to acoustic features.

4. 4.

   Multilingual Language-Family Embedding (MLFE): We extend MLE by additionally introducing a language-family embedding matrix $E^{(f)}\in\mathbb{R}^{n_{\text{fam}}\times 32}$. The acoustic representations are augmented as:

   $$\mathbf{h}^{\prime}_{t}=\mathbf{h}_{t}+\mathbf{e}_{\ell}+\mathbf{e}_{f},\quad\forall t,$$

   (2)

   where $\mathbf{e}_{f}$ corresponds to the embedding of the language family.

5. 5.

   Multilingual ASR + LID (M-CTC+LID): A multitask learning setup where a shared encoder is trained jointly with a CTC-based ASR objective and an auxiliary language identification (LID) head. The total loss is:

   $$\mathcal{L}=\mathcal{L}_{\text{CTC}}+\lambda\mathcal{L}_{\text{LID}},$$

   (3)

   where $\lambda$ controls the contribution of the LID objective.¹¹1$\lambda$ was set at 0.1 after ablation over 0.05, 0.1, 0.2, 0.3.

All models were trained with three seeds for 250 (monolingual) or 100 (multilingual) epochs on NVIDIA A100 GPUs (40GB or 80GB, with 80GB used for Conformer models). Training used bfloat16 mixed-precision, a batch size of 32, and identical optimization settings following a publicly available SpeechBrain-based recipe [32].²²2https://github.com/mattmireles/Mamba-ASR We used the AdamW optimizer tuning the learning rate over 1e-3, 8e-4, 5e-4, 2e-4, 1e-4. All models were trained using a vocabulary of 189 characters derived from the Swivuriso combined training split. Although the target South African languages were not expected to require such a large character inventory, we observed that the dataset includes multiple characters with diacritics, as well as additional non-standard symbols, likely due to portions of the data being sourced from the web (Wikipedia). These include, among others, characters such as ß and Greek letters (e.g. $\sigma$ and $\lambda$). For evaluation, we averaged the weights of the five best checkpoints per seed and report the mean performance over three seeds, with punctuation removed and Word Error Rate (WER) as the metric.

Table II presents the monolingual WER comparison between Conformer and Mamba models on both short and long utterances. All models were trained on short utterances and evaluated on the same language. The in-domain evaluation result shows comparable performance across languages for Conformer and ConMamba: on average, ConMamba achieves a 33.43% WER, while Conformer achieves 35.49%. Despite comparable accuracy, ConMamba, with nearly 10M additional parameters, is more efficient in training, inference, and memory usage [6]. In our experiments, Mamba required approximately 18 hours of training per language, compared to 34 hours per language for Conformer. Due to higher memory requirements, Conformer could not be trained on A100 40GB GPUs and therefore required A100 80GB GPUs, whereas Mamba could be trained on A100 40GB GPUs. Furthermore, the results indicate that the Nguni languages (nbl, xho, and zul) consistently yield the lowest performance across both models, followed by tso, a related language. In contrast, ven and tsn achieve the strongest results across both models. All monolingual models were trained under identical experimental conditions, making data imbalance an unlikely explanation. We hypothesize that this pattern may be linked to the complex agglutinative morphology of Nguni languages. Furthermore, not including English in our training could affect performance, especially in code-switching scenarios where English is often used [15]. More analysis is needed to better understand these challenges and improve performance.

Evaluation on long utterances outside the training length distribution shows that WER is similar to or slightly higher than for short utterances in most languages, except xho, where it is lower. The average WER difference of $\sim 2.5\%$ indicates comparable length generalization across models, with similar degradation trends likely due to the CTC decoder. In  [33] it was shown that incorporating Mamba in encoder-decoder architectures helps reduce degradation for long-form ASR, and we investigate if the same holds true for encoder-only ASR.

To investigate this, we performed an ablation experiment by sampling and concatenating dev_test utterances of up to 60s, inserting 0.12 s of silence between them, to create sequences spanning progressively longer ranges (i.e. 60-90s, 90-120s, …, 210-240s). Scripted and unscripted speech from the same speaker was kept separate and concatenation was restricted to utterances from that speaker.

Figure 1 shows the performance degradation of the monolingual models for nbl, tsn, ven, and zul, using inference on the original short utterances (0–30 s) as baselines. We observe that both architectures experience increased degradation as the utterance duration grows; however, ConMamba exhibits slightly less degradation overall compared to Conformer. Neither model is fully robust to longer utterances, and the degradation patterns appear similar for both architectures. Future work should evaluate these architectures on authentic long utterances to better assess their robustness as we already observe degradation when simply concatenating short utterances into ranges of at least 30s.

Thus far, we have shown that Mamba matches Conformer across all seven languages while being more computationally efficient, although both models degrade on long-form ASR. We evaluate Mamba in the multilingual setting by comparing training strategies and assessing cross-corpus generalization.