The Interlingua Hypothesis: LLMs Translate
via a Latent Task-agnostic Feature Space

Machine translation has long been an influential area of natural language processing. The attention mechanism was first proposed to improve machine translation performance (Bahdanau et al., 2015; Luong et al., 2015; Vaswani et al., 2017); one consequence of attention was the emergence of language models that could be efficiently trained on massive corpora (Devlin et al., 2019; Radford et al., 2018). Recently, large language models (LLMs) have revolutionized many areas of natural language processing, but have been relatively slowly adopted in machine translation (MT).

A reason for the slow adoption of LLMs in MT has been a focus on low-resource languages, where LLMs face significant challenges (Hendy et al., 2023; Robinson et al., 2023). Because LLMs require large training corpora, they are difficult to train effectively on low-resource languages. However, they also hold significant promise: even when not trained on a given language, LLMs can be prompted to translate from or into a language they have seen very little of in their training data (Cahyawijaya et al., 2024)—e.g., by prompting with a grammar book (Tanzer et al., 2024). Moreover, their representations of high-resource languages could enable cross-lingual transfer (Conneau et al., 2020).

Whether LLMs can deliver on this promise depends in part on the mechanisms underlying how they translate. Specifically, do LLMs rely on task- and language-agnostic mechanisms to translate, or do they rely on more specialized translation mechanisms or language pair–specific mechanisms? If the former, this suggests clear paths toward improving MT performance, potentially without large parallel corpora. Much recent work provides mechanistic evidence supporting the existence of massively multilingual representations in language modeling contexts (Wendler et al., 2024; Brinkmann et al., 2025; Wu et al., 2025).

We therefore hypothesize that LLMs perform MT in large part by reusing computational mechanisms for monolingual language modeling. The “stages of inference” hypothesis (Lad et al., 2024) holds that LLMs devote the first half of their layers to reading an input and composing increasingly abstract concept representations. Then, the latter half of the model reads these concept representations, and uses them to decide which token should be predicted given prior context. If language models perform translation this way (what we call the interlingua hypothesis), then translation would not necessarily require language pair–specific mechanisms; instead, it only requires that a model be capable of reading abstract features from the source language, and generating text for the target language conditioned on those features.

If this hypothesis is true, it would entail the following predictions: (1) Given language pair $(S,T)$, translation performance should be predictable from monolingual capabilities in $S$ and $T$ in isolation without cross-linguistic interaction terms. (2) There should exist task-agnostic representations that are causally relevant for predicting correct outputs in both machine translation and monolingual task settings. Finally, (3) adding translation capabilities for a new language should primarily require improvements to monolingual capabilities for that language---i.e., fine-tuning on monolingual corpora should recover a substantial proportion of the improvement in translation performance as fine-tuning on parallel corpora, assuming the model can already effectively handle the other language in the language pair.¹¹ 1 Note that our claims relate to the mechanisms a fully trained LLM uses to perform translation. We do not make claims regarding what is contained in the pretraining data; it is possible that parallel pretraining data is necessary for multilingual representations to be learned.

We investigate each of these three implications, and find positive evidence for each. While no single experiment definitively confirms the interlingua hypothesis, each provides a different type of evidence in support of it. These findings could provide a preliminary explanation for the value of exposure to (largely monolingual) documents in producing models more effective at translation.

How much of a model’s ability to translate can be explained by its monolingual language modeling capabilities? If a model uses an interlingua to perform translation, then much of its translation capabilities should be explainable as a function of its ability to parse latent features from an input, and produce coherent text in that language. In other words, one should not need language pair–specific terms to explain translation performance, unless a model deploys some separate translation mechanism that does not involve going through an interlingua. To investigate, we define a linear model that predicts translation performance from monolingual competencies.

We use several measures of monolingual competence. First, to measure a model’s ability to distinguish grammatical from ungrammatical sentences, we use MultiBLiMP (Jumelet et al., 2026), a multilingual version of the BLiMP (Warstadt et al., 2020) benchmark. For a given language, MultiBLiMP accuracy is the fraction of minimal pairs for which the model assigns higher full-sentence log-probability to the grammatical completion over its ungrammatical counterpart. We average this over 200 randomly-selected samples per language. Because Llama and Aya saturate this benchmark for many of the languages in our analysis, we also compute the log-probability margin, which is the mean $\log p(\text{grammatical})-\log p(\text{nongrammatical})$ per minimal pair. To overcome issues related to benchmark saturation, we also include accuracy on GlobalMMLU (Singh et al., 2025), a multilingual multiple-choice question answering dataset based on MMLU (Hendrycks et al., 2021). For each task, we filter the set of languages to those in common between the monolingual dataset and FLORES (Table 1).²² 2 The following languages are supported across all datasets we use in this section: ara, ces, deu, ell, eng, fas, fra, heb, hin, ita, nld, pol, por, ron, rus, spa, tur, ukr

For translation, we have language pair $(S,T)$, where $S$ is the source and $T$ is the target language. Translation competence $t_{ST}$ is quantified as a BLEU score for $(S,T)$. To perform translation, we use a 2-shot prompt containing two randomly sampled sentences from FLORES.³³ 3 We ensure that the examples do not overlap with the test example. We use sacrebleu (Post, 2018) to compute BLEUs.

Let $l$ be a monolingual competence estimate using MultiBLiMP or GlobalMMLU. We have $l_{S}$ and $l_{T}$ for each language pair. We then learn coefficients $\beta_{S}$ and $\beta_{T}$ as well as bias term $\beta_{0}$ to maximize predictive accuracy on BLEU scores $t_{ST}$ in the following function:

We also train a bilinear model that is nearly identical, but also contains a multiplicative interaction term:

If the interlingua hypothesis holds, then the bilinear model should not have significantly greater predictive power than the linear model. This would imply that machine translation performance is better explained by monolingual terms, rather than by the existence of a translation mechanism (which should use terms specific to particular language pairs).

Our previous results show that monolingual capabilities are predictors of translation performance, but also that language pair interactions are not strong predictors. This provides correlational evidence in support of our hypothesis, but not causal evidence. To obtain causal evidence, we now perform an analysis based on causal mediation analysis (Pearl, 2001; Vig et al., 2020).

Past work has found that language models use the first half of their layers to form progressively more abstract representations of the latent features in a given input (Lad et al., 2024). We hypothesize that language models could repurpose this language modeling machinery to perform translation by reading the source language into a latent feature space, and then using the latter half of its layers to read from this feature space and produce the translation in the output language. If this is true, then we should be able to find model representations or components that are causally influential in both machine translation and language modeling settings. To obtain causal evidence in support of this view, we now perform mechanistic experiments by patching model components. We show that the most influential attention heads for producing correct translations also strongly mediate a language model’s ability to produce grammatical outputs in monolingual settings.

If LLMs translate via task-agnostic internal representations, translation quality should depend in part on the model’s monolingual competence in the source and target languages. For example, if we wish to translate between a low-resource and high-resource language, then we should be able to improve translation performance given access only to data that improves language modeling quality on the low-resource language, assuming that we preserve capabilities in the high-resource language. Under this view, low translation performance may reflect a failure to encode the source sentence into an adequate internal representation, or to decode the target sentence fluently from it. Recent work has shown that bilingual or mixed-language signals during pretraining can be important for acquiring translation capabilities in LLMs (Briakou et al., 2023; Qorib et al., 2025; Shao et al., 2026). We ask the complementary question of whether, given a fully pretrained model, improvements in monolingual language modeling can (at least in part) transfer to translation even without parallel data.

We test this idea using Llama-3.1-8B and TinyAya-3B.⁵⁵ 5 We use TinyAya instead of Aya-23-8B because Aya-23-8B only provides an instruction-tuned model, and no base model. In pilot experiments, we found it difficult to achieve good performance after fine-tuning in any language (including high-resource languages) with instruction-tuned models. For each model, we compare how fine-tuning on an underrepresented language (Xhosa) affects translation performance when the training data consist of either parallel data or monolingual text. Specifically, in the monolingual setting, we train on unaligned text consisting of 80% Xhosa and 20% English, French, and German data to avoid catastrophic forgetting of the model’s existing language abilities. In the parallel setting, we use paired Xhosa and English sentences from OPUS MT560 (Gowda et al., 2021), presented in both translation directions.

Both settings use a next-token prediction loss and rank-16 LoRA adapters (Hu et al., 2022). The reported runs use a learning rate of $3\times 10^{-5}$, a maximum sequence length of 512 tokens, and one epoch over 100 million tokens. We evaluate translation with few-shot prompting on the FLORES devtest split (Goyal et al., 2022), using examples from the dev split as in-context demonstrations. Thus, the models are prompted to translate at evaluation time even though the monolingual condition contains no translation examples during training.

Table 3 shows that monolingual fine-tuning recovers most of the improvement obtained with parallel data. For Llama, monolingual fine-tuning raises BLEU from 16.84 to 24.81, compared with 24.88 under parallel fine-tuning. This recovers 99% of the improvement over the base model. For Aya, monolingual fine-tuning raises BLEU from 23.46 to 26.64, compared with 27.80 under parallel fine-tuning, recovering 73% of the improvement. For both models, both fine-tuning conditions largely preserve translation performance from French and German into English, although monolingual fine-tuning remains closer to the base performance.

These results are consistent with the view that once an LLM has been pretrained and acquired multilingual representations, improving its ability to model an underrepresented language can produce most of the available translation gains. Parallel data may still perform better because it can improve language modeling while also strengthening translation-specific mechanisms. Our results therefore do not imply that parallel data are unimportant, particularly during pretraining. They show that, after pretraining, a large share of the gains from parallel fine-tuning can be achieved using only unaligned data.

We provide results for additional languages (German and Thai) and translation directions in Appendix C. In short, changes in performance are smaller for higher-resource languages, but parallel and monolingual fine-tuning still achieve largely comparable results.

We have provided evidence that machine translation capabilities in LLMs are mediated in significant part by multilingual representations that are also relevant for some monolingual tasks. Across three complementary analyses, we find evidence consistent with this view: Translation performance is largely explained by source- and target-language competence, translation-relevant components overlap with components used for monolingual grammatical behavior, and monolingual Xhosa fine-tuning recovers most of the gains from parallel fine-tuning across both models. Together, these results provide support for the hypothesis that LLMs translate by mapping source-language input into a latent feature space (that can also in theory be used for monolingual next-token predictions), and then generating target-language text from that shared representation.

This interpretation helps explain why monolingual competence is strongly associated with translation performance. If a model translates through a partially language-agnostic feature space, then improving its ability to read or write a language should improve translation involving that language. The observation that target-language competence explains more variance than source-language competence may suggest that fluent and grammatical generation is often the bottleneck. Under this view, poor translation into a language need not imply the absence of a dedicated translation circuit for that language pair; instead, it may reflect weak target-language decoding capabilities given otherwise usable latent representations of the source sentence.

We emphasize that translation-specific mechanisms not based on interlingua are also likely to exist; indeed, Todd et al. (2024) find that there are components selective for word translation. We do not claim that an interlingua is the only means by which LLMs translate, but rather, that it is a significant mechanism that controls a large proportion of model performance on MT tasks. It is likely that LLMs use a mixture of mechanisms (Gur-Arieh et al., 2026) to achieve their machine translation capabilities.

The interlingua hypothesis is one plausible explanation for what we have observed, but our experiments do not rule out the existence of other mechanisms. The finding that the target language yields more predictive power than the source language in predicting BLEU score may be an artifact of the $n$-gram-based computation of BLEU scores. It is also possible that the features underlying high-quality translations are not well represented in the BLEU score.

Our experiments were limited to two 8B-parameter models and one 3B-parameter model, and may not generalize to smaller or larger LLMs. Future work should investigate whether similar trends hold for a wider variety of language models, and whether recent developments in thinking models affect these findings.

For our GCM experiments, we focus on components that mediate at the last token, missing computations that happen at earlier positions.

Finally, our translation experiments are limited to a relatively small number of language pairs. Future work could scale up this experiment to investigate whether these trends hold across a large number of language pairs, and to what degree monolingual fine-tuning can recover the performance of parallel fine-tuning across many language pairs (and what other factors explain when this works well and when it does not).

We are grateful to the members of the BAAIGL lab at Boston University for helpful comments on an earlier iteration of this work. The computational work reported on in this paper was performed largely on the Shared Computing Cluster, which is administered by Boston University’s Research Computing Services. Jannik Brinkmann is supported by the German Federal Ministry for Economic Affairs and the German Federal Ministry of Research, Technology and Space.