Subword Segmental BabyLMs:
Learning to Tokenise for Sample-Efficient Pretraining

Without being motivated by developmental plausibility, several works have explored learning to segment language inputs during training, for tasks such as handwriting recognition (Kong et al., 2016), speech recognition (Wang et al., 2017), and translation (Kreutzer and Sokolov, 2018). These efforts equip models with the ability to discover optimal segmentation units for a given task, rather than relying on a pre-determined segmentation scheme.

Sun and Deng (2018) propose the segmental language model (SLM): an LSTM LM that computes the likelihood of a sentence by marginalising over all possible segmentations of the character sequence. Their aim is not improved performance, but rather unsupervised word segmentation for Chinese, which lacks explicit word boundaries. Kawakami et al. (2019) augment the SLM with a lexicon of high-frequency segments, improving unsupervised word segmentation. Downey et al. (2022) propose the masked SLM, a bi-directional, transformer-based SLM that outperforms recurrent SLMs on unsupervised word discovery. SLMs are trained on raw character sequences without word boundaries (whitespaces) and, as a by-product of optimising the LM training objective, they partially recover word-level segmentation.

Meyer and Buys (2022) propose the subword segmental language model (SSLM), which adapts the SLM to model subword tokenisation. SSLM assumes access to word boundaries, constrains segments to subword units (they cannot cross word boundaries), and learns subword tokenisation to optimise the LM training objective. The original SSLM (Meyer and Buys, 2022) is LSTM-based (Hochreiter and Schmidhuber, 1997). Meyer and Buys (2025) propose a transformer (Vaswani et al., 2017) SSLM, primarily as a tool to study subword learning dynamics. For both models, evaluation is limited to the Nguni languages: low-resource, agglutinative languages, for which tokenisation was hypothesised to have an outsized impact. SSLM outperformed tokenisation-based LMs on perplexity and sequence-to-sequence tasks, and performed strongly as an unsupervised morpheme segmenter.

It is unknown whether learnable subword tokenisation can improve pretraining sample-efficiency beyond the narrow linguistic scope of previous work. The BabyLM Challenge provides the ideal setting to test this. We propose two new SSLM variants, SubSegGPT and SubSegDeBERTa, which incorporate learnable subword tokenisation into modern, BabyLM-style architectures. Understanding our models requires familiarity with the subword segmental framework, so in the next subsection we provide a technical overview of SSLM.

SSLM (Meyer and Buys, 2022) establishes a framework for learning tokenisation during training. Any LM architecture can be adapted to the framework, removing its reliance on a fixed tokeniser. The key idea is that tokenisation is cast as a latent variable, inferred jointly with model parameters to optimise the LM objective. Adapting a model for subword segmental modelling requires augmenting its architecture with additional subnetworks, which we describe for our models in Section 3, and adopting the training algorithm of Meyer and Buys (2022), which we now summarise.

For a training sequence $S$, SSLM still minimises the standard LM loss $\mathcal{L}(\theta)=-\log p(S)$. However, whereas a vanilla LM computes $p(S)$ with the chain rule over a single pre-determined subword token sequence, SSLM computes $p(S)$ as

where $\pi(S)$ is the set of all candidate tokenisations of $S$: every possible way that the sequence of words $S$ can be segmented into subword units (word boundaries are enforced, so subword units cannot span across whitespaces). Each $p(T)$ is still computed with the chain rule over the token sequence $T$, but the overall sequence probability $p(S)$ now incorporates multiple potential tokenisations of the training example $S$.

Marginalising over $\pi(S)$ is intractable for long sequences, so SSLM introduces two constraints:

1. 1.

   Subword segments cannot exceed a maximum character length $L$, which is a hyperparameter.

2. 2.

   In computing the probability of a candidate tokenisation $T=\{t_{1},t_{2},...,t_{|T|}\}$ with the chain rule, each next-token probability is conditioned on the untokenised autoregressive (preceding) character-level context, so

   $\displaystyle p(T)=\prod_{i=1}^{|T|}p(t_{i}|t_{<i})\approx\prod_{i=1}^{|T|}p(t_{i}|c_{<t_{i}}),$

   (2)

   where $c_{<t_{i}}$ is the character sequence in $S$ that precedes $t_{i}$. This approximation discards tokenisation history, but enables tractable conditioning via a character-level context encoder.

Finally, to compute Equation 1 efficiently, Meyer and Buys (2022) use a dynamic programming algorithm that iteratively computes $p(S_{1:k})$, the marginal probability of the sequence up to each character position $k$, for $k=1,...,|S|$ (we refer the reader to Meyer and Buys (2022) for a detailed presentation of the algorithm).

SubSegGPT is a decoder-only SSLM with a GPT-2 backbone. It is a straightforward adaptation of the original, LSTM-based SSLM (Meyer and Buys, 2022) to GPT-style language modelling and makes use of the same training algorithm outlined in Section 2.2. We now describe its architecture, which closely mirrors the transformer-based SSLM of Meyer and Buys (2025), but is parameterised by the GPT-2 architecture to incorporate more recent architectural conventions and match the setup of competitive decoder-based BabyLMs.

We propose SubSegDeBERTa, an encoder-based, masked SSLM with a DeBERTa-v2 backbone. Meyer and Buys (2022) introduce subword segmental modelling for decoder-only LMs. Their framework is inherently autoregressive, so its extension to masked language modelling is non-trivial. In SubSegDeBERTa, we develop the first masked SSLM, incorporating learnable subword tokenisation into encoder-based pretraining. We encode bi-directional character-level context with a DeBERTa-v2 backbone, randomly mask a subset of input words, and jointly learn to generate and tokenise masked words into subword units.

We follow the guidelines of the 2026 BabyLM Challenge (Choshen et al., 2026) to pretrain SubSegGPT and SubSegDeBERTa for Strict and Strict-small. We use the text-only datasets released by the organisers and pretrain for 10 epochs.

To test the impact of learnable tokenisation, we compare our models to fixed-tokenisation BabyLMs based on the corresponding backbone architectures and trained on the same data. For SubSegGPT, we use the GPT-2 (Radford et al., 2019) baselines released by the BabyLM Challenge organisers. For SubSegDeBERTa, we pretrain our own DeBERTa-v2 (He et al., 2021) models as baselines, matching the backbone architectural configurations of SubSegDeBERTa for comparability. Table 4 in the appendix details our baseline setup.

Our backbone architectures match the size of the BabyLM GPT-2 baseline, which corresponds to Base configurations of GPT-2 and DeBERTa. We tune pretraining hyperparameters as detailed in Appendix A. Our backbone architectures are character-based, so their embedding matrices contribute negligibly to overall parameter count. However, our subnetworks for subword segment scoring introduce additional parameters. The model sizes and hyperparameter settings of our submissions and baselines are reported in Tables 4 and 5 in the appendix.

We evaluate on the official 2026 BabyLM Challenge evaluation pipeline, which includes three types of tasks (full list in Appendix B). (1) Zero-shot tasks test linguistic knowledge directly from model probabilities via minimal pair evaluations. (2) Finetuning tasks test natural language understanding on (Super)GLUE (Wang et al., 2018; Wang et al., 2019) with task-specific finetuning. (3) Human-likeness tasks evaluate how well model predictions align with human psycholinguistic data (we report these results in Appendix D, as neither our models nor baselines perform well on these tasks).

The BabyLM evaluation pipeline is designed for standard LM architectures. To evaluate our models on finetuning tasks, we can apply the pipeline without change (final-layer representations are passed to classification heads). However, for zero-shot tasks, the pipeline expects per-position logits, which our models do not emit: SubSegGPT outputs a per-sentence marginal (Equation 1) and SubSegDeBERTa outputs a per-word marginal (Equation 4). We implement model-specific wrappers that transform our outputs into the quantities required for each task. We leave the official pipeline unchanged, except for one line of code (see Appendix  C.1 for details). Appendix C describes the wrappers we implement to evaluate our models.

We compare tokenisations of the 2022 SIGMORPHON English test set (Batsuren et al., 2022) across regular interval checkpoints (every 1M words until 10M words, every 10M words until 100M words). To extract the learned tokenisation of a sentence, we use the Viterbi algorithm to extract the highest-probability tokenisation (replacing the sum in Equations 1 and 4 with an argmax).

For each checkpoint, we quantify the properties of its subwords with three metrics. (1) Boundary flip rate measures the rate of change in tokenisation as the fraction of possible subword boundaries that change from one checkpoint to the next. (2) Fertility (Ács, 2019) is the average number of subwords per word, reflecting the granularity of subword tokenisation. (3) Morpheme boundary identification F1 measures the overlap between learned subword boundaries and ground truth morpheme boundaries, as annotated in the SIGMORPHON dataset.

Figure 2 plots the learning dynamics of SubSegGPT and SubSegDeBERTa during Strict-small pretraining. The two models exhibit similar learning trajectories. After an initial period of rapid changes, subword learning gradually stabilises and converges on a settled tokenisation scheme. This convergence is characterised by a steady increase in fertility: words are tokenised into more subword units. As a comparison, the fertility of the BabyLM baseline tokeniser (16k-vocabulary BPE) on this dataset is 1.23, so our models learn more aggressive segmentation. The subword boundaries of both models shift towards greater alignment with morphological boundaries. This alignment remains weak compared to a dedicated unsupervised morphological segmenter like Morfessor (Smit et al., 2014), which achieves 42.9% F1 on the same evaluation set, but is well above random boundary insertion at the same fertility as our models, which would achieve around 11.6% F1.

The subword units learned by SubSegGPT and SubSegDeBERTa reflect the tokenisation demands of sample-efficient language modelling. Our quantitative analysis suggests that this balances linguistic plausibility (partial morphological alignment) against frequency-based criteria (finer-grained segmentation). To study this trade-off qualitatively, Table 3 presents CHILDES (MacWhinney, 2000) utterances tokenised by our models, highlighting subwords corresponding to morphemes.

The examples show several instances of our models discovering morphemes as subword units, such as suffixes (“–ing”, “–s”) and compound constituents (“sand–box”). They also show instances of morphologically unsound tokenisation, some of which are model-specific: SubSegGPT performs worse as a morphological segmenter (see Figure 2) and this is reflected in examples like “playi–ng” and “jumpi–ng”. Other morphological violations can be attributed to biases introduced by our architecture design (e.g. our mixture model for subword generation) and constraints imposed by modelling assumptions. For example, subword segments cannot exceed 5 characters, so words like “kitchen” and “little” cannot be left untokenised. Removing these constraints would allow us to study truly unrestricted subword learning, but marginalising over unbounded segment lengths is computationally infeasible in our current setup.