German MaxText SLMs

This repository contains training code for pretraining small German language models from scratch on TPUs using the MaxText library.

Features

• Full pretraining pipeline for German SLMs on TPU infrastructure
• Recipes for training various tokenizers optimized for German
• Comprehensive tokenizer evaluation across multiple metrics

❤️ Acknowledgements

Huge thanks to Google for providing TPU resources through the TPU Research Cloud (TRC) program!

Made from Bavarian Oberland with ❤️ and 🥨.