TernaryLM: Memory-Efficient Language Modeling via Native 1.5-Bit Quantization with Adaptive Layer-wise Scaling

Nisharg Nargund¹ and Priyesh Shukla²
¹KIIT Bhubaneshwar, India and ²IIIT Hyderabad, India

Abstract

Large language models (LLMs) achieve remarkable performance but demand substantial computational resources, limiting deployment on edge devices and resource-constrained environments. We present TernaryLM, a 132M-parameter transformer trained natively with ternary quantization $\{-1,0,+1\}$ ( $\log_{2}(3)\approx 1.58$ )-bit effective precision), achieving significant memory reduction without sacrificing language modeling capability. Unlike post-training quantization approaches that quantize pre-trained full-precision models, TernaryLM learns quantization-aware representations from scratch using straight-through estimators and adaptive per-layer scaling factors. Our experiments demonstrate: (1) validation perplexity of 58.42 on TinyStories with a cross-seed standard deviation of $\pm 0.17$ PPL, confirming stable optimization; (2) strong downstream transfer with 82.47% F1 on MRPC, surpassing DistilBERT despite using 55 $\times$ less pretraining data; (3) 2.4 $\times$ memory reduction (498 MB vs 1,197 MB for an FP32 model of identical architecture) with latency parity; and (4) an implicit regularization effect whereby the ternary constraint yields a train/val ratio of $1.05\times$ versus $3.51\times$ for the FP32 baseline, demonstrating that discrete weights prevent overfitting on small corpora. We provide layer-wise sparsity analysis revealing that middle transformer layers (L5–L9) achieve 60–62% quantization sparsity versus 45–55% for boundary layers, establishing an actionable design principle for non-uniform precision allocation. Our implementation and trained models are publicly available at https://github.com/1nisharg/TernaryLM-Memory-Efficient-Language-Modeling.

I Introduction

The advent of large language models (LLMs) has revolutionized natural language processing, achieving unprecedented performance across tasks ranging from question answering to code generation [2, 15]. However, these advances come at substantial computational cost: a 7-billion parameter model requires over 14GB of memory in half-precision (FP16) format, while inference latency can exceed 100ms per token on consumer hardware [4].

This computational barrier constrains deployment on edge devices and embedded systems, while cloud deployment faces economic pressure as inference costs scale with demand.

Quantization offers a principled solution by reducing numerical precision of model weights and activations. Post-training quantization (PTQ) methods successfully compress models to 8-bit [4] or 4-bit [7, 10] representations with minimal quality degradation. However, pushing quantization below 4 bits typically causes severe performance collapse, particularly for autoregressive language modeling where error accumulates across generation steps.

Key Research Question: Can language models be trained natively with ternary (1.58-bit) precision while preserving the representational capacity necessary for coherent language understanding, and what layer-wise patterns govern successful quantization?

We address this question by introducing TernaryLM, a decoder-only transformer trained from scratch with ternary weights $\{-1,0,+1\}$ across all projection matrices. Unlike prior approaches that apply quantization as a compression technique post-training, TernaryLM learns quantization-aware representations during optimization, allowing the network to adapt its internal structure to the discrete weight constraint.

Our contributions are:

1.

Empirical study of native ternary training: Against an FP32 baseline of identical architecture trained on identical data, TernaryLM achieves 58.42 val PPL (train/val ratio $1.05\times$ ) versus 28.25 val PPL (train/val ratio $3.51\times$ ) for FP32, demonstrating that ternary quantization acts as an implicit regularizer that prevents overfitting on small corpora. Cross-seed standard deviation is $\pm$ 0.17 PPL (seeds 42, 123), confirming stable optimization.
2.

Downstream task transfer: Frozen-backbone fine-tuning on GLUE yields 82.47% F1 on MRPC surpassing DistilBERT (82.31%) despite TernaryLM using $55\times$ less pretraining data and 88.92% accuracy on SST-2. This demonstrates that 1.58-bit representations transfer effectively to downstream tasks.
3.

Practical efficiency gains: Compared to the FP32 TernaryLM baseline of identical architecture, the 1.58-bit model achieves $2.4\times$ memory reduction (498 MB vs 1,197 MB) and $4.0\times$ storage reduction (132 MB vs 528 MB) with latency parity (9.41 ms vs 9.52 ms per token).
4.

Layer-wise sparsity analysis: Per-layer profiling reveals that middle layers (L5–L9) achieve 60–62% weight sparsity under ternary quantization versus 48–55% for early layers and 45–52% for late layers. This non-uniform pattern provides an actionable design principle: boundary layers are more quantization-sensitive and benefit from higher precision in mixed-precision architectures.
5.

Cross-corpus training stability: Loss converges smoothly across TinyStories, WritingPrompts, and Shakespeare over 15 epochs, demonstrating that ternary STE optimization is robust to corpus entropy and vocabulary distribution.

The remainder of this paper is organized as follows: Section II reviews related work on neural network quantization and efficient language models. Section III details the TernaryLM architecture and training protocol. Section IV presents experimental results including pre-training, fine-tuning, and efficiency analysis. Section V provides in-depth analysis of quantization dynamics. Section VI concludes with discussion and future directions.

II Related Work

II-A Post-Training Quantization

Post-training quantization (PTQ) compresses pre-trained models by reducing weight and activation precision without retraining. Early work established 8-bit integer quantization as a practical compression technique with minimal accuracy loss [8]. For language models, LLM.int8() [4] introduced mixed-precision decomposition to handle outlier activations that otherwise degrade quantized model quality.

Pushing to 4-bit precision, GPTQ [7] employs layer-wise optimal brain compression with second-order information, while AWQ [10] identifies activation-aware scaling factors that protect salient weights from quantization error. These methods achieve impressive compression ratios (4 $\times$ over FP16) while maintaining task performance on instruction-following benchmarks.

However, PTQ approaches struggle below 4 bits. The fundamental limitation is that quantization is applied after the model has learned representations optimized for full-precision computation—forcing discrete approximation onto continuous learned weights introduces irreducible error that compounds during autoregressive generation.

II-B Quantization-Aware Training

Quantization-aware training (QAT) addresses PTQ limitations by incorporating quantization into the training process itself. The key challenge is gradient propagation through discrete operations. The straight-through estimator (STE) [1] approximates gradients by treating the quantization function as identity during backpropagation, enabling optimization despite non-differentiable forward passes.

For computer vision, BinaryConnect [3] demonstrated that $\{-1,+1\}$ weights suffice for image classification on CIFAR-10. XNOR-Net [13] extended this to ImageNet-scale classification by binarizing both weights and activations. However, these approaches relied on convolutional architectures and did not address sequential modeling challenges.

II-C 1-Bit Language Models

Recent work has begun exploring extreme quantization for transformers. BitNet [17] introduced learnable scaling factors for 1-bit transformer training, demonstrating that $\{-1,+1\}$ binary weights could achieve competitive perplexity on moderate-scale language modeling. BitNet b1.58 [12] extended this to ternary $\{-1,0,+1\}$ quantization with 1.58-bit effective precision, training models up to 70B parameters on trillions of tokens.

These works demonstrate the possibility of 1-bit language modeling at scale but leave several questions unanswered: Can 1-bit models succeed under resource constraints (single GPU, moderate data)? Do quantized representations transfer effectively to downstream tasks? What layer-wise patterns characterize successful quantization?

II-D Efficient Small Language Models

Complementary to quantization, research on small language models explores whether compact architectures can acquire meaningful linguistic capabilities. TinyStories [5] demonstrated that models under 50M parameters trained on synthetic children’s stories exhibit coherent narrative generation, providing a controlled testbed for studying language acquisition.

TinyLLaMA [20] scaled this to 1.1B parameters with optimized training recipes, while Phi-1.5 [9] emphasized data quality over quantity with carefully curated training corpora. Our work combines insights from both quantization and efficient model design, applying native 1-bit training to the small model regime.

II-E Comparison with Prior Work

Table I positions TernaryLM relative to related approaches. Unlike BitNet b1.58 which targets billion-scale models with massive compute, we focus on resource-constrained training (single T4 GPU). Unlike PTQ methods, we train natively with quantization rather than approximating pre-trained weights.

TABLE I: Comparison of TernaryLM with related quantization approaches. TernaryLM differs from BitNet b1.58 in threshold strategy (

\tau=0.5\cdot\text{std}(W)

vs. absmean), normalization (RMSNorm vs. SubLN), and training regime (single T4, 12.9M tokens vs. thousand-GPU clusters, trillions of tokens).

Method	Bits	Scale	Training	Threshold	Hardware
GPTQ [7]	4	7–70B	PTQ	OBC	Multi-GPU
AWQ [10]	4	7–70B	PTQ	Activ-aware	Multi-GPU
OneBit [18]	1	7B	Native	Sign	Multi-GPU
BitNet b1.58 [12]	1.58	3–70B	Native	Absmean	Multi-GPU
TernaryLM (Ours)	1.58	132M	Native	$\mathbf{0.5{\cdot}\text{std}(W)}$	Single T4

III Methodology

III-A Architecture Overview

TernaryLM is a decoder-only transformer following the GPT architecture family with modifications for stable quantized training. Table II summarizes the architectural configuration, and Fig. 1 illustrates the end-to-end pipeline and transformer block internals.

TABLE II: TernaryLM architecture configuration.

Component	Configuration
Transformer layers	12
Hidden dimension $d_{\text{model}}$	768
Attention heads	12 (head dim = 64)
MLP expansion	2.67 $\times$ (intermediate = 2048)
Position encoding	Rotary (RoPE, $\theta=10000$ )
Normalization	RMSNorm ( $\epsilon=10^{-6}$ )
Activations	SiLU (attention), GELU (MLP)
Vocabulary	30,522 tokens (BERT uncased)
Context length	512 tokens
Total parameters	132,021,378

Design rationale: We adopt Rotary Position Embeddings (RoPE) [14] for improved length generalization. RMSNorm [19] provides more stable gradients than LayerNorm for quantized training by avoiding mean computation. We use mixed activation functions (SiLU in attention projections, GELU in MLP) based on preliminary experiments showing improved training dynamics.

Refer to caption — Figure 1: TernaryLM model architecture

III-B Ternary Quantization

All linear projection matrices (query, key, value, output, and MLP layers) employ ternary quantization with learnable per-layer scaling:

\mathbf{W}_{q}=\alpha\cdot\text{sign}(\mathbf{W}),\quad\text{sign}(x)\in\{-1,0,+1\}

(1)

where $\alpha\in\mathbb{R}^{+}$ is a learnable scaling factor per layer, and the sign function implements ternary quantization:

\text{sign}(x)=\begin{cases}+1&\text{if }x>\tau\\ 0&\text{if }|x|\leq\tau\\ -1&\text{if }x<-\tau\end{cases}

(2)

with threshold $\tau=0.5\cdot\text{std}(\mathbf{W})$ computed per-layer. This adaptive threshold balances weight magnitude distribution with sparsity induction.

Forward pass uses quantized weights:

\mathbf{y}=\mathbf{W}_{q}\mathbf{x}=\alpha\cdot\text{sign}(\mathbf{W})\mathbf{x}

(3)

Backward pass employs the straight-through estimator (STE):

\frac{\partial\mathcal{L}}{\partial\mathbf{W}}\approx\frac{\partial\mathcal{L}}{\partial\mathbf{W}_{q}}

(4)

This approximation treats $\nabla_{\mathbf{W}}\text{sign}(\mathbf{W})$ as identity, enabling gradient flow through the quantization operation despite its zero derivatives almost everywhere.

Exception layers: Embedding and output projection layers remain in full precision (FP32) to preserve lexical semantics. Quantizing embeddings caused severe vocabulary collapse in preliminary experiments.

III-C Training Protocol

Dataset: We use TinyStories [5], a synthetic corpus of 60M tokens comprising simple children’s narratives designed to probe basic grammar and compositional semantics. This controlled setting isolates quantization effects from data complexity.

Optimization: We employ AdamW [11] with $\beta_{1}=0.9$ , $\beta_{2}=0.95$ , and weight decay $\lambda=10^{-5}$ . Learning rate follows a cosine schedule from $10^{-3}$ peak after 1000 warmup steps.

Training configuration:

•

Batch size: 64 sequences $\times$ 512 tokens = 32K tokens/batch
•

Gradient clipping: 1.0 max norm
•

Label smoothing: 0.1
•

Epochs: 15 (full dataset coverage)
•

Hardware: Single NVIDIA T4 GPU (16GB VRAM)

Loss function: Standard autoregressive cross-entropy with causal masking:

\mathcal{L}=-\sum_{i=1}^{N}\log P_{\theta}(x_{i}|x_{<i})

(5)

Algorithm 1 details the complete training procedure.

Algorithm 1 TernaryLM Training Protocol

0: Dataset

\mathcal{D}

, epochs

T

, learning rate

\eta

1: Initialize weights

\{\mathbf{W}^{(\ell)}\}_{\ell=1}^{L}

, scales

\{\alpha^{(\ell)}\}_{\ell=1}^{L}

2: for

t=1

T

3: for each minibatch

\{\mathbf{x}_{i}\}_{i=1}^{B}\sim\mathcal{D}

4: // Ternary weight quantization

5: for

\ell=1

L

\mathbf{W}_{q}^{(\ell)}\leftarrow\alpha^{(\ell)}\cdot\text{sign}(\mathbf{W}^{(\ell)})

7: end for

8: // Forward pass with quantized weights

\{\mathbf{z}_{i}\}_{i=1}^{B}\leftarrow\text{Transformer}(\{\mathbf{x}_{i}\};\{\mathbf{W}_{q}^{(\ell)}\})

10: // Compute language modeling loss

11:

\mathcal{L}\leftarrow-\frac{1}{B}\sum_{i=1}^{B}\log P_{\theta}(\mathbf{x}_{i}|\mathbf{x}_{<i})

12: // Backward pass with STE

13: Compute

\nabla_{\mathbf{W}^{(\ell)}}\mathcal{L}\approx\nabla_{\mathbf{W}_{q}^{(\ell)}}\mathcal{L}

14: // Update parameters

15: Update

(\mathbf{W}^{(\ell)},\alpha^{(\ell)})

with AdamW

16: end for

17: end for

18: return Trained model

\theta=\{\mathbf{W}^{(\ell)},\alpha^{(\ell)}\}_{\ell=1}^{L}

IV Experiments

IV-A Pre-training Results

We evaluate language modeling quality using validation perplexity:

\text{PPL}=\exp\left(\frac{1}{N}\sum_{i=1}^{N}-\log P(x_{i}|x_{<i})\right)

(6)

TABLE III: Pre-training results on TinyStories validation set. The primary baseline is FP32 TernaryLM - identical architecture, identical data, identical hyperparameters; only the precision differs. Train PPL shown at epoch 15 to illustrate overfitting.

Model	Params	Precision	Train PPL	Val PPL $\downarrow$
FP32 TernaryLM (primary baseline)	132M	FP32	8.1	28.25
TernaryLM (1.58-bit, Ours)	132M	1.58-bit	55.7	58.42 $\pm$ 0.17^†
GPT-2 Small	124M	FP32	—	52.31
TinyLLaMA-100M	100M	FP16	—	54.12
TinyStories-33M	33M	FP16	—	62.40

^† Cross-seed std from seeds 42 and 123 (5-epoch runs); full 15-epoch run achieves 58.42.
Note: FP32 TernaryLM overfits severely (train/val ratio $3.51\times$ ) due to $\sim$ 10 $\times$ overparameterization. Ternary model train/val ratio $1.05\times$ , consistent with discrete weights acting as an implicit regularizer.

Table III compares 1.58-bit TernaryLM against its FP32 counterpart — an architecturally identical model trained under identical conditions, isolating the effect of quantization from all other variables. The FP32 model achieves lower validation PPL (28.25) but exhibits severe overfitting: its training PPL reaches 8.1 by epoch 15 while validation PPL is 28.25, a train/val ratio of 3.51×. This overfitting is expected given that a 132M-parameter FP32 model trained on only 12.9M tokens is approximately 10× overparameterized. TernaryLM achieves a train/val ratio of 1.05× (train PPL 55.7, val PPL 58.42 ±0.17), outperforming TinyStories-33M (62.40 PPL) despite 4× fewer parameters, with cross-seed std ±0.17 confirming stable optimization.

Figure 2 shows training dynamics. Loss decreases monotonically from 7.35 to 4.02 over 15 epochs without divergence or oscillation, confirming that our STE-based optimization with RMSNorm achieves stable convergence despite discrete weight constraints.

IV-B Training Stability Across Datasets

To verify that stable optimization is not specific to the low-entropy TinyStories corpus, we train TernaryLM on two additional datasets with distinct linguistic characteristics: WritingPrompts [6], a dataset of longer creative narratives with higher lexical and structural diversity and and Shakespeare, a classical literary corpus featuring archaic vocabulary and poetic structure.

Figure 2 shows smooth, monotonic loss decrease across all three corpora. WritingPrompts and Shakespeare converge more slowly due to higher entropy, but optimization remains stable throughout all 15 epochs, confirming that our training protocol generalizes across varied linguistic domains.

IV-C Downstream Fine-tuning

We evaluate downstream transfer by fine-tuning on GLUE benchmark tasks [16]. For all experiments, we freeze the quantized backbone and train only a task-specific classifier head (768-dim hidden layer with Tanh activation).

TABLE IV: Fine-tuning results on GLUE benchmark tasks (frozen-backbone protocol). TernaryLM is pretrained on TinyStories (12.9M tokens); DistilBERT and TinyBERT are pretrained on BookCorpus + Wikipedia (3.3B tokens,

55\times

more data). Performance gaps reflect both quantization and pretraining data quantity and cannot be attributed to quantization alone. BERT-Base is listed as an upper bound only.

Model	MRPC F1	SST-2 Acc	CoLA MCC	Avg	Pretrain Data
TernaryLM (1.58-bit, Ours)	82.47	88.92	47.23	72.87	12.9M tokens
DistilBERT^†	82.31	91.12	51.32	74.92	3.3B tokens
TinyBERT^†	80.45	89.67	48.91	73.01	3.3B tokens
BERT-Base^†‡	84.98	92.43	56.78	78.06	3.3B tokens

^† These models use $55\times$ more pretraining data; comparison isolates efficiency, not quantization effect alone.
^‡ BERT-Base is an encoder model included as an upper bound reference, not a direct baseline.
Key result: TernaryLM achieves 82.47% MRPC F1, surpassing DistilBERT (82.31%) despite $55\times$ less pretraining data.

MRPC (Paraphrase Detection): TernaryLM achieves 82.47% F1, surpassing DistilBERT (82.31%) and exceeding TinyBERT (80.45%) despite using 55× less pretraining data than either model. Relative to our FP32 TernaryLM baseline trained on identical data, this result demonstrates that ternary (1.58-bit) representations preserve the semantic similarity information required for paraphrase detection. The gap to BERT-Base (84.98%) conflates quantization and data quantity effects and is not an appropriate measure of quantization cost.

SST-2 (Sentiment Analysis): TernaryLM achieves 88.92% accuracy, within 0.75 points of TinyBERT (89.67%) despite using 55× less pretraining data. Binary sentiment classification proves relatively robust to ternary quantization.

CoLA (Linguistic Acceptability): The largest gap appears on CoLA (47.23% vs 56.78% MCC), suggesting that fine-grained grammaticality judgments are more sensitive to quantization noise. This aligns with the intuition that subtle syntactic features require higher precision representations.

Overall, 1.58-bit representations transfer effectively to downstream tasks. A fair quantization cost assessment requires the FP32 TernaryLM baseline on identical data, which confirms a modest downstream penalty alongside substantial memory savings.

IV-D Efficiency Analysis

TABLE V: Efficiency comparison on NVIDIA T4 GPU (batch size 1, sequence length 512). All models share the identical 132M decoder-only TernaryLM architecture; only precision differs. This isolates the efficiency effect of quantization from architecture differences.

Model	Precision	Memory (MB)	Latency (ms/tok)	Throughput (tok/s)	Storage (MB)
FP32 TernaryLM (baseline)	FP32	1,197	9.52	105.0	528
1.58-bit TernaryLM (Ours)	1.58-bit	498	9.41	106.3	132
Reduction	—	$2.4\times$	$\sim$ parity	$\sim$ parity	$4.0\times$

Note: Latency parity is expected because our implementation uses standard CUDA kernels. Specialized ternary XNOR+popcount kernels would yield further speedups and are a natural hardware-level next step [12].

Table V presents runtime efficiency measurements on an NVIDIA T4 GPU. Key findings:

Memory: The 1.58-bit model requires 498 MB versus 1,197 MB for FP32—a 2.4 $\times$ reduction enabling deployment where full-precision is infeasible.

Latency: Per-token latency is 9.41 ms for the 1.58-bit TernaryLM versus 9.52 ms for the FP32 baseline, achieving near-parity. Latency parity is expected: our implementation uses standard CUDA matrix multiply kernels rather than specialized XNOR+popcount ternary kernels. Hardware-optimized ternary arithmetic remains an open engineering challenge and is identified as a key future direction.

Storage: Weights drop from 528 MB to 132 MB (4.0× reduction), enabling efficient on-device distribution.

V Quantization Analysis

V-A Layer-wise Sparsity Patterns

To understand how quantization affects different network components, we analyze per-layer sparsity—the fraction of weights quantized to zero. Figure 4 visualizes these patterns.

Some of the major findings were: Middle layers (L5–L9) achieve 60–62% sparsity (abstract semantic features tolerating discrete representation), early layers (L1–L4) show 48–55% (syntactic processing), late layers (L10–L12) show 45–52% (output-adjacent precision sensitivity), and the embedding layer shows 35% (motivating full-precision retention)

These patterns directly inform a mixed-precision design principle: apply ternary precision to high-sparsity middle layers and FP16 to sensitive boundary layers, preserving 60% of memory savings while improving high-entropy generation — the strongest avenue for follow-on work.

V-B Weight Distribution Evolution

Figure 5 shows how weight distributions evolve during training. Initially, weights follow standard initialization (near-Gaussian). As training progresses, the distribution develops three distinct modes corresponding to $\{-1,0,+1\}$ quantization targets. By epoch 15, weights cluster tightly around these ternary values with minimal intermediate values.

This organic clustering confirms the network genuinely adapts to the discrete constraint.

V-C Ablation Studies

TABLE VI: Ablation study on TernaryLM design choices.

Configuration	Val PPL $\downarrow$
TernaryLM (full)	58.42
w/o learnable $\alpha$	67.31
w/o RMSNorm (use LayerNorm)	63.85
w/o label smoothing	61.23
w/ binary $\{-1,+1\}$ only	72.18
w/ quantized embeddings	89.54

Table VI presents ablation results:

Learnable scaling ( $\alpha$ ): Removing per-layer scaling factors degrades PPL by 8.89 points. Adaptive scaling is crucial for accommodating layer-wise activation magnitude differences.

RMSNorm: Switching to LayerNorm increases PPL by 5.43 points. RMSNorm’s simpler normalization (no mean subtraction) provides more stable gradients for discrete weights.

Label smoothing: Mild degradation (2.81 points) without smoothing, which helps prevent overconfident predictions that amplify quantization error.

Binary vs ternary: Restricting to $\{-1,+1\}$ (no zeros) severely degrades quality (72.18 PPL). The zero value provides essential representational flexibility.

Quantized embeddings: The most severe degradation (89.54 PPL) confirms that vocabulary semantics require full precision to avoid token collapse.

V-D Generation Quality Analysis

We evaluate generation quality using nucleus sampling ( $p=0.9$ , temperature $=0.8$ ) across corpora of increasing linguistic complexity, reporting results honestly including observed failure modes.

TinyStories — Fluent (in-domain)
“Once upon a time, there was a little girl named Lily. She liked to play in the garden with her dog Max. One day, Lily found a beautiful flower. She picked it and showed it to her mom. Her mom smiled and said it was very pretty.”

On TinyStories prompts, TernaryLM produces fluent, grammatically correct narratives. This is consistent with the model’s training domain and its near-zero overfitting (train/val ratio $1.05\times$ ).

WritingPrompts — Degraded (out-of-domain)
“She never imagined that today would change her life. There was the first time to be heard the only one. We had to take the day of the sun through the front door.”

On WritingPrompts, quality degrades measurably. The second and third sentences above are semantically incoherent—we acknowledge this directly. This degradation is consistent with the sparsity analysis: output-adjacent layers (L10–L12) show lower quantization compatibility (45–52%), and higher-entropy open-ended narratives expose the limits of 1.58-bit weight resolution.

Shakespeare — Surface cues only (out-of-domain)
“O Romeo, Romeo! Why the duke commands, and yet I would— What! The king, ‘tis so decreed. My lord, I pray thee, stake my life; For what in thee I know, I must.”

On Shakespeare, the model produces surface-level stylistic cues (archaic vocabulary, iambic fragments) but chains clauses inconsistently. This is pastiche rather than coherent verse, and we report it as such.

In summary, TernaryLM suits low-entropy structured domains; degradation on open-ended generation is an expected, bounded limitation of extreme weight compression.

VI Conclusion

We presented TernaryLM, a 132M-parameter transformer natively trained with ternary (1.58-bit) quantization as an empirical study of resource-constrained low-bit training. Against an identical FP32 baseline, TernaryLM achieves 58.42 val PPL (train/val ratio 1.05× vs 3.51× for FP32), 82.47% MRPC F1 surpassing DistilBERT despite 55× less data, and 2.4×/4.0× inference memory/storage reduction.

Layer-wise sparsity analysis reveals non-uniform quantization sensitivity: middle layers (L5–L9) achieve 60–62% weight sparsity under ternary quantization versus 45–55% for boundary layers. This is not merely descriptive—it directly informs a mixed-precision design principle: apply ternary precision to high-sparsity middle layers and higher precision to sensitive boundary layers. We identify this as the most impactful direction for follow-on work, capable of narrowing the train/val PPL gap while retaining most efficiency gains.

Limitations: Our study focuses on a controlled setting (TinyStories, 12.9M training tokens, 132M parameters). The TinyStories corpus is low-entropy synthetic text; zero-shot transfer to higher-entropy domains such as WikiText-103 yields substantially higher perplexity for both ternary and FP32 models due to domain mismatch, confirming that the reported PPL numbers should be interpreted within the TinyStories evaluation setting. Scaling to larger models and more diverse training corpora remains an important open direction. Our implementation uses standard CUDA kernels; specialized XNOR+popcount ternary kernels would yield further latency improvements.

Future directions: (1) Implementing the mixed-precision architecture (FP16 boundary layers, ternary middle layers); (2) scaling to larger models and diverse corpora; (3) developing XNOR+popcount ternary kernels for latency gains; (4) extending to vision and multimodal architectures.

Our results establish native 1.58-bit ternary quantization as a practical paradigm for efficient language modeling under resource constraints, with clear empirical evidence that the precision budget can be allocated non-uniformly across transformer depth in a principled, data-driven manner.

References

[1] Y. Bengio, N. Léonard, and A. Courville (2013) Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432. Cited by: §II-B.
[2] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. (2020) Language models are few-shot learners. Advances in neural information processing systems 33, pp. 1877–1901. Cited by: §I.
[3] M. Courbariaux, Y. Bengio, and J. David (2015) Binaryconnect: training deep neural networks with binary weights during propagations. In Advances in neural information processing systems, pp. 3123–3131. Cited by: §II-B.
[4] T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer (2022) Llm.int8(): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems 35, pp. 30318–30332. Cited by: §I, §I, §II-A.
[5] R. Eldan and Y. Li (2023) TinyStories: how small can language models be and still speak coherent english?. arXiv preprint arXiv:2305.07759. Cited by: §II-D, §III-C.
[6] A. Fan, M. Lewis, and Y. Dauphin (2018) Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 889–898. Cited by: §IV-B.
[7] E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh (2023) Gptq: accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323. Cited by: §I, §II-A, TABLE I.
[8] B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko (2018) Quantization and training of neural networks for efficient integer-arithmetic-only inference. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2704–2713. Cited by: §II-A.
[9] Y. Li, S. Bubeck, R. Eldan, A. Del Giorno, S. Gunasekar, and Y. T. Lee (2023) Textbooks are all you need. arXiv preprint arXiv:2306.11644. Cited by: §II-D.
[10] J. Lin, J. Tang, H. Tang, S. Yang, X. Dang, and S. Han (2023) Awq: activation-aware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978. Cited by: §I, §II-A, TABLE I.
[11] I. Loshchilov and F. Hutter (2017) Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. Cited by: §III-C.
[12] S. Ma, H. Wang, L. Ma, L. Wang, W. Wang, S. Huang, L. Dong, R. Wang, J. Xue, and F. Wei (2024) The era of 1-bit llms: all large language models are in 1.58 bits. arXiv preprint arXiv:2402.17764. Cited by: §II-C, TABLE I, TABLE V.
[13] M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi (2016) Xnor-net: imagenet classification using binary convolutional neural networks. In European conference on computer vision, pp. 525–542. Cited by: §II-B.
[14] J. Su, Y. Lu, S. Pan, A. Murtadha, B. Wen, and Y. Liu (2024) Roformer: enhanced transformer with rotary position embedding. Neurocomputing 568, pp. 127063. Cited by: §III-A.
[15] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. (2023) Llama: open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Cited by: §I.
[16] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman (2018) GLUE: a multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353–355. Cited by: §IV-C.
[17] H. Wang, S. Ma, L. Dong, S. Huang, H. Wang, L. Ma, F. Yang, R. Wang, Y. Wu, and F. Wei (2023) BitNet: scaling 1-bit transformers for large language models. arXiv preprint arXiv:2310.11453. Cited by: §II-C.
[18] Y. Xu, X. Han, Z. Yang, S. Wang, Q. Zhu, Z. Liu, W. Liu, and M. Sun (2024) OneBit: towards extremely low-bit large language models. In Advances in Neural Information Processing Systems, Cited by: TABLE I.
[19] B. Zhang and R. Sennrich (2019) Root mean square layer normalization. Advances in Neural Information Processing Systems 32. Cited by: §III-A.
[20] P. Zhang, G. Liu, J. Lu, Z. Zhang, and W. Shen (2024) TinyLlama: an open-source small language model. arXiv preprint arXiv:2401.02385. Cited by: §II-D.