nanochat-d18-base

A small language model pretrained from scratch with nanochat. This is the base (pretraining) checkpoint — a raw next-token predictor. It has not been fine-tuned for chat: it completes text but does not follow instructions or use conversation turns. For the conversational version see nanochat-d18-sft.

Model details

Training infrastructure

This model was trained on a single consumer GPU (no multi-GPU node):

Because of the single-GPU / 16 GB memory budget, training used device_batch_size=1 with gradient accumulation, and was run as a resumable job. Pretraining used FP8 (tensorwise) matmuls.

Important: format

This is not a 🤗 transformers model — you cannot load it with AutoModelForCausalLM.from_pretrained. It is a raw nanochat checkpoint and must be run through the nanochat codebase.

Repository contents

base_checkpoints/d18/
  model_003712.pt     # pretrained weights
  meta_003712.json    # model config + training args
tokenizer/
  tokenizer.pkl       # tokenizer
  token_bytes.pt      # token byte lengths (used for bpb eval)

The optimizer state is intentionally omitted (inference does not need it).

Usage

# 1. Clone nanochat and set up the environment
git clone https://github.com/karpathy/nanochat
cd nanochat
uv venv && source .venv/bin/activate
uv sync --extra gpu

# 2. Download this repo into the nanochat cache directory
export NANOCHAT_BASE_DIR="$HOME/.cache/nanochat"
hf download nitishpandey04/nanochat-d18-base --local-dir "$NANOCHAT_BASE_DIR"

# 3. Generate text completions from the base model
python -m scripts.chat_cli -i base -p "Once upon a time"

License

MIT, following the nanochat project.