Russian version: README_RU.md

Capstone: Samsung Innovation Campus · Deep Learning / NLP
Author: Bogdan Savelyev
Task: polarity (pos/neg) on mixed 2GIS reviews that switch between Russian and Kazakh in one text. You can't build that sample honestly without a reliable ru / kz / mixed filter: auto-LID on Telegram tags almost everything as mixed, while most of those lines are Kazakh with a Russian loanword.

Full narrative and numbers for reviewers: STORY.md · web version web/story.html (build: python scripts/build_story_landing.py).

Name note: Nazarbayev University already ships an open toolkit also called KazNLP. This repo is a separate student/capstone project. For papers and outreach, prefer a distinct short name (e.g. ShalaLID / kk-ru SwitchFilter) so the two are not confused.

Sources: main.ipynb (cells 45, 173, 237, 255, 268), docs/capstone/Final_Report.md, data/processed/metrics_tone_test.json.

Tone 97.33% caveat: ~94% of tone gold comes from llm_composer; the hold-out metric is agreement with LLM drafts, not a blind human eval. Details in Final Report §3.3.3.

On-disk counts: if CSVs were rebuilt after cell 237, disk row counts can drift (e.g. main.csv >331k). For defense and papers, trust cell 237 and the table above.

Not in git (GitHub size limits). Hugging Face:

https://huggingface.co/datasets/naadgob/kaznlp-weights

pip install huggingface_hub
python scripts/download_kaznlp_weights.py   # download models.zip and unpack
python scripts/setup_demo_models.py         # symlink RU/KZ + path checks

Locally: unpack models/models.zip into the repo root (next to README.md).

pip install -r inference/requirements.txt
python run_demo.py

Windows: start_demo.bat. Open http://127.0.0.1:8000/ (web/index.html + POST /analyze).

The first request may return 503 for 30–60 s while four models (~8.5 GB) load. Download weights first (above) or drop models.zip in place by hand.

pip install -r requirements-labeler.txt
ollama pull qwen2.5:3b    # for LLM-batch language drafts
python run_labeler.py

Default port 8000 (next free port if busy). See labeling_service/README.md.

python scripts/eval_tone_v1.py

KazNLP/
├── main.ipynb                      # main pipeline (274 cells)
├── README.md · README_RU.md · STORY.md · STORY_RU.md
├── run_demo.py · run_labeler.py · start_demo.bat
├── collect_2gis_reviews.py         # 2GIS scraper (CLI)
├── essential_ru_kaz.py · popular_ru_kaz.py
├── requirements.txt · requirements-labeler.txt
├── .env.example                    # secrets template (do not commit .env)
│
├── data/
│   ├── raw/                        # telegram_code-switch_dataset.csv, 2gis_*.csv
│   ├── processed/                  # gold_v1, main.csv, tone_*, metrics_tone_test.json
│   │   └── synthetic_tone/         # tone synthetic batches (after generate_tone_synthetic.py)
│   ├── manual labeling/            # history of manual LID labeling
│   │   └── language/               # kaspi-telegram_part, lingua_candidates_*, mixed-heuristic-seed
│   └── training/
│       ├── filter/v1/              # LID train · val · test (gold)
│       └── tone/v1/                # tone splits
│
├── models/
│   ├── fasttext/                   # fasttext_v1.bin · fasttext_v2.bin
│   ├── xlm-roberta/                # xlm-r_v2.pt · tone_v1.pt · lid/ · tone/
│   └── tone_pretrained/            # ru_rubert_rureviews · kz_kazakh_sentiment_bert
│
├── training/                       # fasttext_synthetic.txt · train_v2.txt · test_v2.txt · …
├── inference/                      # FastAPI: app.py · pipeline.py · config.py · test_api.py
├── labeling_service/               # labeler UI + LLM batch (see README inside)
│   ├── main.py · manual_labeling.py · text_heuristics.py
│   ├── templates/index.html
│   └── static/                     # app.js · manual.js · style.css
│
├── web/
│   ├── index.html                  # landing + live demo
│   ├── story.html                  # jury narrative (build_story_landing.py)
│   ├── assets/
│   │   ├── labeler/                # 01–03 PNG for story carousel
│   │   └── _verify-sources.png
│   └── README.md
│
├── scripts/                        # eval · export · deck · synthetic (see table)
│   └── archive/                    # old LID synthetic/EDA (not defense path)
│
├── docs/
│   ├── README.md
│   ├── tone_dataset_overview.md
│   ├── outreach/                   # professor outreach drafts
│   └── capstone/                   # SIC paperwork (below)
│
└── sessions/                       # Telethon *.session (gitignored)

Live defense narrative: web/story.html (does not replace presentation.pptx in paperwork).

Baseline ladder on one shared test (n=461): §10 (cells 265–275) — FastText → HeLI raw/neutral/windows-grid → char-3gram NB → Lingua → XLM-R.

Regenerate synthetic:

python scripts/generate_tone_synthetic.py
python scripts/merge_tone_synthetic.py

Gold tone path: 2GIS → LID v2 → language == mixed → run_labeler.py (manual mode + llm_composer draft) → balance pos/neg.

TEXT → XLM-R LID v2 → ru | kz | mixed
  ru    → RuReviews RuBERT (pretrained)
  kz    → Kazakh Sentiment BERT (pretrained)
  mixed → Mixed Tone v1 (fine-tuned XLM-R)

python scripts/download_tone_pretrained.py
python scripts/verify_tone_pretrained.py

See models/tone_pretrained/README.md.

Historical batch scripts: scripts/archive/ (not needed for the defense path).

Defense path (minimum): cells 45, 173, 237, 268 + eval_tone_v1.py + run_demo.py.

SIC presentation: docs/capstone/presentation.pptx · live: web/story.html · rebuild HTML: python scripts/build_self_deck.py

• The language column in raw Telegram is the collector label (FastText), not ground truth. Under the is_real_mixed() heuristic only ~1.66% of FT-mixed lines are real code-switches.
• The 16,364 mixed rows in main.csv are LID v2 predictions, not a full human audit of the corpus.
• main.ipynb does not replay via Run All; do not re-run Telethon cells 31, 35, 41, 88.
• Do not commit: .env, sessions/, labeling_service/uploads/.