LangChef

You changed the model, the retriever, or the fine-tune. Is the app better or worse?

Most teams shipping an LLM feature cannot answer that with a straight face. LangChef is a command-line tool that answers it properly — and, just as often, tells you honestly that your test set was never big enough to tell.

It is built for the engineer who maintains a retrieval app, a classifier or an agent, has no evaluation background, and has no evaluation team to ask. Your provider retires the model you shipped on; you swap an embedding model; you distil to a fine-tuned small model to cut the bill. Same question every time. You label about forty examples once, and from then on it tells you whether your judge can be trusted, and whether each change was a real regression, a real improvement, or noise.

Documentation: https://deepskandpal.github.io/LangChef/ — start with your first evaluation.

Already have a labelled test set in a spreadsheet? Point at it instead: bring your own data. Name the columns in evals/config.toml and start. For classification, retrieval and reranking there is no rubric to write and nothing to label, because those have a hard target — so setup is minutes rather than an afternoon. The trade is honest and stated on that page: with a hard target there is no judge to calibrate, so what you get is the paired comparison, the detection limit and the experiment discipline rather than "the judge you can trust".

Trusting a grader nobody graded. Your judge marks 95% of answers good. If only 5% of your answers are genuinely bad, a judge that marks everything good scores 95% too — and from that number alone you cannot tell them apart.

Reading meaning into a three-point swing. 83% to 80% on ninety examples is well inside what randomness produces. Nothing in a spreadsheet says so.

Comparing two runs measured differently. Edit the grading prompt between runs and the two numbers were never measuring the same thing — but they still line up on a chart.

Averaging retrieval and generation into one number. When quality drops after an embedding swap, one pass rate cannot tell you whether the generator got worse or is being handed worse context. Those have different fixes.

Every product in this market is capable and most of them are cheap. They also all assume a human eval engineer exists to design the rubrics, interpret the numbers and maintain the suites. Below a certain size that person does not exist, which is why only about a third of teams running AI in production evaluate it online at all. LangChef is the missing person, not another dashboard.

The whole loop runs: score a suite, choose what a person should label, take their labels back, report how far the judge can be trusted, compare two arms, and write the memo. No API key, no network, no model — the default judge is deterministic, so a fresh clone runs the entire flow on any machine.

Works today

• init, approve rubric, judge run, label plan, label import, calibrate report, calibrate diff, baseline set | show, compare, memo render, ledger append | query, power, experiment design | approve | check | readout | list, plus doctor, contract, packs list
• Calibration statistics — Cohen's kappa with an interval, TPR/TNR/PPV/NPV with Wilson intervals, MCC, and a disagreement taxonomy that only flags a slice when its interval clears the base rate
• Rubric iteration that costs one pass — calibrate diff re-scores a revised rubric against the labels you already have and reports the change in kappa and in both rates, paired, because both rubrics judged the same examples
• Paired experiment comparison — exact McNemar, bootstrap interval, and a minimum detectable effect on every inconclusive result
• Per-criterion attribution — which rubric criterion the loss landed on, Holm-corrected across criteria, so "groundedness fell, correctness held" is a finding rather than five uncorrected tests
• compare --tolerance — non-inferiority in three words, held, failed or unresolved, with unresolved explicitly not a pass and the detection limit printed beside it
• langchef power — is forty examples enough? Answered with no workspace, because requiring one to do arithmetic is ceremony
• A judgement cache keyed on content, rubric hash and model pin, so a rerun is free; two-tier judging, with a strong model re-scoring only the unsure cases
• Gate one and gate two, enforced: an unapproved or edited rubric exits 2, a comparison across moved pins exits 5, a readout with no pre-registration exits 2, and a --tolerance that disagrees with a registered margin is refused rather than preferred
• The waiter: experiment design proposes a sized experiment before anything runs, sizing a continuous outcome from the spread of paired differences and a binary one from the discordant rate, and refusing rather than guessing at a shape it has no rule for
• A dogfood app with six planted regressions and a self-test that asserts the harness finds the ones it can and honestly reports that it cannot resolve the one deliberately planted below the detection limit. That last property is the point: the tool does not claim to find what it cannot

Not built yet — production connectors and sampling, scheduling and unattended operation, eval suites and triage, experiment pre-registration. See the roadmap below and docs/AGENT-CONTRACT.md for where each command lands.

pip install langchef

Or without installing anything permanently:

uv run --with langchef langchef --version

Python 3.12 or 3.13. Both are tested in CI on every push.

For the dogfood app and the test suite, which are not in the wheel. uv is the only prerequisite; it fetches the interpreter itself, so no system Python is involved and nothing is installed globally.

curl -LsSf https://astral.sh/uv/install.sh | sh      # if you don't have it yet

git clone https://github.com/deepskandpal/LangChef.git langchef
cd langchef
uv sync                                     # Python 3.12 + dependencies
uv run langchef doctor

A green doctor means the interpreter is the pinned one, an expertise pack resolves, and no provider credential is sitting in your environment.

Nothing is on PyPI yet — see docs/RESERVE-NAMES.md.

To try the whole thing on an app whose failures are already known, run the dogfood:

uv run python -m dogfood.build
uv run pytest tests/test_dogfood.py -v

One command runs every check, and it is the same script CI runs — there is no second list to drift:

$ ./scripts/verify.sh
1. no provider credentials present        PASS
2. interpreter 3.12                       PASS
3. dependencies match the lock            PASS
4. lint                                   PASS
5. format                                 PASS
6. agent contract in sync                 PASS
7. documentation site in sync             PASS
8. tests                                  PASS
9. distribution builds                    PASS
10. wheel runs from a clean env           PASS

10 passed, 0 failed

Step 1 comes first on purpose: the suite exercises the deterministic core and replays recorded judge responses, so a provider key in the environment means a test could quietly start spending money. Any failure prints the last 25 lines of that step and exits non-zero.

CI runs this same script on both 3.12 and 3.13, the range pyproject.toml declares, because advertising support for a version nothing runs is a claim rather than a fact. Both must pass before main will accept a merge.

$ langchef calibrate report       # stderr — written for you
calibration for support-baseline on 40 labelled example(s)
  kappa      0.68  0.44..0.92
  TPR        80.0%  (12/15)
  FPR        12.0%
  disagreed  6 ({'false_alarm': 3, 'miss': 3})

$ langchef compare --variant support-stale-index
support-baseline -> support-stale-index on 90 shared golden(s)
  baseline 83.3%   variant 63.3%
  difference -20.0% [-27.8%, -12.2%]  p=0.0000
  REGRESSION

$ langchef compare --variant support-truncated-context
  difference +0.0% [+0.0%, +0.0%]  p=1.0000
  INCONCLUSIVE
  (smallest effect this run could have seen: 6.0%)

That last one is the product in one screen. There is a regression in that arm — we planted a 3.3-point one — and the honest answer at this sample size is not "no regression found", it is "nothing we could have seen".

Which half broke. One verdict is a fact; the criterion it landed on is the actionable half, and for a RAG app it tells you whether to look at the retriever or the generator:

$ langchef compare --baseline base --variant hedge-run
base -> hedge-run on 90 shared golden(s)
  baseline 83.3%   variant 72.2%
  difference -11.1% [-17.8%, -5.6%]  p=0.0020
  REGRESSION
  attribution over 2 criterion(s), Holm-corrected — not 2 separate findings:
    Directness     -16.7% [-24.4%, -10.0%]  p=0.0001  MOVED WORSE
    Correctness    +5.6% [+1.1%, +11.1%]  p=0.0625  inconclusive
                   (nothing under 12.0% was in reach for this criterion)

Holm-corrected, because reporting k uncorrected tests is how you find an effect that is not there. And a criterion is only given a direction when the adjusted p clears alpha and the interval lies wholly one side of zero, so the output can never print a direction its own interval contradicts.

Did quality hold? The question a cheaper model actually asks:

$ langchef compare --baseline base --variant hedge-run --tolerance 0.15
  against a 15.0% tolerance: QUALITY UNRESOLVED
    margin came from the command line, not a pre-registration, so it
    constrains nobody: it could have been chosen after seeing the interval.
    unresolved is not held. This run could not resolve 15.0%;
    it needed to see 13.0% or larger.

Is forty examples even enough? Needs no workspace — it is arithmetic, and asking early is the whole point:

$ langchef power --n 90 --effect 0.05 --per-week 40
90 examples detect a difference of 13.2% or larger (binary).
  basis: assumed default (20%) — no prior comparison here
  5.0% needs 628 examples. You have 90, so 538 short.
  At 40 a week that is about 13.4 weeks away.

$ langchef doctor 2>/dev/null     # stdout — written for the agent (abridged)
{
  "checks": "… five checks, each with name / ok / required / detail …",
  "credentials_present": [],
  "ok": true,
  "pack_search_path": [
    "/home/dk/code/github/langchef/evals/packs",
    "/home/dk/code/github/langchef/packs"
  ],
  "packs": [
    "classification@0.1.0",
    "genai-rag@0.2.0"
  ],
  "python": "3.12.14",
  "version": "0.1.0"
}

Two streams, always. The agent parses stdout; you read stderr. There is no --format flag, and --help is the only thing that ever prints something else to stdout.

The interface between a model that decides and a binary that computes is written down, generated from the code, and readable at runtime:

langchef contract      # the same document as JSON

Two rules carry most of the weight.

The CLI produces every number. The agent chooses what to look at and says what it means. Model spend goes on judgement and synthesis, never on arithmetic — a run that costs $0.40 in judge calls should cost nothing in reasoning tokens to add up. A corollary the linter enforces: no number without a run artifact.

Refusals are exit codes. The approval gates and the pre-registration rule cannot live in a prompt; a model asked nicely not to read out early will, eventually, read out early. So:

An agent cannot argue with a non-zero exit.

src/langchef/
  cli/          typer commands — thin, no logic
  core/         statistics, calibration, metrics — no I/O, no LLM, no network
  judge/        rubric, providers.py, cache, two-tier runner
  connect/      duckdb-first read-only connectors      (M5)
  workspace/    paths, formats, config, runs, ledger, scaffold
  render/       decision memos
  packs/        loader + manifest schema — the boundary
packs/genai-rag/       expertise pack: qna and generation, both judged
packs/classification/  expertise pack: a hard target, no judge, no rubrics
adapters/claude-code/  skill + commands — disposable packaging
dogfood/               RAG app with three planted regressions

One rule holds the design together: core/ imports nothing from judge/, connect/ or packs/, and nothing third-party beyond numpy and scipy. That is enforced by tests/test_boundaries.py, which is what makes every number in the product testable with no API key and no network. A matching test keeps litellm inside judge/providers.py, so there is exactly one file to rewrite if it goes bad.

The other marked directory is packs/. It stays separable because the core is Apache-2.0 and the expertise packs are not; if pack logic leaks into the core, that split becomes impossible. See DECISIONS.md #5.

A pack is where a task class is defined — the fields an example carries, what one example's outcome is, the metrics reported for it, and whether it needs a judge at all. qna and generation score free text and are judged; classification has a hard target, so its outcome is predicted == ideal, it ships no rubric, and there is nothing to calibrate (DECISIONS.md #12).

What crosses into core/ is the shape of the outcome and nothing else: binary or continuous, which is what compare and design size an experiment from. The class is named in the manifest, the shape is declared beside it, and none of those three class names appears anywhere in core/. tests/test_boundaries.py plants a leak into core/ to prove that rule is checked rather than merely written down. Adding a fourth class is a directory on the pack search path, not a patch.

./scripts/verify.sh is the whole thing. While iterating, the individual pieces:

uv run pytest                               # the whole suite, with no API key
uv run pytest tests/test_boundaries.py -q   # just the layering rules
uv run ruff check . && uv run ruff format .
uv run python scripts/render_contract.py    # regenerate docs/AGENT-CONTRACT.md
uv run langchef doctor                      # what the agent sees

Judge calls are recorded once and replayed forever, so no test can ever spend money — and CI asserts the absence of a key rather than assuming it.

That leaves one awkward path: the litellm backend is the one every real user runs and the one a keyless build machine cannot run, so for a while it was the only part of the codebase nothing had ever executed. It is covered now by faking the socket rather than the credential — tests/test_litellm_path.py hands litellm an httpx.MockTransport through its own client_session seam and replays recorded chat-completion bytes through it, so litellm's request building, its response parsing, its retries and every line of the shim above them genuinely run. Recording a session against a live model is still worth doing and still needs a person with a key (#31).

Calibration comes first on purpose. A judge is a measuring instrument, not a metric; an eval suite built on an uncalibrated judge produces confident garbage, and no amount of downstream statistics repairs it.

Documentation: https://deepskandpal.github.io/LangChef/ — overview, quickstart, concepts, and a command reference generated from the contract.

• Issues — where the work is, including what is already done: the closed issues are the build log
• The board — the same work by priority, area, size and ownership
• AGENTS.md — the working agreement: the lifecycle, the area boundaries, and the constraints that are not negotiable in a pull request
• CONTRIBUTING.md — how to start, in about a minute
• DECISIONS.md — eleven calls, each with the reasoning and the date
• NON-GOALS.md — what is deliberately not built, and why
• TRACKER.md — the map: where the build is and where to look
• docs/AGENT-CONTRACT.md — generated; what the agent may read, write, spend and decide
• dogfood/README.md — the planted regressions and what they prove

The market analysis, the PRD and the engineering plan behind this repository are working documents and are not published. Their conclusions are in the docs above; DECISIONS.md carries the parts that constrain the code.

The 1.0 platform is not published either. Its vocabulary carried over; its code did not.

Apache-2.0 for the CLI and the workspace format — see LICENSE. Expertise packs are separately licensed.