@plune-ai/cli

AI-powered assertion testing for LLM apps — a test runner for model behaviour.

Plune runs an assertion suite against an LLM provider and gives you a pass/fail report — locally, in CI, or as a regression diff between two runs. You describe the checks in one plune.yaml; Plune calls the model, evaluates each assertion, caches results, and reports token cost. Ten built-in assertion types cover plain text, JSON-schema, LLM-as-judge, and RAG metrics (faithfulness, answer-relevance, context-precision).

Links: plune.ai | npm | GitHub Action on Marketplace

npm install -g @plune-ai/cli      # or: pnpm add -g @plune-ai/cli
plune --version

Or run it without installing:

npx -y @plune-ai/cli run

Requires Node.js ≥ 20.

# 1. Scaffold plune.yaml, an example dataset, and .env.example
plune init

# 2. Add your provider key (read from the environment / .env — never written to disk)
echo 'ANTHROPIC_API_KEY=sk-ant-...' >> .env

# 3. Run the assertions
plune run
# → 1/1 passed · 0 failed · 0 errored · $0.0008

# 4. Re-render the last run, or diff two runs to catch regressions
plune report --format markdown
plune diff baseline.json current.json --fail-on-regression

Each run writes its full result to .plune/last-run.json.

Plune reads a single plune.yaml, discovered by walking up from the working directory (or passed with -c <path>). A minimal example:

version: 1
provider:
  type: anthropic            # anthropic | openai | openrouter
  model: claude-3-5-sonnet-latest
evals:
  - id: example
    prompt: "Answer concisely. {{question}}"   # {{vars}} come from each dataset row
    dataset: datasets/example.jsonl            # a file path, or an inline `examples:` list
    assertions:
      - type: contains
        value: "Paris"

Datasets are JSONL, one row per line, shaped { "vars": { ... }, "expected"?: "..." }. The provider API key is read from the environment based on provider.type:

Global flags: -c, --config <path> · -v, --verbose · --no-color.

Exit codes: 0 everything passed · 1 an assertion failed · 2 configuration or execution error.

Everything above works with no account, no network, and no token — that does not change. If you also want run history, trends, and a shared dashboard, the three cloud commands push your local runs to the Plune platform:

plune login          # paste the API token from your platform settings page
plune run            # exactly as before — the run is saved locally
plune sync           # upload .plune/last-run.json, print the read-back URL

The token is stored at ~/.config/plune/credentials.json (mode 0600, honours XDG_CONFIG_HOME) and is never printed or logged. PLUNE_API_URL points sync at a different server — useful for a self-hosted backend.

The same engine that powers plune run is exported for use from your own code. Unlike the CLI, the library does not parse argv or auto-load .env — set the provider key in process.env yourself.

import { run } from '@plune-ai/cli';
import type { RunResult } from '@plune-ai/cli';

const result: RunResult = await run({ dryRun: false, configPath: 'plune.yaml' });
console.log(result.summary); // { total, passed, failed, errored, ... }

Run Plune on every pull request and post a regression diff as a sticky comment with the companion GitHub Action, plune-ai/eval-action:

- uses: plune-ai/eval-action@v1
  with:
    config: plune.yaml
    fail-on-regression: true

Bug reports and pull requests are welcome — see CONTRIBUTING.md. For security issues, see SECURITY.md.

MIT © Plune Contributors