Introducing Plune CLI

LLM features regress quietly. A sharper prompt, a model bump, a refactor of your retrieval step — any of them can change what the model says, and nothing turns red. Unit tests check that code runs; they don’t answer the question that actually matters: is this output still good?

Plune CLI (@plune-ai/cli) is assertion testing for exactly that question. You describe the checks for each case in a single plune.yaml; Plune calls the model, evaluates every assertion, caches the result, and prints a pass/fail summary with the token cost — locally, in CI, or anywhere Node runs.

It ships ten built-in assertion types, so you can grade outputs the way a reviewer would:

• plain text — contains, contains-any, contains-all, exact-match
• structure — json-schema
• judgement — llm-judge, semantic-similarity
• RAG — faithfulness, answer-relevance, context-precision

Each run writes a full JSON result, and plune diff compares two runs and flags only genuine passed → failed regressions — the signal you care about when you change a prompt.

npm i -g @plune-ai/cli
plune init                 # scaffolds plune.yaml + a sample dataset
plune run                  # → 1/1 passed · 0 failed · $0.0008
plune diff base.json head.json --fail-on-regression

Provider keys (Anthropic, OpenAI, OpenRouter) are read from the environment and never written to disk. Exit codes are CI-friendly: 0 passed, 1 an assertion failed, 2 a config or execution error — so a failing suite blocks a merge without extra glue.

Plune CLI is the Evaluate step in Plune’s Generate → Evaluate → Gate flow: write the checks once, run them everywhere. When you want them running automatically on every pull request, eval-action wraps this same engine.

@plune-ai/cli is MIT-licensed and needs Node ≥ 20.