The Plune eval platform · MIT

Evaluate and gate your LLM app — locally and in CI.

One product, two surfaces. Write assertion suites once in plune.yaml, run them anywhere with the CLI, and catch regressions on every pull request with the GitHub Action. Provider-agnostic — and the mock provider keeps CI free.

$ npm i -g @plune-ai/cli
Add the GitHub Action
Read the docs
Local · CLI
$ plune run
✔ faq-grounding   · 3/3 assertions
✔ tone-safety     · 2/2 assertions
2/2 passed · 0 failed · 0 errored · $0.0019
CI · eval-action gate passed
One product, two surfaces

Write evals once. Run them everywhere.

The same plune.yaml drives a local run on your machine and the gate on your pull requests — no second config, no drift.

Local · your machine & any CI MIT
Plune CLI @plune-ai/cli

AI-powered assertion testing for LLM apps. Describe the checks in one plune.yaml; Plune calls the model, evaluates each assertion, caches results, and reports pass/fail with token cost.

  • plune init · run · report · diff
  • 10 built-in assertion types
  • Programmatic API — import { run } from "@plune-ai/cli"
$ plune run
2/2 passed · 0 failed · $0.0019
Docs npm GitHub
CI · every pull request MIT
eval-action plune-ai/eval-action

A thin GitHub Action that wraps the CLI. On every PR it runs your evals, compares against the base branch, and leaves a single sticky comment with the diff — with optional fail-on-regression.

  • Sticky PR diff — which evals regressed, which improved
  • fail-on-regression gates merges (opt-in)
  • Mock provider by default — zero cost, fork-safe
- uses: plune-ai/eval-action@v1
with: { fail-on-regression: true }
Docs Marketplace GitHub
Evaluate → Gate

From plune.yaml to a green gate.

Define your checks once. Run them locally while you build, then let the Action re-run them on every pull request and block the merge if quality drops.

  1. 1 Define

    Describe the checks in one file

    A provider, a prompt with {{vars}} from your dataset, and the assertions each row must pass. 

    version: 1
provider:
  type: anthropic
  model: claude-3-5-sonnet-latest
evals:
  - id: faq-grounding
    prompt: "Answer using only the provided context. {{question}}"
    dataset: datasets/faq.jsonl
    assertions:
      - type: llm-judge
        criteria: "Grounded in the context; invents no facts."
      - type: contains-any
        values: ["refund", "return"]
  2. 2 Evaluate

    Run it locally while you build

    plune run calls the model on every dataset row, evaluates each assertion, caches results, and reports pass/fail with token cost. Exit code 1 on any failure.

    $ plune run
    ✔ faq-grounding  · 2/2 assertions · 12 rows
    12/12 passed · 0 failed · 0 errored · $0.0142
  3. 3 Gate

    Catch regressions on the pull request

    eval-action re-runs the suite on the PR head and the base branch, diffs them, and posts a sticky comment. With fail-on-regression a passed → failed transition turns the check red. 

    # .github/workflows/plune-eval.yml
- uses: plune-ai/eval-action@v1
  with:
    config: plune.yaml
    fail-on-regression: true
    ◆ Plune eval diff 1 regression · 0 improvements
    evalbase → PR
    faq-grounding passed → failed ⬇ regression
    ✕ Plune detected a quality regression (pass→fail) — check failed.
Why it matters

Regressions stop slipping through.

Without Plune

You tweak a prompt to fix one case. Three other cases quietly get worse. Nothing fails — there's nothing checking — so it merges. You find out from a user, in production, days later.

silent quality drift
With Plune

The same change re-runs the whole suite on the PR. The diff shows exactly which evals went passed → failed, the check turns red, and the regression never reaches main.

caught on the PR
Assertion power

Ten ways to assert behaviour.

From plain string checks to JSON-schema, LLM-as-judge, and RAG metrics — mix and match per eval, or set defaults.assertions across the whole suite.

Text
  • exact-match output equals value (optional trim, ignore_case)
  • contains output contains value
  • contains-any output contains at least one of values
  • contains-all output contains every one of values
Structured
  • json-schema output validates against the JSON schema
Model-graded
  • llm-judge an LLM grades the output against criteria
  • semantic-similarity embedding similarity to reference ≥ threshold
RAG
  • faithfulness output is grounded in the context
  • answer-relevance output actually answers the question
  • context-precision context is relevant to the question
Full assertion reference
Provider-agnostic

Bring your own model. Pay nothing in CI.

Switch backends with one line in plune.yaml — the API key is read from the environment by provider type, never written to disk. And the built-in mock provider runs your suite with no key, zero cost, and fully deterministic output — so pull-request checks stay free.

Anthropic OpenAI OpenRouter mock · $0
plune.yaml
provider:
  type: anthropic # | openai | openrouter
  model: claude-3-5-sonnet-latest
# zero-cost, deterministic — no key needed:
$ PLUNE_MOCK_PROVIDER=1 plune run
Apache-2.0

Don't want to write every test by hand?

Cairn is our open-source QA agent. It logs in, explores your app, designs methodology-based test cases, and generates self-repairing @playwright/test specs — the front of the funnel into the Plune quality stack.

Explore Cairn

Ship LLM features with confidence.

Install the CLI, write your first plune.yaml, and add the Action to your repo. Both MIT, both free.

$ npm i -g @plune-ai/cli
CLI docs Action docs
CLI on npm Action on Marketplace cli repo eval-action repo
@plune-ai/cli v0.3.0 · eval-action v1.2.0 · MIT