Eval diffs on every PR with eval-action

Evals only protect you if they run on every change. eval-action is the GitHub Action that puts the Plune CLI on autopilot: on each pull request it runs your suite, compares it to the base branch, and tells you — right in the PR — exactly what changed.

It contains no evaluation logic of its own. It orchestrates the CLI, so it always behaves exactly like the @plune-ai/cli version it pins. On every PR the Action:

1. runs plune run on the PR head,
2. runs plune run on the base ref in a detached worktree, so your checkout is untouched,
3. runs plune diff between the two, and
4. posts (or updates) one sticky comment with the diff — which evals regressed, which improved.

The diff is honest about what counts. Only a real passed → failed transition is a regression; an eval already failing on the base branch won’t suddenly turn your check red. Flip fail-on-regression: true and that regression blocks the merge.

# .github/workflows/plune-eval.yml
name: Plune eval diff
on:
  pull_request:
permissions:
  contents: read
  pull-requests: write        # so the Action can post the sticky comment
jobs:
  plune:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - uses: plune-ai/eval-action@v1
        with:
          config: plune.yaml
          # use-mock defaults to true — zero cost, deterministic, no API key
          # fail-on-regression: true   # block merges on a pass→fail regression

By default it runs on the built-in mock provider: no API key, zero cost, fully deterministic — so you can wire it up today and every PR gets a diff comment without spending a cent. Point it at a real model by setting use-mock: false and supplying a provider key from a repository secret. On fork PRs the token is read-only, so the Action runs the evals and skips the comment with a notice instead of failing.

eval-action is the Gate step in Generate → Evaluate → Gate. It’s MIT-licensed; pin it with plune-ai/eval-action@v1.