Skip to main content
Caronte
Docs

Admin guide · Platform

Evaluations

Score Prompt Vault versions against test datasets. The Evaluations page has four tabs: Datasets, Evaluators, Runs and Schedules.

Datasets

The test cases a run is scored over. Each dataset is a named set of rows; the list shows its size, who created it and when it last changed.

The Datasets tab listing one dataset with its row count, creator and last update. The creator's name is redacted.
Datasets — the inputs a run replays (creator name redacted).

Runs

Each run executes a prompt version over a dataset with a chosen model, tracking running / pending / completed / failed counts. Start one with New run.

The Evaluations Runs tab listing a run with its prompt, dataset, model and completion counts.
Runs — one prompt version × dataset × model per run.

Evaluators

The scorers applied to each output: exact match, fuzzy match, or an LLM judge with its own prompt.

The Evaluators tab listing an LLM-judge evaluator. The creator's name is redacted.
Evaluators — exact, fuzzy or LLM judge (creator name redacted).

Schedules

Run an evaluation automatically on a cadence — e.g. a nightly accuracy check.

The Schedules tab listing a monthly guardrail analysis with its frequency, run limit, next and last run and status.
Schedules — the cadence, the stop condition and where it got to.
  • Frequency — every N day(s)/…, at a given time; the form previews the UTC cron and the next five firings in your timezone.
  • Stop condition — never, after N runs, or until a date.

Enabling the feature (admin)

Evaluations must be switched on once per organization, under Admin → Evaluations.

The Admin Evaluations page with the feature toggle and the provisioned runner API key.
Admin → Evaluations — the master switch and the runner key.

note · The runner key

Enabling provisions a dedicated runner API key on the internal Evaluations project (encrypted, decrypted only by the worker at run time) — runs execute through the gateway like any other traffic. When off, run executions are blocked and the Evaluations UI is read-only.