Docs

Evaluations

Score every model output automatically. Attach evaluators to production traces or run them over a test set — rule-based checks are unlimited on Pro; LLM graders draw from your monthly AI-run pool.

Build an evaluator Jump to custom code

The evaluator library

Seventeen evaluators — plus human labels — grouped by what they do.

Rule-based

— Deterministic, fast, and unlimited on Pro.
Regex
Output matches a pattern
Keyword
Required / forbidden words present
Length
Character / word-count bounds
JSON Schema
Output is valid JSON matching a schema
Semantic
Similarity to a reference (cosine / jaccard / levenshtein)

LLM-as-judge

— A model scores the output. Metered against your monthly AI-run pool.
AI Grader
Your own rubric — pick provider, model, and binary / 5- / 10-point scale
Hallucination
Flags fabrication, contradiction, unsupported claims
RAG
Faithfulness, relevance, recall against retrieved context

Safety

— Dedicated graders for the risky stuff.
Toxicity
Harmful / unsafe content
Bias
Biased or discriminatory content
Prompt Injection
Jailbreak attempts and whether the model complied
PII Leakage
Personal data leaked in the output

Agentic

— Score how an agent got there, not just the final answer.
Tool Call
Correct tool selection and arguments
Transcript
The whole trajectory / reasoning path
Outcome
Did the run achieve its goal

Custom & integration

— When the built-ins are not enough.
Custom Code
Your own JS function, run in a secure sandbox
Webhook
Delegate scoring to your own endpoint
Human
Reviewer labels — feed judge↔human alignment

Custom code evaluator

Write a JavaScript function that scores an output. It runs in a QuickJS WebAssembly sandbox with no network or filesystem access and a 1-second time limit — so your code is safe to run, and a runaway function can never hang the platform.

Evaluator function
sandboxed
// Runs in a secure sandbox — no network, no filesystem, 1s time limit.
({ input, output, expectedOutput, metadata }) => {
  const required = ['refund', 'policy']
  const text = output.toLowerCase()
  const missing = required.filter((w) => !text.includes(w))
  return {
    score: missing.length === 0 ? 1 : 0,        // 0..1
    passed: missing.length === 0,               // optional (defaults to score >= threshold)
    reasoning: missing.length ? 'Missing: ' + missing.join(', ') : 'All terms present',
  }
}

The contract

Your function receives { input, output, expectedOutput, metadata } and returns { score, passed?, reasoning? }. score is clamped to 0–1; if you omit passed it defaults to score ≥ threshold.

Add one in the dashboard

1

Pick “Custom Code”

In build an evaluator, choose Custom Code from the type list.
2

Start from a template

Choose a starter — exact-match, matches-regex, valid-JSON, contains-keywords, word-count — then edit it.
3

Test against a sample

Paste a sample output and hit Run test. It executes the real sandbox and shows the exact pass/fail, score, and reasoning production will compute.
4

Attach & save

Add it to a test run or a production trace’s evaluator set. Done.

Run it from the API

POST /api/dashboard/evaluators/run
curl -X POST https://enprompta.com/api/dashboard/evaluators/run \
  -H "Content-Type: application/json" \
  -d '{
    "type": "CUSTOM",
    "input": { "output": "Our refund policy allows 30 days." },
    "config": { "evaluatorFunction": "({ output }) => ({ score: output.includes(\"refund\") ? 1 : 0 })" }
  }'

Is your grader as good as your reviewers?

Label some traces by hand, then measure how well an AI grader agrees with your reviewers — agreement, Cohen’s κ, precision/recall on the fail class, and inter-annotator agreement (the ceiling a judge can reach). When it’s off, the Optimize step rewrites the grader’s criteria and proves the gain on a held-out split.

Open judge alignment
Evaluations - Enprompta