The Only 4 Metrics That Matter for Production Prompts

There are dozens of metrics you could track for your production prompts. Latency, token count, perplexity, BLEU score, ROUGE score, human preference ratings, cost per call, error rate, cache hit rate, and on and on. Most of them are noise.

After working with hundreds of teams and analysing millions of prompt runs, we have cut the list down to four. These are the metrics that actually drive decisions — the ones where a change in the number leads to a change in what you do.

Metric 1: Task completion rate

This is the north star. Everything else is secondary. A prompt that is fast, cheap, and eloquent but fails to complete the task 30% of the time is worse than a slow, expensive prompt that works every time.

Measurement is straightforward for structured outputs: did the JSON parse? Did all required fields get populated? Did the values fall within expected ranges? For unstructured outputs, use an LLM-as-judge approach with a rubric specific to the task.

Metric 2: Cost per successful completion

Note the emphasis on "successful." If your prompt costs $0.02 per call but fails 40% of the time (requiring retries), your real cost per successful completion is $0.033. Teams that track cost per call instead of cost per completion systematically underestimate their spending.

This metric combines model cost, prompt efficiency, and reliability into a single number. When you optimise for cost per successful completion, you naturally make the right trade-offs — you will not switch to a cheaper model if it tanks your completion rate.

Metric 3: P95 latency

Median latency is misleading. If your median is 800ms but your P95 is 4 seconds, one in twenty users is having a terrible experience. P95 latency is the metric that reflects reality for your tail users.

Latency matters more than most teams think. In user-facing applications, every additional second of wait time increases abandonment. In batch processing, high P95 latency creates bottlenecks that slow the entire pipeline.

Metric 4: Output consistency

A prompt that produces excellent output on Monday and mediocre output on Thursday is unreliable, even if the average quality is high. Consistency matters because your downstream systems (and your users) need predictability.

Measure consistency by running a fixed evaluation dataset weekly. Track the standard deviation of your quality score over time. If the standard deviation is increasing, something is drifting — either the model was updated, your input distribution shifted, or the prompt has accumulated conflicting instructions.

Putting it together

These four metrics — task completion rate, cost per successful completion, P95 latency, and output consistency — give you a complete picture of prompt health. Track them on a dashboard, set alerts on regressions, and review weekly.

Enprompta tracks all four metrics automatically for every prompt on our platform. If you are building your own monitoring, start with task completion rate — it is the easiest to measure and the most impactful to optimise.