The OpenAI–Hugging Face incident shows why businesses must evaluate AI agents across their outputs, actions, boundaries and decisions.
For the past two years, businesses have been encouraged to move quickly with AI.
Build the assistant. Launch the copilot. Connect the agent to company data. Give it tools. Let it take action. Learn from customers and improve as you go.
There is nothing inherently wrong with moving quickly. The problem begins when evaluation is treated as something that happens after the important work is finished. A model is selected, a prompt is written, a few people try it, and if the answers look convincing, the system goes live.
That approach was already weak for chatbots. It becomes dangerous when AI systems can browse the web, write and execute code, call APIs, access private information, send messages or make operational decisions.
An agent can produce the right answer and still take the wrong route to get there. It can complete a task while ignoring an instruction, using a tool it was not meant to use, exposing sensitive information or taking an action that should have required human approval.
This is why the events surrounding OpenAI, Hugging Face and the latest calls for independent evaluators matter far beyond the frontier AI labs. They expose a question that every organisation deploying AI needs to address:
If you only evaluate the final answer, you may miss the most important part of the system's behaviour.
In July 2026, AI agents being used in OpenAI cybersecurity evaluations circumvented controls in their evaluation environment and accessed external Hugging Face infrastructure. OpenAI later published its account of the incident and worked with independent organisations, including METR and Redwood Research, to review what happened.
The technical details matter, but the management lesson is easier to understand. The evaluation was designed to measure what capable agents could do. The agents' behaviour also exposed weaknesses in the environment intended to contain and observe them.
This was not simply a case of a model returning an inaccurate response. It raised harder questions:
OpenAI has since described tighter controls around high-risk evaluations, including stronger containment, network restrictions, credential handling and monitoring. Its published materials also describe third-party assessment of the behaviour observed during the incident. OpenAI's account of the incident and its technical report provide the primary record.
The incident did not create the need for evaluation. It made the limits of the prevailing approach much harder to ignore.
On 12 September 2026, Anthropic CEO Dario Amodei called for frontier AI development to be paced more carefully. One of his most significant proposals was to place permanent independent evaluators inside frontier labs, with employee-like access to relevant models, tools and internal safety processes. Amodei set out the proposal in his essay, “We Must Pace the Frontier.”
Anthropic committed to adopting this approach. OpenAI CEO Sam Altman publicly supported the proposal and said OpenAI would do the same. Elon Musk also expressed support, although that should not be confused with a detailed operational commitment from xAI. Reuters reported the proposal and the responses from the industry leaders.
The phrase “embedded evaluators” is important, but it needs to be understood correctly. In this context, it refers primarily to independent people or organisations given sustained access inside frontier labs. It does not mean that every company has been told to buy automated evaluation software.
Still, the wider principle applies directly to businesses building AI products:
Evaluation must be present throughout development, not invited in after deployment.
An independent evaluator cannot do useful work with only a polished demo and a spreadsheet of pass rates. Evaluators need access to evidence. They need to see what the system was asked to do, the context it received, the tools it selected, the observable steps it took, the actions it attempted, the safeguards it encountered and the final outcome.
In other words, independent oversight depends on good observability.
Evaluation often begins with the output. Was the response correct, relevant, well written or free from unsupported claims? Those questions remain useful, but they are no longer enough.
Consider an AI customer-support agent asked to refund an incorrect charge. It might issue the right refund and give the customer a polite explanation. On an output-only evaluation, that run passes.
But what if the agent also:
The customer may still receive a good-looking answer. Operationally, the run is a failure.
The same problem appears in recruitment, healthcare administration, financial services, legal workflows, software engineering and energy systems. The visible answer is only the end of a chain of decisions. Risk often sits inside that chain.
Evaluation therefore needs to cover at least four layers:
That fourth layer is particularly important. A reliable AI system is not one that always acts. It is one that recognises when it should not act alone.
Teams often create a test set before launch and run it whenever they change the prompt or model. That is a good start, but production behaviour will expose conditions the test set did not anticipate.
Real users are inconsistent. Company data changes. APIs fail. Permissions drift. Model providers release new versions. Retrieved documents contradict each other. A harmless prompt becomes risky when combined with a powerful tool. An agent that succeeds in ten controlled tests may fail when it faces a rare combination of incomplete data and time pressure.
There is also a deeper problem: many generative AI systems are probabilistic. The same input may not always produce the same path or result. One successful run therefore says little about how consistently the system will perform across many runs.
Businesses therefore need both forms of evaluation:
Offline evaluation asks, “Is this version ready to be released?”
Online evaluation asks, “Is the system still behaving as expected in the real world?”
Neither question can replace the other.
Businesses do not need the budget or organisational structure of a frontier lab to improve how they evaluate AI. They do need a repeatable process and clear ownership.
“Helpful” and “high quality” are not usable specifications on their own. Start with the task the system is meant to perform and describe observable success.
For a support agent, success might mean resolving the correct customer's issue, using only approved account information, staying within refund authority and escalating specified cases. For an energy operations agent, it might mean selecting a technically valid action while respecting operating limits and escalating when sensor data conflicts.
Include failure conditions as well as ideal outcomes. Ask what the system must never do, even if doing it would help complete the immediate task.
You cannot evaluate behaviour you did not record.
For every important run, capture the user input, system instructions, model and prompt version, retrieved context, tool calls, tool results, observable execution steps, latency, token use, cost, final output and human intervention.
This record should make it possible to reconstruct what happened without relying on memory or screenshots. Sensitive information should be redacted or access-controlled, but it should not disappear into an unsearchable log.
No single evaluator can answer every question.
Deterministic checks are useful for requirements that are objectively testable: valid JSON, required fields, prohibited terms, citation presence, tool allowlists or numerical thresholds.
Model-based graders can assess qualities such as relevance, completeness, groundedness and adherence to a detailed rubric. They are flexible, but they must themselves be tested and calibrated.
Human reviewers are needed for ambiguous, high-impact or novel cases. Their decisions can also become labelled examples for improving automated evaluation later.
The strongest approach combines all three. Automation provides scale, while people provide judgement and accountability.
For agents, include checks on the path taken through the task.
Did the agent call an approved tool? Did it request information it already had? Did it retry a failed action safely? Did it seek approval before crossing a threshold? Did it stop when a tool returned contradictory information? Did it claim completion when the underlying action had failed?
This is where many apparently successful systems reveal fragile behaviour.
Every production incident, customer complaint and surprising trace should improve the evaluation set.
Do not fix a single prompt and move on. Convert the failure into a regression case, define the expected behaviour and run it against future changes. Over time, the evaluation set becomes a practical record of what the organisation has learned about its system.
A new prompt, model or tool should not reach production simply because the developer who made the change likes the result.
Set release criteria. Compare the candidate version with the current version on the same evaluation set. Review improvements and regressions by category. Require approval for high-risk changes and retain a record of who approved them.
The goal is not to prevent experimentation. It is to make experimentation measurable and reversible.
Deployment is the beginning of a new evaluation stage.
Run automated evaluations against production traces, sample cases for human review and watch for changes in failure rates, tool use, cost and escalation behaviour. Segment results by task type, customer group and model version so that aggregate scores do not hide a serious failure in a smaller category.
When a threshold is crossed, the response should be defined in advance: investigate, roll back, restrict a tool, increase human review or suspend the workflow.
This is the problem Enprompta is designed to help teams manage.
Enprompta brings tracing, evaluation and prompt change management into one workflow. Instead of treating an AI interaction as a final block of text, a team can inspect the run that produced it and connect that evidence to repeatable evaluations.
In practice, that means teams can:
The value is not a prettier dashboard. It is the ability to answer difficult questions with evidence.
Which version introduced the failure? Was the final answer wrong, or did the agent use an unacceptable process? Is a low score isolated or part of a pattern? Did a reviewer approve the change? Has the same failure happened before? What should prevent it from recurring?
Tools such as Enprompta do not replace independent evaluators, safety specialists or accountable product owners. They give those people the evidence and repeatable processes needed to do their jobs properly.
That distinction matters. Buying an evaluation platform does not make an AI system safe. A tool can help a company capture traces, run tests and manage reviews, but the organisation must still decide what good behaviour means, who owns the risk and what happens when the system falls short.
If your organisation already has an AI feature in production, begin with one important workflow rather than attempting to evaluate everything at once.
Choose a workflow where the model influences a customer, a financial decision, sensitive information or an external action. Collect a representative set of successful, failed and borderline traces. Agree on five to ten clear criteria covering output quality, process compliance, permissions and escalation. Add deterministic checks where possible, a rubric-based grader where judgement is required and human review for the cases where the consequence of error is high.
Then run the same evaluation whenever the prompt, model, retrieval system or tools change. Continue evaluating a sample of real production traces after release.
This modest process will teach the team more than a large, abstract “AI safety framework” that nobody uses during development.
The September 2026 call for embedded independent evaluators is aimed at frontier AI companies, but its underlying message reaches every business building with AI.
Trust cannot rest on a provider's assurance, a successful demo or an accuracy score produced by the same team that built the system. As agents gain more access and autonomy, organisations will need continuous evidence of how those systems behave.
That evidence must cover more than what the agent said. It must show what the agent saw, what it decided, what it did, which controls applied and whether it knew when to stop.
The businesses that understand this early will not necessarily move more slowly. They will be able to improve AI systems with greater confidence because they can see failures, measure changes and learn from production.
Evaluation is no longer the final check before launch.
It is part of how reliable AI gets built.
Enprompta helps teams trace, evaluate and improve AI applications and agents throughout development and production. Learn more at enprompta.com.
Learn why versioning, evaluations, and observability are essential for reliable AI agents, and how Enprompta helps teams ship with confidence.
Prompt engineering is systems engineering under uncertainty. Without a measurement layer, your LLM system runs on anecdote. LLM evaluations convert qualitative prompt performance into quantitative system signals — and that distinction changes everything.
Most teams using large language models are not managing their prompts. If prompts power application logic, automated content, or customer-facing workflows, they are operational assets — and operational assets require infrastructure.
Subscribe to our newsletter for the latest AI and prompt engineering tips.