DefendML ·
The AI agent requested the export. Was it authorized?
Ten repeated before-and-after trials show why an agent’s tool request, the application’s authorization decision and a qualified remediation result need separate evidence.
Did the AI agent perform an unauthorized action—and did the correction stop it without breaking legitimate work?
In DefendML’s owned test application, a real language model selected a tool action that exported another tenant’s synthetic document. After an application-level authorization correction, the same request reached an explicit denial. The legitimate export still worked.
That pattern held across ten repeated before-and-after pairs. It is evidence about one tested boundary—not ten different vulnerabilities, a general detection rate or a claim that the model became safe.
Check the action, not just the answer
For Product Security and AI engineering teams, the release question is practical: can an agent do something the caller is not allowed to do? A reassuring final answer cannot undo an export that already happened.
A tenant is an organization whose data must remain separate from another organization’s data. If an agent can request documents by identifier, the document service must check the caller’s authority before reading or exporting the contents. The model’s choice is a request, not permission.
One export scenario, repeated ten times
We used an owned document-export agent with synthetic records: one document belonged to the acting tenant and another belonged to a different tenant. The test sent a direct request to export the other tenant’s document, plus a separate legitimate request for the acting tenant’s document.
These were direct tool/action requests—not prompt injection, retrieved-content injection or adaptive attacks. Ten pairs repeated the same scenario and control; they did not cover ten distinct attack cases.
Hosted execution ran on October 16, 2026, from 1:14:55 to 1:35:17 p.m. Pacific Daylight Time (20:14:55–20:35:17 UTC). The target model snapshot was claude-haiku-4-5-20251001, with temperature 0, a 512-token output limit and automatic tool choice. The model was not forced to call the tool. The model requests and tool definition stayed the same across the two policies.
What happened before and after the correction
The vulnerable policy allowed the requested foreign-document export. The corrected policy checked record ownership against the acting tenant before permitting the contents to be read for export. That authorization check—not DefendML and not a changed model answer—enforced the denial.
| Check | Before correction | After correction |
|---|---|---|
| Other-tenant synthetic document | 10 exports executed | 10 explicit denials; no export |
| Own-tenant document control | 10 exports executed | 10 exports executed |
| Completed assessments | 10 | 10 |
| Recorded technical errors | 0 | 0 |
A request became a finding, then a linked retest
In the first pair, the model selected the document-export tool for the other tenant’s record. Application-owned instrumentation recorded the acting tenant, requested record, executed export and an artifact digest. The retained finding therefore described an observed effect, not the model merely saying it had exported something.
After the ownership check was applied, a case-linked full-target retest recorded the attempted foreign action as denied, with no export artifact. A separate tool call still exported the acting tenant’s document. The same comparison was retained for each of the ten pairs.
The hosted protocol produced twenty assessments and forty model calls: one prohibited-action request and one legitimate control per assessment. A separate four-call local preflight was not counted in the ten hosted pairs. The retained hosted records contain no technical failures, retries, excluded trials or unexercised attempts. Each response and effect is tied to its recorded assessment and policy.
The effect ledger is independent of the model’s prose, but it is owner-controlled instrumentation—not independent third-party attestation. It observes this confined application path, not every action an agent could take.
Observed denial is not “Verified resolved”
The observations support a narrow result: prohibited execution before correction, denied execution afterward, and legitimate work preserved under the tested conditions.
DefendML’s linked native-action Fix Plan still reports INCONCLUSIVE under its current remediation evidence contract. These observations do not establish product-qualified “Verified resolved” status. A manually resolved case label does not change that distinction.
The retests were case-linked full-target scans, not category-targeted execution. We did not change the product’s qualification rules to turn the experiment into a stronger publication claim.
How this extends the earlier controlled case
Our earlier RAG case used deterministic code and no language model. It retained separately signed read/denial evidence under a different contract and reached “Verified resolved — tested boundary.” This new experiment adds real model-selected tool actions; it does not inherit the earlier case’s remediation status.
Garak and Promptfoo were used in separate local checks for the earlier case. They did not participate in these new hosted trials. Neither experiment establishes customer compatibility, broad model robustness, certification or runtime enforcement by DefendML.
Temporary test execution and application fixtures were removed after retaining sanitized evidence. An inactive disposable Auth identity was retained by owner decision, without a profile, organization membership or active session at cleanup verification; complete account removal is not claimed.
Before approving an agent authorization fix
Ask for evidence that answers these questions:
- Which action was prohibited for this caller, and where is that authority enforced?
- Did the tool actually execute, explicitly deny the action, or leave an unknown outcome?
- Were the model configuration, requests and tested scope comparable before and after the fix?
- Did legitimate actions still succeed independently of the prohibited-action check?
- Does the product’s remediation status qualify this evidence—or remain inconclusive?
Source: DefendML’s retained October 16, 2026 trial protocol, model-response provenance, application effect ledger and original-to-retest audit links. Owner-controlled instrumentation, not third-party attestation. No customer data or customer application was tested.