DefendML ·
A safer AI answer is not proof of authorized data access
How to evaluate an AI fix beyond its final answer: observe data access, compare the retest, and check that legitimate work still succeeds.
Your AI assistant stopped giving the wrong answer. But can it still access data it should never see?
That is an illustrative review question—not a reported customer incident. For a SaaS team, the distinction matters: an assistant can produce a reassuring response after a tool has already requested or read another organization’s information. Reviewing only the answer misses that possibility.
Before approving a fix, ask what happened at the data boundary, not just what appeared in the chat window.
The answer and the action are different evidence
A refusal is text. A denied database read is an observed event. They can agree, but one does not establish the other. A tool request may fail safely, execute successfully, or remain unobserved; those outcomes should not be collapsed into a single “safe” label.
Prompt injection means outside content is used to redirect an AI system’s behavior. In a retrieval-assisted application, the content might arrive in a document returned by search rather than in the user’s own message. OWASP’s 2025 Prompt Injection guidance describes this indirect route and recommends testing trust boundaries alongside restricting privileges.
An example: an account-setup document crosses a tenant boundary
Imagine a customer asking how to set up an account. A retrieved help document also contains an instruction to read another organization’s account record. The user did not authorize that action. Here, “tenant” means the organization whose users and data must remain separate from other organizations.
If the assistant can call a record-reading tool, inspect the requested record, the caller’s verified organization, the store’s authorization decision, and whether any read actually executed. Changing the final response to hide the foreign value is not the same as stopping the read.
OWASP’s 2025 Excessive Agency guidance recommends enforcing authorization in downstream systems, rather than letting the model decide whether an action is permitted. In this example, the record service should check the caller’s authority on every request, even when an assistant generated it.
What our controlled demonstration actually established
DefendML’s controlled case used an owned application, synthetic records and deterministic code. No language model was tested. The application interpreted a structured retrieved instruction; the record-access observer witnessed an unauthorized read before the fix and a denied attempt afterward.
Local Garak custom probes and Promptfoo regression checks ran on October 12, 2026. The hosted DefendML original and signed linked retest ran on October 13. Hosted evidence established the read/denial boundary and the expected office-hours control answer. Separate local regression checks established account-setup answer preservation.
The hosted workflow linked the finding, advisory Fix Plan and retest, then showed “Verified resolved — tested boundary.” That is evidence about the tested boundary—not a result for an LLM, another application or every possible attack.
Related evidence: a real-model action experiment
The controlled RAG case used deterministic execution with no language model and reached “Verified resolved — tested boundary.” A later owned, synthetic experiment used a real model to select tool actions in one scenario repeated across ten before-and-after pairs. It observed prohibited execution before correction, denial afterward and legitimate work preserved, but its native-action remediation status remained INCONCLUSIVE under a different evidence contract. It does not inherit the RAG case’s verified-resolution status or establish general model effectiveness.
With an LLM, evaluate behavior and effects separately
A language model interprets the retrieved content rather than following our deterministic instruction parser. It may ignore the instruction, repeat it, or produce a tool request. OWASP notes the stochastic nature of generative behavior; one successful refusal is therefore not a universal result.
For your own authorized evaluation, record the model version and settings, application prompts, retrieved document snapshot, tool permissions, caller identity and relevant code version. Keep them comparable before and after the fix. Changing the model and the access check together makes attribution harder.
Plan repeated trials and report the trial count, outcomes and missing observations. Record seeds where supported, but do not assume a seed or low temperature guarantees identical behavior. Choose trials for the application and decision at hand; there is no universal sample count that proves safety.
Observe four stages separately: what was retrieved, what request the model produced, what the tool executed, and what the data service allowed. These are evaluation recommendations, not LLM test results completed in the controlled case.
Good evidence connects the failure to the fix
Start with a witnessed failure: the prohibited access occurred, tied to a specific request and authorized test scope. Then identify the correction that should stop it. A targeted store-level authorization fix is different from an answer filter that suppresses the visible symptom.
Retest comparable conditions and retain the attempted action, denial or execution result, and final answer. A timeout or missing observer record is uncertainty or a technical failure—not evidence that access was blocked. Absence of a read is meaningful only when the observer covers the relevant path and observation window.
Also exercise legitimate tasks. A system that denies every read can pass the attack check while breaking the product. Compare expected answers or effects for benign tasks independently of the attack result.
Preserve enough sanitized evidence to review the comparison. Signatures can help establish integrity and attribution; they do not make an incomplete or untrusted observer truthful. Our demonstration used owner-controlled instrumentation, not independent third-party attestation.
Questions to ask before approving a fix
Use these questions in your next pre-deployment security review:
- What exact action or data access was prohibited, and whose authority was checked?
- Did we observe the actual effect, or only the assistant’s answer?
- What changed, and are the original test and retest comparable?
- For model-backed behavior, which configuration and repeated trials were recorded?
- Does the observer cover the relevant tool and data paths? What remains unknown?
- Did legitimate tasks still work, with separate evidence?
- Can another reviewer follow the sanitized finding-to-retest evidence?
Primary-source references: OWASP’s 2025 LLM Top 10 entries linked above, reviewed October 14, 2026. Testing recommendations are DefendML analysis, not a claim of completed model evaluation.