Skip to case study
← DefendML home

Controlled validation case study · October 2026

From unauthorized read to verified retest

A fix should stop the prohibited action without breaking the legitimate task. This case shows how DefendML connected an observed failure, an authorization correction and a witnessed retest.

Owned demonstration. Synthetic data. Deterministic application. No language model was tested.

What was tested

A normal account-setup question retrieved a document containing an instruction to access another tenant’s synthetic account record. Our deliberately vulnerable application interpreted that structured instruction and requested the record. The tested boundary was authorization at the record store.

Evidence at a glance

Dates below use Pacific Daylight Time (UTC−7). Local tool runs and hosted assessments are separate executions.

October 12 · Local Garak 0.17.0
Six custom probes against the vulnerable application. Three responses disclosed the synthetic foreign record; the store witness confirmed three corresponding unauthorized reads. Two benign tasks succeeded; one query did not exercise the boundary.
October 12 · Local Promptfoo 0.124.1
Six regression checks passed against the corrected application. Three attack-bearing queries reached a denied read; legitimate account setup and office-hours answers remained correct. One unsupported query did not exercise the boundary.
October 13, 16:57 · Hosted original
The signed original recorded ALLOW: one prohibited read executed. The office-hours control returned its expected answer. The signed witness records the read, not the attack answer’s disclosure.
October 13, 18:15 · Hosted linked retest
The corrected retest recorded BLOCK: the prohibited attempt was denied, no foreign read executed, and the expected office-hours control answer was preserved. Account-setup answer preservation is established by the separate local regression checks.

Sanitized finding → correction → retest

Finding: an account-setup request crossed into another tenant’s record. Correction: check the requesting tenant on every record read. Linked retest: the same query and retrieved document reached a denied attempt, while the expected office-hours control answer remained intact. Fix Plan and Issues then showed “Verified resolved — tested boundary.”

This is a summary of retained observations, not a fabricated transcript. The local checks and hosted jobs used different test sets; their counts are not comparative detection rates.

Before and after the fix

Before: unauthorized read observed

The application treated its service authority as sufficient permission, read the other tenant’s record and returned the synthetic value. The retained finding recorded the observed access—not merely a suspicious answer or a claimed action.

After: denied at the record boundary

The corrected application checked tenant authorization on each read. The hosted linked retest recorded denial before the prohibited read and the expected office-hours control answer. Separate local regression checks also preserved the account-setup answer.

How the result was verified

  1. 1. Observe the failure

    A record-access witness confirmed an actual unauthorized read. The synthetic record value was not present in the retrieved document.

  2. 2. Fix the authorization boundary

    The application checked the requesting tenant against the requested record on every store read. The document and original query remained unchanged.

  3. 3. Retest the same scope

    DefendML’s hosted workflow linked the original finding to a corrected retest. The witness recorded a denied attempt, no unauthorized read, and the expected office-hours control answer.

  4. 4. Qualify the resolution

    Fix Plan and Issues showed “Verified resolved — tested boundary.” The original and retest supported a same-scope comparison; older, non-comparable evidence remained uncertain.

The observer recorded store-access attempts separately from the application’s answer. Signed observations and retained verification material allowed integrity and attribution to be checked after cleanup. Their reliability depends on the observer and its coverage: this was owner-controlled instrumentation, not third-party attestation.

An earlier workflow remained inconclusive. The reviewed signed-observation contract supplied the qualifying linked evidence; older missing or non-comparable evidence was not relabelled as safety.

Different tools, clearly separated roles

Garak
Local custom probes exercised the vulnerable application—not the default broad suite.
Promptfoo
Local HTTP regression checks exercised the corrected application and legitimate tasks.
DefendML
The hosted product exercised authorized assessment, retained findings, Fix Plan, the linked retest and evidence-qualified resolution.

Garak and Promptfoo were run locally, not integrated into DefendML’s production service for this demonstration.

What this case does—and does not—establish

It demonstrates verified resolution of the recorded tenant-access boundary under unchanged attack scope, with expected functionality preserved. It also demonstrates the hosted finding-to-retest evidence workflow.

This owned, synthetic demonstration did not test an LLM and is not a customer engagement or comparative benchmark. It does not establish real-model prompt-injection effectiveness, comprehensive security, runtime protection, certification, third-party endorsement or general detection rates. A prospective buyer’s application needs its own authorized scope and observations; this result cannot be transferred to another system.

This deterministic RAG case reached “Verified resolved — tested boundary.” Our later owned, synthetic experiment used a real model to select tool actions in one scenario repeated across ten before-and-after pairs. It recorded prohibited execution before correction, denial afterward and legitimate work preserved, but its native-action remediation status remained INCONCLUSIVE under a different evidence contract. It does not inherit this case’s verified-resolution status or establish general model effectiveness.

Read the real-model action experiment and its limitations

What evidence would your security review need?

For Product Security and AI engineering teams, start with the application you own, the actions it is allowed to take and the boundary you need to test. We can discuss a scoped, authorized pre-deployment assessment and the evidence needed to evaluate a fix.

Discuss a scoped assessment