AI Inference Data Leakage

An enterprise's customer support AI platform had been configured with zero-retention settings , customer conversations were processed through the vendor's LLM inference infrastructure and no conversation data was retained after each session ended. The enterprise had verified this configuration through the vendor's DPA and admin console settings. What the enterprise had not assessed was the in-session data exposure during the inference call itself. The vendor's AI platform loaded customer context at the start of each support session , the customer's account history, open tickets, subscription details, and previous interactions , and maintained this context in the model's context window throughout the session. A security researcher testing the platform demonstrated that under certain conditions, carefully crafted user inputs could cause the model to reproduce content from earlier in the same session context that had not been part of the explicit conversation , specifically, account details and ticket history that had been loaded as system context at session initialisation but not displayed to the user. The zero-retention configuration correctly prevented data from persisting after the session. It did not prevent data that was present in the session context from being accessible to in-session prompt manipulation. The session window was the data exposure window.

What is AI Inference Data Leakage, Really?

AI inference data leakage is the exposure of data that is present in an AI model's inference environment , the context window, the retrieval-augmented system prompt, connected data sources loaded as context, and the model's intermediate processing states , to extraction through user inputs, prompt manipulation, or side-channel techniques during the inference session. Unlike post-session data retention, which persists data after the session ends, inference-time data leakage occurs during the active session and may be exploitable regardless of retention configuration.

The context window exposure problem is the primary mechanism. Modern AI assistants and customer-facing LLM applications load significant context into the model's context window at session initialisation , customer profiles, account histories, permissions, knowledge base content, and system instructions. This context is used by the model to generate informed, personalised responses. It is also present in the model's processing environment and is potentially accessible to prompt manipulation techniques that cause the model to reference or reproduce context content it was not explicitly asked about.

The system prompt extraction dimension is a related attack surface. AI applications typically include system prompts , instructions to the model that define its role, constraints, and access to connected resources. These system prompts may contain sensitive configuration information, security constraints, or references to connected data sources. Prompt injection attacks that cause the model to reproduce or describe its system prompt expose configuration information that was intended to be opaque to end users. In multi-tenant AI deployments, system prompts may contain tenant-specific configuration that one tenant should not be able to extract.

The cross-session contamination risk applies to AI deployments where conversation history is maintained across sessions , long-term memory systems, conversation summarisation, and persistent context features. In these configurations, data from one session may be present in the context of subsequent sessions. Cross-session contamination occurs when data from a previous session becomes accessible in a later session through the persistent context mechanism , potentially exposing data from one user's session to a different user who interacts with the system in a subsequent session.

The multi-tenant isolation problem is the specific enterprise risk in shared AI deployment models. Vendor AI platforms that serve multiple enterprise customers on shared infrastructure create cross-tenant inference data leakage risks , specifically the risk that context loaded for one tenant's session is accessible to another tenant's inference calls through shared model state, insufficient isolation between tenant contexts, or prompt injection that crosses tenant boundaries. These cross-tenant risks are analogous to cross-tenant database isolation failures but applied to the AI inference layer.

Why this matters

AI inference data leakage matters for TPRM because zero-retention configuration , while necessary , is not sufficient to protect sensitive data that passes through the AI inference environment. Customer data loaded as session context, system prompts, and retrieved knowledge base content are all present in the inference environment during the session and require protection at the inference layer as well as the retention layer.

Where most teams get this wrong

The most consistent failure is treating zero-retention configuration as the complete data protection answer for AI deployments , without assessing the in-session exposure risks that retention controls do not address.

• Zero-retention equated with complete data protection
• In-session context window exposure not assessed
• System prompt confidentiality not evaluated
• Cross-session contamination in persistent context deployments not assessed
• Multi-tenant inference isolation not assessed for shared platform deployments

What good looks like

Mature AI inference data protection programmes address both retention controls and in-session exposure controls , specifically context window scoping, system prompt confidentiality testing, and multi-tenant isolation validation alongside standard retention configuration.

• Context window scope minimisation , load only necessary context for each session
• System prompt confidentiality testing , specifically testing whether prompts can be extracted
• Multi-tenant isolation validation , confirming cross-tenant context isolation
• In-session prompt injection testing , testing for context extraction via in-session prompts
• Cross-session contamination assessment , for deployments with persistent context

Tooling

LLM Security Testing , Garak, PromptBench for in-session extraction testing

LLM security testing tools that specifically test for context extraction and system prompt disclosure provide the in-session security assessment that retention audits do not. For TPRM practitioners, asking whether the vendor has conducted in-session prompt injection testing specifically targeting context window extraction provides a specific inference-time security question.

Governance challenges

The governance challenge with AI inference data leakage is the inference-time visibility gap , the enterprise cannot directly observe what data is present in the model's context window during inference. The governance resolution is vendor transparency about context loading practices and independent in-session security testing that validates isolation between context layers.

• Ask what data is loaded into the context window at session initialisation
• Request in-session prompt injection test results for context extraction
• Ask about multi-tenant inference isolation on shared infrastructure
• Assess system prompt confidentiality , can prompts be extracted by users
• Evaluate cross-session contamination controls for persistent context deployments

If you are a small team

For any vendor AI platform where sensitive customer data is loaded as session context, ask one question that zero-retention assessment does not answer: what customer data is loaded into the model's context window at session initialisation , and has the platform been tested specifically to confirm that in-session prompt manipulation cannot cause the model to reproduce context content that was not explicitly part of the user-facing conversation? That question reveals whether the inference-time exposure has been assessed alongside the post-session retention.

• Ask what customer data is loaded into the context window at session start
• Ask whether in-session prompt injection testing for context extraction has been conducted
• Ask about multi-tenant inference isolation on shared infrastructure
• Ask about system prompt confidentiality testing

What to require

Ask directly:

"What customer data is loaded into the model's context window at session initialisation , and has your platform been tested specifically to confirm that in-session prompt manipulation cannot reproduce context content that was not part of the explicit user conversation?"

Expect as evidence

• Context window content description , what is loaded at session start
• In-session prompt injection test results for context extraction
• Multi-tenant inference isolation validation
• System prompt confidentiality test results

A vendor who confirms zero-retention configuration should be asked about in-session context exposure. Zero retention addresses post-session persistence. In-session testing addresses the exposure window during the session itself. Both are required.

How to evidence it

• Context window content documentation
• In-session security testing records
• Multi-tenant isolation validation records
• System prompt confidentiality assessment

Key Takeaway

Zero retention: configured. Customer account context in the context window: present throughout the session. In-session prompt manipulation extracting that context: possible under specific conditions. The retention control correctly addressed post-session persistence. The inference-time exposure was a different risk that the retention assessment did not cover. Zero retention is a necessary control. It is not sufficient when the data that should not persist is present in the inference environment during the session and accessible to in-session extraction. Context window scope minimisation, system prompt confidentiality testing, and in-session injection testing are the controls that address what retention controls do not.

More on AI governance

• AI API Key Exposure
• AI Adversarial Attacks
• AI Attack Surface Expansion