Supply Chain AI Trust Boundaries

An enterprise deployed an AI assistant for their finance team , a RAG-based system that answered questions from the enterprise's internal financial policies, procedures, and approval matrices. The tool had been intended to reduce the finance team's internal support burden by enabling employees to self-serve answers to policy questions rather than querying the finance team directly. The AI had been configured with appropriate access controls: it could only be accessed by authenticated enterprise users, and it only answered from documents within its configured knowledge base. An attacker who had compromised the account credentials of a low-privilege employee used the AI assistant to conduct a reconnaissance operation , asking detailed questions about financial approval thresholds, dual authorisation requirements, CFO review triggers, and international wire transfer procedures. The AI answered all questions accurately and helpfully, drawing from the enterprise's policy documents exactly as designed. The attacker used the information to calibrate a business email compromise attack , crafting a fraudulent payment request for an amount precisely below the dual authorisation threshold that would have required a second approver. The AI assistant had been trusted to serve the finance team. It had also served an attacker with a compromised credential. The AI's trust boundary was authentication. It could not assess whether the authenticated user was acting within the legitimate scope that the trust was intended to cover.

What are AI Trust Boundaries, Really?

AI trust boundaries define the scope of what an AI system should and should not do, reveal, or enable based on who is asking and in what context. Trust boundary failures occur when AI systems are trusted to perform sensitive functions , answering sensitive policy questions, executing privileged actions, retrieving confidential information , without adequate controls to verify that the requesting user's intent is within the scope that the trust was intended to extend to. Authentication establishes identity. Trust boundaries determine whether that identity's request falls within the appropriate scope.

The authentication-as-trust-boundary limitation is the core problem. Enterprise AI deployments commonly use authentication as the primary trust boundary , if the user is authenticated, the AI serves their requests. This is appropriate for AI systems serving general information needs. It is inadequate for AI systems that answer sensitive questions, execute privileged actions, or retrieve confidential policy information that could be used to calibrate attacks against the enterprise's controls. Authentication confirms identity. It does not confirm intent or appropriate scope of information access.

The reconnaissance threat is the specific AI trust boundary risk. AI assistants that consolidate sensitive operational information , approval thresholds, security controls, system architectures, personnel procedures , into a queryable knowledge base create a reconnaissance capability for any authenticated user. A legitimate employee uses this for self-service policy answers. An attacker with a compromised credential uses it to gather the specific operational intelligence needed to calibrate attacks that circumvent the enterprise's controls. The AI's helpfulness is the attacker's advantage.

The context-aware trust boundary problem is the governance solution and its complexity. Implementing trust boundaries that reflect the context and intent of requests , rather than only authentication , requires understanding what constitutes a legitimate versus suspicious request pattern for the specific AI deployment. A finance employee asking about approval thresholds in isolation may be a legitimate policy query. A sequence of queries about approval thresholds, dual authorisation requirements, and payment procedures from an account that has not previously accessed the finance AI is a different pattern. Context-aware trust requires pattern analysis, not just credential verification.

Why this matters

AI trust boundaries matter for TPRM because vendor AI products that aggregate sensitive operational information into queryable knowledge bases create reconnaissance capabilities that extend to any user who can authenticate , including users with compromised credentials, social engineering attacks, and insider threats with credentials exceeding their intended access scope. Authentication is necessary but not sufficient as the sole trust boundary for sensitive AI deployments.

• Authentication accepted as complete trust boundary
• Reconnaissance threat , AI aggregates sensitive policy info accessible to any authenticated user
• Context-aware trust not implemented , single queries vs reconnaissance patterns not distinguished
• Sensitive information category not assessed for AI knowledge base inclusion
• Query pattern monitoring for reconnaissance-indicative sequences not implemented

What good looks like

Mature AI trust boundary programmes assess what information the AI knowledge base contains against what should be accessible to all authenticated users , specifically identifying sensitive operational information that should require additional authorisation or context beyond authentication , and implement query monitoring for patterns indicative of reconnaissance rather than legitimate policy self-service.

• Knowledge base content assessment , what sensitive information is accessible to any authenticated user
• Context-aware access controls , sensitive information requiring additional authorisation beyond authentication
• Query pattern monitoring , reconnaissance-indicative query sequences flagged for review
• Sensitive category exclusion , specific sensitive operational information excluded from AI knowledge base
• Trust boundary scope documentation , what the AI is trusted to answer and to whom

Tooling

AI Trust , LLM gateway with query monitoring, Microsoft Purview for sensitive content controls in Copilot

LLM gateway and AI governance platforms can implement query monitoring that flags sequences of queries about sensitive operational information , specifically identifying patterns consistent with reconnaissance rather than single-question policy self-service. For TPRM practitioners, asking whether the vendor's AI assistant has query pattern monitoring alongside authentication provides a specific trust boundary question.

Governance challenges

The governance challenge with AI trust boundaries is the helpfulness-security tension. AI assistants deployed for self-service policy questions are most valuable when they answer fully and accurately. Restricting sensitive information from the knowledge base reduces the AI's utility. The governance resolution is segmented knowledge bases , general policy information accessible to all authenticated users, sensitive operational information accessible through additional authorisation , and query monitoring that flags unusual access patterns.

• Assess AI knowledge base for sensitive operational information
• Segment knowledge base access by information sensitivity
• Implement query pattern monitoring for reconnaissance-indicative sequences
• Define trust boundary scope , what the AI is trusted to answer and to whom
• Include credential compromise scenario in AI security assessment

If you are a small team

Ask the compromised credential scenario question: if an attacker compromised a low-privilege employee's account and queried your finance AI assistant about approval thresholds, dual authorisation requirements, and payment procedures, what would they learn , and would any monitoring or control prevent or detect that reconnaissance? That question reveals the trust boundary gap between authentication and appropriate information access scope.

• Ask the compromised credential reconnaissance scenario question
• Assess what sensitive operational information is in the AI knowledge base
• Ask about query pattern monitoring for reconnaissance sequences
• Evaluate knowledge base segmentation by information sensitivity

What to require

Ask directly:

"If an authenticated user queried your AI assistant about financial approval thresholds, dual authorisation requirements, and payment procedures in sequence , would any monitoring or control flag that query pattern as potentially reconnaissance rather than legitimate policy self-service?"

Expect as evidence

• Query pattern monitoring capability
• Knowledge base sensitive content assessment
• Context-aware access controls beyond authentication
• Trust boundary scope documentation

A vendor who confirms AI authentication controls should be asked about trust boundary scope. Authentication confirms identity. Trust boundary scope determines whether the authenticated identity's request falls within the appropriate access scope for sensitive information.

How to evidence it

• Knowledge base content assessment records
• Query pattern monitoring implementation
• Trust boundary scope documentation
• Sensitive information access control records

Key Takeaway

Finance AI. Policy questions answered accurately. Attacker with compromised credential: also answered accurately. Approval thresholds: disclosed. Dual authorisation requirements: disclosed. Payment procedures: disclosed. BEC attack calibrated below dual authorisation threshold: executed. Authentication confirmed the credential was valid. The trust boundary that could have distinguished the legitimate policy query from the attacker reconnaissance was not implemented. Authentication is the identity boundary. Trust boundaries are the scope boundary. For AI systems that aggregate sensitive operational information, both are required. Query pattern monitoring is the mechanism that detects the reconnaissance pattern that single-query authentication cannot distinguish from legitimate use.

More on AI governance

• AI API Key Exposure
• AI Adversarial Attacks
• AI Attack Surface Expansion