Breach Simulation Gaps

A financial services vendor commissioned a breach simulation exercise following a regulatory requirement that their IR capability be regularly tested. The simulation vendor provided a structured exercise: a scripted ransomware scenario injected at defined steps, with the IR team notified at each step about what had occurred and directed to execute their response procedures. The simulation was conducted over two days, with the simulation team and the IR team in regular communication throughout. The IR team's performance was assessed against defined response criteria: notification timeline, escalation accuracy, containment speed, and communication quality. The IR team performed well across all criteria , the notification was timely, escalation followed procedure, containment was initiated within the defined SLA, and communications were professional and complete. The simulation report confirmed that the vendor's IR capability met the defined response criteria. The financial services company accepted the report as evidence of IR capability. Six months later, the vendor experienced an actual breach , a sophisticated supply chain compromise that had been developing for thirty-one days before detection. The detection was ambiguous , anomalous network traffic that initially appeared to be a misconfigured monitoring agent. The scope was unclear , the investigation took four days to establish the breach perimeter. The attacker was active , containment actions were partially countered by the attacker's lateral movement during the investigation period. The IR team had performed excellently in a scripted exercise where they knew what was happening at each step. The real breach required detecting an ambiguous signal, investigating an unclear scope under attacker counterpressure, and making containment decisions with incomplete information. The simulation had not exercised any of these capabilities.

What are Breach Simulation Gaps, Really?

Breach simulation gaps are the differences between what a breach simulation exercise tests and what a real security incident requires , specifically the gap between a scripted, cooperative simulation where the IR team is informed of each step and a real breach where detection is ambiguous, scope is uncertain, the attacker is active and adaptive, and decisions must be made with incomplete information. Breach simulations that are designed to demonstrate successful IR execution in controlled conditions do not necessarily prepare an IR team for the conditions of real incidents.

The information completeness gap is the primary simulation limitation. In scripted breach simulations, the IR team is typically informed of what has occurred at each step , the simulation team injects the scenario and the IR team responds to known information. In real incidents, the IR team must first detect that something has occurred , often from an ambiguous signal , and then establish what has occurred through investigation before they can respond. The detection and scoping phases , the most challenging aspects of real incident response , are frequently absent or simplified in breach simulations.

The adversarial adaptability gap is the secondary simulation limitation. Breach simulations use a scripted attacker whose behaviour is determined in advance by the simulation team. Real attackers are adaptive , they respond to IR team actions by changing techniques, moving to unmonitored environments, and specifically countering containment measures. An IR team that performs well against a scripted attacker has not been tested against an adaptive adversary who actively works to counter their response. Red team exercises , with skilled penetration testers actively countering the blue team's response , test this adversarial adaptability in a way that scripted simulations cannot.

• Scripted scenario , IR team informed of each step vs real breach where detection is ambiguous
• No detection challenge , simulation injects known information rather than ambiguous signals
• No adversarial adaptability , scripted attacker vs adaptive real attacker
• Scope certainty , simulation defines perimeter vs real breach requiring investigation
• Performance criteria designed for simulation conditions rather than real breach requirements

Why this matters

Breach simulation gaps matter for TPRM because the simulation report that confirms IR capability to defined criteria may be testing a different set of capabilities than a real breach requires. A vendor who performs well in a scripted simulation and poorly in a real breach has demonstrated scripted response capability, not real incident response capability.

Where most teams get this wrong

The most consistent failure is accepting scripted simulation performance as evidence of real breach response capability without assessing whether the simulation tested the detection, scoping, and adversarial adaptability challenges that real incidents present.

• Scripted simulation accepted as real breach capability evidence
• Detection challenge absent from simulation
• Adversarial adaptability not tested , scripted vs adaptive attacker
• Simulation criteria designed for simulation not real breach conditions
• No purple team or live-fire exercise testing real detection and response

What good looks like

Mature breach simulation programmes combine scripted exercises for procedure validation with red team or purple team exercises that test detection under realistic adversarial pressure , providing both procedure confirmation and adaptive response capability demonstration.

• Scripted exercise for procedure validation , IR procedures executed under known conditions
• Red or purple team for adaptive response testing , blue team detecting and responding to active skilled red team
• Ambiguous signal injection , exercises beginning with detection challenge rather than known scenario
• Attacker counterpressure simulation , red team adapting to containment actions
• Real breach metrics included , MTTD, MTTC measured in realist conditions

Tooling

Purple Team Services , Mandiant, CrowdStrike, SANS purple team programmes

Purple team exercises combine red team attack execution with blue team detection and response , providing real-time feedback on which attacks are detected and which are missed, and testing the blue team's adaptive response to active attacker pressure. For TPRM practitioners, asking whether the vendor's IR testing includes purple team exercises , where the blue team must detect and respond to active red team activity without being informed of the scenario , provides a specific adversarial testing quality question.

Governance challenges

The governance challenge with realistic breach simulation is cost and operational disruption. Full red team exercises are expensive and may disrupt operations if realistic attack techniques are used in production environments. The governance resolution is tiered testing: scripted exercises for procedure validation, purple team for detection and adaptive response, and periodic full red team for comprehensive attack surface validation.

• Ask what detection challenge the simulation included , ambiguous signal or known injection
• Ask whether red or purple team testing supplements scripted simulation
• Ask about adversarial adaptability testing , scripted vs adaptive attacker
• Ask for MTTD and MTTC from most recent red or purple team
• Ask about simulation criteria , designed for simulation conditions or real breach conditions

If you are a small team

Ask your highest-risk vendor two questions about their breach simulation: first, in your most recent simulation, was the IR team told what had occurred at each step, or did they need to detect and investigate to understand what was happening? Second, has your team participated in a purple team or live-fire red team exercise where they needed to detect and respond to active skilled attacker activity without being informed of the scenario? Those two questions reveal whether the simulation tested scripted response capability or adaptive real incident response capability.

• Ask whether IR team was told what occurred or needed to detect and investigate
• Ask whether purple team or live-fire exercise has been conducted
• Ask about adversarial adaptability testing
• Ask for MTTD from most recent red or purple team exercise

What to require

Ask directly:

"In your most recent breach simulation , was the IR team informed of what had occurred at each step, or did they need to detect and investigate independently? And have you supplemented scripted exercises with purple team or live-fire testing where the team must detect and respond to active skilled attacker pressure without scenario pre-briefing?"

Expect as evidence

• Simulation methodology , scripted vs detection-first
• Purple team or live-fire exercise history
• MTTD and MTTC from adversarial exercises
• Adversarial adaptability testing confirmation

A vendor who confirms breach simulation completion should be asked about the simulation methodology. Scripted simulation tests procedure execution. Adversarial testing tests detection capability and adaptive response. Both are required for a complete IR effectiveness assessment.

How to evidence it

• Simulation methodology documentation
• Purple team exercise records
• Adversarial exercise MTTD and MTTC
• Simulation criteria assessment

Key Takeaway

Scripted scenario. IR team informed at each step. Cooperative simulation team. Defined response criteria. Performance confirmed. Six months later: ambiguous detection, uncertain scope, active attacker, incomplete information. The scripted exercise and the real breach required different capabilities. The scripted exercise tested procedure execution under known conditions. The real breach required detection from ambiguity, scoping through investigation, and containment against an adaptive adversary. Procedure execution under known conditions is a prerequisite. Detection capability under adversarial pressure is the capability that matters. Test both. Scripted for procedures. Adversarial for detection and response.

More on Security

• API Identity Misuse
• Access Certification Effectiveness
• Access Policy Misconfigurations