Vendor IR Testing Frequency

A logistics software vendor's TPRM questionnaire response described their incident response capability as regularly tested through tabletop exercises and red team assessments. The TPRM practitioner at a major retailer who was assessing the vendor accepted this description as confirmation of mature IR testing. When a follow-up interview was conducted as part of an enhanced assessment, the specifics of the IR testing programme became clear. The tabletop exercises: two in the last three years, both facilitated by the same internal security manager, both using a ransomware scenario with minor variations between iterations, both involving only the security team and the IT operations team. The legal, communications, customer success, and finance teams , all of whom would have material roles in a real incident response , had not participated in either exercise. The red team assessments: one penetration test in 2022 scoped to the external perimeter, one internal network assessment in 2023 scoped to the corporate network. Neither included the vendor's cloud infrastructure, their customer portal, or their supply chain integrations. The most recent joint exercise with a customer: never. The IR capability was described as regularly tested. The testing that had occurred in three years covered a narrow scenario scope, involved a fraction of the people a real incident would require, and had not covered most of the vendor's attack surface. Regularly tested covered both a programme with twelve exercises per year and a programme with two exercises per three years. The description was accurate. The programme it described was not what the description implied.

What is the Vendor IR Testing Frequency Problem, Really?

Vendor IR testing frequency encompasses not just how often exercises occur, but how broadly they cover the attack scenarios, technology environments, and organisational participants that a real incident would involve. Testing frequency is a necessary but insufficient indicator of IR programme effectiveness. Two comprehensive, cross-functional, full-scope exercises per year provide stronger IR preparation than eight narrow, security-team-only tabletops using the same scenario with minor variations.

The scenario diversity problem is the first dimension of testing quality. IR exercises that repeatedly use the same or similar scenarios build familiarity with a specific response pattern rather than the adaptive response capability that varied incidents require. A vendor whose tabletop exercises have always used ransomware scenarios has not exercised their response to a data exfiltration, a supply chain compromise, a cloud infrastructure breach, or an insider threat. When a different incident type occurs, the familiarity built through repeated ransomware exercises does not transfer.

The participant coverage problem is the second dimension. Real incidents require coordinated response from multiple organisational functions , security, legal, communications, customer success, operations, and finance all have material roles in a significant breach response. Exercises that involve only the security team and IT operations build response capability for those teams while leaving the coordination across functions , the most challenging aspect of real incident response , unexercised. The teams that struggle most during real incidents are frequently the non-technical functions whose roles are clear in documentation but never practiced in exercises.

• Low exercise frequency , two tabletops in three years
• Repeated scenario , same or similar ransomware scenario with minor variations
• Narrow participant scope , security and IT only, no legal, communications, customer success
• Red team scope gap , cloud infrastructure and supply chain integrations not covered
• No joint customer exercises , coordination with customer IR teams never tested

Why this matters

Vendor IR testing frequency matters for TPRM because the IR capability that matters most , the coordinated response involving legal, communications, operations, and security working together under incident pressure , is only built through exercises that practice exactly that coordination. A vendor whose exercises are limited to the security team has not built the cross-functional coordination capability that a real incident requires.

Where most teams get this wrong

The most consistent failure is accepting IR testing confirmation without requesting the specifics , exercise frequency, scenario diversity, participant coverage, and red team scope. The specifics reveal whether the testing builds the capability the vendor claims.

• IR testing confirmed without specifics , frequency, scenario, participants, scope
• Exercise frequency accepted without scenario diversity assessment
• Participant coverage not assessed , security only vs cross-functional
• Red team scope gaps not assessed , cloud, supply chain, customer integrations
• Joint customer exercises not asked about

What good looks like

Mature IR testing programmes conduct at least annual tabletop exercises with cross-functional participation , specifically including legal, communications, and customer-facing teams , use varied scenarios that cover different incident types and attack vectors, and commission red team assessments that cover the full attack surface including cloud, supply chain, and customer integration environments.

• Annual cross-functional tabletop , legal, communications, customer success, operations included
• Scenario diversity , different incident types tested across exercise programme
• Full attack surface red team , cloud, supply chain, customer integrations in scope
• Joint customer exercises for critical vendor relationships
• Exercise findings incorporated into IR plan updates

Tooling

IR Exercise , FEMA Homeland Security Exercise Evaluation Program, Crisis simulation platforms

Structured crisis simulation platforms provide facilitated IR exercises with realistic scenario injection and systematic evaluation. For TPRM practitioners, asking whether the vendor uses a structured exercise programme with scenario diversity and cross-functional participation provides a specific exercise quality question.

Governance challenges

The governance challenge with IR testing frequency and quality is organisational bandwidth. Cross-functional exercises require time commitments from legal, communications, and senior leadership , functions with competing priorities. The governance resolution is executive-level commitment to IR exercise participation as a required activity, not an optional one.

• Ask for exercise log , frequency, dates, scenarios, participants for last three years
• Ask about participant coverage , who attends, beyond security team
• Ask about scenario diversity , what incident types have been exercised
• Ask about red team scope , what environments are covered
• Ask about joint exercises with critical customers

If you are a small team

Ask your highest-risk vendor for their IR exercise log for the last two years , a simple list of each exercise, date, scenario, and participant roles. That list reveals scenario diversity, participant coverage, and exercise frequency in a single document. If the list shows the same scenario repeated with only security team participants, you have identified the quality gap that the description regularly exercised did not reveal.

• Ask for IR exercise log , dates, scenarios, participants for last two years
• Assess scenario diversity from the log
• Assess participant coverage , security-only vs cross-functional
• Ask about red team scope and most recent assessment date

What to require

Ask directly:

"Can you provide your incident response exercise log for the last two years , including exercise dates, scenarios used, and participant roles , and can you confirm whether your most recent exercises included your legal, communications, and customer-facing teams alongside the security team?"

Expect as evidence

• Exercise log , dates, scenarios, participants
• Scenario diversity confirmation
• Cross-functional participant confirmation
• Red team scope documentation

A vendor who confirms regular IR testing should be asked for the exercise log. The log reveals what the confirmation does not: frequency specificity, scenario diversity, participant coverage, and red team scope. The exercise log is the evidence. The confirmation is the description.

How to evidence it

• Vendor exercise log records
• Cross-functional participation confirmation
• Scenario diversity assessment
• Red team scope verification

Key Takeaway

Two tabletops. Three years. Same scenario. Security team only. Regularly tested. Both statements are true simultaneously. The testing frequency is real , exercises occurred. The testing quality and coverage are not what regularly tested implies to the customer who uses it as an IR maturity indicator. Exercise logs reveal what the description does not. Frequency, scenario diversity, participant coverage, and red team scope together describe the IR testing programme. Any one of them in isolation describes a fraction. The exercise log contains all four. Request it.

More on Security

• API Identity Misuse
• Access Certification Effectiveness
• Access Policy Misconfigurations