Control vs Implementation Gap

During a TPRM assessment for a software vendor handling healthcare payment processing, the assessors asked for evidence supporting the questionnaire response confirming an annual penetration testing programme. The vendor's response had confirmed: annual penetration tests by an external firm, findings remediated according to a defined SLA, and results reviewed by the security committee. The evidence requests revealed a different picture. The most recent penetration test report was dated fourteen months prior , outside the stated annual cadence. The remediation tracking document showed forty-three open findings, of which thirty-one had passed their SLA date without remediation completion. The security committee meeting minutes showed the last meeting was six months ago , with penetration test findings as an agenda item that had been deferred to the next meeting, which had not yet occurred. Each element of the questionnaire response was accurate as a description of the programme that existed in policy. None of it reflected the current operational state of the programme. The penetration testing programme was real. Its implementation at the time of assessment was not consistent with the programme's stated requirements.

What is the Control vs Implementation Gap, Really?

The control versus implementation gap is the difference between what a security control is designed to do , as specified in policy, programme documentation, and procedure , and what it actually does in practice , as reflected in execution records, findings data, and operational metrics. Controls that exist at the policy level describe the intended security posture. Controls that are implemented and operating describe the actual security posture. The gap between the two is the difference between compliance documentation and operational security reality.

Control existence is relatively easy to establish and document. An organisation can create an information security policy, a penetration testing programme, a vulnerability management process, and an access review procedure in a matter of days. These policies accurately describe the controls the organisation intends to operate. Whether those controls are actually operating , whether the penetration tests are being conducted on schedule, whether vulnerabilities are being remediated within SLA, whether access reviews are producing genuine decisions , is a different question that requires operational evidence rather than policy documentation.

The questionnaire-as-policy-documentation problem is the mechanism through which control-implementation gaps persist undetected in TPRM programmes. Questionnaires that ask 'do you have a penetration testing programme' are asking a policy existence question. The accurate answer , 'yes, we have a programme that requires annual tests, SLA remediation, and committee oversight' , accurately describes the policy. It does not describe the implementation state. A TPRM programme that accepts questionnaire responses as evidence of control implementation has accepted policy documentation as operational evidence , and the gap between the two is invisible to the assessment.

• Policy existence accepted as implementation evidence , questionnaire responses describing programme design treated as evidence of operational execution
• SLA overruns not visible through questionnaire , delayed remediation not reflected in questionnaire responses confirming SLA remediation exists
• Cadence failures not detectable from policy description , overdue assessments, delayed reviews described correctly as programmes rather than incorrectly as current
• Committee governance described in present tense , security committee oversight described as existing even when the committee has not met recently
• Evidence requests distinguishing policy from implementation , the audit step that most questionnaire assessments skip

Why this matters

Control implementation gaps matter for TPRM because the risk reduction provided by a security control depends entirely on the control's operational state , not its policy state. A penetration testing programme that has not produced a test in fourteen months provides zero vulnerability discovery from that programme. Forty-three open findings with overdue remediation SLAs represent known vulnerabilities that exist in the vendor's environment and have not been remediated. The programme policy is in place. The programme's risk reduction is absent.

The customer's exposure from control implementation gaps is direct: the vendor's actual security posture is weaker than the questionnaire-confirmed programme posture suggests. A TPRM risk rating based on questionnaire responses that confirm programme existence is calibrated to the intended security posture. The actual security posture , the penetration test overdue, the findings unremediated, the committee inactive , is the posture that determines the customer's actual risk exposure. The rating is overstating the vendor's security.

Where most teams get this wrong

The most consistent failure is not requesting evidence to distinguish policy from implementation. Most TPRM assessments include evidence requests , asking vendors to provide supporting documentation for selected questionnaire responses. The gap is in what is accepted as evidence. A penetration testing programme policy document or a list of test parameters confirms programme design. The most recent test report and the remediation tracking spreadsheet confirm operational state. Accepting the former as evidence of the latter is the gap.

• Accepting policy documentation as implementation evidence
• Evidence requests not including operational metrics , SLA performance, cadence adherence, and finding age
• No evidence review for implementation state , only policy documentation reviewed
• Questionnaire framing enabling policy-accurate but implementation-inaccurate responses
• No follow-up on evidence gaps , missing or vague evidence accepted without challenge

What good looks like

Mature TPRM programmes distinguish policy evidence from implementation evidence in their evidence request design , asking for the artefacts that reflect operational state rather than policy documentation, and specifically requesting the metrics that reveal cadence adherence, SLA performance, and programme execution quality.

• Evidence requests specifying implementation artefacts , most recent test report, current finding age, remediation SLA performance rate
• Cadence verification , dates of most recent programme executions compared against stated cadence
• SLA performance metrics , percentage of findings remediated within SLA rather than confirmation that SLA exists
• Meeting minutes for governance bodies , confirmation that oversight committees are meeting on stated cadence
• Finding age tracking , oldest open findings as an implementation quality indicator

Tooling

GRC Platforms , ServiceNow GRC, MetricStream, RSA Archer

GRC platforms that track control implementation metrics alongside control documentation provide the operational state visibility that policy documentation cannot. For TPRM practitioners, asking whether vendors use a GRC platform that tracks implementation metrics , finding age, remediation SLA performance, assessment cadence adherence , provides a specific operational state monitoring question.

Automated Compliance , Drata, Vanta, Secureframe

Automated compliance platforms collect implementation evidence continuously , monitoring that penetration tests were performed, tracking vulnerability remediation SLA performance, and alerting when controls fall out of operating effectiveness. For TPRM practitioners, asking whether vendors use automated compliance monitoring that would detect and alert on the implementation failures described in the hook provides a specific detection capability question.

Governance challenges

The governance challenge with control implementation gaps is the evidence request design problem. Asking for evidence is straightforward. Knowing which evidence distinguishes policy from implementation , and not accepting policy documentation as implementation evidence , requires understanding the difference between the two and designing evidence requests that specifically surface the implementation metrics.

• Design evidence requests for implementation artefacts , dates, metrics, and operational records rather than policy documents
• Request the most recent execution for all periodic controls , most recent test report, most recent review output
• Request SLA performance data , percentage compliant, not confirmation that SLA exists
• Track programme execution cadence , when was each periodic control last executed
• Challenge vague evidence , accepting 'evidence is available upon request' as non-responsive

If you are a small team

For your three highest-risk vendor relationships, change your evidence requests for three common controls from policy documentation to implementation artefacts. Instead of 'penetration testing programme documentation,' request 'most recent penetration test report and current open finding list with ages.' Instead of 'vulnerability management policy,' request 'current open vulnerability count by age band and SLA performance rate for the last quarter.' Instead of 'security committee charter,' request 'meeting minutes from the last three committee meetings.' Those three evidence redesigns will immediately reveal whether control implementation matches control documentation.

• Request most recent test report and current open finding list , not policy documentation
• Request SLA performance metrics , not SLA policy
• Request meeting minutes , not committee charter
• Track execution cadence for all periodic controls

What to require

Ask directly:

"For your penetration testing programme, please provide: the date of your most recent external penetration test, the number of findings from that test, and the current number of open findings with the age of your oldest open finding."

"For your vulnerability management programme, please provide your remediation SLA performance rate for the last quarter , specifically, what percentage of vulnerabilities were remediated within your defined SLA timeframe."

Expect as evidence

• Most recent penetration test date and current open finding count with oldest finding age
• Remediation SLA performance rate , percentage compliant, not SLA policy
• Most recent security committee meeting date
• Finding age distribution , how long open findings have been open

A vendor whose questionnaire response confirms an annual penetration testing programme should be asked the date of the most recent test. Fourteen months is outside annual cadence. The programme is real. The implementation is not current. Both are assessable. Ask for the date.

How to evidence it

• Implementation evidence records for key vendor controls
• Cadence verification records , most recent execution dates
• SLA performance data requests and results
• Evidence redesign documentation

Key Takeaway

The programme that exists on paper and the programme that operates in practice are two different security postures. The questionnaire describes the paper programme accurately. The evidence reveals the operating programme. Fourteen months since the last penetration test is a programme that has not tested the environment in fourteen months. Thirty-one findings past their remediation SLA are thirty-one known vulnerabilities that have not been fixed. A security committee that has not met in six months is a governance body that has not performed its oversight function. All of these facts are compatible with an accurate questionnaire response that confirms the programme's existence. The questionnaire describes the design. The evidence describes the operation. Request the evidence that reveals the operation.

More on Compliance

• Assurance vs Validation
• Audit Evidence Quality
• Audit Scope Limitations