Audit Evidence Quality

A cloud services vendor's annual security assessment included an evidence request for MFA enforcement. The vendor provided a screenshot of their identity platform's administrative panel showing the MFA settings with 'Required' status for the Administrator role. The TPRM assessor logged the evidence as satisfying the MFA enforcement requirement and moved to the next item. An experienced auditor reviewing the evidence package later noted four issues with the screenshot. The screenshot showed one role , Administrator , with MFA Required. The identity platform had twelve roles, and the screenshot did not show the MFA settings for the other eleven. The environment shown in the screenshot's URL bar appeared to be a test environment based on the subdomain structure, not the production environment. The screenshot's metadata showed a creation date eleven months prior to the assessment. And the MFA enforcement for the Administrator role was configured as 'Required' for new authentications , but the platform allowed existing sessions to continue without reauthentication, meaning established sessions were not enforced. The screenshot was genuine and accurately showed what it depicted. What it depicted was not what the evidence request was intended to demonstrate.

What is the Audit Evidence Quality Problem, Really?

Audit evidence quality is the degree to which evidence provided for a security control actually demonstrates what it is represented as demonstrating , that the control is implemented, operating effectively, and applied to all relevant systems and users, not just that a setting exists in a captured configuration at a particular time. High-quality evidence provides strong assurance that the control operates as intended. Low-quality evidence provides confirmation that some evidence was provided without providing meaningful assurance about control effectiveness.

Screenshots are the most commonly provided and most commonly insufficient evidence type in vendor security assessments. Screenshots confirm a configuration at the moment of capture, in the environment from which the screenshot was taken, for the user or role visible in the capture. They do not confirm the setting's persistence over time, its application to all relevant users and systems, or its presence in the environment where customer data is actually processed. A screenshot can accurately depict exactly what the vendor says it depicts while not demonstrating anything about the actual security posture of the systems relevant to the customer's risk.

The coverage gap is the most significant screenshot limitation. Security controls applied to one role, one system, or one configuration panel while other roles, systems, and configuration panels are not shown in the evidence do not demonstrate comprehensive control coverage. An assessor who accepts a screenshot showing MFA enabled for the Administrator role as evidence that MFA is enforced has not confirmed that MFA applies to the other eleven roles, to service accounts, or to the authentication pathways that may bypass the standard authentication flow. The screenshot proves the setting for what is shown. It says nothing about what is not shown.

• Single-role or single-system screenshots , partial coverage presented as comprehensive evidence
• Test environment screenshots , non-production environment configurations presented as production evidence
• Aged screenshots , screenshots taken months before the evidence request not confirming current state
• Session vs new authentication enforcement , configurations that enforce for new authentications but not existing sessions
• Evidence accepted without quality assessment , screenshots logged as satisfying requirements without evaluating their coverage and currency

Why this matters

Evidence quality matters for TPRM because the assurance provided by a completed assessment is only as strong as the evidence that supports it. An assessment that accepts low-quality evidence , partial screenshots, aged configurations, test environment captures , produces a risk rating that is supported by documentation that does not actually demonstrate the controls it is supposed to evidence. The risk rating appears well-supported. The underlying controls may not be in the state the evidence suggests.

The audit examination consequence is direct. Regulatory examiners who review evidence packages evaluate evidence quality , whether the evidence demonstrates what it is represented as demonstrating. An evidence package of partial screenshots from test environments will not satisfy an examiner's evidence quality expectations regardless of how the evidence was rated in the TPRM assessment. The examiner will ask the evidence quality questions that the assessment did not.

Where most teams get this wrong

The most consistent failure is accepting evidence that demonstrates something , a setting exists somewhere , without confirming that the evidence demonstrates what it was requested to demonstrate , that the setting applies comprehensively and is currently maintained. Evidence acceptance without quality evaluation confirms documentation completeness rather than control assurance.

• Accepting evidence without quality evaluation , confirming evidence was provided rather than what it demonstrates
• Coverage not assessed , single-system or single-role evidence accepted for comprehensive coverage claims
• Environment not verified , screenshots not confirmed to depict production environment
• Currency not confirmed , aged evidence accepted without confirmation of current state
• Evidence limitations not documented , low-quality evidence accepted without recording its limitations

What good looks like

Mature evidence quality programmes define evidence quality criteria for each requested control , specifying what coverage, currency, and environment confirmation is required , and evaluate provided evidence against those criteria before accepting it as satisfying the control requirement.

• Evidence quality criteria defined per control , what coverage, currency, and environment the evidence must demonstrate
• Coverage confirmation required , all relevant roles, systems, and pathways shown or explicitly confirmed
• Production environment confirmation , URL, environment indicator, or explicit confirmation that evidence depicts production
• Currency confirmation , evidence dated within defined recent period or accompanied by currency attestation
• Evidence limitations documented , where evidence is accepted despite gaps, limitations are recorded with compensating context

Tooling

Automated Compliance Evidence , Drata, Vanta, Secureframe

Automated compliance platforms collect evidence programmatically rather than through screenshots , querying configuration APIs to confirm current settings across all relevant systems and roles. For TPRM practitioners, asking whether vendors can provide automated compliance tool output rather than manual screenshots provides a specific evidence quality improvement question.

Configuration Reports , AWS Config, Azure Policy compliance reports

Cloud configuration management reports provide coverage-complete, current, and production-verified evidence , showing all resources' compliance with defined policies rather than a screenshot of one resource's settings. For TPRM practitioners, requesting cloud compliance reports rather than screenshots provides evidence that is superior on all three quality dimensions.

Governance challenges

The governance challenge with evidence quality is the time investment required to evaluate quality rather than confirming receipt. TPRM programmes under volume pressure , assessing many vendors in a defined timeframe , default to confirming that evidence was provided rather than investing the time required to evaluate its quality. The governance resolution is tiering evidence quality review , applying rigorous quality evaluation to the evidence for the highest-risk controls and highest-risk vendor relationships, and accepting briefer evidence review for lower-risk items.

• Define evidence quality criteria for highest-risk controls , MFA, encryption, access review, penetration testing
• Require production environment confirmation for all configuration screenshots
• Set maximum evidence age , screenshots older than six months require currency confirmation
• Require coverage confirmation , all relevant roles or systems confirmed, not a single example
• Request automated compliance output rather than screenshots for comprehensive controls

If you are a small team

For your next evidence review, apply three quality questions to every screenshot provided as evidence. First: does this screenshot show the complete scope of the control , all roles, all systems, all access pathways , or just one example? Second: is this the production environment? Third: when was this screenshot taken, and does it represent the current state? Those three questions, asked consistently, will immediately surface the coverage, environment, and currency gaps that screenshots routinely have. Documents that fail any of the three should be flagged for follow-up before the control is logged as satisfied.

• Ask: does the screenshot show the complete scope, or one example?
• Ask: is this the production environment?
• Ask: when was this taken, and does it represent the current state?
• Request automated compliance output instead of screenshots for comprehensive controls

What to require

Ask directly:

"For the MFA enforcement evidence you provided , the screenshot shows the Administrator role. Can you confirm whether MFA is required for all roles in your platform, not just Administrator, and whether this screenshot depicts your production environment rather than a test environment?"

Expect as evidence

• Coverage confirmation , all roles confirmed or comprehensive policy screenshot
• Production environment confirmation
• Currency confirmation , evidence represents current state
• Automated compliance output preferred over manual screenshots

A vendor who provides a screenshot of MFA enabled for the Administrator role should be asked three questions: does MFA apply to all roles, is this the production environment, and when was this captured? The screenshot proves what it shows. The questions prove whether what it shows is what matters.

How to evidence it

• Evidence quality criteria documentation
• Coverage and currency assessment records for key controls
• Production environment confirmation records
• Automated compliance output requests and results

Key Takeaway

The screenshot showed MFA enabled for the Administrator role in a test environment, captured eleven months ago, with a session continuation exception that means existing sessions are not re-enforced. The screenshot was genuine. What it demonstrated was not what it was represented as demonstrating. Evidence quality is the degree to which evidence actually demonstrates what it is claimed to demonstrate , not whether evidence exists, but whether the evidence that exists proves what the control assessment requires. Screenshots prove a setting existed in the depicted environment at the time of capture. They do not prove comprehensive coverage, production deployment, or current maintenance. Define evidence quality criteria. Evaluate coverage, environment, and currency. Request automated compliance output where screenshots are insufficient.

More on Compliance

• Assurance vs Validation
• Audit Scope Limitations
• Compliance Automation Gaps