Control Testing Depth

A healthcare software vendor provided their most recent penetration test report as evidence of security testing maturity. The report showed no critical findings and two medium-severity findings, both related to TLS configuration on a staging endpoint. The vendor's security team presented this as evidence that their environment had been tested and found secure. A thorough review of the report's scope section told a different story. The test had been conducted with an external black-box methodology , simulating an unauthenticated external attacker , against the public-facing web application only. The scope exclusions listed: the internal corporate network, all database servers, all API endpoints requiring authentication, the AWS cloud infrastructure hosting customer data, the development environment, and the vendor's own administrative access systems. The two-day test had assessed the login page, the unauthenticated portions of the API, and the public documentation portal. The environments most relevant to the customer's risk , the authenticated API where customer health data was processed, the AWS infrastructure where it was stored, and the administrative systems through which vendor staff accessed it , were all outside the scope. The clean report described a clean external perimeter. The internal environment and cloud infrastructure had not been tested.

What is the Control Testing Depth Problem, Really?

Control testing depth is the degree to which security testing exercises the controls, systems, and attack paths that are most relevant to the actual threat model , rather than the controls, systems, and attack paths that are easiest to test within a defined budget and timeline. Testing depth has two dimensions: scope depth, which refers to how many of the relevant systems and environments are included in the test, and technique depth, which refers to how thoroughly the included systems are tested using the techniques that actual attackers would employ.

Scope limitation is the most consequential testing depth problem. Penetration tests with narrow scope , external only, web application only, unauthenticated pathways only , produce findings that accurately reflect the security of the tested scope while providing no assurance about the untested systems. The attack surface that sophisticated attackers prioritise , authenticated API endpoints, cloud infrastructure configuration, internal lateral movement pathways, and administrative access systems , are frequently the systems excluded from penetration test scope to contain cost and complexity. A clean test of a narrow scope is not evidence of a secure environment; it is evidence that the tested portion of the environment has no detected issues.

The technique limitation problem is the secondary depth dimension. Even within a defined scope, the depth of testing determines what the test can detect. An automated vulnerability scan detects known CVEs and common misconfigurations. A manual penetration test by an experienced tester can find logic vulnerabilities, authentication bypasses, and privilege escalation paths that automated tools miss. Business logic vulnerabilities , authorisation gaps that allow one customer to access another customer's data through legitimate-seeming API calls , require application-specific understanding that automated tools cannot provide. Tests that rely primarily on automated scanning within a limited scope will find the vulnerabilities that automated tools detect. They will miss the attack classes that require human expertise and application-specific knowledge.

• Scope limited to external perimeter , internal network, cloud infrastructure, and authenticated pathways excluded
• Authenticated API not tested , the environment where customer data is most accessible excluded from scope
• Cloud infrastructure excluded , AWS, Azure, or GCP environment where data resides not assessed
• Black-box only testing , no authenticated or white-box testing of internal systems
• Automated scanning without manual testing , technique depth insufficient to detect complex vulnerabilities

Why this matters

Control testing depth matters for TPRM because penetration test reports are among the most commonly requested and most heavily weighted evidence items in vendor security assessments. A clean penetration test result that covers a narrow scope creates strong positive signal in the assessment while providing no assurance about the systems most relevant to the customer's risk. The customer who relies on the test report to understand the vendor's security posture is relying on evidence that was deliberately scoped away from the highest-risk systems.

The authenticated API testing gap is specifically important for vendors processing customer data through APIs. Most modern data processing relationships involve authenticated API calls , the customer's system calls the vendor's API with authentication credentials, and the vendor's system processes the request and returns data. The authentication and authorisation controls on those API endpoints determine whether customer A's data can be accessed by customer B, whether a compromised customer credential can access more data than intended, and whether the vendor's data access controls are working as designed. Testing the unauthenticated external surface while excluding authenticated API endpoints describes a test that assessed the lock on the front door and did not test the room where the customer's data is kept.

Where most teams get this wrong

The most consistent failure is reviewing penetration test reports for finding severity without reviewing scope coverage. A report with no critical findings from a narrow scope provides less assurance than a report with medium findings from a comprehensive scope. Scope determines what the test was able to find. Finding severity describes what it did find within the scope.

• Reviewing finding severity without reviewing scope coverage
• Scope exclusions not evaluated , scope section not reviewed for relevant excluded systems
• Authenticated API coverage not confirmed
• Cloud infrastructure scope not verified
• Testing technique not assessed , automated vs manual vs application-specific

What good looks like

Mature penetration testing governance programmes define scope requirements that cover the systems most relevant to the customer's risk , authenticated API environments, cloud infrastructure, and internal lateral movement pathways , and require testing techniques appropriate to the vulnerability classes in scope.

• Scope requirements aligned to threat model , authenticated API, cloud infrastructure, and internal pathways included
• Authenticated testing required , testing of systems requiring authentication, not just public-facing perimeter
• Cloud infrastructure in scope , AWS, Azure, or GCP configuration assessment included
• Manual testing required for critical systems , business logic and authorisation bypass testing by experienced testers
• Scope coverage review before accepting test report , verifying that relevant systems are in scope

Tooling

Penetration Testing Standards , PTES, OWASP Testing Guide, NIST SP 800-115

Penetration testing methodology standards define comprehensive testing scope and techniques. OWASP's Testing Guide specifically addresses authentication and authorisation testing for web applications and APIs , the technique depth required to detect the vulnerabilities that shallow scanning misses. For TPRM practitioners, requiring that vendor penetration tests follow OWASP or PTES methodology provides a baseline for technique depth.

Cloud Security Testing , AWS Inspector, Azure Security Center assessments, third-party cloud pen test

Cloud-specific security assessments cover the configuration attack surface that traditional penetration tests typically exclude , IAM misconfigurations, storage access policies, network security group settings, and cloud-native authentication vulnerabilities. For TPRM practitioners, requiring that vendor cloud infrastructure assessments are conducted in addition to application penetration testing provides a specific cloud coverage question.

Governance challenges

The governance challenge with control testing depth is the cost-scope trade-off. Comprehensive penetration testing that covers authenticated APIs, cloud infrastructure, internal networks, and multiple testing techniques costs significantly more than a two-day external black-box test. Vendors who control their own testing scope will default to the minimum scope that satisfies the contractual requirement to conduct annual penetration testing. Customers who want broader scope need to specify it , either in contract requirements or in evidence requests that include scope coverage questions.

• Review scope section before finding severity in any penetration test report
• Ask specifically whether authenticated API endpoints were tested
• Ask whether cloud infrastructure was in scope
• Require scope coverage that includes the systems most relevant to customer data
• Distinguish test methodology , automated scanning vs manual testing vs application-specific testing

If you are a small team

For your highest-risk vendor's most recent penetration test report, read the scope section before reading the findings. List the systems and environments explicitly included and explicitly excluded. Then ask: is the environment where my organisation's data is processed and stored within the included scope? If the answer is no or unclear, the test findings , regardless of their severity , do not describe the security of the systems you care about most.

• Read scope section before findings in any vendor penetration test report
• Confirm authenticated API environments are in scope
• Confirm cloud infrastructure is in scope
• Ask about testing methodology , automated vs manual

What to require

Ask directly:

"For your most recent penetration test, was the scope limited to external-facing systems, or did it include authenticated API endpoints, cloud infrastructure configuration, and internal network pathways , specifically, were the systems that process and store our organisation's data within the test scope?"

Expect as evidence

• Penetration test scope documentation including in-scope and out-of-scope systems
• Confirmation that authenticated API environments are in scope
• Cloud infrastructure assessment confirmation
• Testing methodology description , automated and manual components

A vendor who provides a clean penetration test report should be asked specifically whether the systems handling the customer's data were within the test scope. The report describes what was found in the tested systems. The scope defines which systems were tested.

How to evidence it

• Scope coverage assessment for vendor penetration test reports
• Authenticated API and cloud infrastructure testing confirmation
• Methodology assessment records
• Scope requirements in vendor contracts

Key Takeaway

No critical findings from a narrow scope is not the same as no critical vulnerabilities in the environment. The penetration test found nothing critical because the authenticated API where customer data is processed, the AWS infrastructure where it is stored, the internal network, and the administrative systems were all outside the scope. The test was clean. The relevant systems were untested. Penetration testing assurance is bounded by scope. Reading the scope section before the findings section describes what the test can tell you about the vendor's security posture. Reading only the findings tells you what was found in the systems the vendor chose to have tested.

More on Compliance

• Assurance vs Validation
• Audit Evidence Quality
• Audit Scope Limitations