AI Governance Gaps

A consumer lending company's internal AI governance programme had matured significantly over three years , an AI Ethics Committee with cross-functional membership, a model risk management framework adapted from SR 11-7 guidance, an AI inventory with risk classifications, and a vendor AI assessment process that required review before any vendor AI product was deployed in customer-facing workflows. The programme was genuinely mature by the standards against which it had been built. The governance framework had been designed in 2022 against the regulatory environment of that time , before the EU AI Act had been finalised, before the UK's AI regulation approach had been clarified, and before the US federal agencies had issued specific AI guidance for the financial services sector. When the consumer lending company's compliance team conducted an EU AI Act readiness assessment in 2024, they identified that three vendor AI systems in active production deployment met the EU AI Act's definition of high-risk AI systems , specifically systems making or meaningfully influencing credit decisions for individuals. High-risk AI systems under the EU AI Act have specific requirements: fundamental rights impact assessments, human oversight mechanisms, transparency obligations to affected individuals, registration in the EU AI Act database, and conformity assessments. None of those requirements had been part of the AI governance framework when those systems were approved. The governance framework was mature for its design basis. The regulatory environment had moved significantly past it.

What are AI Governance Gaps, Really?

AI governance gaps are the deficiencies in an organisation's AI governance framework , policies, processes, review committees, and compliance frameworks , that result in AI systems being deployed or continuing to operate without adequate oversight, transparency, or compliance with applicable regulatory requirements. AI governance gaps can arise from governance frameworks that are outdated relative to the evolving regulatory environment, AI use cases that were not anticipated when the governance framework was designed, or governance processes that assess the wrong risk dimensions for the AI systems they are reviewing.

The regulatory evolution gap is the primary driver of AI governance failures in enterprise environments. The AI regulatory landscape has evolved rapidly since 2022 , the EU AI Act, the NIST AI Risk Management Framework, financial services sector AI guidance from the OCC and FRB, and the UK's sector-specific AI regulatory approach. AI governance frameworks that were mature against the 2022 regulatory environment may be substantially non-compliant with current requirements. The governance framework's maturity is relative to the regulatory environment it was designed for , not the regulatory environment that currently applies.

The risk classification accuracy problem is the second governance gap dimension. AI risk classification , high-risk, medium-risk, limited-risk, minimal-risk , determines which governance requirements apply to each AI system. Misclassification of AI systems , classifying a high-risk system as medium-risk, or failing to reclassify a system when its use case expands , means that the governance requirements that should apply to the system are not applied. The EU AI Act's high-risk AI system definitions are specific to use case and sector , a system that is not high-risk in one use case may become high-risk if its application is expanded.

The vendor AI governance coverage problem is the TPRM dimension. Enterprise AI governance frameworks that review internal AI systems may not systematically apply equivalent governance to vendor AI systems. The enterprise's governance framework may have an AI Ethics Committee review for internal development but a simplified vendor assessment process for procured AI products. The vendor's AI product may carry higher risk , because it operates at scale, because it is used for more consequential decisions, or because its training data provenance is less transparent , than internal AI systems that receive more thorough governance review.

The human oversight implementation problem is a specific EU AI Act gap for high-risk systems. The Act requires that high-risk AI systems be designed to allow appropriate human oversight , specifically that humans are able to understand the system's capabilities and limitations, monitor its operation, and override or intervene in the system's outputs. High-risk AI systems deployed in workflows where human oversight is nominal rather than substantive , where humans review AI outputs at a speed and scale that makes meaningful oversight impossible , may not meet the Act's human oversight requirements even if a human technically reviews each decision.

Why this matters

AI governance gaps matter for TPRM because the enterprise's regulatory compliance obligations apply to vendor AI systems that perform regulated functions in the enterprise's workflows , regardless of whether those systems were procured rather than internally developed. A credit decision AI system that meets the EU AI Act's high-risk definition triggers compliance obligations for the enterprise that deploys it, whether that enterprise built the system or purchased it from a vendor.

Where most teams get this wrong

The most consistent failure is applying an AI governance framework designed for a prior regulatory environment to current vendor AI deployments, and treating vendor AI assessment as separate from the enterprise's internal AI governance requirements. The regulatory obligations apply to the deployment, not to who developed the system.

• Governance framework not updated for current regulatory environment
• Vendor AI assessment not aligned with enterprise governance requirements
• Risk classification not reviewed when use case or regulatory environment changes
• EU AI Act high-risk classification not assessed for applicable vendor systems
• Human oversight implementation not validated , nominal vs substantive

What good looks like

Mature AI governance programmes conduct annual regulatory environment reviews , updating the governance framework for new regulatory requirements , and apply consistent governance standards to vendor AI systems as to internal systems, with specific EU AI Act high-risk classification assessment for all AI systems in regulated use cases.

• Annual regulatory environment review , governance framework updated for new requirements
• EU AI Act high-risk classification assessment for all AI in regulated use cases
• Vendor AI under equivalent governance as internal AI
• Human oversight validation , substantive oversight confirmed, not nominal
• AI inventory maintained with current regulatory classification

Tooling

AI Governance , IBM OpenScale/Watson OpenScale, Microsoft Responsible AI Dashboard

AI governance platforms provide model monitoring, fairness assessment, and explainability tools that support the technical compliance requirements for high-risk AI systems. For TPRM practitioners, asking whether the vendor's high-risk AI systems use an AI governance platform that supports the technical requirements of applicable regulations , specifically EU AI Act transparency and monitoring obligations , provides a specific compliance infrastructure question.

Regulatory Compliance , EU AI Act compliance frameworks, NIST AI RMF, SR 11-7 for financial services

The NIST AI Risk Management Framework provides a structured approach to AI risk management that aligns with emerging regulatory requirements across multiple jurisdictions. For TPRM practitioners, asking whether the vendor's AI governance programme is aligned with NIST AI RMF and current EU AI Act requirements provides a regulatory currency question.

Governance challenges

The governance challenge with AI governance gaps is the regulatory velocity problem. The AI regulatory environment is evolving faster than most governance frameworks are updated. The governance resolution is treating AI governance framework updates as a continuous process , triggered by regulatory developments , rather than a periodic review on a fixed cycle.

• Assess vendor AI governance against current regulatory environment , not just framework existence
• Ask about EU AI Act compliance assessment for applicable systems
• Ask about human oversight implementation , substantive vs nominal
• Ask whether vendor's governance framework has been updated for current regulations
• Require contractual compliance with applicable AI regulations for high-risk systems

If you are a small team

For any vendor AI system used in a regulated context , credit, healthcare, employment, education, law enforcement , ask one EU AI Act question: has this system been assessed against the EU AI Act's high-risk AI system criteria for your sector, and if it qualifies as high-risk, do you have the required transparency, human oversight, and conformity assessment documentation? That question reveals whether the vendor's AI governance framework has been updated for the current regulatory environment or is still operating against a prior baseline.

• Ask whether system has been assessed against EU AI Act high-risk criteria
• Ask about fundamental rights impact assessment for high-risk systems
• Ask about human oversight implementation , substantive confirmation
• Ask whether vendor's AI governance framework has been updated for current regulations

What to require

Ask directly:

"Has your AI system been assessed against the EU AI Act's high-risk AI system criteria for our use case , and if it qualifies as high-risk, can you provide the fundamental rights impact assessment, human oversight documentation, and conformity assessment evidence required by the Act?"

Expect as evidence

• EU AI Act high-risk assessment for applicable systems
• Fundamental rights impact assessment
• Human oversight implementation documentation
• Conformity assessment evidence

A vendor who confirms AI governance maturity should be asked about current regulatory alignment. Governance maturity describes the framework. Regulatory currency describes whether the framework reflects the regulations that currently apply.

How to evidence it

• EU AI Act compliance assessment records
• AI governance framework regulatory update records
• Human oversight validation records
• Regulatory classification for AI inventory

Key Takeaway

AI Ethics Committee review: complete. EU AI Act high-risk classification: not assessed. The governance framework was mature for the 2022 regulatory environment. The EU AI Act's high-risk system requirements , fundamental rights impact assessment, human oversight, transparency obligations, conformity assessment , apply to this credit decision system and were not part of the framework when the system was approved. AI governance maturity is relative to the regulatory environment the framework was designed for. The regulatory environment has moved. The framework must move with it. Annual regulatory environment review is the mechanism that keeps governance frameworks current. The EU AI Act question , has this system been assessed against high-risk criteria , is the specific question that reveals whether the framework has kept pace.

More on AI governance

• AI API Key Exposure
• AI Adversarial Attacks
• AI Attack Surface Expansion