In December 2023, US insurance regulators drew a line most mid-market operators still haven’t absorbed: you are accountable for what your AI does, including the models your vendors run. Lending and healthcare regulators have since said versions of the same thing. AI governance for regulated industries is no longer a policy nicety; it is the gap between a defensible operation and one improvising its compliance after an audit has already started. The difficulty for a company at this scale is that the line has been drawn around AI that is already running inside the business, mostly without anyone having approved it.
Shadow AI is already inside your operation
The assumption behind most AI risk planning is that the risk arrives with the first board-approved project. In regulated mid-market operations it arrives months earlier, carried in by employees solving their own workload problems with whatever tool sits one browser tab away. Underwriters lean on it to draft correspondence. A servicing team runs borrower call notes through it and produces summaries nobody downstream realizes were machine-written. None of this shows up in a budget line or a vendor contract, which is why leadership rarely knows the true scale of it until someone goes looking.
The National Institute of Standards and Technology published its AI Risk Management Framework in January 2023, and the first of its four functions is named “Govern,” ahead of Map, Measure, and Manage. The ordering is deliberate. Before you can measure or manage an AI system’s behavior, someone has to decide which systems are permitted, who is accountable for them, and what rules they operate under. That deciding function is governance, and it exists whether or not a company has formalized it. In most mid-market operations it simply has not been formalized, so the decisions get made informally by whoever installs the tool.
That informal state is the problem this article is about. A regulated operation carries obligations for every system that touches customer data or influences a regulated decision, and a tool nobody approved is still a tool the regulator will hold you accountable for.
Why this matters now, not next year
For most of the last decade, AI regulation in the US was a set of principles and voluntary frameworks. Between 2023 and 2026 that changed, and the obligations became concrete enough to name.
The insurance bulletin that set this template made insurers accountable for outcomes from AI systems, including those their third-party vendors build and run, and a majority of states have since adopted some version of it. The exact count keeps moving, so the direction matters more than the number. Colorado went further with a comprehensive AI law signed in 2024, imposing duties on companies that deploy high-risk AI systems in consequential decisions such as insurance and lending, with obligations phasing in during 2026. On the lending side, the Consumer Financial Protection Bureau issued guidance in 2023 confirming that a lender using a complex model still has to give applicants specific, accurate reasons for an adverse decision. A model too opaque to explain is not a defense; it is a compliance failure waiting to be cited. And in healthcare, the Department of Health and Human Services extended its nondiscrimination rules in 2024 to cover patient care decision support tools, which brings algorithms and AI directly into scope for discriminatory outcomes.
The through-line across all of these is that accountability sits with the operator, not the tool vendor. You cannot outsource the obligation by buying software, and you cannot escape it by not knowing what your teams have already installed. For a mid-market company without a dedicated compliance-plus-technology function, that is a genuinely difficult position, because the obligations are enterprise-grade and the resources are not.
What is AI governance for regulated industries?
Governance is the set of rules that decides what AI is allowed to do inside your operation before any specific tool is chosen or built. It answers four questions in advance: which uses of AI are permitted, who signs off on a new one, what data those uses may touch, and how the outputs get checked before they act on a regulated decision.
That is a narrower and more practical definition than the one most vendor material offers. Governance is not an ethics statement or a policy document sitting in a shared drive nobody opens; it is an operating discipline with named owners and a paper trail. When a regulator asks how your operation decided that a particular model could touch borrower data, governance is what produces an answer that is not a shrug.
For a regulated operator, the check on outputs is where governance earns its keep. A model that drafts a denial rationale is fine as a drafting aid and dangerous as an unreviewed decision. What governance asks is less about whether the tool is smart and more about whether a qualified person reviews the output before it becomes an action the company is accountable for, and whether that review is documented well enough to survive an audit.
Why does shadow AI appear in mid-market operations first?
Enterprises have procurement gates and a legal function that catches new tools before they spread. Mid-market operations at 200 to 1,000 employees usually have none of that friction, which is a strength when the company is moving fast and a liability when the tools moving fast are ungoverned models handling regulated data.
The pressure that drives it is the same pressure that drives every workaround in a scaling service business. Teams are handling more volume than their systems were built for, and they reach for whatever closes the gap. We have written before about the Human API problem, where staff manually bridge the gaps between disconnected systems because no integration does it for them. Shadow AI is the newest form of that same instinct. Instead of copying a value between two screens by hand, an analyst now asks a model to do the interpretation, and the manual bridge becomes an unmonitored automated one. The gap the person was covering is still there, except a black-box model now sits inside it, touching regulated data, with no approval and no output check.
This is why governance cannot be bolted on after an AI rollout. The exposure exists before the rollout, sitting in the daily habits of people trying to get their work done.
How is governance different from AI adoption?
Adoption is the decision to deploy a specific tool against a specific workflow. Governance is the decision about what deployment is even allowed to consider. The two get conflated constantly, and the conflation is expensive, because a company that jumps to adoption without governance ends up deploying tools onto workflows nobody mapped and under rules nobody wrote.
We covered the mapping failure in detail in why AI deployment leaves human bridging intact: a tool automates the documented version of a process and leaves the undocumented exception-handling exactly where it was. Governance is the upstream discipline that would have caught the gap, because a real governance review asks what data the tool touches and what decisions it influences before anyone signs a vendor contract. Skip it, and you optimize the visible half of the work while quietly expanding the compliance surface at the same time.
For a regulated operator, the sequence has to run governance first. Decide the rules, then choose the tools that fit inside them. Running it the other way means retrofitting rules onto tools already in production, which is slower and far more likely to leave something uncovered.
What does an AI governance framework actually include?
A workable framework for a mid-market operator does not need to look like a bank’s model risk management program. It needs four things that a COO can actually maintain.
First, a permitted-use list: the specific tasks AI is allowed to perform, written plainly enough that an analyst knows whether their intended use is on it. Second, an approval owner: a named person, not a committee that meets quarterly, who can add a use to the list and is accountable for that decision. Third, a data-boundary rule that states which categories of data may and may not enter a model, with protected health information and non-public financial data treated as off-limits unless a specific approved system handles them under contract. Fourth, an output-check step for any use that touches a regulated decision, so a qualified human reviews and signs before the output becomes an action.
None of this requires new software. It requires a decision about who owns the rules and the discipline to route new AI uses through them. The hard part is not writing the framework but finding someone in a mid-market company whose actual job is to own it.
Who owns AI governance when there is no CDO?
Most mid-market service companies have a CTO or a head of IT whose mandate is keeping systems stable and evaluating technology on technical grounds. That is a real function, and it is the wrong one for this job. Governance is an operational and regulatory question first and a technical question second, and asking the technology function to own it produces tool evaluations where policy decisions should be.
This is the CDO gap we see across the regulated mid-market: companies have someone who runs the technology, but no one whose mandate is to translate business and regulatory priorities into decisions about which systems are allowed and in what order. When that role is absent, AI governance has no natural home, so it either falls to an overstretched compliance officer who does not control the technology or to a technology leader who does not own the regulatory exposure, and neither arrangement holds up under the accountability regulators now expect.
At Digital Forms, this is the function we describe as an External CDO: someone who owns the operational design decision, including which AI is permitted, measured against the company’s actual regulatory and cost structure rather than against a vendor demo. The title matters less than the fact that governance needs a single accountable owner, and in most mid-market operations that owner does not currently exist on the org chart.
What this looks like in practice
Consider a mid-market lender in mortgage servicing, a vertical where the regulatory surface is dense and the manual workload is heavy. A servicing operation of this size typically runs on several disconnected systems, with staff moving borrower information between them by hand. When AI tools appear, they appear the same way they do everywhere: informally, tool by tool, with a servicing agent using a model to summarize a hardship call or draft a borrower notice.
The governance exposure here is specific and serious. Adverse-action and loss-mitigation decisions carry explanation requirements, and a model drafting borrower communications is influencing regulated territory. Without a permitted-use list and an output check, the operation has automated part of a compliance-sensitive workflow with no record of who approved it or whether the output was reviewed. The same pattern shows up in insurance claims automation, where the difference between a governed and an ungoverned deployment is whether anyone decided the rules before the tool went live. In our experience, the ungoverned version is the default state until someone deliberately changes it.
What to do about it
Start by finding out what is already running. Most leadership teams underestimate their shadow AI footprint because it never appears in a contract. A short, honest survey of what tools people actually use, without blame attached, usually surfaces more than expected and gives you the real starting position.
From there, the fastest path is to write the four-part framework before you evaluate a single new tool, and to name one accountable owner for it. If no one internally can hold that role, that gap is worth surfacing on its own, because it will block every AI decision that follows. Quantifying where ungoverned AI and manual work are already costing you, in specific workflows with specific numbers, is exactly what our Profit Leak Diagnostic and the AI Potential Analysis inside it are built to surface, and it is the natural place to start when the exposure is real but unmeasured.
None of this is a reason to slow down on AI. Governed AI in a regulated operation is faster than ungoverned AI, because a governed operation can deploy with confidence rather than discovering its exposure during an audit, and clear rules are what let a team move at that speed.
What you can answer now
The test of whether your operation has AI governance is not the existence of a policy document but whether you can answer, today, which AI tools are touching regulated data inside your business and who approved them. If that question produces a clear answer, you have governance. If it produces a pause, the shadow AI is already there, and the only choice left is whether you find it before your regulator does.
