How The Joint Commission & CHAI Are Quietly Building A Parallel FDA For Hospital AI & Why The Governance Infrastructure Layer Will Eat More Of The Health AI Market Than The Models Themselves

Video Preview

🎧 Part I Podcast free on Spotify.

Part I: How The Joint Commission & CHAI Are Quietly Building A Parallel FDA For Hospital AI & Why The Governance Infrastructure Layer Will Eat More Of The Health AI Market Than The Models Themselves

🎧 Part II Podcast episode for paid subscribers only. Also available on Spotify.

Part II: How The Joint Commission & CHAI Are Quietly Building A Parallel FDA For Hospital AI & Why The Governance Infrastructure Layer Will Eat More Of The Health AI Market Than The Models Themselves

To listen to paid episodes in Apple or Spotify, link your Substack subscription via the show settings on those platforms (instructions inside the Substack app under Subscriptions → Podcast).

Abstract

Most healthcare AI commentary still assumes the rulebook gets written in DC. Wrong building. The real operational rulebook for hospital AI is being drafted by accreditation bodies, standards consortiums, quality orgs, and private guidance frameworks. The September 2025 Joint Commission plus CHAI announcement is the cleanest signal yet that governance is shifting into a decentralized accreditation-driven model. Quick map of what gets covered:

• Why hospital accreditation has more enforcement teeth than most federal AI proposals

• How the Joint Commission and CHAI framework reshapes procurement gates

• The 7 operational layers hospitals now need to stand up

• Why ambient scribes were the warmup act for a much bigger governance problem

• The category map for the governance stack (GRC, observability, drift monitoring, red teaming, validation-as-a-service)

• Winners, losers, and where the picks-and-shovels capital should flow

TLDR thesis: the next gen of healthcare AI winners may not be the smartest models. They may be the companies hospitals can actually operationalize without losing accreditation, attracting plaintiffs, or showing up in a Becker’s headline.

Table of Contents

1. Everyone thinks Washington will regulate healthcare AI

2. The real regulators are emerging somewhere else

3. Why accreditation bodies have more power than people realize

4. The Joint Commission plus CHAI partnership changes the market

5. From AI tool to AI governance surface

6. The seven operational layers hospitals now need

7. Why ambient scribes were just the beginning

8. The rise of the hospital AI governance stack

9. The coming explosion of AI committees, audits, and internal controls

10. Why health systems are unprepared operationally

11. The new vendor category map that emerges

12. How this becomes a massive procurement cycle

13. Why startups focused only on models may lose

14. The shadow regulatory state of healthcare AI

15. The next battleground, continuous monitoring and AI drift

16. The accreditation flywheel that could reshape the industry

17. Winners, losers, and investment implications

18. Conclusion

Everyone thinks Washington will regulate healthcare AI

Conventional wisdom on healthcare AI regulation sounds about right at the cocktail party level. FDA handles devices, including the growing list of AI/ML SaMDs operating under predetermined change control plan frameworks. CMS handles reimbursement and conditions of participation. ONC handles interoperability and certified health IT under HTI-1 and HTI-2. Congress eventually shows up and codifies whatever has already become consensus. So the assumption is that hospital AI policy gets written in those four buildings, then trickles down through guidance, rulemaking, and enforcement actions.

That mental model is going to age poorly. The actual operational governance of hospital AI is forming somewhere else entirely, inside a network of private institutions that already shape hospital behavior more aggressively than most federal agencies do. Anyone trying to figure out where the real action is should stop watching the Federal Register and start watching the Joint Commission’s R3 reports, CHAI’s framework releases, NCQA HEDIS specs, and HL7 Da Vinci implementation guides. That is where the binding constraints on hospital AI are actually being set. The question is not whether the FDA approves an algorithm. The question is whether a hospital’s AI governance committee can defend its deployment to its surveyors, its board, its insurers, and its plaintiff’s bar.

The argument here is that healthcare AI regulation will end up looking a lot more like the hospital quality and accreditation regime than like the pharmaceutical approval regime. And the investment implications of that shift are larger than most people are pricing in.

The real regulators are emerging somewhere else

Look at the institutions actually moving on AI governance in 2024 and 2025. The Joint Commission, which accredits roughly 80% of US hospitals, partnered with the Coalition for Health AI in June 2025 and released initial guidance in September 2025 around responsible AI use. CHAI itself has rolled out an applied model card framework, an assurance lab concept, and a series of best practice documents covering fairness, transparency, and trustworthiness. NCQA published HEDIS MY 2026 changes that bake digital quality and FHIR-based measurement deeper into accreditation. NCQA’s Digital Quality Transition program has been pushing health plans and providers toward digital measurement infrastructure that any AI tool will eventually have to plug into. HL7’s Da Vinci Project has been quietly building the FHIR rails for prior authorization, coverage requirements discovery, and documentation templates for years. AMA has updated its augmented intelligence policy and published research on physician concerns about AI-driven prior authorization denials. The National Academy of Medicine has been running an AI Code of Conduct workstream that increasingly looks like the template a lot of these other groups are borrowing from.

None of these organizations have rulemaking authority in the federal sense. None of them can fine a hospital directly. But collectively they set what hospitals must do to maintain accreditation, qualify for payer contracts, satisfy purchaser RFPs, survive malpractice discovery, and keep board risk committees off their backs. That is not soft power. That is the actual operating envelope of an American hospital. Federal rulemaking sets the floor. These groups set the procedures.

Why accreditation bodies have more power than people realize

The Joint Commission deserves special attention because the leverage it has over hospitals is genuinely under-appreciated outside the industry. Joint Commission accreditation is a CMS-deemed status, meaning a hospital that maintains accreditation can demonstrate Medicare Conditions of Participation compliance without going through separate state surveys for that purpose. Lose Joint Commission accreditation and a hospital risks losing Medicare and Medicaid reimbursement, which is roughly 60% of revenue at most acute care facilities. There is no business model that survives that hit. Boards know it. Insurers know it. Bond raters know it. Operations teams plan their entire lives around survey readiness.

Joint Commission standards do not have to be law to be operationally binding. They function more like the building code than like criminal statutes. Hospitals operationalize thousands of these standards across patient safety, medication management, infection control, environment of care, and emergency management. Compliance staff document everything. Mock surveys are routine. Failing a tracer methodology audit creates real consequences. The framework is mature, deeply embedded, and culturally enforced.

If AI governance gets folded into that survey framework, and the September 2025 guidance is the leading edge of exactly that, then hospitals will operationalize AI oversight the same way they operationalize infection control or medication reconciliation. Which is to say, with mandatory committees, written policies, documented training, periodic audits, evidence binders, and a willingness to fire people who do not comply. That is a fundamentally different posture from the current hospital AI playbook, which is closer to enthusiastic chaos.

The Joint Commission plus CHAI partnership changes the market

The September 2025 announcement was easy to miss in a news cycle full of model launches and FDA guidance documents, but for hospital procurement teams it was a regime change. The initial guidance lays out expectations across governance frameworks, local validation, bias monitoring, transparency, clinician oversight, incident reporting, post-deployment monitoring, and accountability structures. None of that language is exotic to anyone who has read CHAI’s prior work. What is new is the implied enforcement mechanism. The Joint Commission has signaled that responsible AI use will eventually be part of accreditation expectations, and the clock from initial guidance to expected practice is short by hospital planning standards. Twelve to twenty four months is roughly how this organization tends to move when it commits.

That single shift transforms AI from an innovation initiative into enterprise risk infrastructure. Innovation officers will still pilot things, but the procurement gate is moving to legal, compliance, IT security, and clinical risk. The CMIO and CMO will still champion specific tools, but the AI governance committee will have a veto. Vendors who cannot produce documentation that maps to the guidance will get filtered out before pricing conversations even happen. That filter is being built right now inside health systems, and the early movers are visibly different from the late movers in their RFP language.

From AI tool to AI governance surface

The conceptual shift that matters most is that an AI deployment is no longer just a tool. It is a governance surface area. Every model in production creates exposure across multiple risk dimensions at the same time. Compliance exposure shows up through HIPAA, OCR enforcement, state privacy laws, and increasingly state-specific AI laws like the Colorado AI Act and the Texas TRAIGA framework. Legal exposure shows up through medical malpractice doctrines that are starting to address algorithmic standard of care. Audit exposure shows up through internal audit, external audit, and now Joint Commission surveyors looking for AI governance documentation. Patient safety exposure shows up through sentinel event reporting and FDA postmarket surveillance for any tool that crosses into SaMD territory. Cybersecurity exposure shows up through ransomware vectors, model poisoning, and data exfiltration concerns. Workflow exposure shows up when a flawed recommendation propagates across an entire shift before anyone catches it. Documentation exposure shows up when a draft note that was supposed to be reviewed gets autosigned and ends up as the legal record.

Every deployed model is, in effect, a small clinical program that has to be governed like one. Health systems that figured out years ago how to operationalize new drug protocols, blood product policies, or surgical safety checklists already have the muscle for this. They just have not pointed it at AI yet. The Joint Commission and CHAI guidance is the prompt that points it.

The seven operational layers hospitals now need

Once AI deployment is treated as a governance surface, the operational stack a hospital actually has to stand up gets fairly concrete. There is an enterprise AI inventory, because no one can govern what they cannot see, and shadow AI is already a measurable problem in nearly every system that has bothered to count. There is a governance approval workflow with intake forms, risk scoring, committee review, sign offs, and renewal cadence. There is a local model validation pipeline, since national vendor claims rarely match local population performance, and the guidance is explicit that hospitals should validate against their own data before deployment. There is AI observability infrastructure covering input distribution monitoring, output distribution monitoring, latency, refusal rates, and downstream clinical signals. There is incident reporting, because surveys will eventually ask hospitals to produce an AI incident log the way they already ask for medication error logs. There is a policy enforcement layer, ideally technical rather than just paper, so a model that loses its approval can actually be turned off. And there is rollback and orchestration, because hospitals will run dozens of models concurrently and need to manage them like services rather than one off implementations.

Almost no hospital has all seven of those layers in mature form. Most have one or two, sometimes built ad hoc by an enthusiastic data science group or stitched together by a CMIO with a part time analyst. Closing that gap is a five year program with significant capital and operating expense, and the build versus buy decisions across each layer are going to be major budget events. That is a vendor market that did not really exist in 2022 and is now visible in pitch decks every week.

Why ambient scribes were just the beginning

The current generation of hospital AI is mostly low autonomy and low decisional weight. Ambient documentation tools like Abridge, Suki, DAX, and Augmedix listen, draft, and let the clinician sign. Coding assistance tools propose ICD-10 and CPT picks for human review. Administrative copilots help with prior auth letter drafting, scheduling support, and inbox triage. These tools are valuable, frequently popular with clinicians, and reasonably easy to govern because the human stays in the loop on the final decision. Most boards are comfortable signing off on them. Most malpractice carriers do not flinch.

The next generation is qualitatively different. Autonomous triage agents, AI-driven utilization management, care navigation that initiates outreach without human approval, nurse workflow orchestration that reprioritizes patient assignments, ambient clinical decision support that pushes recommendations into the encounter, and administrative agents that complete end to end tasks across multiple systems all change the governance calculus. Once a model is taking actions or strongly shaping decisions without a clinician in the loop on every turn, the surface area gets bigger fast. Local validation matters more. Drift detection matters more. Bias monitoring matters more. Audit trails matter more. Failure modes get more interesting and more dangerous. The Joint Commission and CHAI guidance is written in a way that scales with autonomy. The more decisional weight a tool carries, the more governance infrastructure the deploying hospital is expected to wrap around it.

This is the part of the cycle where governance complexity rises faster than capability. Capability is exponential in compute and data. Governance is linear in headcount, process maturity, and software tooling. The gap between what a model can do and what an enterprise can responsibly let it do is going to be the binding constraint on hospital AI for years.

The rise of the hospital AI governance stack

Step back and the category map for healthcare AI infrastructure starts to look familiar to anyone who lived through the cybersecurity buildout post HIPAA enforcement, or the IT service management buildout in the ServiceNow era. AI observability platforms log inputs, outputs, latency, and behavior across deployed models. AI governance SaaS handles intake, risk scoring, approvals, attestations, renewals, and surveyor-ready evidence. AI audit logging captures who did what to a model, when, and with what data. Synthetic testing platforms generate adversarial and edge case prompts to probe failure modes before deployment. Bias monitoring tools track subgroup performance against fairness criteria over time. Explainability middleware sits between black box models and clinical end users to surface salient features and confidence signals. Model lifecycle management tools handle versioning, deployment, and rollback. Policy orchestration layers translate written governance policies into actual technical controls. AI access management defines who in the organization can deploy, modify, or query which models against which data. Workflow monitoring catches the second order effects of AI on operational metrics like turnaround time, length of stay, and staff productivity.

That looks a lot like a Datadog, Okta, ServiceNow, and Splunk equivalent layered into hospital AI operations. None of those analogs were obvious categories in 2010. By 2020 they were each multibillion dollar businesses. The same trajectory is plausible here, with a healthcare-specific twist because the data, the workflows, the regulators, and the buyers are all different from the general enterprise market.

The coming explosion of AI committees, audits, and internal controls

The organizational chart of a typical health system is about to gain some new boxes. AI steering committees are already common, but they are about to grow teeth. Model review boards modeled on IRBs are emerging at academic medical centers and will spread. Enterprise AI governance councils will sit above the steering committees, pulling in legal, compliance, security, clinical leadership, finance, and operations. Board level oversight committees, often nested under existing audit or quality committees, will start requesting AI risk reports at quarterly cadence. CMIO offices will absorb new staff focused on clinical AI governance. Clinical AI risk scoring will become a routine intake artifact, the way HIPAA risk assessments became routine after Omnibus.

The closest analogy is the SOX compliance buildout in public companies after Enron and WorldCom. Internal controls, formal documentation, defined accountability, periodic testing, and external attestation reshaped the finance function across the entire S and P 500. The cost was significant and the cultural change was real. Hospital AI is heading somewhere similar in operational impact, on a faster timeline, because the technology is moving faster and the governance frameworks are converging faster than SOX did.

Why health systems are unprepared operationally

The gap between what the September 2025 guidance expects and what most hospitals can actually do today is wide. Procurement is fragmented across service lines, with rev cycle, IT, clinical innovation, and individual departments all signing AI contracts independently. Shadow AI is rampant, with clinicians using consumer chatbots, browser plug ins, and free trials of vendor tools without IT or compliance review. Governance, where it exists, is inconsistent across facilities even inside the same system. Technical oversight is thin, with a typical 500 bed hospital having maybe one or two staff capable of evaluating model performance claims. Vendor opacity is the norm rather than the exception, with model cards, performance data, and bias evaluations frequently incomplete or absent. Internal AI expertise is concentrated in a handful of academic centers and a few large IDNs that have built dedicated data science teams. The rest of the country is staffing up from a small and expensive pool of qualified candidates.

Adoption velocity has been outpacing governance capacity for at least two years. The September 2025 guidance is the moment that misalignment becomes a documented problem rather than a quiet one. Anyone reading the surveyor’s prep materials over the next year should expect AI governance to move up the priority list quickly.

The new vendor category map that emerges

For investors and operators, the implication is a category map that did not really exist as a coherent market two years ago. Healthcare AI GRC platforms, covering governance, risk, and compliance, sit at the top of the stack and are currently dominated by point tools and adjacent expansion from broader enterprise GRC vendors. Validation-as-a-service offerings, where third parties handle local validation against a hospital’s own data, can compress months of work into weeks for under-resourced systems and may end up looking like a Press Ganey or Vizient style recurring relationship. Synthetic clinical testing fills the gap between vendor claims and real world performance with controlled adversarial probing. AI procurement diligence services help legal and supply chain teams evaluate vendor documentation against the emerging frameworks. AI governance infrastructure includes the workflow tools that route models through approval, monitoring, and decommissioning. AI compliance automation maps controls to specific accreditation, regulatory, and contractual requirements. AI red teaming applies offensive testing methodologies, borrowed from cybersecurity, to clinical AI failure modes. Drift monitoring systems specialize in detecting performance degradation post deployment, which is the hardest governance problem in the entire stack.

A consistent investment thesis across all of these categories is that the picks and shovels layer may end up larger than the model layer in healthcare specifically, because healthcare AI is the least black-box-tolerant market in the world. The combination of patient safety stakes, regulatory exposure, professional liability, and accreditation pressure produces a buyer who will pay for trust infrastructure at a premium.

How this becomes a massive procurement cycle

Health system AI budgets in 2024 were largely innovation budgets. Soft money, one off pilots, conference travel, and occasional consulting engagements. The Joint Commission and CHAI guidance, combined with NCQA’s digital push and CMS’s interoperability and prior authorization rule finalized as CMS-0057-F in early 2024, are turning those innovation budgets into operating budgets with compliance line items. Dedicated AI governance budgets are showing up in 2025 and 2026 financial plans. Enterprise governance software purchases are starting to appear in capital plans for systems that previously left this work to ad hoc spreadsheets. Consulting waves are building, with the big four already standing up dedicated healthcare AI governance practices. Implementation services growth follows the consulting because someone has to actually stand up the committees, write the policies, and configure the tools. Internal compliance staffing is expanding across the job postings tracked by the major health system associations.

The closest cycle to learn from is the cybersecurity buildout after the early 2010s ransomware attacks and HIPAA Omnibus enforcement. Cybersecurity went from a CIO line item to a CISO function with its own committee, its own budget, its own vendors, its own consultants, and its own accreditation hooks. AI governance is heading in the same direction with a similar shape but probably a faster curve, because the underlying technology is moving faster and the surface area is broader.

Why startups focused only on models may lose

The contrarian implication for healthcare AI startups is uncomfortable. Model quality matters, but it is increasingly a necessary condition rather than a sufficient one. The next layer of competitive advantage is documentation, integration, governance readiness, and trust signals. Auditability beats marginal accuracy at the procurement gate. Explainability is now table stakes for any tool that touches clinical decisioning. Workflow integration is the difference between a pilot that ends in a Becker’s case study and a deployment that survives the second renewal cycle. Governance readiness, which includes model cards, validation packets, monitoring plans, incident response protocols, and bias evaluations, is rapidly becoming a binary filter in RFPs. Rollback capability matters because hospital risk committees want to know they can turn a model off in an hour, not a quarter. Continuous monitoring matters because the question is no longer just how well a model performs on day one, but how to detect when it starts performing differently on day three hundred. Trust infrastructure, broadly defined, will be more durable as a moat than benchmark scores.

That favors vendors who built governance and workflow as first class features rather than as afterthought layers. It also favors infrastructure plays that any vendor can use to look more governable, which is part of why the picks and shovels thesis has so much momentum among investors who have spent time inside hospital procurement.

The shadow regulatory state of healthcare AI

Putting all of this together, healthcare AI is being governed by what amounts to a shadow regulatory state. Accreditation bodies set expectations that hospitals treat as mandatory. Payer contracts encode requirements that translate accreditation language into commercial obligations. Procurement rules pulled from those contracts and from internal risk frameworks filter vendors before any government agency ever sees the deployment. Malpractice doctrine, plaintiff’s bar pressure, and insurance underwriting create another layer of operational governance that is functionally regulatory without being formally regulatory. Implementation guides from HL7 Da Vinci shape what AI tools can actually do because they shape the underlying data flows for prior auth, coverage determination, and documentation. Insurer requirements, including from medical professional liability carriers, push hospitals toward documented governance practices. Operational governance frameworks like CHAI’s translate all of the above into something a CMIO can actually implement.

The most important AI regulators in healthcare may not be government agencies at all. They may be a small cluster of nonprofit accreditation bodies, standards organizations, and quality groups that already shape hospital behavior more deeply than most federal agencies do, plus the insurers and lawyers who price the consequences of getting it wrong. Federal action will still happen and will still matter, especially around CMS conditions of participation, FDA SaMD oversight, and ONC certification rules. But the binding day to day constraints on hospital AI deployment are being written in the language of accreditation, not the language of statute.

The next battleground, continuous monitoring and AI drift

The hardest operational problem in the emerging governance stack is post deployment monitoring. Predeployment validation is conceptually familiar. Hospitals have been doing local validation of risk scores, sepsis algorithms, and clinical decision support tools for years. The new problems show up after go live. Clinical drift, where the patient population changes and the model’s training distribution no longer matches reality, is hard to detect without specific tooling. Model drift, where the same input starts producing meaningfully different outputs over time, is similarly hard to catch by eye. Hallucination detection in generative tools is an open research problem with real clinical stakes. Workflow degradation, where the existence of an AI tool changes how clinicians document, decide, or escalate, can show up months after deployment as a quality signal that no one initially attributes to AI. Patient population shifts during seasons, payer mix changes, and service line expansions all stress models in ways that are invisible without monitoring infrastructure. Escalation systems need to be in place so that when drift is detected, someone with authority can actually act on it. Post deployment surveillance, in the spirit of FDA postmarket adverse event reporting, is the regulatory direction of travel.

Drift is where governance gets technically hard, and it is where the next round of vendor differentiation lives. Anyone who can credibly monitor a hospital’s full AI portfolio in production, surface meaningful alerts, and integrate with the governance workflow becomes very sticky very fast.

The accreditation flywheel that could reshape the industry

Once the cycle is running, the dynamic is self reinforcing. Joint Commission and CHAI publish guidance. Hospital procurement teams encode it into RFPs. Vendors adapt their documentation, monitoring, and integration capabilities to meet the new criteria. Insurers, both payers and medical liability carriers, observe the trend and incorporate governance maturity into underwriting and contracting. NCQA folds related expectations into HEDIS and accreditation for plans, which flows back into provider contracting. CMS observes the consensus emerging from the private side and starts referencing it in guidance, conditions of participation language, or quality measurement programs. State legislatures pull from the same frameworks when drafting AI laws. The Joint Commission updates its standards to formalize what has become consensus practice, and the loop runs again at a higher level.

That is how soft guidance becomes hard operational necessity in American healthcare. It happened with patient safety after the IOM To Err Is Human report. It happened with cybersecurity after the early ransomware waves and HIPAA Omnibus. It happened with health information exchange after Meaningful Use, and is happening again with TEFCA. Hospital AI is on the same kind of trajectory, faster, and the September 2025 guidance is one of the inflection points the histories will look back on.

Winners, losers, and investment implications

Likely winners include governance infrastructure vendors with healthcare-specific depth, model orchestration platforms that can manage portfolios across modalities, AI observability companies that built for clinical workflows rather than for general enterprise IT, middleware providers that bridge model output to clinician interface, workflow-native AI vendors who shipped governance features alongside model features, and enterprise infrastructure layers that integrate with EHRs and FHIR-based exchange. Validation-as-a-service businesses with credible methodology and access to diverse hospital data should command durable relationships. Drift monitoring specialists with strong clinical signal libraries should become acquisition targets. Healthcare-specific GRC platforms that can map controls to Joint Commission, CHAI, NCQA, and state AI law requirements should grow into recurring spend categories.

Likely losers include opaque black box AI vendors who cannot produce surveyor-ready documentation, point solutions lacking governance controls or monitoring hooks, non-integrated copilots that cannot demonstrate workflow safety, and vendors without rollback or monitoring capabilities. Companies that built brilliant models but assumed the buyer would handle governance are about to find out that hospitals do not have that capacity and will not buy under those terms much longer. Companies that assumed federal regulation would set the bar and then stalled until it materialized are about to discover the bar got set somewhere else without them.

For investors, the practical reframing is simple. Underwriting hospital AI investments based on model performance, addressable market, and design partner momentum is no longer enough. The diligence questions that matter now include how the company documents itself for accreditation, how it integrates with hospital governance workflows, how it monitors performance after deployment, how it supports local validation, how it handles incident reporting, and how it talks to surveyors and lawyers. Companies that can answer those questions credibly will get into hospitals. Companies that cannot will get filtered out before pricing.

Conclusion

The healthcare AI race is being narrated as a competition between models, foundation labs, and hyperscalers. That framing misses where the actual constraint sits. The binding constraint on hospital AI in the United States is operational governance, and the operational governance is being defined by a quiet cluster of accreditation bodies, standards organizations, quality groups, and private frameworks that already shape hospital behavior more deeply than any federal agency does on a day to day basis.

The next generation of healthcare AI winners may not be the companies with the smartest models. They may be the companies hospitals trust enough to operationalize at scale without losing accreditation, attracting plaintiffs, or surprising their boards. That trust will increasingly be shaped less by FDA letters and CMS rules and more by Joint Commission surveys, CHAI frameworks, NCQA specs, Da Vinci implementation guides, AMA policy, and the private governance infrastructure already restructuring American healthcare from underneath. The companies and investors who see that shift early get to build the operating system for hospital AI. The ones who keep watching Washington will end up reading about it.