Healthcare already had the strictest data rules in the economy. AI adds new questions about who makes the clinical decision, and what the patient is told.
AI is in ambient scribes, prior authorization, patient messaging, coding and clinical decision support. Each of those touches a rulebook that predates generative AI, and several regulators have updated their rules specifically for it.
PHI follows the prompt
When a clinician pastes a note into an AI tool, that text is protected health information. The vendor needs a business associate agreement, and you need to know where the data goes, whether it trains a model and how long it’s kept. Many “quick pilot” tools fail this test on day one.
Decision support comes with a fairness duty, and soon less built-in transparency
Since 2025, certified health IT that offers predictive decision support has had to show information about how each model was developed and evaluated. ONC’s HTI-5 rule, published in October 2026, removes those requirements from 28 December 2026. They still apply until then; after that, the transparency will increasingly have to come from your contracts and your own review.
The fairness duty hasn’t moved: covered entities must make reasonable efforts to identify and mitigate discrimination risks in the patient care decision support tools they use.
Some AI is a medical device
Software that diagnoses or treats can be regulated by the FDA. For AI-enabled devices, a predetermined change control plan lets manufacturers specify in advance how a model may be updated without a new submission. That’s change control written into regulation.
In the EU, AI that is a safety component of a medical device is high-risk under the AI Act from August 2028, on top of the Medical Device Regulation.
States are writing rules for AI in care
- Patient communications: California requires a disclaimer when generative AI writes clinical messages to patients without clinician review.
- Utilization review: California requires that medical necessity decisions be made by a licensed physician or other qualified licensed health care professional, not by an algorithm alone.
- More states are moving on AI in prior authorization and insurance decisions.
The compliance question moved from “is the data protected?” to “who decided, what were they shown, and what was the patient told?”
What good looks like
- An inventory of every AI tool touching PHI or care, with its BAA status and data flows.
- Clinical owners for decision-support tools, with documented review of fairness and performance.
- Change control for model and prompt updates, especially for anything patient-facing.
- Evidence of human review where the law requires a clinician decision.
Where AgentFlow fits
AgentFlow keeps a single register of clinical and operational AI, records who reviewed and approved each change, maps use cases to federal and state rules, and stores the evidence your compliance team and auditors need. See it in action →