I build AI systems. Not in the abstract — I run a small fleet of autonomous agents that handle scheduling, research, document generation, and publishing.
That work has taught me something clinical AI is about to teach everyone: AI doesn't fail the way you expect it to. It fails confidently, in ways that look correct.
Documentation tools, triage assistants, sepsis alerts, ambient scribes, patient-message drafting. Clinical AI is arriving whether or not anyone asked. Nurses will meet it first, at the point of care — which means nurses, not vendors, will be the ones who notice when it's wrong.
The failure that looks like success
The failure mode isn't usually a machine that says "I don't know." It's a machine that produces something plausible, well-formatted, and wrong — in the same tone it uses when it's right.
In the systems I run, the errors I catch are almost never dramatic. They're small substitutions: a date that shifted by one day, a name normalised to the more common spelling, a number that appears in two places and disagrees with itself. Everything around the error is correct. That context is what makes it dangerous — it reads as trustworthy.
My scheduling agent once booked two of my own meetings into the same block and reported both as confirmed. No error. No flag. Two records that each looked fine on their own. I caught it because I happened to look at the day, not because the system told me.
A system that fails loudly is manageable. A system that fails quietly is the one that hurts someone.
The questions to ask before a tool touches a patient
If your unit is evaluating an AI tool, these are the ones that separate a real product from a demo.
What does it do when it can't ground an answer in the source data? This is the important version of "what happens when it isn't sure," and it's harder than it sounds. In most deployed systems you get a fluent sentence either way, and the confidence signal a developer sees in testing often isn't surfaced to the user at all. So don't ask to see the low-confidence path. Ask them to show you a case where it declined to answer. If they can't produce one, it doesn't decline.
Where does the output go when nobody checks it? Trace the whole flow, because the risk isn't where you'd expect. Signed notes require authentication, so a drafted note reaching the chart unreviewed is a narrow path. The wider one is auto-populated flowsheet and vitals fields — data that flows into the record without a person ever affirming it. Find out what the tool writes and who has to approve it, field by field.
What is it actually reading? Ask what data source it draws on and, more usefully, what it ignores. A tool that can't see the medication list is making recommendations with a blind spot — and it won't tell you which one.
Who is accountable when it's wrong? Ask the vendor to put it in writing. You'll usually get a version of "the clinician." Get them to say which clinician, for what decision, and under what conditions — because that answer determines how much the tool should be doing unsupervised.
Can you show me the misses? Any vendor can demo a system being right. Ask what a false negative looks like, and how often one has happened in the field. The quality of that answer tells you more than any accuracy figure on a slide.
Your new role: the last check
There's an uncomfortable truth here. AI systems are often deployed precisely because nurses are too busy to do everything — and the reviewing work lands on the same nurses.
That's not a reason to refuse the technology. It's a reason to be deliberate about what you accept. If a tool saves you four minutes of typing but adds two minutes of verification, you haven't gained two minutes. You've gained two minutes and a new failure mode you're now responsible for noticing.
The nurses who do well in this shift will treat AI output the way they treat a phone order: plausible, probably fine, and confirmed before anyone acts on it.
What to do when you catch it
Use the formal channel first. Report it through whatever safety-reporting system your organisation has, in writing, the same way you'd report any other concern. The pattern worth flagging is a wrong output that a colleague accepts as correct — that's a system-level hazard, not a user error.
And note the regulatory layer. If the tool is an FDA-cleared device and it contributes to a death or serious injury, the facility has mandatory reporting obligations — medical device reports are due within about ten days. That's not something to route through a suggestion box.
The part that matters
I'm not warning you off AI. I use it daily, and it has changed what I can build. But the usefulness comes from knowing where the edges are, and right now those edges are nurses.
Every deployment I've seen has a person handling the output. The technology will improve. Until it does, knowing how the machine fails is what turns "the computer said so" into nursing judgement.
Clinician Approved
One note a month on clinical AI — what I saw, what broke, what I'd sign.