Clinician Signature

Clinical AI

A Nurse's Guide to Reading an AI Tool's Evidence

How to read a vendor's evidence the way you read a lab value.

By Nnamdi Okorafor

When a vendor walks in with an AI tool, the demo is clean and the promises are broad. The evidence is where the tool either holds up or does not. Reading it is a skill nurses already have. It is the same way you read a lab value: you do not ask whether it looks good, you ask what it was measured against and what it does not show.

First, ask what the tool was measured against. A vendor will say the tool was accurate on some dataset. Ask which patients were in that dataset, and whether they look like your patients. A model that does well on a large hospital's records may not do well on a small clinic's.

Second, ask what the number means in your unit. Accuracy is not the same as usefulness. A tool can be accurate and still add no value if a nurse has to redo the work anyway. Ask what a nurse does with the output, and whether it saves a step or adds one.

Third, ask who is responsible when it is wrong. The answer matters more than the accuracy number. If the vendor will not put it in writing, that is the answer.

Fourth, ask to see a failure. Not the clean demo case, but a real one where the tool got it wrong. What did it get wrong, and how did a person catch it. If the vendor cannot show you one, they have not looked.

None of this requires a statistics degree. It requires the same skepticism a nurse already applies to a monitor that might be wrong.