Nightmare because users approach LLMs with the false confidence that they're always right, and present LLM outputs as fact to Doctors who have to waste time explaining that it's wrong most of the time. It hurts more than it helps.
Hurts who ? Yes the doctor is super stressed and has maybe 10 minutes for you that's the actual problem, it's not like before LLMs they were super glad to sit there and answer all your questions.
Well it was a nightmare for my mother's do-nothing GP surgery in the UK. She had several conditions which were being handled completely separately without central coordination, and her health was in serious decline. We went in with a list of 20 AI-generated questions based on her conditions and treatment (which I was able to screen as I have a bio postgrad, but not medical training), including those related to NICE guidelines and procedure, and, frankly the GP bricked it and ordered a load of new interventions. My mother started to get proper treatment.
I wouldn't trust AI to make a diagnosis, but I would absolutely trust it to notice where procedure hasn't been correctly followed, where a treatment is counter-indicated because someone has missed a line on a health record, or where there's a clear potential alternate diagnosis which has been missed for spurious reasons. Also, unfortunately, where doctors aren't doing a decent job - often because they're overworked or underfunded.
UK has probably the worst healthcare in the developed world. In part perhaps due to UK blindly accepting any kind of medical degree (doctors and nurses) from all over the globe. Yes, you heard that right, they verify the validity of the degree but there is no formal standardized exam to sit to practice in the UK.
Her nurses frequently couldn't speak English. I had to spell out her medical conditions letter by letter so that the nurse could input it onto the medical system. She was entering gibberish / wrong info until I stepped in, as I was watching the PAS out of interest.
They effectively had tourist English. They're probably in through an agency so the financial incentive is to find an examiner who will give the right answer.
There’s more than two options here. It was already difficult to deal with self diagnosis for doctors, now we have a machine that outputs recommendations, and does it with confidence whether it’s correct or not.
The same issues that were present with search-engine self diagnosis are still present with LLMs. If you provide Google with an incomplete list of symptoms and can’t interpret the information you find correctly, you will likely get an incorrect diagnosis. The same is true for LLM output.
Everyone on the internet loves to put doctors on a pedestal, but I think upwards of 30% of my doctor visits have been misdiagnosed.
There's a reason I ask AI about absolutely everything medical and there's a reason I keep extra quantities of prescription medications around for emergencies. I've saved my own ass a lot more times than the doctors have, thanks to good doctors not being available.
That works in your case because you likely already have an analytical mindset, can reliably discard or research information further. Same thing why AI is in an enabler for people having already some skill. But it can be very dangerous or misleading for people blindly believing all output.
> It was already difficult to deal with self diagnosis for doctors
I get it. But the current system is also super difficult for the patient: getting time to ask questions, get clear answers, get the best possible diagnosis taking into account your history, symptoms etc and all that in 5-10 minute checkup when your doctor sees 50 patients a day and has very little time for you; this doesn't scale well. Patients run to A.I for a reason.
There are quite a few disclaimers everywhere that soften confidence: "always ask a medical specialist", "I'm not a doctor", "this could have been this or that but really not sure", etc.
Its a nightmare because it erodes trust. Doctors are not "always right" which is why "always get a second opinion" is codified in culture.
But AI's problem is that its completely full of shit, sometimes, and the people most qualified to evaluate whether its full of shit are the doctors, not the patients, but just like OP's original article, patients are left feeling like their second opinion from AI might be more trustworthy than their doctors opinion.
> But AI's problem is that its completely full of shit, sometimes
It's now quite unusual that it's "Completely full of shit". If it contradicts something your doctor said I don't see why you should feel ashamed to bring it up.
Sure it complicates the doctor's work, having ignorant obedient patients must be more comfortable for the doctor, but the end result could be more accurate diagnosis.
But those are obvious errors. What if the AI tells you to up your intake of X and it seems plausible? So you up your intake of X by taking some supplement, but upping the intake of X makes your body deplete more of Y and now you have a new or compounding problem.
A doctor might have never recommended upping X, because they would know what it does to your body. Or they might have suggested additional supplementation to avoid this.
The fact that LLMs are trained on all public knowledge is a huge red flag, because there are more wrong infos out there than right ones. Especially about health, diet, etc.
I was about to respond but your last statement implies fundamental misunderstanding of how LLMs work. I don’t think you even know about RLHF and you think good and bad ideas are spread in proportion to how much they are seen on the internet.
They are using the “gold standard for the evaluation of expert medical computing systems” not a proxy for what a doctor actually does when diagnosing someone.
It may have some utility after diagnosis, but this test doesn’t demonstrate utility for patients.
Not quite. An LLM generates text that would likely follow. The sky is… “blue”. A patient in pain with a bone protruding from their shin has a… “broken leg”.
The more training data, the more questions it can answer with a reasonable degree of probability of accuracy.
Throwing away a potentially useful analysis just because it’s probabilistic seems a bit like throwing the baby out with the bath water.
Yes that’s my point. An LLM can clearly accurately predict obvious cases, so it’s reasonable to assume that it can predict less obvious cases with somewhat less accuracy.
The real question is where’s the cut-off point between accuracy and utility.
Remember: a second human opinion can also be wrong, and even a wrong opinion can still be useful (especially in medicine where differential diagnoses are a common practice - if the LLM gives you a useless opinion, you rule it out and move on).
I don’t think it’s particularly unreasonable to think that an LLM would have enough literature, or enough reasoning ability, to be able to generate a plausible interpretation of the data. A human can then review and say either “yeah that’s clearly not the case here” or “hmm, actually that could explain it, maybe we should order another test”.
