So, I'm in the medical field building an EMR and LLMs have obviously been a really important topic in the industry the last few years. We're still not even sure that giving LLM-assisted suggestions TO ACTUAL DOCTORS AND CLINICIANS will be helpful let alone to the patient themselves.

It's breaking the golden rule of these tools which is to have someone with enough knowledge to verify the accuracy of the data it spits out. Patient's famously don't. Hell, even the actual staff don't really understand or know how these tools work (or the ways in which you can/can't trust them).