As more people turn to generative artificial intelligence for health information, a new paper in Frontiers in Public Health argues that the central question should not be whether users trust these systems, but whether their reliance on the advice is appropriate given its accuracy and the potential risks of acting on it.
The perspective article, authored by researchers at Huzhou Normal University in China, introduces the concept of "calibrated reliance" as a standard for evaluating how laypeople use AI health advice. The authors argue that four possible actions — adopting the advice, verifying it against another source, rejecting it, or escalating to professional medical care — should be judged together based on whether the advice is correct and what an error would cost, particularly in terms of clinical risk.
According to the paper, surveys indicate that many people already use large language models (LLMs) for health information, self-diagnosis, and decisions about whether to see a doctor. The authors cite a case report in which a man replaced dietary sodium chloride with sodium bromide after consulting ChatGPT, leading to hospitalization with bromism, paranoia, and hallucinations. They also reference an experimental trial in which members of the public using LLMs to work through medical scenarios identified the correct course of action no more often than a control group, even though the same models performed well when tested alone.
The paper reviews a set of human-participant studies published between 2023 and 2026, drawn from targeted searches of PubMed, Europe PMC, and the human-computer interaction and psychology literatures. The studies span models from GPT-3 to GPT-4o and include preregistered randomized experiments with large samples, including two experiments with 2,280 participants on source labels and a trial of care decisions with 1,298 UK adults.
The evidence, the authors write, shows both types of reliance errors. Underreliance occurs when identical advice is discounted simply because it is labeled as coming from AI rather than a physician. Two preregistered experiments with 2,280 participants found that medical advice labeled as AI-generated, or as from a physician assisted by AI, was rated less reliable and less empathic than the same advice labeled as from a physician alone. Participants were also less willing to follow it. The authors note that this pattern fits known phenomena such as algorithm aversion and expertise-weighted advice taking, but caution that it is not necessarily proof of miscalibration — a user who discounts AI advice because no one is accountable for it may be reasoning sensibly.
The stronger concern, according to the paper, is overreliance: laypeople often fail to distinguish correct from incorrect advice. In three experiments with about 300 participants, researchers presented physician responses and GPT-3 responses that physicians had rated as high or low in accuracy. Participants could not tell AI responses from physicians' responses, and they rated low-accuracy AI responses similarly to physician responses. In another trial, lay users assisted by large language models chose care dispositions no better than controls, suggesting they were not effectively using the AI to improve their decisions.
The authors argue that both patterns suggest lay reliance responds more to source and fluency cues — such as whether the advice sounds confident or comes from a trusted label — than to the actual correctness of the content. They note that interventions that require users to act, such as verification affordances, cognitive forcing, and expressed uncertainty, show more promise than passive labels, but no study has yet manipulated correctness and graded clinical risk together while observing verification and escalation behaviors.
The paper states five testable propositions with operational definitions and proposes a factorial design that would test them. The authors call for future research to move beyond measuring trust as an attitude and toward measuring reliance as a behavior that can be scored against an outcome standard.
For university students, who are among the heaviest users of generative AI tools, the findings carry a practical warning: the fluency and confidence of an AI-generated health answer are not reliable signals of its accuracy. The paper suggests that students — and all lay users — should develop habits of verification, especially for high-stakes health questions, and should not assume that an AI's advice substitutes for professional medical judgment.