A cross-sectional study published in Frontiers in Public Health has evaluated how well four leading large language model (LLM) chatbots perform when providing melanoma-related health information to the public. The research, conducted by a team of Chinese medical researchers, tested ChatGPT-5.6, DeepSeek-V4, Claude Sonnet 5, and Gemini 3.5 Flash on both medical knowledge and patient-oriented communication quality.
According to the study, the chatbots were assessed using 200 melanoma-specific multiple-choice questions (MCQs) drawn from established medical exam datasets, including MedQA and MedMCQA, as well as Chinese national medical examinations. Each chatbot answered the same set of questions across three independent trials to account for variability in their outputs.
ChatGPT-5.6 achieved the highest mean accuracy at 94.0%, followed closely by Gemini 3.5 Flash at 92.5%. Claude Sonnet 5 scored 87.2%, while DeepSeek-V4 lagged behind at 79.3%. The differences between models were statistically significant, the researchers reported.
Beyond factual accuracy, the study also examined how well the chatbots could address real-world public concerns. The researchers developed 15 core patient-oriented questions by analyzing global Google Trends data from July 2021 to July 2026, combined with clinical experience. These questions covered topics such as early signs of melanoma, risk factors, and treatment options.
Responses to these questions were evaluated using four established tools: DISCERN (for treatment choice quality), the Ensuring Quality Information for Patients (EQIP) tool, the Global Quality Scale (GQS), and JAMA benchmark criteria (which assess authorship, attribution, disclosure, and currency). ChatGPT-5.6 scored highest on DISCERN (52.07), EQIP (81.67), and GQS (4.40 out of 5). However, JAMA transparency scores were low across all models, indicating that the chatbots rarely disclosed their sources or limitations clearly.
Readability was assessed using six different indices, including the Flesch-Kincaid Grade Level (FKGL) and the Simple Measure of Gobbledygook (SMOG). The study found that Gemini 3.5 Flash had the most favorable overall readability profile. Nevertheless, all models produced responses at a mean grade level exceeding the sixth-grade reading level recommended for patient-facing health information. Similarly, all mean Flesch Reading Ease Scores (FRES) fell below the recommended range, suggesting that the language used remains too complex for many members of the general public.
For university students, these findings carry several implications. Students who turn to AI chatbots for quick answers about melanoma or other health conditions should be aware that while the factual accuracy of top models like ChatGPT-5.6 is high, the information may not always be presented in an easily understandable way. Moreover, the lack of source transparency means students cannot easily verify where the chatbot's information comes from. The study's authors caution that LLM chatbots should remain a supplementary resource, not a primary source of medical advice, until improvements in transparency and health-literacy-appropriate language are made.
The researchers noted that previous studies have shown variability in the quality of online melanoma information, and that even well-written websites often exceed recommended reading levels. The current study extends this concern to AI chatbots, which, despite their conversational fluency, still struggle to communicate complex medical information at a level accessible to all readers.
A key limitation of the study, as reported by the authors, is that the chatbots were evaluated in a controlled setting without access to web browsing or external search functions. In real-world use, students might ask follow-up questions or seek clarification, which could affect the quality of the information received. Additionally, the study focused on English and Chinese language queries, and results may differ for other languages.
The study was published on November 16, 2026, in Frontiers in Public Health and was conducted by researchers from Central South University and the University of South China.