Can ChatGPT help patients understand their andrological diseases?

¿ChatGPT puede ayudar a los pacientes a comprender sus enfermedades andrológicas?

We aimed to assess the reliability of Chat Generative Pre-training Transformer (ChatGPT)’s andrology information and its suitability for informing patients and medical students accurately about andrology topics. We presented a series of systematically organized frequently asked questions on andrology topics and sentences containing strong recommendations from the European Association of Urology (EAU) Guideline to ChatGPT-3.5 and 4.0 as questions. These questions encompassed Male Hypogonadism, Erectile Dysfunction and Sexual Desire Disorder, Disorders of Ejaculation, Penile Curvature and Penile Size Abnormalities, Priapism, and Male Infertility. Two expert urologists independently evaluated and assigned scores ranging from 1 to 4 to each response based on its accuracy, with the following ratings: (1) Completely true, (2) Accurate but insufficient, (3) A mixture of accurate and misleading information, and (4) Completely false. A total of 120 questions were included in the study. Among these questions, 50.0% received a grade of 1 (completely correct) (55.4% for 4.0 version). The combined rate of correct answers (grades 1 and 2) was 85.2% for frequently asked questions (88.8% for 4.0 version) and 81.5% for questions obtained from the guideline. The rate of completely incorrect answers (grade 4) was 1.8% for frequently asked questions (0% for 4.0 version) and 5.2% for questions based on strong recommendations. The response rate of version 4.0 to questions created from sentences containing strong recommendations from the EAU guideline was the same as version 3.5. ChatGPT provided satisfactory answers to the questions asked, although some responses lacked completeness. It may be beneficial to utilize ChatGPT under the guidance of a urologist to enhance patients’ comprehension of their andrology issues.

Resumen

El uso de Internet y de sistemas de información basados en inteligencia artificial está aumentando en muchos ámbitos. Nuestro objetivo era evaluar la confiabilidad de la información de andrología de Chat Generative Pre-training Transformer ChatGPT y su idoneidad para informar a pacientes y estudiantes de medicina con precisión sobre temas de andrología. Presentamos una serie de preguntas frecuentes organizadas sistemáticamente sobre temas de andrología y oraciones que contienen recomendaciones sólidas de la Guía de la Asociación Europea de Urología (EAU) para ChatGPT-3.5 y 4.0 como preguntas. Estas preguntas abarcaban hipogonadismo masculino, disfunción eréctil y trastorno del deseo sexual, trastornos de la eyaculación, curvatura del pene y anomalías del tamaño del pene, priapismo e infertilidad masculina. Dos urólogos expertos evaluaron de forma independiente y asignaron puntuaciones que van del 1 al 4 a cada respuesta en función de su precisión, con las siguientes calificaciones: (1) Completamente cierto, (2) Preciso pero insuficiente, (3) Una mezcla de información precisa y engañosa, y (4) Completamente falso. En el estudio se incluyeron un total de 120 preguntas. Entre estas preguntas, el 50.0% recibió una calificación de 1 (completamente correcta) (55.4% para la versión 4.0). La tasa combinada de respuestas correctas (grados 1 y 2) fue del 85.2% para las preguntas frecuentes (88.8% para la versión 4.0) y del 81.5% para las preguntas obtenidas de la guía. La tasa de respuestas completamente incorrectas (grado 4) fue del 1.8%para las preguntas frecuentes (0% para la versión 4.0) y del 5.2% para las preguntas basadas en recomendaciones sólidas. La tasa de respuesta de la versión 4.0 a las preguntas creadas a partir de oraciones que contienen recomendaciones sólidas de la guía EAU fue la misma que la de la versión 3.5. ChatGPT proporcionó respuestas satisfactorias a las preguntas formuladas, aunque algunas respuestas no fueron completas. Puede resultar beneficioso utilizar ChatGPT bajo la guía de un urólogo para mejorar la comprensión de los pacientes sobre sus problemas andrológicos.

ChatGPT; Andrology; Artificial intelligence; Education

Palabras Clave

ChatGPT; Andrología; Inteligencia artificial; Educación

[1] De Rose AF, Gallo F, Bini PM, Gattuccio I, Chiriacò V, Terrone C. Epidemiology of sexual disorders in general medical practice: an italian survey. Urologia. 2019; 86: 79–85.

[2] Eardley I. The incidence, prevalence, and natural history of erectile dysfunction. Sexual Medicine Reviews. 2013; 1: 3–16.

[3] Feldman HA, Goldstein I, Hatzichristou DG, Krane RJ, McKinlay JB. Impotence and its medical and psychosocial correlates: results of the massachusetts male aging study. Journal of Urology. 1994; 151: 54–61.

[4] Althof SE, McMahon CG, Waldinger MD, Serefoglu EC, Shindel AW, Adaikan PG, et al. An update of the international society of sexual medicine’s guidelines for the diagnosis and treatment of premature ejaculation (PE). Sexual Medicine. 2014; 2: 60–90.

[5] Stuntz M, Perlaky A, des Vignes F, Kyriakides T, Glass D. The prevalence of peyronie’s disease in the United States: a population-based study. PLOS ONE. 2016; 11: e0150157.

[6] Rowe PJ, Comhaire FH, Hargreave TB, Mahmoud AMA. WHO manual for the standardized investigation and diagnosis of the infertile couple. 1st edn. Cambridge University Press: Cambridge. 2000.

[7] OpenAI. ChatGPT: optimizing language models for dialogue. 2023. Available at: https://help.openai.com/en/articles/6825453-chatgpt-release-notes (Accessed: 03 August 2023).

[8] Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor RA, et al. How does ChatGPT perform on the United States medical licensing examination (USMLE)? The implications of large language models for medical education and knowledge assessment. JMIR Medical Education. 2023; 9: e45312.

[9] Caglar U, Yildiz O, Meric A, Ayranci A, Gelmis M, Sarilar O, et al. Evaluating the performance of ChatGPT in answering questions related to pediatric urology. Journal of Pediatric Urology. 2024; 20: 26.e1–26.e5.

[10] Yeo YH, Samaan JS, Ng WH, Ting PS, Trivedi H, Vipani A, et al. Assessing the performance of ChatGPT in answering questions regarding cirrhosis and hepatocellular carcinoma. Clinical and Molecular Hepatology. 2023; 29: 721–732.

[11] EAU Guidelines. Edn. presented at the EAU Annual Congress Milan (Period:10 March 2023–13 March 2023). 2023. Available at: https://uroweb.org/guidelines (Accessed: 01 April 2023).

[12] Ergul A. Quality and reliability of YouTube Videos on surgical treatment of Uterine Leiomyomas. Cureus. 2021; 13: e20044.

[13] Alsyouf M, Stokes P, Hur D, Amasyali A, Ruckle H, Hu B. ‘Fake News’ in urology: evaluating the accuracy of articles shared on social media in genitourinary malignancies. BJU International. 2019; 124: 701–706.

[14] Moons P, Van Bulck L. Using ChatGPT and Google Bard to improve the readability of written patient information: a proof of concept. European Journal of Cardiovascular Nursing. 2024; 23: 122–126.

[15] Cakir H, Caglar U, Yildiz O, Meric A, Ayranci A, Ozgor F. Evaluating the performance of ChatGPT in answering questions related to urolithiasis. International Urology and Nephrology. 2024; 56: 17–21.

[16] Abi-Rafeh J, Hanna S, Bassiri-Tehrani B, Kazan R, Nahai F. Complications following facelift and neck lift: implementation and assessment of large language model and artificial intelligence (ChatGPT) performance across 16 simulated patient presentations. Aesthetic Plastic Surgery. 2023; 47: 2407–2414.

[17] Hofmann HL, Guerra GA, Le JL, Wong AM, Hofmann GH, Mayfield CK, et al. The rapid development of artificial intelligence: gpt-4’s performance on orthopedic surgery board questions. Orthopedics. 2024; 47: e85–e89.