Journal Article

DKFZ-2025-02230

http://join2-wiki.gsi.de/foswiki/pub/Main/Artwork/join2_logo100x88.png Readability of Chatbot Responses in Prostate Cancer and Urological Care: Objective Metrics Versus Patient Perceptions.

Maywald, L. ; Nguyen, L. ; Winterstein, J.DKFZ* ; Hetz, M. J.DKFZ* ; Mangold, M. H. ; Renner, L. V. ; Brinker, T. J.DKFZ* ; Wessels, F. ; Carl, N. (Last author)DKFZ*

2025
Multimed Toronto, Ontario

Abstract: Large language models (LLMs) are increasingly explored as chatbots for patient education, including applications in urooncology. Since only 12% of adults have proficient health literacy and most patient information materials exceed recommended reading levels, improving readability is crucial. Although LLMs could potentially increase the readability of medical information, evidence is mixed, underscoring the need to assess chatbot outputs in clinical settings. Therefore, this study evaluates the measured and perceived readability of chatbot responses in speech-based interactions with urological patients. Urological patients engaged in unscripted conversations with a GPT-4-based chatbot. Transcripts were analyzed using three readability indices: Flesch-Reading-Ease (FRE), Lesbarkeitsindex (LIX) and Wiener-Sachtextformel (WSF). Perceived readability was assessed using a survey covering technical language, clarity and explainability. Associations between measured and perceived readability were analyzed. Knowledge retention was not assessed in this study. A total of 231 conversations were evaluated. The most frequently addressed topics were prostate cancer (22.5%), robotic-assisted prostatectomy (19.9%) and follow-up (18.6%). Objectively, responses were classified as difficult to read (FRE 43.1 ± 9.1; LIX 52.8 ± 6.2; WSF 11.2 ± 1.6). In contrast, perceived readability was rated highly for technical language, clarity and explainability (83-90%). Correlation analyses revealed no association between objective and perceived readability. Chatbot responses were objectively written at a difficult reading level, exceeding recommendations for optimized health literacy. Nevertheless, most patients perceived the information as clear and understandable. This discrepancy suggests that perceived comprehensibility is influenced by factors beyond measurable linguistic complexity.

Keyword(s): Humans (MeSH) ; Male (MeSH) ; Prostatic Neoplasms: therapy (MeSH) ; Prostatic Neoplasms: psychology (MeSH) ; Comprehension (MeSH) ; Health Literacy (MeSH) ; Middle Aged (MeSH) ; Aged (MeSH) ; Patient Education as Topic: methods (MeSH) ; Adult (MeSH) ; Perception (MeSH) ; Generative Artificial Intelligence (MeSH) ; GPT-4 ; artificial intelligence ; chatbots ; health literacy ; large language models ; patient education ; readability ; urology ; urooncology

Note: #LA:C140#

---

Contributing Institute(s):

1. Digitale Prävention, Diagnostik und Therapiesteuerung (C140)

Research Program(s):

1. 313 - Krebsrisikofaktoren und Prävention (POF4-313) (POF4-313)

Appears in the scientific report 2025

---

Database coverage:
; ; ; ; Article Processing Charges ; Clarivate Analytics Master Journal List ; DOAJ Seal ; Ebsco Academic Search ; Essential Science Indicators ; Fees ; IF < 5 ; JCR ; SCOPUS ; Science Citation Index Expanded ; Web of Science Core Collection

---