Evaluating Large Language Model Compliance With AUA Guidelines on Muscle-Invasive Bladder Cancer in Addressing Female Sexual Dysfunction Following Radical Cystectomy

Document Type

Conference Proceeding

Publication Date

12-9-2025

Publication Title

J Sex Med

Keywords

Urology & Nephrology

Abstract

Introduction: Compared to men, women present with more advanced bladder cancer and have poorer outcomes than men following radical cystectomy. Radical cystectomy often leads to sexual dysfunction in women, yet many women feel unprepared for these changes in sexual health due to the limited pre- and postoperative counseling. Despite the impact of these issues, female sexual dysfunction (FSD) remains understudied. Objective: The use of artificial intelligence (AI) and Large Language Models (LLMs) has become increasingly common in clinical practice. While previous research suggests that AI has been relatively accurate in analyzing psychological conditions and offers average accuracy in addressing Female and Male sexual dysfunction, its reliability in generating information regarding FSD remains poorly understood. To address this gap, our study aims to evaluate the quality of AI-generated FSD content. Methods: Five commonly asked patient questions were presented to three LLMs (ChatGPT-4, Claude, and Gemini) in separate, independent queries for each question. The com monly asked patient questions were selected using peer-reviewed literature addressing FSD post-cystectomy and common concerns of patients presented on validated health websites and professional organizations. A custom guideline adherence rubric was developed based on AUA guidelines for the treatment of muscle-invasive bladder cancer. Each AI response was evaluated by independent reviewers for the inclusion (yes/no) of each of the following five components: (1) counseling on complications and quality of life implications (e.g., sexual function, fertility); (2) discussion of potential sexual function-preserving surgical options (e.g., nerve-sparing procedures); (3) inclusion of supportive care and mental health counseling; (4) lifestyle modifications to support long-term health (e.g., smoking cessation, physical activity); and (5) discussion of potential bladder-preserving therapies when clinically appropriate. An adherence score (0-100%) was calculated for each output. Future mean adherence scores for each LLM will be compared via one-way ANOVA for statistical significance. Inter-rater reliability will be determined using Cohen’s kappa statistic to assess agreement between the two independent reviewers for the binary inclusion (yes/no) of each guideline component. Results: We had 5 ChatGPT-4 responses evaluated on a list of common patient questions addressing FSD following a radical cystectomy. Each ChatGPT-4 response was rated on the use of five components of the AUA guidelines using a checklist developed by the researchers for this purpose. The five responses had a mean adherence score of 58% and consistently followed guidelines components 1(100%), 2(80%), and 3 (100%), but did not follow guidelines components 4 and 5. Conclusions: ChatGPT-4 demonstrated moderate compliance (58%) with AUA guidelines when addressing patient questions regarding female sexual dysfunction following radical cystectomy, routinely addressing the anatomical and psychosocial ramifications of surgery (Guideline Component Inclusions 1,2,3). Still, omitting guideline compliance components relating to lifestyle change or discussing potential bladder-preserving methods (Guideline Component Inclusions 4,5) highlights the need for refinement of AI-generated content to ensure alignment with guidelines for muscle-invasive bladder cancer. Enhancing AI outputs may help bridge gaps in patient education and support more informed, holistic care for women navigating treatment options for muscle-invasive bladder cancer, including those at risk for or considering cystectomy.

Volume

22

First Page

iv203

Last Page

iv204

Find It @ Sladen

Share

COinS