Do bots provide correct and adequate guidance regarding acidity: A blinded comparison rated by patients and physicians

Table 4 Physician assessment of large language model responses to patient education questions on acidity

Characteristics	ChatGPT-5a	Gemini-2.5a	Claude-4a	P value
				A vs B	A vs C	B vs C
Overall
Comprehensiveness	2.86 ± 0.49	3.36 ± 0.52	3.43 ± 0.58	0.0001	0.0001	0.5763
Accuracy	2.89 ± 0.53	3.30 ± 0.51	3.34 ± 0.56	0.0008	0.0005	0.7425
Actionability	2.73 ± 0.62	3.67 ± 0.74	3.45 ± 0.53	0.0001	0.0001	0.1353
Empathy	2.64 ± 0.59	3.33 ± 0.57	3.38 ± 0.55	0.0001	0.0001	0.6945
Domain wise
Domain 1: General understanding
Comprehensiveness	2.55 ± 0.51	3.22 ± 0.43	3.33 ± 0.48	0.0001	0.0001	0.2898
Accuracy	2.66 ± 0.48	3.16 ± 0.51	3.43 ± 0.67	0.0001	0.0001	0.0488
Actionability	2.55 ± 0.51	3.39 ± 0.61	3.44 ± 0.51	0.0001	0.0001	0.6956
Empathy	2.61 ± 0.50	3.11 ± 0.67	3.33 ± 0.48	0.0004	0.0001	0.0996
Domain 2: Symptoms and diagnosis
Comprehensiveness	2.71 ± 0.46	3.43 ± 0.51	3.43 ± 0.67	0.0001	0.0001	1.0000
Accuracy	2.81 ± 0.51	3.28 ± 0.64	3.23 ± 0.62	0.0006	0.0016	0.7270
Actionability	2.76 ± 0.70	3.90 ± 0.70	3.47 ± 0.51	0.0001	0.0001	0.0027
Empathy	2.90 ± 0.53	3.42 ± 0.59	3.33 ± 0.57	0.0001	0.0009	0.4954
Domain 3: Causes and risk factors
Comprehensiveness	2.84 ± 0.42	3.56 ± 0.49	3.61 ± 0.50	0.0001	0.0001	0.6568
Accuracy	2.94 ± 0.23	3.44 ± 0.51	3.44 ± 0.51	0.0001	0.0001	1.0000
Actionability	2.83 ± 0.70	3.83 ± 0.98	3.50 ± 0.51	0.0001	0.0001	0.0660
Empathy	2.56 ± 0.61	3.33 ± 0.48	3.50 ± 0.51	0.0001	0.0001	0.1337
Domain 4: Dietary considerations
Comprehensiveness	2.83 ± 0.39	3.42 ± 0.51	3.50 ± 0.52	0.0001	0.0001	0.4948
Accuracy	2.75 ± 0.45	3.33 ± 0.49	3.41 ± 0.51	0.0001	0.0001	0.4821
Actionability	2.67 ± 0.49	3.67 ± 0.78	3.50 ± 0.53	0.0001	0.0001	0.2638
Empathy	2.50 ± 0.52	3.42 ± 0.51	3.47 ± 0.47	0.0001	0.0001	0.6538
Domain 5: Treatment and management
Comprehensiveness	2.90 ± 0.53	3.42 ± 0.47	3.52 ± 0.51	0.0001	0.0001	0.3707
Accuracy	3.05 ± 0.50	3.52 ± 0.51	3.42 ± 0.51	0.0001	0.0018	0.3893
Actionability	2.89 ± 0.48	3.67 ± 0.65	3.47 ± 0.51	0.0001	0.0001	0.1347
Empathy	2.71 ± 0.56	3.47 ± 0.52	3.47 ± 0.51	0.0001	0.0001	1.0000
Domain 6: Lifestyle factors
Comprehensiveness	3.11 ± 0.51	3.26 ± 0.59	3.29 ± 0.67	0.2334	0.1859	0.8343
Accuracy	3.03 ± 0.71	3.11 ± 0.69	3.22 ± 0.64	0.6153	0.2183	0.0001
Actionability	2.67 ± 0.68	3.59 ± 0.69	3.37 ± 0.63	0.0001	0.0001	0.1456
Empathy	2.52 ± 0.70	3.26 ± 0.59	3.26 ± 0.65	0.0001	0.0001	1.0000

^aP < 0.05.

P values indicate pairwise comparisons (ChatGPT-5 vs Gemini-2.5, ChatGPT-5 vs Claude-4a, Gemini-2.5 vs Claude-4a). Scores represent mean ± SD across three blinded gastroenterologists. Responses were evaluated for comprehensiveness, accuracy, actionability, and empathy using predefined ordinal scales (1-4 for comprehensiveness, accuracy, and empathy; 1-5 for actionability). Comparisons were made between ChatGPT-5, Gemini-2.5, and Claude-4 overall and across six thematic domains (general understanding, symptoms and diagnosis, causes and risk factors, dietary considerations, treatment and management, and lifestyle factors). A: ChatGPT-5; B: Gemini-2.5; C: Claude-4a.