Multimodal artificial intelligence and large language models in gastrointestinal endoscopy: From lesion detection to workflow-integrated report generation

Table 1 Distinction among multimodal artificial intelligence concepts in gastrointestinal endoscopy

Concept	Direct model input	Key distinction	Typical outputs	Key limitations
Multimodal learning	≥ 2 heterogeneous data sources	Broad fusion/alignment paradigm; not a specific architecture	Prediction, cross-modal representation, decision support, structured output	Missing and asynchronous data
VLM	Image/video + text	Learns visual–language representations or alignment	Image-text alignment, retrieval, annotation, visual description	Weak temporal modeling and limited grounding
MLLM	Visual + text + multimodal input	LLM-centered generation conditioned on visual input	Visual question answering, draft report generation	Hallucination and long-video challenges
Text-based LLM	Text	No direct access to endoscopic visual input	Summarization, information extraction, patient communication	Inability to directly interpret images

VLM: Vision-language model; MLLM: Multimodal large language model; LLM: Large language model.

Table 2 Representative clinical and diagnostic-validation studies of artificial intelligence across gastrointestinal endoscopy tasks and workflows

Ref.	Region/setting	Population/sample size	AI function	Study design/comparator	Evaluation unit	Key findings	Validation/main limitation
Wang et al[7], 2019	China; colonoscopy	1058 patients; 522 CADe, 536 control	Real-time CADe	Prospective randomized trial; CADe vs standard colonoscopy	Patient	ADR: 29.1% vs 20.3%; adenomas/patient: 0.53 vs 0.31	Single-center, non-blinded trial; benefit mainly driven by diminutive adenomas
Byrne et al[8], 2019	Colonoscopy video	125 diminutive-polyp videos; 106 analyzed	CADx for adenoma vs hyperplastic polyp	Independent video test set; histopathology reference	Lesion/video	Sensitivity: 98%; specificity: 83%; accuracy: 94%; NPV: 97%	Offline video evaluation; 19/125 lesions lacked sufficient prediction confidence; no live clinical trial
Wu et al[9], 2019	China; EGD	324 randomized; 303 analyzed	WISENSE blind-spot monitoring	RCT; WISENSE-assisted vs conventional EGD	Patient/procedure	Blind spot rate: 5.86% vs 22.46%, P < 0.001	Single-center trial; procedure-quality endpoint
Gong et al[23], 2020	China; colonoscopy	704 randomized; 355 ENDOANGEL, 349 control	Real-time CADe/withdrawal-quality support	Randomized trial; ENDOANGEL vs unassisted colonoscopy	Patient	ADR: 16% vs 8%; odds ratio: 2.3, P = 0.001	Single-center study; relatively low baseline ADR
Wang et al[24], 2020	China; colonoscopy	1010 randomized; 962 analyzed	Real-time CADe	Double-blind randomized trial; CADe vs sham system	Patient	ADR: 34% vs 28%; odds ratio: 1.36, P = 0.03	Single-center; interval-CRC benefit not evaluated
Repici et al[25], 2020	Italy; colonoscopy	685 participants; 341 CADe, 344 control	GI Genius CADe	Multicenter randomized trial; CADe vs standard colonoscopy	Patient/procedure	ADR: 54.8% vs 40.4%; APC: 1.07 vs 0.71	Multicenter clinical evidence; no long-term cancer-outcome assessment
Shaukat et al[26], 2022	United States; screening/surveillance colonoscopy	1359 analyzed; 682 CADe, 677 standard	SKOUT CADe	Multicenter randomized trial; CADe vs standard colonoscopy	Patient/procedure	ADR: 47.8% vs 43.9%, P = 0.065; APC: 1.05 vs 0.83, P = 0.002	Multicenter United States evidence; ADR difference not statistically significant
Spada et al[36], 2024	14 European centers; small-bowel CE	137 enrolled; 133 analyzed	AI-assisted capsule reading	Prospective multicenter paired comparison; AI-assisted vs standard reading	Patient	Diagnostic yield: 73.7% vs 62.4%; reading time: 3.8 minutes vs 33.7 minutes	Strong workflow benefit; downstream management and clinical outcomes not evaluated
Seager et al[27], 2024	United Kingdom; 12 NHS hospitals	2032 randomized	GI Genius CADe	Pragmatic multicenter RCT; CADe vs standard colonoscopy	Patient/procedure	Adenomas/procedure: 1.56 vs 1.21; ADR: 56.6% vs 48.4%	Strong routine-practice evidence; open-label design
Maas et al[28], 2024	Europe/Canada; 7 centers	581 enrolled; 497 analyzed	CADe	International randomized trial; CADe vs conventional colonoscopy	Patient/procedure	ADR: 38.4% vs 37.7%; APC: 0.66 vs 0.66	Experienced endoscopists; not statistically significant
Yao et al[47], 2024	China; screening colonoscopy	393 patients	AI bowel-preparation assessment	Prospective observational validation; AI e-BBPS with tandem colonoscopy outcome assessment	Patient/procedure	> 5-mm AMR: 35.71% in unqualified-AI vs 13.19% in qualified-AI	Single-center observational study; no randomized AI-vs-standard intervention

Outcomes are reported as task-specific findings from the original studies and are not intended for direct comparison across different tasks, study designs, or evaluation metrics. AI: Artificial intelligence; CADe: Computer-aided detection; CADx: Computer-aided diagnosis; EGD: Esophagogastroduodenoscopy; GI: Gastrointestinal; ADR: Adenoma Detection Rate; APC: Adenomas per colonoscopy; CE: Capsule endoscopy; RCT: Randomized controlled trial; AMR: Adenoma miss rate; NPV: Negative predictive value; NHS: National Health Service.

Table 3 Representative primary studies of artificial intelligence-assisted endoscopic documentation and report generation

Ref.	System/input	Setting/sample	Clinical function	Design/comparator	Evaluation unit	Key findings	Validation/main limitation
Qu et al[18], 2020	ISRGS; endoscopic images/video	46987 images; prospective set: 5699 images	Site/lesion recognition and structured report generation	Multicenter development and prospective validation	Image/patient	Overall diagnostic accuracy: 88.41% multicenter; 89.65% prospective	Multicenter data, but structured/template-based output
Dong et al[65], 2022	EAIRS; real-time EGD video	210198 development images; 161 prospective patients	Automated photo-documentation	Internal, external and prospective testing; system vs endoscopists	Patient/procedure	Landmark capture accuracy 95.2%; completeness: 93.1% vs 88.8%; detected-lesion capture: 99% vs 98%	Multicenter development; mainly image documentation rather than full narrative report
Yokota et al[67], 2022	Voice Capture/speech input	356 patients; 101 propensity-matched pairs	Speech-assisted structured reporting	Retrospective propensity-matched comparison; voice recognition vs manual entry	Procedure	Median report preparation time: 311 s vs 383 s, P = 0.009	Single-center retrospective study
Zhang et al[19], 2023	AI-EARS; EGD video/image	252111 images for training, 62706 images, and 950 videos for testing	Automatic image capture, diagnosis and textual description	Multireader multicase randomized crossover; AI-EARS vs conventional reporting	Report/procedure	Text completeness: 79.03% vs 51.86%; accuracy: 64.47% vs 42.81%; reporting time: 46.47 s vs 80.13 s	Relatively small clinical crossover sample
Naito et al[20], 2025	AR-C1; video/procedural events	40 endoscopic mucosal resection procedures	Procedure-event capture and report support	Retrospective clinical comparison	Procedure/report	Report generation time: 186.6 s vs 258.4 s	Single-center and single examiner
You et al[66], 2026	AI gastric-lesion reporting system	276 video clips; prospective clinical analysis: 100 patients	Structured lesion documentation	Prospective AI-assisted vs routine reporting	Lesion/report	Overall lesion-report completeness: 86.5% vs 75.11%, P = 0.001	Prospective but single-center
Kawagoe et al[68], 2026	EVAS voice recognition	21 cases; 5 experienced endoscopists	Voice-assisted colonoscopy reporting	Pilot comparison; EVAS vs manual typing	Report	Mean reporting time: 29 s vs 56 s; 46.3% reduction	Very small pilot; recognition accuracy/system integration require further evaluation
Jiang et al[21], 2026	Report-Angel; endoscopic images/videos + text	20617 training image-text pairs; prospective internal n = 300 and external n = 300 cohorts	MLLM-generated draft EGD reports	Prospective internal and external clinical validation; expert review of generated drafts	Case/report + lesion	Clinically acceptable draft reports: 79.3% internal, 83.3% external; case completeness 88.51%, accuracy 78.93%	Strongest current prospective MLLM evidence; generated drafts still require clinician review
Massimi et al[71], 2026	Gemini 2.5 Pro; complete EGD video	5 full-length EGD videos	MLLM video analysis and report generation	Retrospective paired pilot; clean video vs same video with CAD overlay	Report/event	Adequate completeness: 56% vs 48%; landmark accuracy: 0.55 vs 0.33	Extremely small pilot; authors concluded performance was inadequate for clinical EGD reporting

Outcomes are reported as task-specific findings from the original studies and are not intended for direct comparison across different tasks, study designs, or evaluation metrics. AI: Artificial intelligence; EGD: Esophagogastroduodenoscopy; LLM: Large language model; MLLM: Multimodal large language model; CAD: Computer‐aided detection/diagnosis.

Table 4 Proposed clinician-in-the-loop evaluation framework for artificial intelligence-generated endoscopy reports

Dimension	Clinician assessment	Scale	Suggested outcome
Factual grounding	Verify findings against source evidence	Item-level correct/incorrect	Accuracy; hallucination rate
Completeness	Check required report elements	Present/absent	Completeness; omission rate
Clinical safety	Grade consequences of errors	Minor/moderate/major	Major-error rate
Overall acceptability	Determine need for substantive correction	Yes/no	Clinically acceptable report rate
Workflow utility	Assess usability and editing burden	Likert scale + time	Utility score; correction time

For clinician-rated endpoints, evaluation by multiple blinded reviewers and reporting of inter-rater agreement (e.g., Fleiss’ κ or intraclass correlation coefficient) are recommended. This framework represents an author-proposed synthesis of current evaluation practices and is not a validated clinical guideline.