Multimodal artificial intelligence and large language models in gastrointestinal endoscopy: From lesion detection to workflow-integrated report generation

Abstract

Artificial intelligence (AI) has rapidly transformed gastrointestinal (GI) endoscopy, with most early applications focusing on computer-aided detection and diagnosis of lesions from endoscopic images. However, real-world endoscopic practice extends beyond image interpretation. Endoscopists must integrate real-time video, anatomical location, lesion morphology, procedural context, histopathological findings, clinical history, and standardized terminology to make diagnostic decisions and generate clinically meaningful reports. Recent advances in multimodal AI and large language models (LLMs) provide an opportunity to move beyond isolated endoscopic image analysis toward more comprehensive endoscopic workflow support. By integrating visual, textual, temporal, and clinical information, these models can support lesion characterization, anatomical navigation, procedure quality assessment, structured documentation, report generation, and patient-centered communication. Nevertheless, clinical translation remains limited by scarce high-quality multimodal datasets, heterogeneous reporting standards, insufficient external validation, hallucination risk, limited interpretability, real-time integration challenges, privacy concerns, and evolving regulation. This review summarizes current applications of AI in GI endoscopy, with particular emphasis on the emerging role of multimodal learning and LLMs in bridging lesion detection, context-aware interpretation, and automated report generation. Furthermore, the article analyzes future challenges and the methodological, clinical, ethical, and regulatory requirements for the reliable integration of these systems into clinical endoscopic practice.

Key Words: Gastrointestinal endoscopy; Artificial intelligence; Multimodal learning; Large language models; Automated report generation; Computer-aided diagnosis

Core Tip: Artificial intelligence in gastrointestinal endoscopy is evolving from single lesion identification and diagnosis to multimodal, workflow-oriented support. The review explores methods to integrate endoscopic images and videos with clinical text, histology, and procedural data using multimodal learning, vision-language models, multimodal large language models (LLMs), and complementary text-based LLMs. The maturity of the clinical evidence varies greatly; however, the current evidence supports applications in lesion classification, procedural quality assessment, systematic documentation and draft report generation. Limited multimodal datasets, hallucination and grounding issues, lack of external validation, workflow integration, privacy and regulatory restrictions are still key challenges for mainstream clinical application.

INTRODUCTION

Gastrointestinal (GI) endoscopy plays a pivotal role in the detection, diagnosis, monitoring, and treatment of various digestive system disorders[1]. Unlike static medical imaging, endoscopy is a dynamic and highly operator-dependent process; its diagnostic efficacy depends not only on the identification of abnormal lesions but is also influenced by multiple factors, including mucosal visualization, anatomical coverage, image quality, withdrawal or observation time, lesion characterization, procedural technique, and the completeness of documentation. Consequently, real-time endoscopy reports should integrate observations from continuous video streams with indications for the procedure, anatomical locations, lesion morphology, interventions performed, specimen collection, preliminary diagnostic impressions, and initial management plans. Upon receipt of histopathological results, a supplementary pathology report can further refine the diagnosis and update the surveillance and follow-up recommendations in the final report. Thus, high-quality endoscopy relies not only on lesion identification but also on the consistency of the entire diagnostic and reporting workflow.

Artificial intelligence (AI), particularly deep learning-based computer vision technology, has made significant strides in the field of GI endoscopy. Computer-aided detection (CADe) systems can identify suspicious lesions during endoscopic procedures, while computer-aided diagnosis (CADx) systems assist in lesion characterization and optical diagnosis. These technologies were first widely investigated in colonoscopy, especially regarding the detection and classification of colorectal polyps. Their applications subsequently expanded to other areas of GI endoscopy, including detection of upper GI neoplasms and Barrett’s esophagus, assessment of inflammatory activity and active bleeding, bowel-preparation quality evaluation, anatomical landmark recognition, and capsule endoscopy (CE)[2-6]. In fact, there is evidence from randomized trials and meta-analyses suggesting that certain kinds of CADe systems may raise adenoma detection rates (ADR), while real-time systems for upper GI endoscopy demonstrate the potential to reduce blind spots and standardize procedural quality[7-10]. Therefore, some professional organizations see AI as holding great promise because it might make endoscopists more diagnostically and therapeutically powerful and thus improve quality measures in general[3,11,12].

Despite these advances, most systems currently under evaluation remain image-centric. While traditional image analysis models can perform specific tasks, such as polyp detection or the classification of gastric lesions, their outputs are often disconnected from the broader context of the endoscopic procedure. A single image offers only a localized view of the procedure, failing to capture the macro-level clinical context or reliably reflect the completeness of the examination, the morphological characteristics of lesions across different perspectives, or the interventions performed. Furthermore, such images provide limited information regarding the relationship between endoscopic findings and the patient’s symptoms, medical history, pathological results, and prior examinations. In addition, key outputs such as bounding boxes, class probabilities, and quality scores do not, in themselves, constitute a clinically meaningful endoscopy report. This disconnect between algorithmic performance and clinical workflow limits the application and integration of AI into routine endoscopic practice.

In endoscopy, a high-quality report is crucial for the early detection, diagnosis, and treatment of disease. The report serves as the primary vehicle for documenting examination findings, interventions, specimen collection, diagnostic impressions, and follow-up recommendations. However, report completeness and terminology vary among endoscopists and across institutions[13]. A recent quality improvement initiative emphasized that upper GI endoscopy reports should include the preprocedural context, periprocedural findings, anatomical landmarks, pathological sampling, photographic evidence, and postprocedural recommendations[14]. Given the large volume of data generated during endoscopic examinations, manual documentation of all required report elements is time-consuming and susceptible to omissions[15,16]. Therefore, an AI system capable of connecting visual interpretation with procedural context and structured documentation may provide greater clinical value than an isolated detection algorithm.

Multimodal learning, vision-language models (VLMs), multimodal large language models (MLLMs), and text-based large language models (LLMs) offer complementary directions to this goal. However, the nomenclature in this quickly changing sector is not standardized and should be used with caution. Thus, explicit definitions are presented in this review (Table 1). Multimodal learning is a general term for computational methods that fuse or align two or more heterogeneous input sources. At the model level, VLMs aim to learn the relationships between visual and textual data. MLLMs, in contrast, integrate non-textual inputs into generative architectures centered on LLMs, enabling the generation of language-based responses. By comparison, a text-based LLM operates on textual input alone and is not considered multimodal merely because that text describes findings originally observed in endoscopic images.

Table 1 Distinction among multimodal artificial intelligence concepts in gastrointestinal endoscopy.

Concept	Direct model input	Key distinction	Typical outputs	Key limitations
Multimodal learning	≥ 2 heterogeneous data sources	Broad fusion/alignment paradigm; not a specific architecture	Prediction, cross-modal representation, decision support, structured output	Missing and asynchronous data
VLM	Image/video + text	Learns visual–language representations or alignment	Image-text alignment, retrieval, annotation, visual description	Weak temporal modeling and limited grounding
MLLM	Visual + text + multimodal input	LLM-centered generation conditioned on visual input	Visual question answering, draft report generation	Hallucination and long-video challenges
Text-based LLM	Text	No direct access to endoscopic visual input	Summarization, information extraction, patient communication	Inability to directly interpret images

VLM: Vision-language model; MLLM: Multimodal large language model; LLM: Large language model.

These distinctions reflect different roles within the endoscopic workflow. For example, large-scale image-text colonoscopy records have been used to extract training knowledge and generate annotated datasets, illustrating how routinely collected clinical records can support more data-efficient model development[17]. Earlier automatic reporting systems relied on image-recognition modules with predefined templates, whereas later systems incorporated real-time image capture, lesion recognition, and textual description[18-20]. More recently, domain-specific MLLMs have been prospectively evaluated for generating draft upper GI endoscopy reports from procedural videos, providing early evidence that visual recognition and language generation can be integrated within a clinically oriented reporting workflow[21]. These developments indicate a transition from narrowly defined computer-vision tasks toward workflow-oriented endoscopic AI. However, multimodal and generative systems introduce additional risks. These include content-related limitations, such as hallucinated findings, incomplete temporal understanding, and weak grounding between visual evidence and generated text. They also extend to implementation and governance concerns, including variability across devices and institutions, limited external validation, automation bias, privacy concerns, and uncertain accountability. Furthermore, high performance on retrospective image datasets does not necessarily ensure safe or useful performance during real-time endoscopic practice. For this reason, real-world implementation requires evaluation beyond conventional algorithmic metrics. In particular, systems should be assessed for report accuracy and completeness, workflow efficiency, human-AI interaction, patient outcomes, and failure modes.

The progress of multimodal AI in the discipline of gastroenterology has been reviewed extensively. Specifically, Wu et al[22] investigated multimodal AI applications for precision diagnosis and treatment of GI and liver illnesses, including endoscopic imaging, pathological analysis, multi-omics integration, and other heterogeneous clinical data. That review described the application of multimodal AI in digestive medicine from a wide disease-oriented perspective. This article, however, reviews a more specific topic: The evolution of AI in GI endoscopy from performing isolated visual tasks to a technological system that understands the entire procedural workflow, integrates multimodal information, and generates clinically usable documentation and reports. This paper reviews the lesion-level CADe and CADx technologies that form the clinical basis of endoscopic AI, focusing on the shift from lesion identification to the creation of clinically integrated reports. This study will also discuss the present evidence for AI-assisted documentation and report generation, as well as the methodological, clinical, ethical, and regulatory prerequisites for safe deployment. The suggested level-based framework is summarized in Figure 1.

Figure 1 Levels of clinical integration of artificial intelligence in gastrointestinal endoscopy. A: Perception at the lesion level; B: Procedure-level understanding; C: Patient-level multimodal integration; D: Documentation and communication support. These levels are intended to represent the increasing clinical integration rather than a strict chronological sequence. This is an author-proposed conceptual framework intended for illustration and does not represent a validated clinical pathway or clinical guideline. AI: Artificial intelligence; CADe: Computer-aided detection; CADx: Computer-aided diagnosis; LLM: Large language model; BBPS: Boston Bowel Preparation Scale; EHR: Electronic health record; Hb: Hemoglobin; CRP: C-reactive protein; Alb: Albumin; ALT: Alanine aminotransferase.

This narrative mini-review employed a structured literature search strategy covering English-language publications from PubMed and Web of Science databases, spanning the period from January 2019 to June 30, 2026. The final database search and verification were conducted on September 4, 2026; however, to maintain the predefined review period, only publications with a verifiable public version available on or before June 30, 2026, were eligible for inclusion. Publications were eligible if a verifiable version of record, conference proceeding, or preprint, as applicable, was publicly available on or before June 30, 2026. Publications or records first made publicly available after this date were not eligible. Google Scholar was also used to supplement citation tracking.

Task-oriented searches were conducted separately across four domains: (1) CADe and diagnosis for clinical assessment; (2) Procedural quality assessment, anatomical navigation, and workflow support; (3) Multimodal learning, VLM, MLLM, and related language model applications; and (4) Automated documentation and endoscopic report generation. Search terms included combinations of keywords such as “gastrointestinal endoscopy”, “colonoscopy”, “gastroscopy”, “esophagogastroduodenoscopy”, “capsule endoscopy”, “artificial intelligence”, “deep learning”, “computer-aided detection”, “computer-aided diagnosis”, “quality assessment”, “blind spots”, “anatomical landmarks”, “bowel preparation”, “multimodal learning”, “vision-language models”, “large language models”, “automated reporting”, “clinical documentation”, and “report generation”.

Literature meeting the inclusion criteria comprised randomized controlled trials, prospective and retrospective clinical studies, diagnostic accuracy and validation studies, methodologically significant studies, systematic reviews and meta-analyses, and relevant professional guidelines or consensus statements. Studies focusing exclusively on radiology, histopathology, surgery, or robotic navigation-without a direct link to endoscopic workflow-were excluded. Additionally, studies not published in English or those lacking sufficient methodological or outcome information for verification were excluded. Conference papers and preprints were considered only if they addressed rapidly evolving multimodal or generative applications.

Four PubMed searches found 477 records, while similar searches in the Web of Science Core Collection found 559 records, for a total of 1036 database records before deduplication. Records were then deduplicated based on bibliographic identifiers and citation information (i.e., DOIs and matching titles, publication years, and first authors). Following the removal of 395 duplicate items, 641 unique records were screened by title and abstract. 153 studies directly relevant to the predefined scope were retained for full-text assessment. Two authors participated in the screening of titles and abstracts and the assessment of full-text articles. Disagreements regarding study eligibility were resolved through discussion, and a total of 68 studies were ultimately included in the narrative synthesis. As this study is a narrative mini-review rather than a systematic review or meta-analysis, formal assessments of risk of bias or certainty of evidence were not performed. The risks of selection bias and citation bias inherent in the narrative review design are acknowledged as limitations of this study.

FROM LESION DETECTION TO WORKFLOW-ORIENTED ENDOSCOPIC AI

The clinical value of endoscopic AI can be understood by considering the various ways it can support the entire procedure. At a fundamental level, these tools sharpen a clinician’s perception by flagging, classifying, or outlining visible abnormalities. Indeed, when AI penetrates further into practice, it interprets entire procedure videos; checks whether certain important parts of anatomy were sufficiently examined; keeps track of some measure of quality; brings other kinds of patient information into account; produces structured documents on its own; and so on. This section therefore reviews endoscopic AI according to its level of clinical integration.

AI-assisted lesion detection and characterization

Computer-aided lesion detection and characterization represent the earliest and most extensively investigated AI applications in GI endoscopy. In this context, AI systems are commonly divided into CADe and CADx. Together, these approaches established the first clinically visible role of AI in endoscopy through lesion-level assistance to endoscopists.

Colonoscopy has provided the strongest evidence base for AI-assisted lesion detection, and the ADR is a widely accepted quality indicator. Given the central role of premalignant lesion detection and removal in colorectal cancer prevention, colorectal polyp detection became a natural starting point for real-time endoscopic AI. Early randomized trials from China indicate that, compared with standard colonoscopy, a deep learning-based CADe system increases the ADR and the number of adenomas detected per examination, with benefits primarily observed for small or diminutive lesions[7,23]. The subsequent double-blind randomized CADe-DB trial reported an ADR of 34% in the CADe group compared with 28% in the sham-control group[24]. A multicenter randomized trial conducted by Repici et al[25] in Italy, involving 685 subjects, demonstrated that the ADR in the CADe group (54.8%) was significantly higher than that in the control group (40.4%).

Subsequent trials have further enriched the relevant evidence within healthcare systems across different regions. For example, Shaukat et al[26] conducted a multicenter randomized trial in the United States to prospectively evaluate the use of CADe during colonoscopy. The results showed that the use of CADe devices increased the adenomas per colonoscopy (APC) from 0.83 to 1.05, without a concomitant increase in the rate of resection of non-neoplastic lesions. More recently, Seager et al[27] conducted the COLO-DETECT trial across 12 National Health Service (NHS) hospitals in England. CADe-assisted colonoscopy demonstrated superior performance compared to standard colonoscopy in terms of both the mean number of adenomas detected per procedure (1.56 vs 1.21) and the ADR (56.6% vs 48.4%). However, the use of CADe has not yielded ideal results in all clinical settings. Maas et al[28] conducted a randomized trial across seven centers in Europe and Canada and found no significant improvement in ADR (38.4% vs 37.7%) or APC (0.66 vs 0.66) among procedures performed by experienced endoscopists. At the same time, systematic reviews and meta-analyses generally support the finding that AI-assisted colonoscopy improves the detection rate of adenomas or polyps, while also highlighting the heterogeneity across different trials and clinical settings[29-31]. Overall, colorectal CADe represents one of the most mature areas of clinical application for AI in endoscopy.

CADx systems extend lesion-level AI from detection to optical characterization. Existing studies have reported promising diagnostic performance for distinguishing diminutive adenomas from hyperplastic polyps, supporting potential “diagnose-and-leave” or “resect-and-discard” strategies[8,32,33]. However, clinical implementation of CADx requires greater diagnostic certainty than simple lesion detection because misclassification can directly affect decisions regarding resection and subsequent management. Reliable use therefore depends on stable image quality, standardized acquisition, validated decision thresholds, and prospective evidence demonstrating performance under routine clinical conditions.

AI-based lesion detection and characterization have also been investigated in upper GI endoscopy, including applications to early esophageal and gastric neoplasms, Barrett’s esophagus-associated dysplasia, gastric precancerous lesions, inflammatory lesions, and bleeding[34,35]. These applications are clinically relevant because subtle abnormalities in color, surface morphology, vascular patterns, and lesion margins can be difficult to recognize during routine examinations. However, the evidence base remains less mature than that for colorectal CADe. Many studies are retrospective, image-based, or single-center and rely on selected high-quality lesion images, which limits generalizability to routine upper GI endoscopy where lesion prevalence, preparation quality, mucus, bubbles, peristalsis, lighting, and equipment vary substantially.

CE represents another important application for lesion-level AI, since conventional interpretation requires review of thousands of images and imposes a substantial reading burden. Given that this burden exists, AI models for CE may support identification of a broad range of small-bowel findings, including bleeding, vascular lesions, ulcers, erosions, Crohn’s disease-related lesions, tumors, and other abnormalities[36,37]. Moreover, by highlighting relevant frames within lengthy CE recordings, these systems could reduce reading time and might improve sensitivity for human review. Meanwhile, the evidence appears to demonstrate that segmentation has emerged as an important complementary task, particularly for colorectal polyps. Delineation supports size estimation and treatment planning[38]. This task depends on annotation quality, and lesion visibility and boundary information remain key considerations.

Overall, lesion detection and characterization have established the foundation of AI-assisted GI endoscopy. These systems can improve visual attention, standardize selected diagnostic tasks, and provide real-time support for endoscopists. Still, almost every model concerned with individual abnormalities produces relatively circumscribed predictions consisting largely of bounding boxes plus class names, confidence scores, segmentation maps, etc. They do not address procedure-wide aspects such as exam adequacy, region exploration, change over time, entry into formal records, advice about management, and so forth. This limitation shows the need to move from lesion-centered AI toward workflow-oriented endoscopic AI.

Anatomical navigation and procedural quality assessment

AI applications have expanded beyond lesion-level analysis to support assessment of the endoscopic procedure as a whole[39]. Procedural quality depends on factors such as mucosal exposure, anatomical coverage, inspection time, bowel preparation, blind spots, and documentation completeness. Workflow-oriented systems therefore monitor how the examination is performed, rather than only whether an abnormality is present.

Anatomical landmark recognition is an important component of procedural support. Several AI systems have been developed to identify anatomical landmarks in GI images[40-42]. For example, Cogan et al[40] proposed an automated preprocessing module for image enhancement to maximize the accuracy of anatomical landmark recognition in small medical image datasets. Yu et al[41] combined graph convolutional networks and long short-term memory networks to extract label features while associating temporal dependencies, thereby achieving real-time and accurate identification of anatomical locations in gastroscopic videos. AI-based landmark recognition can assist anatomical orientation, automatically label image regions, and support structured report generation[43,44]. Importantly, it can also enable AI systems to assess whether all expected regions have been adequately examined.

Blind-spot monitoring provides a representative example of workflow-oriented AI. Instead of detecting one specific lesion, these systems monitor whether any anatomical regions have not been inspected during the examination. In upper GI endoscopy, real-time quality-improving systems have been developed to identify blind spots, monitor inspection time, and support photo documentation[9,45]. Such systems are clinically meaningful because incomplete visualization of high-risk areas may contribute to missed lesions. AI, by alerting endoscopists to unexamined regions, serves not only as a tool for lesion detection but also as a safety safeguard during the procedure.

AI has also been applied to objective assessment of colonoscopy quality, with automated systems developed to evaluate bowel preparation, withdrawal quality, mucosal exposure, image quality, and inspection completeness[11,46]. For example, Yao et al. validated a quantitative, AI-based bowel preparation assessment system, and the findings suggest that this work provides important evidence for AI’s role in guiding bowel re-cleaning and standardizing future colonoscopy screening[47]. Cold et al[48] likewise reported that AI-assisted bowel-preparation assessment can reduce variability associated with subjective human scoring.

Video AI adds a temporal dimension that static images alone cannot supply and enables a number of tasks including anatomical navigation and procedure quality assessment. This direction constitutes an important step toward workflow-level endoscopic AI. Lesion visibility, anatomical coverage, instrument use, and mucosal visualization all change over the course of an examination. For example, the lesion prior to resection and the wound bed after resection are different clinical states. If this time context is not provided, the report may inadvertently imply that the lesion is still present or fail to detail the intervention conducted. Thus, the analysis of the temporal dimension is fundamental to the interpretation of the procedure. Temporal dependencies for anatomical localization and procedural-event detection have been included in existing endoscopic techniques[20,41]. In this regard, video-based procedural AI closes the gap between early image-based CADe or CADx and more advanced multi-modal workflow solutions.

Table 2 presents the application of AI in GI endoscopy workflows, outlining key areas of application, study designs, and significant results for specific tasks. While the table facilitates a broader interpretation of the relevant evidence, it does not imply that model performance across different studies can be directly compared.

Table 2 Representative clinical and diagnostic-validation studies of artificial intelligence across gastrointestinal endoscopy tasks and workflows.

Ref.	Region/setting	Population/sample size	AI function	Study design/comparator	Evaluation unit	Key findings	Validation/main limitation
Wang et al[7], 2019	China; colonoscopy	1058 patients; 522 CADe, 536 control	Real-time CADe	Prospective randomized trial; CADe vs standard colonoscopy	Patient	ADR: 29.1% vs 20.3%; adenomas/patient: 0.53 vs 0.31	Single-center, non-blinded trial; benefit mainly driven by diminutive adenomas
Byrne et al[8], 2019	Colonoscopy video	125 diminutive-polyp videos; 106 analyzed	CADx for adenoma vs hyperplastic polyp	Independent video test set; histopathology reference	Lesion/video	Sensitivity: 98%; specificity: 83%; accuracy: 94%; NPV: 97%	Offline video evaluation; 19/125 lesions lacked sufficient prediction confidence; no live clinical trial
Wu et al[9], 2019	China; EGD	324 randomized; 303 analyzed	WISENSE blind-spot monitoring	RCT; WISENSE-assisted vs conventional EGD	Patient/procedure	Blind spot rate: 5.86% vs 22.46%, P < 0.001	Single-center trial; procedure-quality endpoint
Gong et al[23], 2020	China; colonoscopy	704 randomized; 355 ENDOANGEL, 349 control	Real-time CADe/withdrawal-quality support	Randomized trial; ENDOANGEL vs unassisted colonoscopy	Patient	ADR: 16% vs 8%; odds ratio: 2.3, P = 0.001	Single-center study; relatively low baseline ADR
Wang et al[24], 2020	China; colonoscopy	1010 randomized; 962 analyzed	Real-time CADe	Double-blind randomized trial; CADe vs sham system	Patient	ADR: 34% vs 28%; odds ratio: 1.36, P = 0.03	Single-center; interval-CRC benefit not evaluated
Repici et al[25], 2020	Italy; colonoscopy	685 participants; 341 CADe, 344 control	GI Genius CADe	Multicenter randomized trial; CADe vs standard colonoscopy	Patient/procedure	ADR: 54.8% vs 40.4%; APC: 1.07 vs 0.71	Multicenter clinical evidence; no long-term cancer-outcome assessment
Shaukat et al[26], 2022	United States; screening/surveillance colonoscopy	1359 analyzed; 682 CADe, 677 standard	SKOUT CADe	Multicenter randomized trial; CADe vs standard colonoscopy	Patient/procedure	ADR: 47.8% vs 43.9%, P = 0.065; APC: 1.05 vs 0.83, P = 0.002	Multicenter United States evidence; ADR difference not statistically significant
Spada et al[36], 2024	14 European centers; small-bowel CE	137 enrolled; 133 analyzed	AI-assisted capsule reading	Prospective multicenter paired comparison; AI-assisted vs standard reading	Patient	Diagnostic yield: 73.7% vs 62.4%; reading time: 3.8 minutes vs 33.7 minutes	Strong workflow benefit; downstream management and clinical outcomes not evaluated
Seager et al[27], 2024	United Kingdom; 12 NHS hospitals	2032 randomized	GI Genius CADe	Pragmatic multicenter RCT; CADe vs standard colonoscopy	Patient/procedure	Adenomas/procedure: 1.56 vs 1.21; ADR: 56.6% vs 48.4%	Strong routine-practice evidence; open-label design
Maas et al[28], 2024	Europe/Canada; 7 centers	581 enrolled; 497 analyzed	CADe	International randomized trial; CADe vs conventional colonoscopy	Patient/procedure	ADR: 38.4% vs 37.7%; APC: 0.66 vs 0.66	Experienced endoscopists; not statistically significant
Yao et al[47], 2024	China; screening colonoscopy	393 patients	AI bowel-preparation assessment	Prospective observational validation; AI e-BBPS with tandem colonoscopy outcome assessment	Patient/procedure	> 5-mm AMR: 35.71% in unqualified-AI vs 13.19% in qualified-AI	Single-center observational study; no randomized AI-vs-standard intervention

Outcomes are reported as task-specific findings from the original studies and are not intended for direct comparison across different tasks, study designs, or evaluation metrics. AI: Artificial intelligence; CADe: Computer-aided detection; CADx: Computer-aided diagnosis; EGD: Esophagogastroduodenoscopy; GI: Gastrointestinal; ADR: Adenoma Detection Rate; APC: Adenomas per colonoscopy; CE: Capsule endoscopy; RCT: Randomized controlled trial; AMR: Adenoma miss rate; NPV: Negative predictive value; NHS: National Health Service.

Limitations of task-specific endoscopic AI

Although AI technologies targeting specific lesions and procedural aspects have advanced the field of GI endoscopy, most existing systems remain limited to specific tasks. These systems are typically designed around single objectives, such as lesion detection, optical characterization, image segmentation, quality assessment, anatomical landmark recognition, or blind-spot monitoring. Furthermore, the development and evaluation of these systems often proceed in isolation, resulting in a lack of consistency across datasets, annotation standards, model architectures, and performance metrics. This fragmentation limits the ability of AI to address the inherent complexities of clinical endoscopy.

Probably one of the largest problems is that there is only so much that models can actually produce. Task-oriented methods produce particular kinds of outputs, namely certain kinds of visuals (cues), certain kinds of labels (diagnoses), certain kinds of measurements (bounding boxes, probabilities), certain kinds of maps (segmentations), and certain kinds of warnings (alerts). Likewise, there may be only one kind of output from a general method (quality scores) that cannot describe procedures in full richness or write reports by themselves. Moreover, most task-specific models lack temporal dimensions and clinical contextual information. They cannot accomplish their goals if they depend only on one frame or several pieces of short videos, since these cannot provide enough information concerning, for example, lesion monitoring, the degree of inspection completion for each site, and decision-making regarding whether an intervention has succeeded. Nor do they incorporate any patient-level information such as indications for this procedure, symptoms, medication history, previous endoscopy results, pathology reports, and other types of risk factors. There is also another drawback regarding generalization: Most AI programs have been produced using retrospective data obtained from certain hospitals, devices, or patients and thus do not work well if they are transferred to other kinds of endoscopes, types of images, degrees of bowel cleansing, lesions, operators’ habits, etc.[11,49].

These limitations motivate the development of more integrated systems that combine visual perception, temporal information, clinical context, standardized terminology, and language generation. Workflow-oriented multimodal and language models are therefore being explored as a means of extending AI beyond isolated lesion-level tasks.

MULTIMODAL AI AND LLMS IN ENDOSCOPIC WORKFLOWS

The shift from task-specific AI to workflow-oriented intelligence requires models that integrate information from multiple sources. In routine practice, clinicians rarely interpret a finding on the basis of a single image. They integrate contemporaneous and preceding procedural observations with subsequent events and patient-specific information. There may also be pathology results and codes from standard ways of recording disease in general. Given that this integration involves diverse data types, this review frames multimodal learning as a broad paradigm for achieving data-level integration, encompassing the fusion or alignment of heterogeneous information such as endoscopic images, time-series data, text, and clinical and histopathological findings. Therefore, the evidence suggests that this paradigm could support a wide range of integration tasks across modalities. VLMs, MLLMs, and LLMs act as key approaches that extend this paradigm and support interpretation, documentation, and communication.

Data-level multimodality in GI endoscopy

From a data perspective, GI endoscopy is inherently multimodal, as the clinical interpretation process integrates information across visual, temporal, textual, clinical, and histopathological domains. Integrating these information streams enables AI systems to move beyond simple visual pattern recognition toward interpretations that are more personalized to the patient[50]. A growing body of research highlights the value of multimodal fusion in GI endoscopy. A study by Ding et al[51] demonstrated that fusing endoscopic images with diagnostic text at the semantic level allows for the extraction of high-level features of upper GI lesions and significantly improves diagnostic performance compared to unimodal methods. Another study found that combining white-light endoscopy with image-enhanced endoscopy techniques facilitates the real-time assessment of colorectal cancer invasion depth[52]. Crucially, multimodal fusion should not be reduced to a mere aggregation of data. Information from different modalities may be incomplete, asynchronous, noisy, or inconsistent. In particular, since histopathological results are typically unavailable at the time of real-time endoscopic report generation, they are best viewed as delayed post-procedural information rather than input data for the initial report. Consequently, multimodal endoscopic AI systems must not only integrate diverse data streams but also possess the ability to identify missing information, align cross-modal evidence, and maintain a clear link between outputs and original source data. Figure 2 summarizes the framework of “modality-specific encoding” and “task-oriented output” in multimodal GI endoscopic AI, illustrating how heterogeneous clinical information is organized, integrated, and transformed into outputs that align with clinical workflows.

Figure 2 Modality-specific encoding and task-oriented outputs in multimodal gastrointestinal endoscopic artificial intelligence. This framework does not imply that all modalities are available simultaneously or that all outputs are generated within a single final report. This is an author-proposed conceptual framework intended for illustration and does not represent a validated clinical pathway or clinical guideline. EHR: Electronic health record; BBPS: Boston Bowel Preparation Scale; MLP: Multilayer perceptron; Hb: Hemoglobin; CRP: C-reactive protein; Alb: Albumin; ALT: Alanine aminotransferase.

Model-level approaches: VLMs, MLLMs, and text-based LLMs

At the model level, VLMs, MLLMs, and text-based LLMs represent model-level approaches that can be employed to leverage visual, textual, and clinical information within endoscopic workflows. Their distinction is important not only at the architectural level but also when interpreting evidence: Performance on image-text representation tasks, multimodal benchmark tasks, and clinically evaluated report-generation workflows represents different levels of evidence and should not be treated as interchangeable. For example, Wang et al[17] proposed EndoKED, which utilized large VLMs to extract knowledge from large-scale image-text colonoscopy records and convert routine clinical records into annotated datasets for downstream analysis. This approach is valuable because expert pixel-level annotation is labor-intensive, whereas routine colonoscopy records contain abundant but underused supervision signals. Endoscopy-specific vision-language pretraining frameworks, such as Endo-CLIP, further suggest that image-text colonoscopy records can be used to improve zero-shot and few-shot lesion detection and classification[53].

MLLMs extend vision-language learning by incorporating non-text biomedical inputs into LLM-centered generative architectures. Tu et al[54] introduced Med-PaLM Multimodal as a proof-of-concept generalist biomedical model capable of processing clinical language, medical imaging, and genomic data within a unified framework. Although it was not developed specifically for GI endoscopy, it illustrates how heterogeneous biomedical inputs can be incorporated into a single generative architecture. In GI endoscopy, MLLMs can support visual interpretation by analyzing images or video clips, answering clinical questions, and describing lesion characteristics. Several benchmarks have also been developed to examine the capabilities of general or domain-specific MLLMs in endoscopic tasks. EndoBench covers multiple endoscopic scenarios and clinical tasks[55], whereas GI-Bench extends this evaluation to a broader GI endoscopic workflow, including anatomical localization, lesion identification, diagnosis, findings description, and management[56]. These benchmarks are valuable because they expand the scope of evaluation beyond simple image categorization, testing the ability of models to do more clinically contextualized tasks. At the same time, they expose key shortcomings, including imperfect geographical grounding, inconsistent factual correctness, and the danger that eloquent language may disguise inaccurate visual interpretation. Video MLLMs also suffer from the computational cost of extended inputs. Processing of entire endoscopic recordings at high spatial resolution results in a huge number of visual tokens. Therefore, general-purpose video MLLMs often resort to sparse frame sampling, temporal aggregation, keyframe selection or spatiotemporal token compression. For example, TimeChat[57] and LongVU[58] have addressed long video difficulties using timestamp-aware encoding, sliding temporal aggregation, adaptive spatiotemporal compression, etc. This poses a vital trade-off for endoscopic applications; strong compression may reduce redundancy but may also miss fleeting events of clinical significance (e.g., transient visual events during biopsies or polypectomies). Thus, the ability of an MLLM to process long videos does not equate to a comprehensive capacity for frame-level temporal understanding.

In contrast, text-based LLMs primarily operate on textual information. Singhal et al[59] introduced Med-PaLM and combined conventional benchmark evaluation with clinician-led assessment of medical responses, providing an important precedent for evaluating medical LLMs beyond multiple-choice accuracy alone. In GI endoscopy, text-based LLMs have been explored for documentation, education, patient communication, research support, and workflow-related tasks[60,61]. Their use is most clinically defensible when generated content is based on verified clinical information and remains subject to clinician review.

In this sense, they remain text-based LLMs. They cannot be considered multimodal unless they are combined with a proven visual encoder in some way, or receive multimodal input in a certain structure. However, a clinically meaningful criterion for distinction lies not in the number of named components included in the model, but rather in whether the final output is based on original endoscopic image evidence and has been appropriately validated for its intended clinical use.

AI-ASSISTED DOCUMENTATION AND AUTOMATED REPORT GENERATION

Endoscopy reporting is a critical but often underestimated component of GI endoscopic practice. A clinically useful report must document key procedural elements, including the indication for the procedure, examination extent, and mucosal visualization. It should also capture lesion-related details, interventions, specimen collection, diagnostic impression, complications, and follow-up recommendations[62-64]. Unlike a simple image label or diagnostic category, an endoscopy report serves as the formal clinical record that supports communication across clinical care, pathology, patient communication, and surveillance programs. Thus, the transition from lesion detection to automated report generation represents an important step toward workflow-integrated endoscopic AI. However, AI-assisted endoscopic documentation includes several heterogeneous technologies that address different parts of the reporting workflow. These systems should not be regarded as directly interchangeable or evaluated using a single set of metrics. Therefore, this section organizes AI-assisted reporting technologies according to their clinical function within the documentation workflow rather than treating them as a single comparable model class.

Structured reporting and workflow-based documentation support

Prior to the use of VLM- and MLLM-based systems for generative reporting, many AI-assisted endoscopic reporting systems focused on structured documentation rather than free-text generation. This approach is clinically important because it directly addresses the practical problems of incomplete documentation and the time burden of report writing.

Qu et al[18] developed an image recognition-based structured report generation system, trained and tested it with 46987 images from nine tertiary hospitals, and conducted a multicenter prospective evaluation on a separate set of 5699 images. This study represents an early attempt to integrate endoscopic image analysis with report generation. The significance of this work lies not merely in the reports produced, but in a conceptual shift: Moving from isolated visual recognition tasks toward image-supported workflows.

Further research was then done on real-time image-based reporting during an upper GI endoscopic surgery. Dong et al[65] designed a deep learning-based system for real-time picture reporting during esophagogastroduodenoscopy (EGD) with the objective of creating high-quality image reports for routine recording. These systems can be considered as an intermediate step between visual recognition and complete report generation; instead of generating a complete narrative report, they help in selecting, annotating and organizing clinically significant images, which improves the quality and completeness of the various parts of the report. In a randomized crossover study, Zhang et al[19] tested an AI-based automatic endoscopy reporting system and showed that it produced more accurate reports than traditional approaches. The system integrated automated image capturing with lesion detection, anatomical location and textual description. These results indicate that the first practical use of automated endoscopic reporting may not be unrestrained free-text creation, but rather standardized picture acquisition, anatomical localization, lesion diagnosis, and structured text assistance. Preliminary prospective examination of AI-assisted gastric-lesion reporting has also demonstrated enhanced report completeness but more multicenter validation is required to ensure generalizability[66].

Voice recognition as a modality-independent tool to accelerate endoscopic documentation has been researched in parallel. For example, Yokota et al[67] developed the Voice Capture system that automatically transcribed verbal descriptions into structured reports for endoscopists. In a single-center retrospective analysis, the median time for report production was significantly shorter in the speech recognition group than in the manual-entry group (311 s vs 383 s, P = 0.009) and the system allowed real-time compilation of preliminary reports that could be subsequently modified manually. In a further pilot investigation of the Endoscopy Virtual Assistant System, 21 cases were recorded by five experienced endoscopists using voice input to document the endoscopic results. The average time to create a report was 29 seconds, a 46.3% reduction compared to 56 seconds for traditional typing, with larger time savings when there were more results and interventions present[68]. In colonoscopy, AI-based report support has also been explored from a procedural standpoint. Naito et al[20] examined AR-C1, an AI-based support system for generating colonoscopy reports. AR-C1 was designed to capture procedural events (insertion, withdrawal, instrument use) instead of simply lesion recognition as other systems did. This is an important direction, as reports of colonoscopy should describe the existence of lesions, as well as procedural background, interventions, and information on specimens.

Taken together, these workflow-based systems show that automated endoscopy reporting does not depend solely on generative language models. Accurate report generation depends on reliable image acquisition, anatomical localization, lesion recognition, and structured findings. In this sense, structured reporting systems provide the foundation for later multimodal and generative approaches.

Generative reporting with VLMs and MLLMs

Recent advances in VLMs have created new opportunities for generating more flexible and clinically meaningful endoscopic reports. Unlike structured documentation systems, generative models can produce natural-language descriptions, summarize findings, and organize information into coherent report sections. For example, transformer-based vision encoder-text decoder frameworks have been proposed to generate GI endoscopy reports from image-text and image-report pairs. The Multi-modal Memory Transformer Network (MMTN) is a representative example of this approach. By employing a specialized encoder to align medical images with clinical terminologies, MMTN reduces image-report inconsistencies and improves word prediction through cross-modal complementarity[69]. Another recent model adopts a two-stage training strategy, in which a transformer-based vision encoder and text decoder are first pre-trained on image-caption pairs before being fine-tuned on image-report pairs to produce clinically meaningful findings[70].

While these VLMs effectively map visual features to textual descriptions, their ability to integrate multimodal clinical context remains limited. Building on these foundations, MLLMs have further extended this capability. In principle, these models can integrate endoscopic images or videos with clinical and procedural information. This integration may support draft report generation for subsequent review and editing by endoscopists.

Report-Angel represents one of the most clinically relevant examples. Jiang et al[21] developed a domain-specific MLLM for automated upper GI endoscopy reporting and evaluated it using multicenter prospective validation. At the case level, Report-Angel achieved a report completeness of 88.51% and a report accuracy of 78.93% in an internal prospective video dataset, with an average processing time of 1.5 s per lesion. Lesion-level reporting accuracy was 91.92% on a retrospective image dataset, 89.07% on a prospective single-center video dataset, and 83.94% on a prospective multicenter video dataset. These results suggest that domain-specific multimodal training and prospective evaluation can make draft report generation feasible, but the performance drop across prospective and multicenter settings also highlights the importance of external validation and workflow-specific assessment. Meanwhile, the system does not directly feed raw, unprocessed full videos into the MLLM. Instead, the original endoscopic video stream is first converted into an image sequence with a frame rate of 7 frames per second and processed by a dedicated model. Subsequently, the selected high-quality frames containing lesions and anatomical landmarks are provided to the MLLM for description.

Massimi et al[71] input complete EGD videos into a general-purpose MLLM, focusing on anatomical landmark recognition, visual documentation, and lesion detection or characterization. However, an analysis based on only five examinations showed poor report generation performance, and their proprietary video processing mechanism made it impossible to determine the extent to which the model retained complete frame-level temporal information. A fine-tuning multimodal large Language Model for Endoscopic Report Generation[72] uses a two-stage parameter-efficient fine-tuning strategy, combining pretraining on public endoscopic datasets with prompt-based LoRA fine-tuning on professionally annotated clinical data. In colonoscopy, LoRA, Direct Preference Optimization, and Prompt Engineering (LDP)[73] applies Qwen2-VL-7B with LoRA and Direct Preference Optimization using the expert-annotated Multimodal Endoscopic dataset. In physician evaluation, LDP achieved a score of 7.2/10 while requiring substantially less training computation than full-parameter fine-tuning. These findings demonstrate the technical feasibility and computational efficiency of parameter-efficient MLLM adaptation for endoscopic report generation.

Despite these technical advances, generative reporting in endoscopy remains less clinically mature than established applications such as colorectal CADe. Endoscopic reporting must account for long procedural videos, variable image quality, multiple anatomical regions, interventions, and delayed information such as histopathology. Clinically useful systems therefore require reliable selection of relevant visual evidence, anatomical and temporal grounding, explicit handling of missing information, and traceability between generated statements and their source evidence. Representative primary studies are summarized in Table 3; because these studies differ substantially in input data, output format, study design, and evaluation endpoints, their reported performance should not be interpreted as directly comparable.

Table 3 Representative primary studies of artificial intelligence-assisted endoscopic documentation and report generation.

Ref.	System/input	Setting/sample	Clinical function	Design/comparator	Evaluation unit	Key findings	Validation/main limitation
Qu et al[18], 2020	ISRGS; endoscopic images/video	46987 images; prospective set: 5699 images	Site/lesion recognition and structured report generation	Multicenter development and prospective validation	Image/patient	Overall diagnostic accuracy: 88.41% multicenter; 89.65% prospective	Multicenter data, but structured/template-based output
Dong et al[65], 2022	EAIRS; real-time EGD video	210198 development images; 161 prospective patients	Automated photo-documentation	Internal, external and prospective testing; system vs endoscopists	Patient/procedure	Landmark capture accuracy 95.2%; completeness: 93.1% vs 88.8%; detected-lesion capture: 99% vs 98%	Multicenter development; mainly image documentation rather than full narrative report
Yokota et al[67], 2022	Voice Capture/speech input	356 patients; 101 propensity-matched pairs	Speech-assisted structured reporting	Retrospective propensity-matched comparison; voice recognition vs manual entry	Procedure	Median report preparation time: 311 s vs 383 s, P = 0.009	Single-center retrospective study
Zhang et al[19], 2023	AI-EARS; EGD video/image	252111 images for training, 62706 images, and 950 videos for testing	Automatic image capture, diagnosis and textual description	Multireader multicase randomized crossover; AI-EARS vs conventional reporting	Report/procedure	Text completeness: 79.03% vs 51.86%; accuracy: 64.47% vs 42.81%; reporting time: 46.47 s vs 80.13 s	Relatively small clinical crossover sample
Naito et al[20], 2025	AR-C1; video/procedural events	40 endoscopic mucosal resection procedures	Procedure-event capture and report support	Retrospective clinical comparison	Procedure/report	Report generation time: 186.6 s vs 258.4 s	Single-center and single examiner
You et al[66], 2026	AI gastric-lesion reporting system	276 video clips; prospective clinical analysis: 100 patients	Structured lesion documentation	Prospective AI-assisted vs routine reporting	Lesion/report	Overall lesion-report completeness: 86.5% vs 75.11%, P = 0.001	Prospective but single-center
Kawagoe et al[68], 2026	EVAS voice recognition	21 cases; 5 experienced endoscopists	Voice-assisted colonoscopy reporting	Pilot comparison; EVAS vs manual typing	Report	Mean reporting time: 29 s vs 56 s; 46.3% reduction	Very small pilot; recognition accuracy/system integration require further evaluation
Jiang et al[21], 2026	Report-Angel; endoscopic images/videos + text	20617 training image-text pairs; prospective internal n = 300 and external n = 300 cohorts	MLLM-generated draft EGD reports	Prospective internal and external clinical validation; expert review of generated drafts	Case/report + lesion	Clinically acceptable draft reports: 79.3% internal, 83.3% external; case completeness 88.51%, accuracy 78.93%	Strongest current prospective MLLM evidence; generated drafts still require clinician review
Massimi et al[71], 2026	Gemini 2.5 Pro; complete EGD video	5 full-length EGD videos	MLLM video analysis and report generation	Retrospective paired pilot; clean video vs same video with CAD overlay	Report/event	Adequate completeness: 56% vs 48%; landmark accuracy: 0.55 vs 0.33	Extremely small pilot; authors concluded performance was inadequate for clinical EGD reporting

Outcomes are reported as task-specific findings from the original studies and are not intended for direct comparison across different tasks, study designs, or evaluation metrics. AI: Artificial intelligence; EGD: Esophagogastroduodenoscopy; LLM: Large language model; MLLM: Multimodal large language model; CAD: Computer‐aided detection/diagnosis.

Post-generation review, patient communication, and human oversight

The role of AI in endoscopic documentation extends beyond the mere generation of structured or free-text reports. Although fully automated report generation technology has not yet matured sufficiently for routine clinical practice, AI can still provide support in various downstream tasks, such as information extraction, content summarization, terminology standardization, identification of omissions, and verification of inconsistencies. Furthermore, once the endoscopist has completed or edited a report, AI can serve as a quality control aid by verifying the documentation of key procedural elements and checking for consistency among image annotations, structured data fields, and textual descriptions. These functions may improve report completeness and reduce documentation burden while avoiding some of the additional risks associated with independent interpretation of raw visual input by a generative model[60]. Further real-world evaluation is required to determine their effect on workflow efficiency and clinical utility[74].

Patient communication represents another application of AI-assisted documentation. Because endoscopy reports often contain terminology that is difficult for patients to interpret, text-based LLMs can be used to generate patient-oriented explanations based on verified clinical information. Calabrese et al[75] evaluated ChatGPT for common patient questions related to GI endoscopy and found that the model could provide informative responses. However, clinician oversight remains necessary because broader evidence on LLM-based patient education continues to identify concerns related to accuracy, readability, bias, and over-reliance[76].

Even AI-enabled documentation and communication require human supervision. AI-generated reports should be reviewed, edited, and approved by endoscopists prior to clinical usage. Generated drafts should be fully editable and clinically essential statements should, when feasible, be linked to supporting imagery, procedural events or structured data to facilitate traceability and auditability. Governance frameworks for health AI emphasize safety, transparency, and appropriate human oversight, which are especially important when generated text may influence clinical decisions and patient understanding[77]. Within such a collaborative model, AI offers assistance with data organization, missing-item detection, report drafting, and patient communication, while the final clinical responsibility lies with the physician. A simplified workflow for AI-assisted endoscopic documentation and report generation is shown in Figure 3. Figure 3 emphasizes that final clinical use should rely on clinician-reviewed reports, and pathology-dependent recommendations should not be generated as definitive outputs before histopathological confirmation.

Figure 3 Artificial intelligence-assisted endoscopic documentation and report generation pathway. The figure illustrates a clinician-supervised documentation pathway rather than a fully automated real-time final reporting system. AI-generated draft reports and follow-up suggestions require clinician review before clinical use. Follow-up recommendations should be determined according to confirmed pathology, applicable clinical guidelines, and patient-specific factors. This is an author-proposed conceptual framework intended for illustration and does not represent a validated clinical pathway or clinical guideline.

CHALLENGES AND FUTURE DIRECTIONS

While the role of AI in endoscopy has evolved from lesion-level help to procedural support, multimodal analysis, and generative reporting, translation to the clinic requires more than technical competence. Critical concerns that demand urgent attention include the quality and representativeness of multimodal data, the reliability and evaluation of model outputs, the strength of clinical evidence and workflow integration, and the ethical and regulatory considerations surrounding deployment. These difficulties are growing in importance as AI-generated outputs reach closer to formal clinical documentation and patient care processes.

Data and annotation challenges

Clinically reliable endoscopic AI depends on high-quality multimodal data, yet most datasets available today remain narrow in scale, diversity, and the range of modalities they cover. The traditional datasets may tend to offer selected still images or short clips labeled for a single task, whereas workflow-oriented AI requires richer, synchronized data. Although large-scale endoscopic video datasets have been assembled, multimodal image-text colonoscopy records also indicate that mining with large VLMs appears feasible. Nevertheless, building comprehensive datasets remains challenging because it requires cross-system integration and extensive expert annotation[17,78,79]. Annotation proves difficult in endoscopy[80]. Trouble arises from visual ambiguity and imperfect alignment between images, videos, and reports. Moreover, the lesion boundaries and anatomical landmarks are difficult to define precisely, and video-level events require dense temporal labels. Furthermore, written descriptions might not map neatly onto specific visual evidence. In light of these findings, the mismatches may suggest that noise enters image-text pairs and weakens the reliability of multimodal learning. In addition, terminology varies from one institution to another, which reduces the utility of free-text reports as supervisory signals[81].

Given that isolated image-label pairs limit progress, future datasets need to shift toward standardized multimodal endoscopy records. Additionally, these important records should bundle together synchronized video, structured report fields, pathology links, procedural metadata, and follow-up outcomes. Notwithstanding this observation, the development and reporting might follow recommendations that support transparency, reproducibility, and trustworthy AI in medical imaging. However, standardized terminology and multicenter collection could prove critical to capture real-world diversity. Nevertheless, the evidence may suggest that this diversity helps prevent deployment failures[82].

Model reliability and evaluation

Generative AI in endoscopy also faces significant challenges regarding reliability. While traditional CADe systems typically err through missed diagnoses or false positives, generative models may produce reports that are linguistically fluent yet clinically flawed-such as misidentifying anatomical locations, proposing diagnoses without sufficient evidence, omitting interventions, or recommending inappropriate follow-up care. This problem is commonly described as hallucination, but in clinical reporting it should be understood more specifically as a failure of evidence-grounded generation. Visual grounding is therefore essential to ensure that every statement regarding a lesion is traceable to corresponding visual evidence. In video-based reporting, grounding must also be temporal. A generated statement may be visually plausible yet temporally incorrect-for example, describing a lesion observed before resection as a persistent finding after the intervention. Therefore, evidence grounding should link report statements not only to individual frames but, where relevant, to timestamps and procedural events.

Existing evaluation systems require adjustment to address the clinical and procedural risks tied to generative AI use in endoscopy. Traditional image metrics like accuracy and sensitivity, along with language metrics such as BLEU and ROUGE, are insufficient for fully assessing clinical correctness, completeness, or safety. In contrast, clinician-based assessment should be multidimensional. We propose that generative endoscopy reports be evaluated across five complementary domains: Factual grounding, completeness, clinical safety, overall clinical acceptability, and workflow utility (Table 4). These domains require different assessment scales rather than a single universal score. Individual report elements can be judged as correct or incorrect and present or absent; clinically important errors can be graded according to their potential consequences; the report as a whole can receive a binary judgment of whether it is acceptable as an editable clinical draft; and subjective usability can be assessed using an ordinal Likert scale. These dimensions should be reported separately rather than combined into a single composite score until clinically validated weighting schemes become available.

Table 4 Proposed clinician-in-the-loop evaluation framework for artificial intelligence-generated endoscopy reports.

Dimension	Clinician assessment	Scale	Suggested outcome
Factual grounding	Verify findings against source evidence	Item-level correct/incorrect	Accuracy; hallucination rate
Completeness	Check required report elements	Present/absent	Completeness; omission rate
Clinical safety	Grade consequences of errors	Minor/moderate/major	Major-error rate
Overall acceptability	Determine need for substantive correction	Yes/no	Clinically acceptable report rate
Workflow utility	Assess usability and editing burden	Likert scale + time	Utility score; correction time

For clinician-rated endpoints, evaluation by multiple blinded reviewers and reporting of inter-rater agreement (e.g., Fleiss’ κ or intraclass correlation coefficient) are recommended. This framework represents an author-proposed synthesis of current evaluation practices and is not a validated clinical guideline.

Recent studies provide practical examples of this approach. Jiang et al[21] used blinded expert adjudication to determine whether AI-generated reports were clinically acceptable, together with case-level completeness and accuracy, severity-based error analysis, and inter-observer agreement. Massimi et al[71] similarly used endoscopist assessment of report adequacy across completeness, visualization, and lesion-description domains. More broadly, clinician evaluation of medical LLMs such as Med-PaLM has incorporated dimensions including incorrect content, omission, agreement with scientific consensus, and potential harm[59]. Taken together, these approaches support a clinician-in-the-loop evaluation strategy in which clinically consequential errors cannot be obscured by high linguistic-similarity scores.

Clinical utility should also be evaluated in actual workflow situations. Relevant endpoints in draft-report systems include the time for review and correction by the physician, the proportion of drafts accepted without clinically substantial revision, the number or proportion of statements requiring correction, and the influence on overall time for documentation. User-facing measures can also assess perceived usefulness, cognitive load and ease of editing. These outcomes of human-AI interaction supplement standard performance measures and align with DECIDE-AI’s focus on clinical value, safety, and human factors in the early-stage evaluation of AI[74].

As systems advance to later stages of clinical validation, prospective studies should follow established reporting guidance for AI interventions. CONSORT-AI[83] and SPIRIT-AI[84] specify requirements concerning AI interventions, human-AI interaction, input data characteristics, and failure-case analysis. External validation is also essential because performance may change across endoscopic equipment, imaging modalities, patient populations, and institutional reporting practices. Prospective multicenter validation and post-deployment monitoring should therefore remain integral to evaluation. The FUTURE-AI guideline further emphasizes fairness, robustness, traceability, and explainability across the AI lifecycle[85].

Clinical evidence, integration, ethical considerations, and regulation

Existing evidence indicates significant disparities in the clinical maturity of various endoscopic AI applications. Lesion-level CADe is supported by the most robust clinical evidence, derived from randomized and multicenter trials conducted across multiple healthcare systems[7,23-27]. However, the magnitude of additional benefit varies. While multiple trials have reported a significant increase in adenoma yield, Maas et al[28] found no significant improvement in the ADR or APC during procedures performed by experienced endoscopists. These findings suggest that the incremental value of CADe depends on baseline procedural proficiency, the clinical setting, the patient population, and the specific system being evaluated. Clinical evidence regarding systems for procedural quality and documentation, while relatively limited, is steadily accumulating. Prospective studies have demonstrated that these systems improve metrics such as procedural completeness, documentation completeness, and reporting efficiency[9,18-20,47,65-68]. However, these endpoints primarily reflect procedural or workflow performance rather than clinical outcomes of critical importance to patients. In contrast, generative AI-based endoscopic reporting systems remain in the early stages of validation. The Report-Angel system has yielded promising results in prospective internal and external validation regarding the generation of clinically acceptable draft reports[21]. However, a pilot study by Massimi et al[71] demonstrated that inputting full video footage into a general-purpose MLLM does not necessarily produce clinically reliable reports. Thus, current research on generative reporting, while confirming technological feasibility and initial clinical utility, has not yet offered proof of benefit supported by randomized controlled trials, as in the case of colorectal CADe. Together, Tables 2 and 3 provide a hierarchy of evidence from quite established applications to those in the early stages of development.

The successful adoption depends on the integration of AI into the endoscopic process along with the maturity of evidence. The solution should integrate with existing reporting platforms and electronic health records without adding extra work for data entry or correction. Alarm fatigue can occur from frequent or misprioritized warnings, and automation bias can occur from over-reliance on AI output. The evaluation should therefore not be limited to correctness and should include time taken by physicians to make changes, burden of warnings, convenience of use, training needs and resources required for deployment. This is particularly important for the deployment of cloud-based MLLMs in resource-constrained contexts. Therefore, clinical implementation methods need to be examined in institutions with diverse capacities of technology and manpower, and it should not be assumed that workflows proven at tertiary care centers are translatable to other settings[12,74,85].

Privacy and data governance pose another set of problems. Multimodal endoscopic AI requires access to sensitive data like pathology and health records, demanding secure storage, de-identification, and explicit guidelines for data use. The privacy hazards are greatest for MLLMs that need to send all endoscopic films or the related multimodal clinical information to external or cloud-based computation environments. Even if one frame does not contain clear personal identifiers, as soon as the endoscopic examination video and its associated metadata are associated with or linkable to a specific individual, they are considered sensitive health information. Health data are a specific category of personal data under the General Data Protection Regulation and their processing must be based on an acceptable legal basis as well as measures aligned with principles such as purpose limitation and security of processing[86]. In the United States, if an entity covered by the Health Insurance Portability and Accountability Act uses a cloud service provider for the transmission of protected health information, the related agreement must be made and a mechanism for risk management must be put in place[87]. Some research shows useful safeguards. For instance, Jiang et al[21] mentioned techniques including obtaining informed consent before analysis and de-identifying endoscopic pictures and films. Whole EGD videos were anonymized by Massimi et al[71] before the analysis with cloud-based MLLMs. However, such safeguards should not be construed as evidence of compliance with the standards of any particular jurisdiction, although they do minimize the privacy risks. Hence, research on cloud-based MLLMs will have to be explicit about data transmission routes, control of access, data retention regulations, and third-party processing arrangements.

Regulatory requirements vary depending on the jurisdiction and the system’s intended clinical use. Recent regulatory developments could demonstrate that generative endoscopic AI will require lifecycle‑oriented oversight rather than one‑time validation. These developments include the United States Food and Drug Administration’s approach to predetermined change control plans and the European AI Act’s emphasis on risk management and human oversight for high-risk medical AI[88,89].

Limitations of this review

This review has several limitations. First, it is a narrative minireview, not a systematic review or meta-analysis. Although a structured search of PubMed and the Web of Science Core Collection was performed, study selection and evidence interpretation are nonetheless prone to selection and citation bias, and no rigorous risk-of-bias or certainty-of-evidence evaluation was conducted. Second, the review was limited to English-language literature published between 2019-2026, with landmark studies from earlier years and rapidly emerging studies selectively included where applicable, which may mean some regional or non-English evidence was not captured. Third, the included literature is very varied in terms of study design, unit of evaluation, comparator, definition of outcomes and clinical context, which does not allow a direct numerical comparison between many studies. Fourth, evidence of newer VLM, MLLM, and generative-reporting applications is limited to technical studies, conference publications, and preprints in fields where peer-reviewed clinical data remains rare. Finally, most of the existing generative endoscopy studies examine technical performance, report quality, or workflow efficiency, rather than patient-important outcomes. The conclusions on the use of workflow-integrated generative AI should therefore be seen as an evaluation of an emerging field, not as an indication of proven clinical efficacy.

Future research priorities

Future work should aim at therapeutically grounded multimodal endoscopic AI rather than isolated benchmark performance. Multicenter datasets should link endoscopic video to structured reports, procedural information, pathology and follow-up outcomes while maintaining temporal links between findings and interventions. Generative reporting systems should provide traceable, editable drafts; indicate material that is absent rather than inferring unsupported content; and link clinically important claims to source evidence. Long-video models should preserve the temporal sequence, efficiently select or compress visual information, and explicitly track clinically relevant state changes, including lesion discovery, biopsy, resection, hemostasis, and post-intervention assessment. In the short run, multimodal AI and LLMs should assist endoscopists, not replace clinical judgment.

CONCLUSION

AI in GI endoscopy is moving away from lesion-level help and toward procedure-level and documentation support, with the maturity of clinical evidence varying widely among applications. Colorectal CADe has the most robust comparative clinical evidence, whereas procedure-quality and structured documentation systems are mostly supported by prospective procedural and workflow outcomes. Generative VLM- and MLLM-based reporting is an emerging application, with studies demonstrating technological feasibility, report quality, and clinical acceptability, but not patient benefit. Further clinical translation requires reliability for visual and temporal grounding, clinician-in-the-loop evaluation, prospective multicenter validation, workflow interoperability, privacy protection, and lifecycle governance. Thus, multimodal and generative AI should be adopted in the short term as helpful tools with physician supervision and not as autonomous reporting or decision-making systems.