Computer vision in laparoscopic gastrointestinal surgery: A review of anatomical recognition and clinical navigation

Abstract

Laparoscopic surgery is constrained by a two-dimensional visual field and the absence of tactile feedback, which pose challenges for the precise intraoperative identification of critical anatomical structures. Recently, rapid progress in computer vision (CV) has offered a new way to address this clinical limitation. This review provides a narrative overview of the latest progress in CV in laparoscopic gastrointestinal surgery, ranging from fundamental temporal surgical phase recognition and visual quality preprocessing to high-precision dynamic segmentation and real-time bleeding warning of critically high-risk tubular structures such as mesenteric vessels, ureters, and the biliary tract. The review also examines how CV is being combined with augmented reality, three dimensional (3D) registration, and multimodal imaging, illustrating the shift in minimally invasive surgery from two dimensional image analysis toward 3D clinical navigation. Although CV technologies have demonstrated promising performance in several core tasks, large-scale clinical translation still faces challenges, including limited algorithmic generalization, reliance on single-center datasets, and a paucity of large-scale real-world evidence-based validation. Bolstered by the establishment of multicenter standardized datasets, introduction of edge computing, refinement of data privacy regulations, and strengthening of ethical oversight, CV may help advance gastrointestinal surgery from passive visual assistance toward more proactive intraoperative support and potentially toward greater intraoperative autonomy.

Key Words: Computer vision; Deep learning; Laparoscopic surgery; Anatomic landmarks; Surgical navigation; Augmented reality

Core Tip: Computer vision is advancing laparoscopic gastrointestinal surgery from two dimensional image recognition toward real-time anatomical navigation and context-aware assistance. This review summarizes progress in surgical phase recognition, visual enhancement, critical-structure identification, three dimensional and augmented-reality navigation, and robotic integration, while emphasizing the gap between technical performance and demonstrated clinical benefit. Key priorities include multicenter validation, procedure-specific datasets, privacy-preserving learning, human-artificial intelligence collaboration, and prospective evaluation of surgeon-, procedure-, and patient-centered outcomes. Semi-autonomous functions should remain under surgeon oversight.

INTRODUCTION

Compared with traditional open surgery, laparoscopy has become the mainstream modality in modern gastrointestinal surgery due to its significant advantages, including minimized surgical trauma and accelerated postoperative recovery[1,2]. Nevertheless, a two-dimensional visual field, the absence of tactile feedback, and complex anatomical variation make it difficult for surgeons to identify critical structures precisely during these procedures. This difficulty in anatomical recognition not only elevates the risk of iatrogenic organ and vascular injuries but also prolongs both operative time and hospital stay[2,3]. In recent years, the rapid advancement in artificial intelligence (AI) and computer vision (CV) technologies has offered an innovative paradigm to overcome this clinical bottleneck. As a core branch of AI, CV focuses on acquiring, processing, and extracting key information from digital images and videos[4]. This technology is increasingly central to the analysis of surgical video, and its clinical applications have expanded from basic surgical-phase recognition and instrument detection to real-time anatomical segmentation and multimodal intraoperative navigation[2,5,6]. This review aimed to provide a narrative overview of recent advances in the application of CV in laparoscopic gastrointestinal surgery. Specifically, we focus on exploring the potential challenges in surgical phase recognition, precise identification of critical tubular structures, integration of multimodal technologies, and future clinical translation.

LITERATURE SEARCH AND SELECTION METHODS

To provide a narrative overview of recent advances in CV in laparoscopic gastrointestinal surgery, a literature search was conducted using PubMed and Web of Science Core Collection databases. The search strategy combined relevant keywords and MeSH terms, including “computer vision”, “deep learning”, “laparoscopic surgery”, “image processing, computer-assisted”, and “anatomic landmarks”. The search was limited to the articles published between January 1, 2022 - June 30, 2026. A Boolean search string applied in PubMed was: (“Artificial Intelligence” OR “Deep Learning” OR “Computer Vision”) AND (“Laparoscopic” OR “Gastrointestinal”) AND (“Recognition” OR “Segmentation” OR “Navigation”).

Studies were considered for inclusion if they (1) Investigated CV- or deep learning-based methods applied to laparoscopic or robot-assisted surgery; (2) Addressed surgical phase recognition, image enhancement, anatomical or tissue recognition and segmentation, intraoperative bleeding detection, image-guided navigation, or vision-guided robotic assistance; and (3) Provided sufficient methodological and performance information to assess their technical or translational relevance. Studies focusing primarily on gastrointestinal surgery have been prioritized. Studies from other laparoscopic specialties were selectively included when their anatomical recognition, tissue segmentation, navigation, or intraoperative safety strategies were considered technically transferable to gastrointestinal surgical scenarios. Studies focusing exclusively on preoperative imaging, unrelated applications, or lacking sufficient data were excluded. To avoid overestimating the independence of evidence derived from overlapping, studies testing various algorithms on the same public datasets were treated as technical comparisons on a single cohort instead of independent clinical validations. When a peer-reviewed version of a study was available, the peer-reviewed publication was prioritized over the preprint version. Studies for which full-text access was unavailable were excluded from consideration. Language restrictions were applied during screening: Journal articles in both English and Chinese were eligible for inclusion. The literature screening and data extraction were performed independently by two cofirst authors. After removing duplicate records across databases, titles and abstracts were screened for relevance. Full texts of potentially eligible studies were then assessed against the predefined inclusion criteria. Any disagreements between the two reviewers were resolved through discussion or by consulting a third senior reviewer.

Following literature screening and assessment of methodological and clinical relevance, representative studies were selected for narrative synthesis. As a narrative review, this study may be subject to selection bias due to the purposive inclusion of representative studies based on clinical relevance. Therefore, the findings should be interpreted as a conceptual overview of current CV development trends rather than an exhaustive assessment of all available evidence.

The included studies were subsequently summarized according to the CV task, surgical target, AI approach, dataset characteristics, validation strategy, key performance metrics, clinical relevance, and translational maturity. For descriptive comparisons, a review-defined four-tiered translational maturity framework was used: T1, technical proof-of-concept or internal validation within the same cohort; T2, independent or external validation using separate cohorts or datasets; T3, prospective feasibility evaluation or real-world intraoperative deployment; T4, comparative evidence of clinical efficacy or patient benefit. This classification was developed for descriptive synthesis in the present review and does not represent an established evidence-based grading or regulatory framework.

BASIC RECOGNITION STAGE

Surgical phase recognition

In the early stage of AI integration into surgery, surgical-phase recognition emerged as a foundational application[5,7]. Real-time, high-precision phase recognition provides the basis for computer systems to interpret surgical workflows and support intraoperative decision-making[2]. To parse complex surgical video, CV frameworks rely mainly on combining two types of information: Static spatial features and temporal dynamic features[8].

Static spatial features

In the intelligent parsing of laparoscopic surgical videos, the extraction of static spatial features serves as the foundation for constructing an entire computer-assisted system, with its core lying in the precise capture of visual information within a single video frame. This encompasses anatomical landmarks, pathological structures, intraoperative instrument combinations, and environmental elements of the surgical scene (such as obscuration by electrocautery smoke, local blood accumulation, and artificial implants, such as metal ligation clips)[8-11]. Traditional research predominantly constructs deep convolutional neural networks (CNNs) to perform image feature extraction, progressing from basic edge textures to complex organ morphologies, primarily serving as a foundational tool for classification and phase recognition tasks[8,9]. Recently, self-supervised attention mechanisms and Vision Transformers, such as Bidirectional Encoder representation from Image Transformers (BEiT) have been widely introduced[8,11]. Through the self-attention mechanism, these architectures achieve more robust pixel-level semantic segmentation when confronted with overlapping anatomical structures or local smoke occlusions[11]. This detailed analysis of single frames provides a reliable basis for subsequent temporal feature extraction and is a prerequisite for real-time surgical warnings and precise anatomical navigation.

Temporal dynamic features

To improve the phase classification accuracy, researchers have employed architectures such as long short-term memory, multi-stage temporal convolutional network (MS-TCN), or temporal relation transformers to capture temporal correlations across consecutive video frames[12,13]. By analyzing instrument trajectories and sequential action logic, these models use the inherent chronological order of surgical workflows to rectify single-frame misclassifications[2,8]. Advanced multi-task models, such as RiT networks, further incorporate “instrument-verb-target” triplet predictions or customized attention modules as auxiliary features. This elucidates the causal link between tool usage and macroscopic surgical phases, significantly minimizing confusion in similar scenes[2,8,12,13-16].

Comparisons of existing architectures indicate that purely spatial deep CNNs lacking temporal context often exhibit limited generalization and cross-institutional transferability, yielding an accuracy of only 79.7% on a single-center cholecystectomy dataset. By contrast, a hybrid ResNet50 and MS-TCN architecture fine-tuned with multicenter data exhibited improved robustness[7]. In laparoscopic gastric cancer surgery, hybrid models combining CNNs and temporal transformers classify the procedure into eight standardized phases. Even in complex surgeries averaging over 2.5 hours with challenging lymph node dissections, the model achieved 90.1% accuracy and an 87.0% F1 score[17], demonstrating the capability of CV in parsing advanced clinical workflows. Beyond the temporal phase recognition of surgical procedures, CV can automatically generate standardized operative notes by parsing information, such as instrument utilization and anatomical manipulations[18]. Multiple public datasets have been developed to facilitate algorithm benchmarking, including the SLAM dataset that covers diverse laparoscopic surgical actions[14]. Ultimately, this technology will extend AI into an objective, standardized tool for automated surgical skill assessment and training[19-22]. A reliable phase awareness can provide a contextual layer for workflow coordination, phase-specific decision support, automated documentation, and postoperative quality assessment. However, current studies have largely evaluated recognition performance rather than whether phase-aware assistance reduces the operative time, documentation burden, and perioperative complications.

Visual quality enhancement and preprocessing

During laparoscopic surgery, obtaining a clear surgical field in real time is a prerequisite for safe surgery and for accurate analysis by CV models. However, smoke generated by electrocoagulation can obscure anatomical structures and instrument textures, whereas motion blurring caused by the rapid movement of instruments can lead to the loss of boundary details, thereby substantially degrading feature extraction[23-25]. Consequently, visual quality enhancement of laparoscopic videos has become important preprocessing module in surgical data science.

Early technologies constructed hybrid desmoking models based on generative adversarial networks and dark channel priors. However, these models often rely on strictly paired image datasets, making them difficult to implement in real-world surgical environments[23,24]. The DeSmoke-LAP algorithm has substantially reduced dependence on strictly paired datasets. When handling complex and locally uneven smoke, it demonstrates superior robustness and preserves authentic anatomical detail and the environmental illumination of the surgical field[25]. Importantly, LVQIS provides an early example in which technical image enhancement was evaluated against workflow-centered outcomes rather than image-quality metrics alone. In a clinical controlled trial involving 100 laparoscopic surgeries, the application of this system not only achieved excellent objective dehazing metrics but also significantly shortened the surgical pause time caused by visual obscuration in terms of clinical outcomes (P < 0.001), substantially reducing the intraoperative anxiety experienced by the primary surgeon[23]. However, whether these workflow benefits translate into shorter total operative time or improved postoperative outcomes remains uncertain.

PRECISE IDENTIFICATION OF TUBULAR STRUCTURES

Precise identification of blood vessels

Accurate recognition of the vascular anatomy is essential for safe lymph node dissection and tumor resection in laparoscopic gastrointestinal surgery, particularly in procedures involving anatomically variable or bleeding-prone vessels. Current studies have primarily focused on real-time identification and visualization of major vessels, including the inferior mesenteric artery (IMA), vascular structures relevant to right hemicolectomy, and the gastrocolic trunk (GCT). In colorectal surgery, a deep-learning-based IMA segmentation model achieved a Dice similarity coefficient (DSC) of 0.798 with a processing speed exceeding 12 fps[26]. Adopting a patient‑independent 5-fold cross‑validation strategy, this work achieved higher validation independence than studies relying on single‑center internal dataset splits, supporting the technical feasibility and potential clinical applicability of real-time intraoperative vascular navigation. In laparoscopic right hemicolectomy, one study reported evaluation metrics exceeding 0.75 for superior mesenteric vein recognition, whereas identification of smaller and anatomically variable vessels remained more challenging[27]. In a separate study focusing on GCT recognition, DSC values of approximately 0.53-0.61 were reported[28]. Collectively, these studies indicate that small and anatomically variable vascular structures remain challenging targets for CV-based recognition. Surgeon-based assessments have also demonstrated the potential clinical applicability of these vessel-recognition systems[27]. However, available evidence remains largely limited to technical feasibility and surgeon-perceived usefulness. Whether AI-assisted vascular recognition can reduce intraoperative vascular injury, decrease blood loss, shorten operative time, or improve other clinically meaningful outcomes requires further validation through prospective clinical studies.

Bleeding detection and warning

In addition to outlining static vessels, CV technology can provide dynamic real-time warnings when vessels are damaged or ruptured intraoperatively. Upon bleeding, spatiotemporal hybrid models and multitask convolutional neural networks (U-Net) can detect blood accumulation events and bleeding spots in real time, achieving an accuracy of 90.63% for blood accumulation event detection and a DSC of 81.89% for the associated segmentation task[29]. Separate object detection approaches have also been investigated for localizing intraoperative bleeding sites[30]. Furthermore, CV can leverage RGB information to classify image pixels into blood and nonblood components, thereby automatically quantifying the area of blood spread in the surgical field[29,31]. These developments extend CV from static anatomical localization toward dynamic intraoperative safety monitoring and may provide a technical basis for the earlier recognition of bleeding events. However, the detection accuracy alone does not establish clinical effectiveness. Prospective studies are needed to determine whether AI-assisted bleeding recognition shortens the time to hemostasis, reduces blood loss or transfusion requirements, decreases conversion to open surgery, and improves postoperative outcomes.

Real-time tracking of the ureter

Ureteral injury is a serious complication of laparoscopic colorectal and gynecological surgery, making accurate intraoperative identification clinically important. In colorectal surgery, the Feature Pyramid Network-based UreterNet achieved a DSC of 0.716 with an end-to-end display latency of 143 ms, thereby supporting the feasibility of real-time ureter visualization[32,33]. Similarly, a gynecological laparoscopic segmentation model achieved a DSC of 0.86 with a mean processing time of 0.008 s and showed a performance comparable to that of experienced surgeons[34]. However, whether this technical performance translates into fewer ureteral injuries or improved surgical outcomes remains unclear. The multicenter ImNavi randomized controlled trial is therefore evaluating clinically relevant outcomes, including organ identification time, organ injury, blood loss, and operative time[35].

Biliary tract and other critical tubular structures

Accurate identification of the cystic duct, hepatic vein, and portal vein branches is important for safe laparoscopic cholecystectomy and anatomical liver resection. However, the semantic segmentation of these structures remains challenging because of intraoperative artifacts and their visual similarity to the surrounding tissues. For the CholecSeg8k dataset, U-Net++ achieved the highest overall performance among the five networks evaluated under the same benchmark with a mean DSC of 0.62[36]. In a separate study, DeepLabv3+ optimized with a cross-entropy Tversky loss reported a mean accuracy of 0.976 and a mean intersection over union of 0.977 within its own testing setting[37]. Because the two studies differed in their datasets, target definitions, and evaluation protocols, their reported performance metrics should be interpreted independently rather than as a direct comparison. Importantly, improved segmentation metrics alone do not establish clinical benefits, and whether these approaches reduce biliary or vascular injuries during surgery remains unclear in prospective clinical studies.

Generalized recognition of adjacent high-risk structures and lesions in the abdominopelvic cavity

Beyond the core tubular structures, CV can also achieve precise intraoperative identification of nerves, lymph nodes, and lesion tissues[35,38-41]. A prospective observational study on laparoscopic colorectal surgery showed that an AI-based real-time recognition system identified key anatomical structures earlier than surgeons, with the greatest advantage observed relative to less-experienced operators, suggesting its potential role in reducing experience-related variability in intraoperative anatomical recognition[40]. However, whether earlier recognition translates to fewer organ injuries, shorter operative times, or improved postoperative outcomes remains unclear. Evidence from other laparoscopy specialties may provide complementary insights into transferable anatomical recognition strategies. In inguinal hernia repair, AI-assisted landmark visualization achieved a 90% correct recognition rate in surgeon assessment, while lesion-detection models have also shown feasibility in endometriosis surgery[42-44]. However, these non-gastrointestinal studies should be regarded as supportive evidence of technical transferability rather than direct evidence for gastrointestinal surgical practice.

Taken together, the objectives of CV vary across surgical settings and are inherently procedure-specific. In gastrointestinal surgery, current applications mainly focus on the recognition and localization of critical tubular structures, such as blood vessels, the biliary tract, lymph nodes, tumors, and other anatomical structures closely related to surgical safety and oncological resection[26-28,40,45]. In complex operative fields, AI-assisted recognition may help reduce the interference caused by tissue occlusion, anatomical variation, and visual similarity between adjacent structures, thereby improving the consistency and accuracy of key anatomical identification. For example, in rectal cancer surgery, recognition of the TME dissection plane, pelvic fascia, and autonomic nerves illustrates the progression of CV from generic anatomical segmentation toward procedure-specific anatomical guidance[46-48]. Similarly, surgical phase recognition and real-time identification of critical peripancreatic vessels have been explored[49,50]. Whether anatomical recognition and navigation strategies developed in other laparoscopic specialties, such as gynecological surgery, can be effectively transferred to gastrointestinal surgery while maintaining reliable performance and clinical applicability, remains to be investigated. Furthermore, although this review summarizes the performance metrics of existing models, the included studies exhibit substantial methodological heterogeneity. This inconsistency precludes direct cross-study comparisons and significantly complicates the subsequent selection and real-world implementation of clinical models. Most early proof-of-concept studies rely on small-sample datasets and utilize only internal splits or cross-validation. Such approaches lack patient independence and frequently lead to model overfitting, thereby artificially inflating performance metrics and introducing a substantial risk of bias. In contrast, studies employing patient-independent external validation or multicenter real-world deployment remain exceptionally scarce. Consequently, high algorithmic accuracy on specific benchmark datasets cannot be directly equated with genuine clinical applicability. Moving forward, there is an urgent need to establish standardized evaluation frameworks that incorporate bias risk control, independent external validation, and the complexities of real-world surgical environments to facilitate the objective evaluation and clinical translation of algorithms in this field.

FROM TWO DIMENSIONAL RECOGNITION TO THREE DIMENSIONAL CLINICAL NAVIGATION

Typically, experienced surgeons can accurately identify anatomical structures using a conventional surgical field of view. However, the true clinical challenge lies in the safe dissection and precise localization of key landmarks, such as blood vessels, within complex anatomical settings, such as those with severe intra-abdominal adhesions. Therefore, the more difficult anatomical recognition becomes, the greater the clinical value of AI-based image navigation. Although two dimensional (2D) image analysis has advanced tissue segmentation considerably, the monocular, planar imaging of laparoscopy limits surgeons' depth perception. To bridge this visual blind spot, the deep integration of CV with augmented reality (AR), three dimensional (3D) registration, and multimodal imaging technologies has become an emerging direction, advancing modern minimally invasive gastrointestinal surgery from planar recognition toward stereoscopic clinical navigation[51].

AR and 3D registration

By leveraging the 2D image-parsing capabilities of CV combined with 3D reconstruction and AR projection technologies, precise registration between preoperative 3D models and real-time intraoperative field of view can be achieved. This approach may help address the limitations of depth perception and improve the visualization of otherwise concealed anatomical structures in laparoscopic surgery[52,53].

By superimposing preoperative 3D organ mesh models onto the 2D intraoperative direct-view screen, AR technology can overlay hidden subsurface lesions and internal vascular networks[53,54]. At the frontier algorithm level, learning strategies based on diffusion models have effectively narrowed the domain gap between synthetic and real images, achieving high-precision visual projection of the liver and blood vessels[53,55]. In clinical applications, combining AI detection with AR projection mapping systems has successfully reduced the mean localization error of the hepatic portal vein from 31.2 mm to 14.5 mm, significantly enhancing the stability of dynamic projection[56]. Furthermore, deep-learning-based monocular visual simultaneous localization and mapping (vSLAM) algorithms can reconstruct a surgical scene in real time by relying solely on monocular endoscopic videos without the need for stereoscopic laparoscopes or additional hardware sensors. This provides a solid foundation for precise AR registration[54].

Whether utilizing point cloud registration centered on preoperative 3D model projection mapping or vSLAM focused on dynamic intraoperative 3D scene reconstruction, the registration accuracy of existing deep learning models drastically declines when severe tissue and organ deformation occurs[52-53,55]. Consequently, future efforts must enhance the robustness of AI algorithms in dynamically deforming environments to resolve the registration challenges posed by soft-tissue deformation.

Multimodal navigation strategies

Traditional indocyanine green near-infrared (NIR) fluorescence imaging is widely used in abdominal, visceral, and transplantation surgeries for quantitative tissue perfusion assessment and organ visualization. Particularly in colorectal surgery, it is used to predict and reduce the incidence of postoperative anastomotic leakage through temporal blood flow intensity measurements[51,57]. One study explored the simultaneous use of the Eureka AI navigation platform and an NIR fluorescent ureteral catheter. Although the AI model was not specifically trained on NIR images, it could highlight the hypogastric nerve, lumbar splanchnic nerves, and loose connective tissue, whereas NIR fluorescence visualized the ureter[58]. These findings support the technical feasibility of combining AI-based anatomical recognition with fluorescence navigation. However, whether this multimodal approach improves intraoperative safety or clinical outcomes remains to be established. Additionally, for patients with contraindications to contrast agents, researchers have integrated a compact multispectral camera with a deep-learning model based on invertible neural networks to develop a real-time multispectral imaging system. This system successfully achieved real-time contrast-free monitoring of tissue ischemia during laparoscopic surgery[59]. This contrast-free approach may avoid contrast agent exposure and provide an alternative for tissue ischemia assessment. However, its effects on clinical decision-making, workflow efficiency, cost, and patient outcomes remain to be established.

Furthermore, CV has evolved from singular image recognition toward the macroscopic perception of the operating room environment. Multimodal graph neural networks are being deployed to dynamically predict the anticipated use of instruments in surgical workflows, achieving an F1 score of 66.86%[12]. This capability may support anticipatory workflow assistance and surgical decision support systems; however, the clinical workflow benefits remain to be validated. On the other hand, to address the lack of physical haptic feedback in robotic and laparoscopic surgery, multi-task deep learning models based on visual features can estimate forceps operating force from video alone by restricting the analysis to the instrument tip’s region of interest. Experimental studies have shown that manipulation forces and excessive-force events can be estimated from visual information alone, thereby suggesting a possible visual surrogate for missing haptic feedback. However, this evidence remains preclinical and the reliability of such systems for preventing tissue injury in human surgery has not been established[60].

DEEP INTEGRATION OF COMPUTER VISION AND ROBOTIC SYSTEMS

Robot-assisted minimally invasive surgery is an important platform for integrating CV-derived perceptions with robotic control. Although current robotic systems, such as da Vinci and Hugo, provide advanced mechanical capabilities, they remain predominantly surgeon-controlled. The integration of CV may extend these systems toward context-aware perception and select automated functions while maintaining surgeon oversight and responsibility for intraoperative decision-making.

Visual feedback-driven autonomous tracking

Traditional laparoscopic surgery relies heavily on tacit cooperation between the primary surgeon and the camera operator. However, prolonged manual camera handling frequently induces visual and muscular fatigue, subsequently causing significant degradation in the accuracy and stability of the surgical field of view[10,61]. To address this clinical challenge, CV-based instrument tracking and 3D pose estimation algorithms allow the robotic system to achieve automatic following and adaptive zooming of the endoscopic view based on real-time spatial coordinates and motion trajectories of the intraoperative instruments[10,61-63]. These systems confirm the technical feasibility of autonomous camera positioning and visualization adjustment, thereby demonstrating the potential to reduce reliance on manual camera holders. However, the current empirical evidence has not yet established significant clinical benefits in mitigating operator fatigue, communication errors, operative time, and complication rates.

3D spatial dynamic safety “no-fly zones”

Combined with multimodal anatomical segmentation technology, the CV system can construct high-precision 3D maps in real time during surgery and establish dynamic restricted areas (or safety “no-fly zones”) for high-risk anatomical structures such as major blood vessels and critical nerves[64]. In experimental active-constraint systems, instrument motion approaching these boundaries can be restricted through mechanisms such as haptic feedback or active braking, thereby providing a technical framework for reducing unsafe instrument-tissue interactions[64]. Such systems provide a conceptual and technical framework for robotic safety. However, evidence that they reduce iatrogenic injury in clinical gastrointestinal surgery remains unavailable.

Autonomous execution

Relying on reinforcement learning and imitation learning, combined with CV’s precise multidimensional perception of sutures, needle tip spatial postures, and tissue tension, recent robotic platforms have initially achieved semi-autonomous knot tying and intestinal anastomotic suturing in specifically constrained environments[65]. This signifies that intelligent surgery is evolving along a pathway from visual navigation to autonomous execution. However, whether a fully autonomous robotic surgery in gastrointestinal practice represents a viable near-term pathway or remains an aspirational vision is still a subject of intense debate[66]. Advancing toward high-level or fully autonomous operations faces multifaceted barriers. First, the real intra-abdominal environment is highly dynamic and unpredictable. Severe deformations of soft tissues and sudden occlusions of the surgical field can easily cause CV algorithms to instantaneously lose track of targets, thereby triggering erratic operations of purely vision-driven robotic arms[3,55]. Second, intricate dissection and anastomosis rely heavily on precise haptic feedback. At present, the integration of vision and haptics remains immature, and visual “pseudo-haptic” estimation cannot fully replace true mechanical sensing, so the system cannot make the real-time adjustments that experienced surgeons perform[60].

Although CV has achieved substantial breakthroughs in image segmentation and recognition, its integration into robotic platforms remains at the nascent stage. Constrained by technological and ethical challenges, current intelligent robotic technologies are predominantly reflected in prospective studies and early proof-of-concept validations and have yet to establish a large-scale quantitative evaluation framework.

Clinical translation of CV performance

Although technical metrics, such as the DSC, F1 score, mean average precision (mAP), and processing latency, are important for evaluating CV performance, they do not directly indicate clinical effectiveness. Most current studies focus on technical feasibility, whereas evidence based on surgeon-, workflow-, procedure-, and patient-centered outcomes remains limited. Some studies have begun to evaluate clinically relevant measures, including procedural interruptions and surgeon-related outcomes, earlier anatomical recognition, multicenter external validation, real-world deployment, and prospective clinical outcomes in ongoing trials such as ImNavi[23,35,40,67]. Overall, the field has advanced rapidly in terms of technical capabilities; however, further prospective and comparative studies are required to determine whether these improvements translate into meaningful benefits for surgical safety, workflow, and patient outcomes.

CHALLENGES AND VALIDATION IN CLINICAL TRANSLATION

Although CV and AI technologies have demonstrated high recognition accuracy and efficacy in retrospective studies, integrating them into routine clinical workflows requires the validation of their clinical reliability, and numerous challenges remain.

Algorithm generalization capability and multicenter validation

Meta-analyses indicate that most existing studies rely heavily on small-sample single-center datasets[6]. This data homogeneity frequently causes severe degradation in the performance of deep learning models when confronted with variations in surgical equipment across different medical institutions, diverse operator habits, and varied patient anatomies. Moreover, as indicated by the distribution of translational maturity in Table 1, the vast majority of current research remains confined to the purely technical evaluation stage, relying on algorithmic metrics, such as the DSC, whereas high-tier clinical evidence demonstrating prospective feasibility and comparative effectiveness remains scarce. Real-world studies should be conducted to substantiate the true clinical value. Recent studies have attempted to develop AI navigation systems for laparoscopic gastrectomy by incorporating multicenter external validation and preliminary deployment in real-world operating rooms. Only by accumulating robust evidence-based medical data through large-scale randomized controlled trials, such as the ImNavi trial, supplemented by targeted fine-tuning strategies can the generalization capability of algorithms be improved to construct robust CV-assisted technologies[6,35,68].

Table 1 Existing applications of computer vision in laparoscopic gastrointestinal surgery.

CV task	Target	AI approach	Dataset and sample characteristic	Study design and validation	Key performance	Clinical relevance	Translational maturity
Surgical phase recognition[17]	Laparoscopic gastric cancer	VTR-Net	Single-center dataset; 100 surgical videos from 100 patients	Retrospective study (80-case training/20-case test split); internal validation	Frame-level phase recognition: Accuracy, 90.1%; F1 score, 87.0%; precision, 87.31%	Workflow recognition and automated documentation; clinical benefit not yet established	T1
Vessel segmentation[26]	IMA in colorectal surgery	DeepLab v3+	Multicenter LapSig300; 1200 annotated images from 60 surgical videos	Retrospective study; case-level 5-fold cross-validation; internal validation	Image-level: DSC, 0.798 ± 0.016; inference speed, > 12 FPS	Real-time vascular visualization and localization	T1
Ureter segmentation[32]	Ureter recognition in laparoscopic surgery	UreterNet	14069 annotated images from 304 laparoscopic colorectal surgery videos	Internal static-split validation	Image-level: Precision, 0.712; DSC, 0.716; inference latency, 143 ms with display	Real-time ureter visualization; reduction in ureteral injury not yet demonstrated	T1
Biliary structure segmentation[36]	Laparoscopic cholecystectomy	U-Net++, UNETR, and DeepLabV3+	CholecSeg8k public dataset; 8080 annotated frames from 17 laparoscopic cholecystectomy	Procedure-level data split; internal validation	Frame-level: Dice, 0.62; mean IoU, 0.55; recall, 0.72; precision, 0.73	Anatomical segmentation; clinical impact on bile duct injury remains uncertain	T1
3D scene reconstruction[54]	Abdominal soft tissue	DROID-SLAM	Experimental animal videos; 6 laparoscopic videos, divided into 18 image sequences	Experimental technical validation against optical motion-capture ground truth under static and dynamic lighting conditions	Sequence-level: Mostly sub-centimeter trajectory error in successfully tracked sequences; dynamic-lighting ATE RMSE, 0.0076-0.0167 m	Technical basis for 3D reconstruction, camera tracking, and future AR registration and navigation	T1
AR-assisted registration[56]	Liver portal segments in laparoscopic anatomical liver resection	Detectron2 + 3D slicer + vuforia-based AR	Clinical laparoscopic liver surgery videos; 380 annotated frames; 5 AR evaluation scenes	Technical feasibility study (evaluation on 5 surgical scenes); technical validation	Scene-level: Mean error, 14.5 mm vs 31.2 mm without AI	Assist laparoscopic anatomical liver resection	T1
Functional tissue assessment[59]	Tissue ischemia	MSI+INN-based out-of-distribution detection	Single-center clinical dataset; 10 patients undergoing partial nephrectomy	In-human feasibility study (5 retrospective/5 prospective patients); ICG-referenced validation	Patient-level: Mean AU-ROC, 1.00/0.9; real-time inference, 140 Hz per ROI	Potential contrast-free ischemia assessment	T3
Real-time anatomical instrument segmentation[67]	Organs and instruments in laparoscopic gastrectomy	YOLOv10	Multicenter clinical dataset; 18 surgical videos (6832 annotated frames) from 4 hospitals; 15 patients for intraoperative deployment	Multicenter study; internal validation and external validation	Frame-level: External mAP@50, 0.832; mask mAP@50, 0.836; real-time inference, 31.8 ± 10.7 ms/frame	Real-time intraoperative navigation and usability	T3

CV: Computer vision; DSC: Dice similarity coefficient; ATE: Absolute trajectory error; RMSE: Root mean square error; AI: Artificial intelligence; AR: Augmented reality; AU-ROC: Area under the receiver operating characteristic curve; ROI: Region of interest; IMA: Inferior mesenteric artery; ICG: Indocyanine green; 3D: Three dimensional.

Real-time requirements and the introduction of edge computing

Laparoscopic surgery is a highly dynamic and time-critical process, and any delay in visual processing can lead to intraoperative navigation failure or even trigger serious surgical errors. Consequently, a high real-time processing rate is a key determinant of the clinical applicability of CV technology. Edge computing may help reduce processing latency by enabling local real-time inference within the operating room. This permits algorithms to perform offline, high-speed, and real-time inferences on local equipment within the operating room. A recent AI navigation system for laparoscopic gastrectomy deploys semantic segmentation models locally in the operating room, supporting the feasibility of low-latency real-time deployment across multiple centers[67]. In a single-center study focusing on intraoperative navigation and bleeding warnings, the model achieved a high mAP and generated real-time multimodal warnings consistent with the performance of experienced surgeons within this single-center dataset in a “human-like capability test”[69]. Although these findings demonstrate the technical feasibility of real-time intraoperative AI assistance under the evaluated conditions, they were primarily derived from a specific dataset within a single institution. Confronted with anatomical variations and equipment heterogeneity in clinical practice, such algorithms must overcome the formidable challenge of transitioning from single-center testing to large-scale, multicenter validation.

Data privacy protection

In the process of promoting AI-assisted navigation toward multicenter, large-scale application, the privacy and data security of patients and medical staff, alongside rigorous clinical validation, constitute non-negotiable ethical and scientific baselines[62,68]. Laparoscopic surgical videos often inevitably capture sensitive extra-abdominal frames, such as the operating room environment or the facial features of personnel. To ensure compliance with multicenter data sharing, a newly developed inside-outside discrimination algorithm can precisely identify and eliminate extracorporeal video frames containing privacy risks. Maximizing the retention of valid intra-abdominal surgical data may facilitate a more privacy-conscious construction and sharing of surgical video datasets, although broader governance and data security requirements remain[70].

CONCLUSION

In conclusion, the application of the CV technology in laparoscopic gastrointestinal surgery has significantly progressed. Deep-learning-based semantic segmentation and object detection algorithms have demonstrated considerable application value. They are capable of achieving high-precision, low-latency identification of major abdominal vessels and ureters and have been reported to reduce registration errors in early technical studies.

However, this field is still in the transitional phase of early translation and clinical validation. In complex surgical environments characterized by bleeding, smoke, and anatomical variability, the robustness of current models in recognizing small vessels and variant anatomical structures requires further improvement. Moreover, due to the limited availability of large-scale, multicenter studies, the generalizability of CV systems in real-world operating room settings and their long-term evidence for improving key clinical outcomes have not yet been fully established. Despite the rapid growth of CV research, relatively few studies have directly evaluated clinical translation and patient-relevant outcomes[6].

Looking ahead, CV technologies may gradually facilitate the evolution of laparoscopic surgery from passive visual assistance toward more active intraoperative navigation, potentially contributing to the future development of semi-autonomous robotic surgery[71]. Achieving this transition will require the establishment of large-scale, multicenter, and well-annotated surgical datasets to improve algorithm robustness and generalizability. Privacy-preserving computational approaches, such as federated learning, should also be explored to overcome data silos while ensuring data security and regulatory compliance, thereby promoting multicenter collaboration. At the application level, the integration of AR-based navigation, multimodal imaging, and robotic control technologies may provide a foundation for developing comprehensive intelligent surgical systems that incorporate both anatomical and functional assessment (Table 2). Meanwhile, broader clinical translation will require parallel advancement of technical, regulatory, and ethical frameworks. Clear definitions of responsibility in human-AI collaborative surgery, establishment of appropriate ethical boundaries, and rigorous protection of patient privacy will be essential for the safe and responsible adoption of CV technologies as reliable surgical assistants. Although widespread fully autonomous robotic surgery remains a distant goal requiring substantial technological development and regulatory evaluation, CV-driven surgical data science may play an increasingly important role in advancing gastrointestinal surgery toward greater digitalization, intelligence, and standardization.

Table 2 Major challenges to clinical translation of computer vision in gastrointestinal surgery and potential solutions.

Challenge	Clinical impact	Potential solution	Validation priority
Single-center datasets	Poor generalizability	Multicenter standardized datasets	External validation
Domain shift	Performance degradation	Domain adaptation/federated learning	Cross-center testing
Soft-tissue deformation	Registration failure	Deformable registration and uncertainty estimation	Real-time validation
Latency	Delayed guidance	Edge computing	OR deployment
Privacy	Limits data sharing	Federated/privacy-preserving AI	Governance validation
Automation bias	Unsafe reliance	Human-in-the-loop design	Human-factors studies
Regulatory/liability uncertainty	Deployment barriers	Defined approval/accountability frameworks	Prospective implementation
Lack of clinical evidence	Uncertain patient benefit	Multicenter prospective trials/RCTs	Clinical effectiveness

AI: Artificial intelligence; OR: Operating room; RCTs: Randomized controlled trials.

ACKNOWLEDGEMENTS

We are grateful to the members of our research group for their valuable discussion and suggestions during manuscript preparation. We also appreciate the editors and anonymous reviewers for their constructive comments to improve this review.