Minimal-invasive surgery (MIS) and robotic surgery (RS) offer multiple advantages over open surgery (Vajsbaher et al. in Cogn Syst Res 64:08, 2020). However, the lack of haptic feedback is still a limitation. Surgeons learn to adapt to this lack of haptic feedback using visual cues to make judgements about tissue deformation. Experienced robotic surgeons use the visual interpretation of tissue as a surrogate for tactile feedback. The aim of this review is to identify the visual cues that are consciously or unconsciously used by expert surgeons to manipulate soft tissue safely during Minimally Invasive Surgery (MIS) and Robotic Surgery (RS). We have conducted a comprehensive literature review with papers on visual cue identification and their application in education, as well as skill assessment and surgeon performance measurement with respect to visual feedback. To visualise our results, we provide an overview of the state-of-the-art in the form of a matrix across identified research features, where papers are clustered and grouped in a comparative way. The clustering of the papers showed explicitly that state-of-the-art research does not in particular study the direct effects of visual cues in relation to the manipulation of the tissue and training for that purpose, but is more concentrated on tissue identification. We identified a gap in the literature about the use of visual cues for educational design solutions, that aid the training of soft-tissue manipulation in MIS and in RS. There appears to be a need RS education to make visual cue identification more accessible and set it in the context of manipulation tasks.

Colorectal cancer (CRC) is the third most common cancer worldwide and responsible for approximately 1 million deaths annually. Early screening is essential to increase the chances of survival, and it can also reduce the cost of treatments for healthcare centres. Colonoscopy is the gold standard for CRC screening and treatment, but it has several drawbacks, including difficulty in manoeuvring the device, patient discomfort, and high cost. Soft endorobots, small and compliant devices thatcan reduce the force exerted on the colonic wall, offer a potential solution to these issues. However, controlling these soft robots is challenging due to their deformable materials and the limitations of mathematical models. In this Review, we discuss model-free and model-based approaches for controlling soft robots that can potentially be applied to endorobots for colonoscopy. We highlight the importance of selecting appropriate control methods based on various parameters, such as sensor and actuator solutions. This review aims to contribute to the development of smart control strategies for soft endorobots that can enhance the effectiveness and safety of robotics in colonoscopy. These strategies can be defined based on the available information about the robot and surrounding environment, control demands, mechanical design impact and characterization data based on calibration.

The introduction of robotic-guided procedures in surgical techniques has brought an increase in the accuracy and control of resections. Surgery has evolved as a technique since the development of laparoscopy, which has added to the visualisation of the peritoneal cavity from a different perspective. Multi-armed robot associated with real-time intraoperative imaging devices brings important manoeuvrability and dexterity improvements in certain surgical fields.

The present study is designed to synthesise the development of imaging techniques with a focus on ultrasonography in robotic surgery in the last ten years regarding abdominal surgical interventions.

All studies involved abdominal surgery. Out of the seven studies, two were performed in clinical trials. The other five studies were performed on organs or simulators and attempted to develop a hybrid surgical technique using ultrasonography and robotic surgery. Most studies aim to surgically identify both blood vessels and nerve structures through this combined technique (surgery and imaging).

Ultrasonography is often used in minimally invasive surgical techniques. This adds to the visualisation of blood vessels, the correct identification of tumour margins, and the location of surgical instruments in the tissue. The development of ultrasound technology from 2D to 3D and 4D has brought improvements in minimally invasive and robotic surgical techniques, and it should be further studied to bring surgery to a higher level.

The game of Jenga is a benchmark used for developing innovative manipulation solutions for complex tasks. Indeed, it encourages the study of novel robotics methods to successfully extract blocks from a tower. A Jenga game involves many traits of complex industrial and surgical manipulation tasks, requiring a multi-step strategy, the combination of visual and tactile data, and the highly precise motion of a robotic arm to perform a single block extraction. In this work, we propose a novel, cost-effective architecture for playing Jenga with e.Do, a 6DOF anthropomorphic manipulator manufactured by Comau, a standard depth camera, and an inexpensive monodirectional force sensor. Our solution focuses on a visual-based control strategy to accurately align the end-effector with the desired block, enabling block extraction by pushing. To this aim, we trained an instance segmentation deep learning model on a synthetic custom dataset to segment each piece of the Jenga tower, allowing for visual tracking of the desired block's pose during the motion of the manipulator. We integrated the visual-based strategy with a 1D force sensor to detect whether the block could be safely removed by identifying a force threshold value. Our experimentation shows that our low-cost solution allows e.DO to precisely reach removable blocks and perform up to 14 consecutive extractions in a row.

Virtual care spread rapidly at the outbreak of the COVID-19 pandemic. Restricting in-person contact contributed to reducing the spread of infection and saved lives. However, the benefits of virtual care were not evenly distributed within and across social groups, and existing inequalities became exacerbated for those unable to fully access to, or benefit from virtual services. This "perspective" paper discusses the extent to which challenges in virtual care access and use in the context of COVID-19 follow the Inverse Care Law. The latter stipulates that the availability and quality of health care is inversely proportionate to the level of population health needs. We highlight the inequalities affecting some disadvantaged populations' access to, and use of public and private virtual care, and contrast this with a utopian vision of technology as the "solution to everything". In public and universal health systems, the Inverse Care Law may manifests itself in access issues, capacity, and/or lack of perceived benefit to use digital technologies, as well as in data poverty. For commercial "Direct-To-Consumer" services, all of the above may be encouraged via a consumerist (i.e., profit-oriented) approach, limited and episodic services, or the use of low direct cost platforms. With virtual care rapidly growing, we set out ways forward for policy, practice, and research to ensure virtual care benefits for everyone, which include: (1) pay more attention to "capabilities" supporting access and use of virtual care; (2) consider digital technologies as a basic human right that should be automatically taken into account, not only in health policies, but also in social policies; (3) take more seriously the impact of the digital economy on equity, notably through a greater state involvement in co-constructing "public health value" through innovation; and (4) reconsider the dominant digital innovation research paradigm to better recognize the contexts, factors, and conditions that influence access to and use of virtual care by different groups.

The dominant visual servoing approaches in Minimally Invasive Surgery (MIS) follow single points or adapt the endoscope's field of view based on the surgical tools' distance. These methods rely on point positions with respect to the camera frame to infer a control policy. Deviating from the dominant methods, we formulate a robotic controller that allows for image-based visual servoing that requires neither explicit tool and camera positions nor any explicit image depth information. The proposed method relies on homography-based image registration, which changes the automation paradigm from point-centric towards surgical-scene-centric approach. It simultaneously respects a programmable Remote Center of Motion (RCM). Our approach allows a surgeon to build a graph of desired views, from which, once built, views can be manually selected and automatically servoed to irrespective of robot-patient frame transformation changes. We evaluate our method on an abdominal phantom and provide an open source ROS Moveit integration for use with any serial manipulator. A video is provided.

The objective of this paper is to present a systematic review of existing sensor-based control methodologies for applications that involve direct interaction between humans and robots, in the form of either physical collaboration or safe coexistence. To this end, we first introduce the basic formulation of the sensor-servo problem, and then, present its most common approaches: vision-based, touch-based, audio-based, and distance-based control. Afterwards, we discuss and formalize the methods that integrate heterogeneous sensors at the control level. The surveyed body of literature is classified according to various factors such as: sensor type, sensor integration method, and application domain. Finally, we discuss open problems, potential applications, and future research directions.

This article designs and analyzes a recurrent neural network (RNN) for the visual servoing of a flexible surgical endoscope. The flexible surgical endoscope is based on a commercially available UR5 robot with a flexible endoscope attached as an end-effector. Most of the existing visual servo control frameworks of the robotic endoscopes or robot arms have not considered either the physical limits of the robot or the remote center of motion (RCM) constraints (i.e., the fulcrum effect). To tackle this issue, this article first conducts the kinematic modeling of the flexible robotic endoscope to achieve automation by visual servo control. The kinematic modeling results in a quadratic programming (QP) framework with physical limits and RCM constraints involved, making the UR5 robot applicable to surgical field. To solve the QP problem and accomplish the visual task, an RNN activated by a sign-bi-power activation function (AF) is proposed. The motivation of using the sign-bi-power AF is to enable the RNN to exhibit an accelerated finite-time convergence, which is more preferred in time-critical applications. Theoretically, the finite-time convergence of the RNN is rigorously proved using the Lyapunov theory. Compared with the previous AFs applied to the RNN, theoretical analysis shows that the RNN activated by the sign-bi-power AF delivers an accelerated convergence speed. Comparative validations are performed, showing that the proposed finite-time convergent neural network is effective to achieve visual servoing of the flexible endoscope with physical limits and RCM constraints handled simultaneously.

Pheochromocytomas (PHEOs) are neural crest cell tumors producing catecholamines. PHEOS need to be early diagnosed and adequately managed. Adrenalectomy is the gold standard treatment of these type of tumors. There has been major improvement of surgical technologies with the development of laparoscopic and robotic systems these past several years. We conducted a review of the literature to evaluate the robotic approach for adrenalectomy for patients with PHEO.

Surgical instrument detection is a significant task in computer-aided minimal invasive surgery for providing real-time feedback to physicians, evaluating surgical skills, and developing a training plan for surgeons. In this study, a multi-scale attention single detector is designed for surgical instruments. In the field of object detection, accurate detection of small objects is always a challenging task. We propose an innovative feature fusion technique aimed at small surgical instrument detection. First, the attention map is created from high-level features to act on the low-level features and enrich the semantic information of the low-level features. The original and processed features are then fused by skip connection. Finally, multi-scale feature maps are created to predict fusion features. The experiments on the ATLAS Dione dataset yielded results with a detection time of 0.066 s per frame and a mean average precision of 90.08%. Our proposed feature fusion module can obtain more semantic information for low-level features and significantly enhance the performance of small surgical instrument detection.

Photoacoustic-based visual servoing is a promising technique for surgical tool tip tracking and automated visualization of photoacoustic targets during interventional procedures. However, one outstanding challenge has been the reliability of obtaining segmentations using low-energy light sources that operate within existing laser safety limits.

We developed the first known graphical processing unit (GPU)-based real-time implementation of short-lag spatial coherence (SLSC) beamforming for photoacoustic imaging and applied this real-time algorithm to improve signal segmentation during photoacoustic-based visual servoing with low-energy lasers.

A 1-mm-core-diameter optical fiber was inserted into ex vivo bovine tissue. Photoacoustic-based visual servoing was implemented as the fiber was manually displaced by a translation stage, which provided ground truth measurements of the fiber displacement. GPU-SLSC results were compared with a central processing unit (CPU)-SLSC approach and an amplitude-based delay-and-sum (DAS) beamforming approach. Performance was additionally evaluated with in vivo cardiac data.

The GPU-SLSC implementation achieved frame rates up to 41.2 Hz, representing a factor of 348 speedup when compared with offline CPU-SLSC. In addition, GPU-SLSC successfully recovered low-energy signals (i.e., ≤268  μJ) with mean ± standard deviation of signal-to-noise ratios of 11.2  ±  2.4 (compared with 3.5  ±  0.8 with conventional DAS beamforming). When energies were lower than the safety limit for skin (i.e., 394.6  μJ for 900-nm wavelength laser light), the median and interquartile range (IQR) of visual servoing tracking errors obtained with GPU-SLSC were 0.64 and 0.52 mm, respectively (which were lower than the median and IQR obtained with DAS by 1.39 and 8.45 mm, respectively). GPU-SLSC additionally reduced the percentage of failed segmentations when applied to in vivo cardiac data.

Results are promising for the use of low-energy, miniaturized lasers to perform GPU-SLSC photoacoustic-based visual servoing in the operating room with laser pulse repetition frequencies as high as 41.2 Hz.

The surgeon is not timely in direct control of his field of view. Autonomous laparoscope control can provide appropriate surgical field of view and facilitate the intelligent level of surgical robot system.

This study explores an autonomous laparoscope control framework for semiautonomous surgery. We propose a novel concept that integrates two forms of Remote Center of Motion (RCM) constraint. We also propose a novel safety-RCM model to cope with the collision condition. We modify the image Jacobian matrix to realize RCM constraint. The two models constitute a dual-RCM constraint for robot laparoscope arm. We perform validation of the algorithm with two experiments.

The experimental results show that the RCM position error can be reduced with the dual-RCM constraint. Owing to safety-RCM model, autonomous surgical field of view control could still be realized in case of collision.

The autonomous laparoscope control could be realized with safety-RCM and dual-RCM algorithms.

The purpose of this paper is to develop an autonomous tracking algorithm based on adaptive fusion kinematics method, the autonomous laparoscope control algorithm and adaptive fusion kinematics method are proposed for semi-autonomous surgery, focus on solving the problems of autonomous laparoscope field of view control for surgical robot system. A novel autonomous tracking algorithm is proposed. To realize more robust tracking, an adaptive fusion kinematics method based on fuzzy logic is proposed, the method adaptive associates the kinematics information of surgical robot system and the laparoscope information. The proposed methods are implemented on the laparoscopic minimally invasive surgical robot system which is developed by our laboratory. Two experiments are carried out, the results indicate that the accurate autonomous field of view control is achieved with the addition of laparoscope information, laparoscopic motion frequency is reduced, the methods can avoid the laparoscope continuous motion and ensure the stability of field of view. The proposed methods improve the intelligence level of surgical robot system.

Intelligent surgical robot has great significance to alleviate the fatigue of surgeons. In the minimally invasive surgery robot system, adding intelligent control method to the laparoscope control has great realizability and significance.

Depth independent image Jacobian matrix was modified to make it suitable for laparoscope trocar constraint. We propose a method for intelligent and autonomous adjustment of surgeon's surgical field of view, enabling it to track and predict the motion trajectory of surgical instruments.

The result of experiment shows that the proposed method could realize tracking the surgical instruments and adjusting the surgical field of view autonomously. In case of occlusion, motion trajectory of surgical instruments can be predicted.

The intelligent laparoscope system could improve the intelligent level of surgical robot system. Given providing "a third hand" for the surgeon, the proposed system is a highly improvement for semi-autonomous surgical robot system.

Minimally invasive surgeries rely on laparoscopic camera views to guide the procedure. Traditionally, an expert surgical assistant operates the camera. In some cases, a robotic system is used to help position the camera, but the surgeon is required to direct all movements of the system. Some prior research has focused on developing automated robotic camera control systems, but that work has been limited to rudimentary control schemes due to a lack of understanding of how the camera should be moved for different surgical tasks.

This research used task analysis with a sample of eight expert surgeons to discover and document several salient methods of camera control and their related task contexts.

Desired camera placements and behaviours were established for two common surgical subtasks (suturing and knot tying).

The results can be used to develop better robotic control algorithms that will be more responsive to surgeons' needs. Copyright © 2015 John Wiley & Sons, Ltd.

Visualization of the vast placental vasculature is crucial in fetoscopic laser photocoagulation for twin-to-twin transfusion syndrome treatment. However, vasculature mosaic is challenging due to the fluctuating imaging conditions during fetoscopic surgery.

A scene adaptive feature-based approach for image correspondence in free-hand endoscopic placental video is proposed. It contributes towards existing techniques by introducing a failure detection method based on statistical attributes of the feature distribution, and an updating mechanism that self-tunes parameters to recover from registration failures.

Validations on endoscopic image sequences of a phantom and a monkey placenta are carried out to demonstrate mismatch recovery. In two 100-frame sequences, automatic self-tuned results improved by 8% compared with manual experience-based tuning and a slight 2.5% deterioration against exhaustive tuning (gold standard).

This scene-adaptive image correspondence approach, which is not restricted to a set of generalized parameters, is suitable for applications associated with dynamically changing imaging conditions. Copyright © 2015 John Wiley & Sons, Ltd.

Intraoperative application of tomographic imaging techniques provides a means of visual servoing for objects beneath the surface of organs.

The focus of this survey is on therapeutic and diagnostic medical applications where tomographic imaging is used in visual servoing. To this end, a comprehensive search of the electronic databases was completed for the period 2000-2013.

Existing techniques and products are categorized and studied, based on the imaging modality and their medical applications. This part complements Part I of the survey, which covers visual servoing techniques using endoscopic imaging and direct vision.

The main challenges in using visual servoing based on tomographic images have been identified. 'Supervised automation of medical robotics' is found to be a major trend in this field and ultrasound is the most commonly used tomographic modality for visual servoing.