Current trends in three-dimensional visualization and real-time navigation as well as robot-assisted technologies in hepatobiliary surgery

Abstract

With the continuous development of digital medicine, minimally invasive precision and safety have become the primary development trends in hepatobiliary surgery. Due to the specificity and complexity of hepatobiliary surgery, traditional preoperative imaging techniques such as computed tomography and magnetic resonance imaging cannot meet the need for identification of fine anatomical regions. Imaging-based three-dimensional (3D) reconstruction, virtual simulation of surgery and 3D printing optimize the surgical plan through preoperative assessment, improving the controllability and safety of intraoperative operations, and in difficult-to-reach areas of the posterior and superior liver, assistive robots reproduce the surgeon’s natural movements with stable cameras, reducing natural vibrations. Electromagnetic navigation in abdominal surgery solves the problem of conventional surgery still relying on direct visual observation or preoperative image assessment. We summarize and compare these recent trends in digital medical solutions for the future development and refinement of digital medicine in hepatobiliary surgery.

Key Words: Hepatobiliary surgery, Three-dimensional visualization, Three-dimensional printing, Electromagnetic tracking, Real-time navigation, Robot-assisted surgery

Core Tip: This paper analyzes the latest trends in three-dimensional visualization, robot-assisted surgery, and electromagnetic intraoperative navigation in hepatobiliary surgery and summarizes the advantages and limitations of existing technologies and potential solution strategies. It also analyzes existing real-time intraoperative navigation, compares optical tracking navigation to electromagnetic tracking navigation with a focus on the advantages and existing limitations, and attempts to improve the program as an educational learning tool for new physicians. Additionally, it aims to popularize hepatobiliary surgery as digital medicine and tries to illustrate a direction for the advancement and development of digital medicine in hepatobiliary surgery.

INTRODUCTION

The safety and effectiveness of hepatobiliary surgery is based on knowledge of the detailed anatomy of the hepatobiliary structures, but the structure of the liver is complex and the vascularity and anatomy of the bile ducts at the hilum are prone to variation[1]. Traditional surgery based on two-dimensional (2D) images to visualize the three-dimensional (3D) spatial relationships of anatomical structures in the mind in order to complete the operation, but it’s a significant challenge for new inexperienced surgeons. 3D visualization, digital imaging and 3D printing can clearly show the 3D spatial relationship of the lesion site, which can help with difficult intrahepatic vein reconstruction and blood supply assessment as well as biliary vein drainage problems, enabling surgeons to better plan their operations and pushing surgery towards precision and minimally invasive surgery. The development of robot-assisted surgery can overcome the disadvantages of traditional laparoscopy in hepatobiliary surgery, such as inadequate depth perception, inevitable hand tremors, and the surgeon's greater susceptibility to fatigue after prolonged surgery, helping surgeons to be more flexible in operating on delicate sites[2,3]. Intraoperative navigation reduces the practical uncertainty of the operation and the deformation and displacement of tissues, and evolving digital medicine is helping surgeons to optimize preoperative planning, perform precise and safe intraoperative procedures and carry out accurate postoperative analysis[4,5].

THREE-DIMENSINAL VISUALIZATION IMAGES AND THEIR EXTENSION

The segmental anatomy of the liver and the anatomy of the blood vessels and bile ducts are diverse and the presence of various anatomical variants requires individualized surgical plans to ensure that the operation is carried out safely. Experienced surgeons can sketch a 3D image in their minds based on preoperative 2D images such as computed tomography (CT) plane magnetic resonance imaging (MRI) to complete the operation successfully, but it is a significant challenge for surgeons new to the profession[6,7]. 3D visualization and 3D printing technologies can clearly show the specific spatial anatomy of a lesion and can help young surgeons optimize their surgical plans, which can be used for liver resection, liver transplantation, radiofrequency ablation, transjugular intrahepatic portosystemic shunts, gallstones, gallbladder cancer and many other diseases[4,8,9]. Especially in the case of malignant liver tumor resection, the application of 3D visualization and 3D printing technology allows for accurate preoperative assessment, simulation and optimization of the surgical plan to ensure that the operation is carried out safely[10].

3D visualization

With the rapid development of digital medicine, 3D visualization images are increasingly used in the diagnosis and treatment of hepatobiliary diseases, and more and more companies are developing 3D visualization software for medical use, such as Liversim, Mint Liver, etc.[11]. A large amount of fine stereoscopic data helps surgeons to clearly identify the anatomical relationship of the lesion before surgery, helping the team to share accurate 3D surgical images[12]. Particularly for surgery on hepatobiliary malignancies, 3D visualization technology also allows for a comprehensive assessment of the vasculature and evaluation of variants, which helps the surgery to unfold safely[13,14].

As shown in Table 1, Miyamoto et al[15] used 3D visualization images to diagnose parabile ducts in patients with cholangiocarcinoma that could not be detected by multilayer spiral CT and magnetic resonance cholangiopancreatography. Zeng et al[16] conducted a retrospective study of patients with type-III hilar cholangiocarcinoma using 3D modelling, demonstrating the safety and efficacy of 3D visualization. Nakayama et al[13] retrospectively analyzed 240 consecutive patients undergoing liver resection and demonstrated the effectiveness of 3D simulation to help surgeons effectively reduce operative time. Lin et al[17] explored the value of 3D visualization in pancreatic resection and validated the effectiveness of 3D visualization images to help surgeons plan surgery.

Table 1 Three-dimensional visualization and robot-assisted surgery in recent years.

	Surgical site	Sample size	Patient type of disease	Imaging systems	Incidence of complications (%)	Summary of technology	Ref.
3D visualization	Bile duct department	1	Extrahepatic cholangiocarcinoma combined with paracolic bile duct	Synapse Vincent	0	Accuracy and reliability	Miyamoto et al[15], 2014
	Hepatic portal	47	Type-III cholangiocarcinoma of the porta hepatis	MI-3DVS		Safety, effectiveness, and feasibility	Zeng et al[16], 2016
	Liver	120	Hepatocellular carcinoma, bile duct cancer, liver transplantation	Synapse Vincent	10.8	Time savings	Nakayama et al[13], 2017
	Pancreas	64	Pancreatic cancer, biliary tract cancer, neuroendocrine tumors, IPMN	Synapse Vincent	14	Safety, effectiveness, and feasibility	Miyamoto et al[100], 2018
	Pancreas	44	Pancreatic cancer	MVT		Safety, effectiveness, and feasibility	Lin et al[17], 2020
Robot-assisted	Major and minor liver resections	40	Hemangioma, HCC, hydatid cyst, cholangiocarcinoma	da Vinci Surgical System	12.5	Safety and feasibility	Troisi et al[37], 2013
	Major liver resection	25	Fatty liver, hepatic hemangioma, giant adenoma, HCC, secondary liver carcinoma	da Vinci Surgical System	9.3	Safety and feasibility	Spampinato et al[33], 2014
	Wedge resection of the liver	20	HCC, secondary liver carcinoma, hepatic hemangioma, liver stones	da Vinci Surgical System	9.5	Safety and feasibility	Felli et al[47], 2015
	Cholecystectomy	38	Benign biliary disease	da Vinci Surgical System	0	Safety and effectiveness	Gustafson et al[51], 2016
	Cholecystectomy	1833	Benign gallbladder disease	da Vinci Surgical System, Zeus system, AESPO	9.3	No superiority over laparoscopy	Han et al[101], 2018
	Major and minor liver resections	1312	Liver tumors	da Vinci Surgical System	17.8	No superiority over laparoscopy	Zhang et al[2], 2020

MVT: A three-dimensional multi-touch visualization table introduced by Sectra in 2010 at the Radiological Society of North America. IPMN: Intraductal papillary mucinous neoplasm; HCC: Hepatocellular carcinoma; 3D: Three-dimensional.

Advantages and limitations of 3D visualization

The development and application of visualization images has changed the paradigm of surgery and can also help inexperienced surgeons to learn with simulation, improved safety, reduced intraoperative risk and to some extent reduced postoperative complications[12,17,18].

However, the current 3D visualization techniques still have some limitations[18]. First, the process of medical image reconstruction mainly includes image data pre-processing, segmentation and annotation, alignment and fusion, 3D reconstruction, visual image display, etc. Each step of the process affects the results of 3D reconstruction, and the quality of the raw data acquired during the process and the different capabilities of the various reconstruction software applications also affect the outcome of 3D reconstruction. Second, although the reconstructed images produced by current visualization software are generally better than the image post-processing software that comes with CT or MRI, they are based on secondary processing of the original CT or MRI images, which inevitably results in partial loss of the original data during the image processing, thus affecting the fineness and clarity of the reconstructed images[18]. Future research should maximize the preservation of raw data, optimize the algorithms of various reconstruction techniques, improve the fidelity of the reconstruction and increase the accuracy of the 3D visualized images. Third, the reconstruction of images is currently time-consuming, taking at least one to two hours, future research could be technically optimized to reduce the reconstruction time[19]. Fourth, soft tissue organs such as the liver surface, intrahepatic structures and the bile duct tree are usually deformed intraoperatively due to changes in position and surgical procedures[20]. Although studies have also described calibration algorithms based on deformed organs, the currently available DIR algorithms still have limitations when dealing with complex deformations including volume changes, and optimization solutions for variable organ alignment remain a difficult area for future research, and further development and testing studies are needed in the future[21].

3D printing

3D printing is an extension and expansion of 3D visualization technology. High-fidelity 3D printed models can realistically reflect the 3D spatial relationships of fine anatomical areas such as lesion sites and blood vessels, allowing for multi-dimensional predictions of surgical procedures before surgery, achieving a leap from 3D images to solid 3D physical models[22,23].

The use of 3D printing in liver surgery has become widespread, and studies have shown good results with negative margins for using this technique in the treatment of small liver cancers[24]. Joo et al[25] applied a 3D-printed transparent liver model. The 3D technique was also applied by Fang et al[26] in surgeries on liver diseases such as intrahepatic bile duct stones and liver malignancies. He et al[27] also applied 3D printing in liver resection and autologous liver transplantation for vesicular encapsulation disease with satisfactory surgical results. Yang et al[28] used HepaRG cells and bioink to construct 3D bioprinted hepatic-like biotin, demonstrating that 3D bioprinting can be used to generate human liver tissue as an alternative transplant donor for therapy.

As shown in Table 2, current 3D printing enables the adjustment and placement of 3D printed models in optimal anatomical positions, facilitating both the placement of surgical instruments and the intuitive real-time navigation of key steps in surgery. It also allows rapid identification and precise positioning of key sites, optimizing the plane of surgical resection, the separation of important vessels and the precise removal of lesions, thereby improving surgical precision and safety and reducing surgical risk[29]. A number of studies have shown that 3D printing can produce implant shapes that precisely match their anatomical characteristics, ensuring that implant surgery is carried out safely[30,31].

Table 2 Advantages and current limitations of existing three-dimensional printing.

Advantages	Limitations
(1) Realistic spatially dissected views	(1) Time-consuming production
(2) Intuitive real-time navigation for rapid identification and location	(2) Rigid model with poor soft tissue compliance
(3) Improved surgical safety	(3) Fragility
(4) Less time consumed and fewer complications	(4) High cost
(5) Novel educational techniques	(5) Issues of specificity, safety, and sustainability of implantable 3D-printed products

3D: Three-dimensional.

Despite these advantages, 3D printing has a number of limitations. First, 3D printing devices take longer to plan and produce, often delays surgery and therefore are unsuitable for emergency surgery. Second, the issue of the material of the model is also a key point to be examined, as the visceral soft tissue organs are deformable and rigid models cannot reproduce the compliance of the tissue[32,33]. Fragile models are also unsuitable for surgery, and certain models cannot be handled by the surgeon during surgery because the particular material cannot be sterilized[34,35]. Third, the design and manufacture of 3D models for transplantation is more challenging, requiring consideration not only of the specificity of soft tissue organs, but also of the safety and sustainability of 3D printed products. Fourth, the high additional cost is also one of the disadvantages of current3D printing that cannot be ignored, of course, it is believed that with the development of bioprinting technology, these issues may be addressed to some extent[34].

ROBOT-ASSISTED HPATOBILIARY SURGERY

Precision and minimally invasive surgery have long been the pursuit of surgical procedures, and with the development of surgical anatomy and perioperative care, enhanced imaging modalities such as 3D visualization, and advances in laparoscopic surgery and robotic devices, minimally invasive surgery is becoming the gold standard in specific areas of gastrointestinal surgery[36-38]. However, the straight instruments of the laparoscope allow only four degrees of freedom, and the surgeon's inevitable physical hand tremors are magnified by the long laparoscopic tube. These factors, combined with the 2D field of view, the narrow space and the lack of depth perception, add to the difficulty of laparoscopic surgery, and prolonged procedures are more likely to lead to surgeon fatigue[2]. The robot-assisted surgical system offers many advantages over laparoscopic surgery, including the filtering out of physiological hand tremors based on simulated surgeon wrist movements, a stable camera platform, a 3D surgical field of view and visual magnification, seven degrees of freedom of dexterity, and reduced surgical fatigue for the surgeon[39].

Operation of robotic surgery: Indications and contraindications

Currently, most robot-assisted minimally invasive surgery is performed using the Da Vinci Si Surgical System telesurgery system, in which the surgeon sits at a console and operates several master robots, with intraoperative manipulation and view capture performed by three robotic instrument arms and one camera arm[40]. The stable platform's 3D field of view and flexible robot arm help surgeons better expose anatomical structures for selective control, dissection, and handling[39]. The robotic platform also enables near-infrared fluorescence imaging using indocyanine green (ICG) to assess tissue perfusion and identify lymphatic structures, distinguishing between healthy liver and tumor tissue[41]. The use of ICG fluorescence imaging also improves the discrimination between biliary tract and vascular structures, facilitating the identification of resection lines and helping the surgeon to maintain an accurate resection plane intraoperatively[42]. The use of these devices together allows for better control of the vascular system and fine structures such as the bile ducts, reducing intraoperative risks and intraoperative complications.

According to the available guidelines, indications for robotic hepatectomy include malignant tumors of the liver such as primary liver cancer, secondary liver cancer, and other rare malignant tumors of the liver, as well as benign diseases including adenomas, cavernous hemangiomas with symptoms or over 10 cm in diameter, focal nodular hyperplasia, cystic diseases such as hepatic echinococcosis, and intrahepatic bile duct stones requiring hepatic resection involving combined organ resection[43]. Indications for machine bile duct resection include intra- and extra-hepatic bile duct stones requiring combined hepatic segmental surgery or lobectomy for gallbladder cancer without abdominal implant metastases or large vessel invasion, type I, II and III cholangiocarcinoma of the porta hepatis, etc.[44-46]. Contraindications for robotic surgery include, in addition to the same contraindications as for open hepatobiliary resection, severe cardiopulmonary disease that does not tolerate pneumoperitoneum, intra-abdominal adhesions that are difficult to separate and reveal the lesion in two or more operations, lesions that are close to or that directly invade large blood vessels, invasion of the hilum, invasion of the portal vein, hepatic artery and other blood vessels, or lesions that require extensive hilar lymph node dissection[32].

Robotic surgery in hepatobiliary surgery

As shown in Table 1, Troisi et al[37] reviewed liver resections in 40 patients, comparing robot-assisted surgery with laparoscopic surgery, where the robotic platform provided some reduction in complications compared to laparoscopic surgery, and in difficult posterior and superior segments, robot-assisted surgery appeared to be more advantageous and confirmed the safety and feasibility of robot-assisted surgery[37]. Spampinato et al[33] conducted a retrospective analysis of the perioperative outcomes of robot-assisted major hepatectomy vs laparoscopic major hepatectomy, which confirmed the safety of robot-assisted surgery. Felli et al[47] demonstrated the safety of robotic surgery through initial experience with 20 consecutive robotic liver resections. Zhang et al[2] conducted a meta-analysis in which robot-assisted surgery had advantages over laparoscopic hepatectomy in major hepatectomy. It has also been shown that the proportion of major resections was higher in the more difficult posterior epigastric group than in the laparoscopic group, and that surgeons subjectively preferred robot-assisted surgery[3]. Kamiński et al[48] compared laparoscopic cholecystectomy with robotic cholecystectomy and showed no statistical difference between the two groups in terms of operative time and major bleeding complications, and found that the robotic approach may help in the management of bile duct injuries.

In addition, single-incision robotic cholecystectomy recapitulates the advantages of single-incision surgery, which is based on the same principles as multi-port laparoscopic cholecystectomy and offers the advantages of high definition and stereoscopic vision[49]. It overcomes some of the limitations of conventional laparoscopy through a clear 3D view, redistribution of instruments and optimized engineering design, making it safe and feasible to operate on different gallbladder lesions[49-51]. Gustafson et al[51] compared 38 laparoscopic procedures with 44 robotic single-incision cholecystectomies and found no significant differences between the two groups in terms of either transit rate, length of stay, incidence of incisional hernias requiring repair, or intraoperative and postoperative complications.

Advantages

With the growing trend towards minimally invasive surgery continues to develop, robot-assisted surgery is increasingly being used in hepatobiliary surgery, where it offers potential advantages over other techniques, and studies have shown its advantages in facilitating bile duct reconstruction and vascular anastomosis, and large hepatectomy, and resection of lesions located in highly complex areas[52-54].

First, robotic-assisted technology has more precise resolution, greater magnification, smaller instruments and greater mobility, making it more advantageous in delicate areas such as the liver portal[3], studies have shown that robotic surgery can reduce abdominal wall trauma and improve post-operative diaphragm function, thereby reducing respiratory complications, among other things. Second, the robotic system reproduces the surgeon's natural movements through a steady camera, reducing surgeon fatigue and filtering out physiological tremors, improving precision, accuracy and safety in surgery[43]. Third, the flexible robotic arm can help surgeons perform more precise and safer dissections and sutures, especially in the event of acute bleeding, and the resting position of the robotic arm to stop bleeding allows for safer transfer of open surgery[47]. Previous studies have also shown that intraoperative blood loss is reduced in robotic surgery compared to traditional laparoscopic or open techniques[52]. Fourth, improved venous drainage and reduced bile duct injury are both potential advantages of robotic surgery, which can reduce postoperative pain and complications such as ascites bile duct injury in cirrhotic patients and effectively improve their postoperative quality of life[55]. Fifth, robotic surgery can be used in conjunction with fluoroscopic techniques, with the robotic console providing fluoroscopic cholangiograms that are more conducive to a safe procedure[49].

In recent years, the use of two important phases of minimally invasive hepatectomy — hilar resection and hepatic cavity resection — has improved with the spread of robotic surgery and surgeons’ increased level of experience[49]. As surgeons gain experience, the learning curve for robotic surgical approaches is likely to decrease[56]. In addition, as robotic surgery and open surgery share a common skill principle, even new surgeons with less experience may have a shorter learning curve on the operating table and a correspondingly shorter operating time[57].

Limitations

Current robotic surgery is not mature and still has many limitations. First, compared to laparoscopic techniques, robotic surgery is not as resource efficient, as robotic surgery lacks compression options to control acute bleeding, it usually requires at least two experienced hepatobiliary surgeons to interact with coordination at the console and around the patient for safety reasons. In this regard, there is a need for a technical solution for simpler and faster instrument changes that can be performed independently by the surgeon at the console, thus increasing the efficiency of surgical resources[3]. Second, tactile sensitivity is also one of the primary issues facing surgeons, as the robotic arm has no tactile feedback, in order to avoid tissue damage, the instruments should always be in the surgeon's field of view as blind movements of the instruments can cause damage to surrounding organs and structures, so robotic surgery requires higher quality intraoperative images[47]. Okuda et al[58] have developed new forceps with force sensors that can analyze the gripping force generated by forceps during laparoscopic surgery and display it graphically on a laptop display, providing real-time feedback to the surgical staff. Experiments have shown that this measurement is accurate and feasible and that this new device with force sensors will also provide real-world feedback during endoscopic surgery, providing practical haptic feedback to aid robot-assisted surgical systems (such as the da Vinci Surgical System) is expected to overcome the lack of haptic feedback in robotic surgery. Third, the choice of anatomical approach is one of the limitations of current robotic surgery. Although the bipolar-based "vascular closure" has multiple degrees of freedom, their branches are too wide for precise and substantial dissection, and the longer time required to change instruments and applicators in robotic surgery compared to open and laparoscopic liver surgery also contributes to the longer operative times, and in these areas there is still a need for some technical adjustments to be made[36,59]. The restricted placement of casing needles is also an issue of concern and solution. Ideally, for optimal setup in the cross-section, four 8 mm robotic trocar needles should be placed in a hypothetical straight line at a distance of approximately 7 cm from each other; however, for setup of the upper segment, especially for severe underlying lesions such as large steatotic livers, trocar needle placement may be limited, increasing the postoperative complications of robotic surgery[60]. Fourth, the long operating time remains a drawback of robotic surgery as the preoperative assembly of the robotic system is very time-consuming[2]. The learning curve for robotic surgery inevitably leads to some increased operative times and the need for resident involvement in all procedures, although the learning curve for robots appears to be faster than for laparoscopy, training in advanced laparoscopic techniques is still required before starting robotic hepatobiliary surgery[60]. Reports of robotic hepatectomy at this stage may be somewhat selective and there may be serious adverse events that are not published. The next step is also the need for standardized training in robotic surgery, such as dedicated robotic surgery training using virtual reality training tables or robotic dual consoles, which is the basis for establishing a successful robotic surgery[61,62]. One of the problems with robotic surgery is its high cost, as many hospitals cannot afford this new technology due to the high cost of purchasing and maintaining robots, but in recent years, as surgeons have gained experience, operating times have been reduced and patient lengths of stay have become less decisive in terms of cost[56]. Despite these reports, more prospective randomized studies are needed to assess the true costs of robotic surgery in different procedures, combining robotic surgery with accelerated recovery and perioperative care could theoretically significantly reduce the patient’s length of stay and therefore offset the existing high costs[3]. In addition, there is a lack of communication between clinicians and those developing the technology. Clinicians should communicate fully with technicians to inform them of their needs and the advantages and disadvantages of the existing technology so that they can target improvements to facilitate continuous technological progress, optimization of image processing, develop new computer interfaces to facilitate interfacing, and even add modules with sensory haptics to overcome the lack of tactile feedback and assess pathology based on accurate 3D reconstruction, etc.

REAL-TIME NAVIGATION

Accurate surgical navigation, which can better guide surgeons and improve surgical safety, has received widespread attention with the development of computer science and imaging technology. Surgical navigation refers to the use of medical imaging equipment and computer image processing methods to visualize the patient's preoperative multimodal image data before surgery, to precisely match the patient's anatomy during surgery using rapid alignment procedures, and to obtain and display the position of surgical instruments in space in real time using a 3D positioning system[63-65].

The accuracy of the tracking technique is an important basis for the reliability of the navigation procedure, and the accuracy of the tracking system largely reflects the quality and performance of the surgical navigation system. To date, optical tracking system (OTS) and electromagnetic tracking system (EMT) are the two main tracking techniques used in surgical navigation. Table 3 compares some of the basic characteristics of OTS and EMT.

Table 3 Comparison of optical and electromagnetic tracking navigation.

Item	Optical tracking	Electromagnetic tracking
Tracking accuracy	High	Low
Robustness relative to environmental conditions	High	Low
Visible line of sight	Need for	No need for
Tracking of rigid objects	Suitable for	Unsuitable for
Electromagnetic field	No need for	Need for
Interference from magnetic field	Nothing	Notable
Common uses in the surgeries:
Neurosurgery	+
Orthopedic	+
Endoscopic abdominal		+

The OTS is used to locate visual markers by means of a camera. Its high tracking accuracy and robustness are widely used to estimate the position of surgical tools relative to the target area, with great accuracy and tracking volume, but its main limitation is that a visible line of sight between the intraoperative marker site and the camera is required. Without a line of sight, optical tracking cannot be achieved, and the tip of the knife is usually the location to be tracked and typically needs to be placed near the end of a rigid instrument. As only rigid instruments can be used due to the possibility of tip shift of the tracker, the use of optical tracking is limited, so optical navigation systems are mainly used to track rigid objects, for example in orthopedic surgery[66,67].

The EMT uses a known magnetic field geometry to determine the attitude of the sensor measuring the magnetic flux or field to achieve attitude measurement and dynamic tracking of the target, with the advantages of real-time positioning, high accuracy and no fear of obstruction[68]. EMT provides a solution for precise positioning when line of sight cannot be established, enabling small electromagnetic (EM) sensors to be positioned independently of line of sight in a given EM field, facilitating fast and accurate tracking[66], this avoids the limitations of line of sight establishment problems, and the small size of the sensor allows it to be embedded in the tip of the surgical instrument, reducing tracking errors caused by the large distance between the sensor and the tip of the positioning instrument[66]. Therefore, EM surgical navigation systems are commonly used in endoscopic surgery and abdominal surgery[69].

Real-time navigation and the applications mediated by EM tracking

The implementation of EM tracking-mediated real-time navigation consists of three important steps: (1) Preoperative acquisition of 3D visualization images of organ tissues; (2) Alignment of the virtual 3D visualization images with the real intraoperative images using real-time EM tracking technology and tracking and matching of the virtual 3D images with the changing real images; and (3) Overlay of the virtual images with the real images through augmented reality so that the real environment and virtual images are superimposed on the same screen in real time. As shown in Figure 1, these superimposed virtual images materialize and visualize the intraoperative hepatobiliary structures, helping the surgeon to better judge their spatial relationships and thus making the operation run more smoothly[70-72]. Augmented reality allows the 3D visualization of the hepatobiliary model to be projected onto the surgical area for precise alignment of the coverage area, avoiding hand-eye coordination problems for the surgeon in traditional laparoscopic surgery[73].

Figure 1 Intraoperative schematic of the electromagnetic tracking procedure. It shows a schematic diagram based on electromagnetic tracking navigation under a developing work by the Ohkohchi team which is used to track the position of the micro electro mechanical system (MEMS) within the magnetic field in real time without the need for line of sight and send the real-time information to a computer workstation, fuse the real-time intraoperative actual procedure and visual images with the preoperative computed tomography or magnetic resonance imaging pictures to form a three-dimensional reconstruction image, and display the real intraoperative actual procedure and visual images and the corresponding reconstruction images side-by-side on a TV monitor to achieve real-time navigation of the surgical site (this is the project of “Development of Real-time Navigation System for Laparoscopic Hepatectomy”, University of Tsukuba, Japan, 2017.4-2020.3).

A real-time ultrasound and preoperative CT or MRI image fusion system has been developed in recent years to construct preoperative CT or MRI image datasets as tomographic images and fuse them with real-time acquired ultrasound images with high precision and dynamics[1]. Although this method of navigation is feasible, the inevitable problem of poor accuracy when tracking and locating ultrasound is due to the effect of temperature and air displacement on ultrasonic positioning.

Today, as EM navigation procedures continue to evolve, a number of manufacturers have developed different stand-alone EM tracking systems for medical applications, with the main commercial EM tracking devices currently used in clinical applications being the NDI Aurora (NDI Medical, Canada), the Polhemus Fastack (Polhemus, Canada), and the Ascension MiniBIRD (Ascension Technologies, United States)[74].

Song et al[75] proposed a magnetic tracking-based planar shape-sensing and navigation system for a flexible surgical robot applied to transoral surgery. The permanent magnets were mounted at the distal end of the robot to provide 3D localization and 2D orientation estimation, so there was no need to mount the sensors on the robot. Navigation validation on an experimental platform showed that the approach was feasible and can work in the surgical environment, despite localization errors within the tracking system and the robot[75].

Kok et al[76] evaluated the feasibility and safety of an internally developed EM navigation system for real-time rectal tumor tracking using the NDI Aurora V2 EM tracking system (Northern Digital Inc, Waterloo, Ontario, Canada), employing a patient tracker with an EM sensor (Philips Traxtal/Percunav, Philips, Best, the Netherlands) patient tracker to determine the patient's position during surgery and to place tracking sensors on the tumor to adjust for real-time tumor motion, providing continuous interpretable navigation data for rectal surgery, this prospective study demonstrates that real-time tumor tracking with EM navigation is feasible, safe and accurate and provides direction for wider clinical implementation and contributes to further research to improve workflow and demonstrate clinical benefit[76].

The Ohkochi team at the University of Tsukuba, Japan, in collaboration with LEXI at University of Tokyo, have developed a new forceps with a powerful sensor that connects a micro electromechanical systems triaxial pressure sensor to the forceps tip to measure the pressure exerted by an endoscopic surgical forceps, the gripping force generated by the forceps with the pressure sensor during laparoscopic surgery was measured and analyzed in real time using quantitative data with temperature-compensated triaxial forces displayed graphically on a laptop computer display, providing real-time feedback to the surgical staff on pressure changes due to complex movements, the results show that this measurement is accurate and feasible, and this is the first study to report on the measurement of complex movements during actual surgery, Okuda et al[58] are working on the development of a position sensor system by combining it with a pressure-sensing system, when the data obtained from the device with the pressure sensor is combined with the real-time navigation system, it can display the magnitude of the grip force based on the information provided about the position of the operating site, helping the surgeon to control the intraoperative operation when the pressure is too high and causes damage to the internal soft tissue organs. This will be a breakthrough in traditional navigation surgery, overcoming the lack of tactile feedback from existing navigation[58].

Advantages

Real-time navigation based on EM tracking offers the possibility of navigation in minimally invasive abdominal surgery without the line-of-sight interference problems of optical systems. It provides real-time accurate spatial 3D measurements in the presence of obstruction, allowing real-time unobstructed tracking of miniaturized sensors embedded in surgical tools, probes, needles, guidewires and catheters, which can even be placed at the tips of flexible machinery, helping surgeons to achieve real-time precise navigation of the surgical area and improve the safety of the procedure[66].

EM tracking in surgical navigation provides a non-invasive, radiation-free way to navigate intraoperatively in real-time without any invasive procedures such as portal venipuncture or hepatic dissection, showing the fine anatomy of the lesion in real time, combines flexibly with surgical instruments, solves the surgeon's hand-eye coordination problem, and improves the accuracy and controllability of surgical navigation[77]. In addition, real-time navigation allows the intraoperative surgical team to share intraoperative information to ensure that the operation is carried out safely.

Limitations

First, EMT are highly sensitive to EM interference and magnetic field distortions[66]. Second, existing EM tracking systems do not provide for accurate position tracking at longer distances from the source, some current studies have confirmed that the stability of EM navigation systems needs further improvement, these systems can operate with a limited amount of tracking but their accuracy decreases as the distance between the transmitter and receiver increases, the accuracy of AR navigation decreases when the EM sensor is far from the magnetic field generator and it is difficult to have systems that can track small sensors with a volume greater than 1 cubic metre[78]. Third, in addition to technical shortcomings, EM tracking technology lacks environmental robustness and accuracy compared to optical tracking navigation, and the robustness of EM tracking can be a problem in some environments, so all systems need to be carefully evaluated in clinical practice[79]. The development of customized systems for different environments and applications may offer some solutions for increasing the robustness of EM tracking technology[66]. Fourth, although the ideal navigation system is easy to use for those unfamiliar with intrahepatic anatomy, current navigation systems sometimes require manual intraoperative adjustment, which takes time and requires an in-depth knowledge of hepatobiliary anatomy, and therefore still requires the surgeon to be very familiar with the anatomy in order to ensure a smooth operation[80]. Fifthly, the issue of alignment in real-time surgery has always been a challenge[70]. Sixth, the time-consuming problem of superimposing reconstructed images onto real-time intraoperative images is also a current technical challenge[81]. The construction of superimposed images is still time-consuming and labor-intensive in routine use, and although currently available simulation software programs have reduced surgery time by up to one hour, skilled surgeons still need three to four hours to construct overlay images[82]. There is therefore an urgent need to develop new techniques to reduce the time taken to superimpose images, and in the future it is also hoped that technicians will be able to provide more information on pathological or biological conditions in addition to the superimposed images to enrich the usefulness of the navigation system. Seventh, in terms of image display technology, although various methods are used in navigational surgery, such as monitor-based video fluoroscopic and projection-based systems, there are still problems to overcome such as limited resolution, overlapping distorted images and cumbersome operation[83]. Eighth, the cost of navigation equipment is relatively high and it is believed that as the price of equipment decreases it will be able to drive more hospitals to perform procedures with real-time navigation and more surgeons to participate. However, many clinicians are not aware of the advances in augmented reality technology, so there needs to be a full exchange of information and communication of needs between clinicians and technicians in the clinical setting to develop technology that meets clinicians' expectations, which will help create new inventions and facilitate the advancement and development of navigation[84].

Problem analysis and anticipation of improvements

The accuracy and distortion of EM tracking has been a central issue of research. The accuracy of EM tracking is affected by a variety of factors, and existing EM tracking systems have multiple sources of error, physical laws, design limitations, and manufacturing imperfections or environmental noise can all lead to positioning errors. The intraoperative alignment of deformed organs is also another challenge in navigation technology due to the effects of intraoperative manipulation and respiratory activity, and the clarity and resolution of reconstructed images based on real-time intraoperative images is also of concern to researchers. Despite some attempts to compensate for the tracking, alignment and reconstruction of images, there are still some issues to be resolved[85,86].

Accuracy and distortion: In EMT, errors can be classified as (1) inherent system errors, (2) field distortion errors, and (3) motion-induced errors. Inherent system errors are static errors that can occur when the sensor is placed at a fixed point or when the system is updated; distortion errors refer to disturbances in the secondary and unwanted magnetic fields which can be caused by eddy currents induced by ferromagnetic or conductive materials or by external currents, and they can also originate from the FG field generator and sensor design; motion-induced errors can be caused by changes in the speed of the sensor and the environment during the measurement[79,87].

Upgrading the system to avoid eddy currents and performing a system calibration function can help to some extent with inherent system errors[79,88]. Static pre-calibration processes are cumbersome and ineffective for most dynamic clinical procedures, and often require too many EM sensors to compensate for field distortion in dynamic environments, making them inefficient. A fusion-based approach has also been applied that combines measurements from multiple redundant EM sensors with the motion model of the instrument being tracked, which uses both localization and mapping (SLAM) algorithms to create field distortion maps and compensate for EM tracking errors in real time, however, it requires a large surgical space to complement the tracking technique or an excessive number of redundant sensors, and increases time of calibration. Too many devices also have an impact on the surgeon's surgical space, and their computational complexity, convergence and performance in dynamic environments and spaces still need to be considered by technicians in the future[89].

Alignment errors: The main problem with navigational surgery in hepatobiliary surgery are the accuracy, complexity and time-consuming nature of alignment. First, the EM transmitter should be placed as close as possible to the operating table to avoid interference with alignment accuracy caused by longer distances, second, a suitable probe point should be selected. For transabdominal scans, the ideal location for the probe point is below the glabella, whereas for intraoperative scans, the probe point should be set on the surface of the liver, preferably in the easily recognizable round ligament at the inferior edge of the fissure, the tip of the main portal vein near the tumor can also be used as an important intraoperative landmark, as can the branching vessels near the tumor for precise intraoperative adjustment[90]. In addition, intraoperative control of ventilation or reduction of tidal volume can reduce respiratory-related alignment errors to some extent, and deformed livers can also be monitored in real time using respiratory gating techniques to compensate for errors in the tracking position of EM sensors[91]. It is also necessary to calibrate the camera and the spatial relationship between the camera lens and the solenoid and to manually verify the reference boundary markers, all of the aforementioned techniques can help to improve the accuracy of the alignment[92].

Image reconstruction has also been investigated using surface data obtained from a flexible liver model that simulates deformation, and this data was then used to construct a sample library to predict liver displacement and deformation in alignment, including changes in the shape and internal relative position of the internal structures of the liver[93,94]. However, due to the movement of the diaphragm during breathing and the pulling of instruments can lead to changes in the position and shape of the liver and the occurrence of biliary tract deformities, this leads to incorrect positioning in the navigation system[90]. Although interactive and automated alignment systems have been developed that allow for periodically repeated real-time image acquisition to accommodate alignment difficulties caused by liver deformation and displacement, these systems require a hybrid operating room with CT or MRI equipment and have not been performed in human hepatobiliary surgery[95]. There are also reports of proposed 3D dense surface reconstruction algorithms that can localize hidden structures in intestinal surgery and gallbladder surgery, as well as enhanced block mapping algorithms and reimage mapping techniques that facilitate the implementation of dynamic alignment and aid in alignment studies of variable organs, although there are reports of these studies using existing engineering techniques and mathematical algorithms to solve organ deformation problems, the required methods and algorithms are complex and still need to be simplified and optimized[95].

Superimposed images: Reconstructed images can be displayed in a variety of ways, either video-based or projection-based. Video-based reconstructed image display is commonly used for laparoscopic, robotic and endoscopic procedures[57]. The external video monitor displays the actual surgical scene, and the virtual 3D reconstructed image in the video has poor resolution, requiring tracking and correction of multiple anatomical structures to compensate for changes in the surgeon's field of view and changes in the projected image due to changes in the curvature of the surface of the organ being tracked, this adds to the complexity of constructing the image[1,96,97]. The projection-based reconstructed image also interferes with the surgeon's depth perception, as the image is disturbed and lost when the projector's beam is interrupted by the surgeon's body or robotic arm, and the constructed image is distorted when the beam is not projected on a flat area[98,99]. The development of 3D future holographic projection technology may address the issues of overlapping image interference and diminished depth perception, thereby improving projection-based displays in intraoperative navigation[70]. There is also a transparent display in use that reflects the image in a translucent mirror, allowing the surgeon to view the reconstructed image while also looking directly into the surgical field. It does not require additional video compositing, making it more convenient than conventional video displays and avoids the problem of distortion of the projected image due to changes in the curvature of the object's surface, in addition it does not require special glasses or sensing devices, future research will require improved transparent display methods and more advanced naked eye 3D to provide doctors with a more accurate display of spatial images[29].

CONCLUSION

In Table 4, we summarize the advantages and existing limitations of the latest trends in existing digital healthcare such as 3D visualization of images as well as robot-assisted surgery and real-time EM-based intraoperative navigation. Visualization techniques are more widely used in clinical practice, providing a 3D view of the lesion area and clearer spatial anatomical relationships through the preoperative sharing of accurate 3D surgical images. By creating conditions for complex and precise procedures, such techniques also help surgeons to optimize their surgical plans before surgery and to carry out preoperative simulations through software, which not only reduces surgery time but also reduces intraoperative risks and postoperative complications, improves patient prognosis, and can be used as a new teaching technique for new doctors. However, it is still time consuming and costly to plan and produce 3D models, and rigid models do not reproduce the compliance of soft tissues, implantable organs, and the specificity of 3D-printed products. The specificity, safety, and sustainability of 3D-printed products remain to be addressed. Robotic surgery, which is more minimally invasive than traditional laparoscopic or open surgery, with smaller instruments and a greater degree of motion, a clearer 3D field of view, more precise resolution, and greater magnification. Additionally, it offers filtering out of natural tremors, better ergonomics for the operator, the advantage of highly complex site resections, and improved venous drainage to reduce postoperative complications and help improve patients' quality of life. However, at this stage, robotic surgery is not mature and still has many limitations. Current limitations of robotic surgery include inefficient surgical resources, lack of tactile feedback, limited choice of anatomical approach, limitations in trocar placement, excessive operative time, long assembly time of the robotic system, time-consuming docking procedures, potential tendency to prolong pulmonary portal Pringle surgery, potential bleeding from the clamp squeeze technique, and high costs. Real-time navigation based on EM tracking has the advantage of not requiring any invasive operations and is not limited by line of sight, allowing for real-time intraoperative tracking and navigation, sharing of intraoperative information in real time, display of intraoperative fine anatomy, identification of lesions that cannot be detected by the naked eye, and lessening of hand-eye coordination issues during laparoscopic surgery. The development of sensors is expected to improve the accuracy of navigation for the safe unfolding of hepatobiliary surgery, but at this stage there are also problems with EM navigation systems that are not very stable, as well as low tracking accuracy, poor robustness to environmental conditions, magnetic field interference and tracking errors, poor navigation accuracy, a time-consuming reconstructed image superimposition process, low resolution of reconstructed images, large distortion, and intraoperative variable organ alignment problems. There are problems to be solved, and insufficient information exchange between the technician and the clinician remains problematic.

Table 4 Advantages and limitations of three-dimensional visualization, robot-assisted surgery, and electromagnetic tracking navigation.

	Advantages	Limitations
3D visualization	Realistic spatially dissected views	Complex and time-consuming reconstruction process
	Accurate 3D preoperative images	Possible loss of raw data due to operational errors
	Possibility of complicated surgery	Distortion in reconstructed images
	Optimization of preoperative assessment	Poor accuracy of reconstructed images
	Time-saving simulation	Complex algorithms and imperfect display techniques
	Less time consumed and fewer complications	Registration of mutable organs
	Novel educational techniques	High cost
Robot-assisted	Better micro-invasiveness	Inefficient surgical resources
	Smaller equipment for wider scope	Lack of tactile feedback
	Larger and clearer 3D views	Limitations in the choice of anatomical methods
	Micro-invasiveness	Restrictions on the placement of casing needles
	Improved venous drainage	Time-consuming operation
	More accurate resolution and greater magnification	Prolonged Pringle operation in the hilar region
	Filtering of natural tremor	Potential bleeding tendency of the clamping and squeezing technique
	Better ergonomics of the operator	High cost
Electromagnetic tracking real-time navigation	No requirement for any other invasive operations	Magnetic field interference and tracking errors
	No line of sight restrictions	Low tracking accuracy and robustness relative to environmental conditions
	Real-time intraoperative tracking and navigation	Low stability of electromagnetic navigation system
	Display of intraoperative fine anatomy	High cost
	Improved safety of surgical operations	Registration of mutable organs
	Identification of lesions that are not visually detectable	Accuracy of navigation issues
	Simultaneous sharing of intraoperative information	Time-consuming reconstruction image overlay
	Increased hand-eye coordination for doctors	Low resolution and distortion of the reconstructed image
		Insufficient communication between technicians and surgeons
		Tedious operation

3D: Three-dimensional.

It is noteworthy that in previous studies we have found that surgeons tend to focus on the surgical procedure to the neglect of post-operative care. The final healing after surgery is the result of a combination of factors such as the quality of surgery, intraoperative blood loss, the size of the resected lesion, the patient's underlying preoperative disease or comorbidities, and the patient's physical condition. The concept of Enhanced Recovery After Surgery was first developed by Danish surgeon Henrik Kehlet, based on the principles of reducing the stress of surgery, shortening the length of hospital stay and reducing perioperative complications, leading to rapid recovery. The concept is also considered to be a safe and effective treatment combining existing surgical options with accelerated recovery perioperative care. This could theoretically significantly reduce the length of a patient's hospital stay, which could to some extent offset the existing high costs. The price of various technologies — be it robotic surgery, 3D printing or EM navigation tracking — will certainly come down in the future. This will require a concerted effort and adequate communication between the entire healthcare industry, corporate bodies and technicians in order to target technological improvements and facilitate the continued progress of digital healthcare. Despite the opportunities and challenges, digital healthcare is sure to flourish in the future.

ACKNOWLEDGEMENTS

We thank Dr. Kim J of University of Tsukuba Faculty of Medicine for designing the figure image.