Current status of deep learning in abdominal image reconstruction

Abstract

Abdominal magnetic resonance imaging (MRI) and computed tomography (CT) are commonly used for disease screening, diagnosis, and treatment guidance. However, abdominal MRI has disadvantages including slow speed and vulnerability to motions, while CT suffers from problems of radiation. It has been reported that deep learning reconstruction can solve such problems while maintaining good image quality. Recently, deep learning-based image reconstruction has become a hot topic in the field of medical imaging. This study reviews the latest research on deep learning reconstruction in abdominal imaging, including the widely used convolutional neural network, generative adversarial network, and recurrent neural network.

Key Words: Abdominal imaging, Reconstruction, Magnetic resonance imaging, Computed tomography, Deep learning

Core Tip: We summarized the current deep learning-based abdominal image reconstruction methods in this review. The deep learning reconstruction methods can solve the issues of slow imaging speed in magnetic resonance imaging and high-dose radiation in computed tomography while maintaining high image quality. Deep learning has a wide range of clinical applications in current abdominal imaging.

INTRODUCTION

The emergence of deep learning has made intelligent image reconstruction a hot topic in the field of medical imaging. The applications of deep learning technology in image reconstruction have the advantages of reduced scan time and improved image quality. Magnetic resonance imaging (MRI) is a critical medical imaging technology with characteristics such as non-invasiveness, non-radiation, and high contrast. However, prolonged scanning time is the main obstacle that restrict the development of MRI technology[1]. Long acquisition time can cause discomfort to the patients and severe artifacts due to the patient's motion. In order to solve this issue, under-sampled k-space data can be acquired by reducing the measuring time during scans, and then an artifact-free image can be obtained through advanced reconstruction. Deep learning reconstruction (DLR) produces high-quality images while reducing scan time and patient discomfort. However, traditional MRI, has problems including low acceleration factor, long calculation time, and variability in parameter selection in the reconstruction algorithm[2]. Deep learning automatically captures high-level features from a large amount of data and builds non-linear mapping between the input and output. Wang et al[3] introduced deep learning into fast MRI reconstruction. The deep learning-based MRI reconstruction avoids the difficulty of parameter adjustment in traditional model-based reconstruction algorithms, which has the potential for a wide range of clinical applications. In addition, deep learning has also been used to solve the problem of abdominal motion. Presently, abdominal MRI reconstruction based on deep learning mainly adopts end-to-end remodeling. The current network structures for MRI reconstruction include the convolutional neural network (CNN)[4], U-net[5], generative adversarial network (GAN)[6], recurrent neural network (RNN)[7], and cascade-net[8].

On the other hand, CT imaging suffers from the problem of radiation. Low-dose CT (LDCT) is achieved by reducing the radiation dose. However, reduced radiation dose decreases the image quality, causing bias in the diagnosis. Therefore, an improved reconstruction algorithm is required for LDCT images. Traditional methods for reconstructing CT images include total variation[9], model-based iterative reconstruction (MBIR)[10], and dictionary learning[11]. However, the performance of LDCT image reconstruction could be improved further by introducing some latest techniques. The emergence of deep learning[12-15] has become the mainstream research of LDCT in recent years.

In this review, we assessed the current status of deep learning in abdominal image reconstruction. Specifically, we reviewed the latest research on deep learning methods in abdominal image reconstruction, attempted to solve the related problems, and address the challenges in this field.

DEEP LEARNING ALGORITHM

The deep learning method is obtained through a simple combination of non-linear layers. Each module can transform the initial low-level features into high-level representation. The core of deep learning is feature representation to obtain information at various levels through network layering. Compared to traditional machine learning algorithms, deep learning improves the accuracy of learning from a large amount of data. Another advantage of deep learning is that it does not require feature engineering. Typically, classic machine learning algorithms require complex feature engineering. Conversely, deep learning algorithms only need to feed data into the network and learn the representation. Finally, the deep learning network is highly adaptable and easily converted into different applications. Transfer learning makes the pre-trained deep networks suitable for similar applications.

At present, several studies have applied deep learning to different aspects of medical imaging, such as image detection[16,17], image segmentation[18,19], image denoising[20,21], super-resolution[22,23], and image reconstruction[3,24,25]. As described above, traditional model-based reconstruction algorithms require manual adjustment of the reconstruction parameters, which results in low reconstruction speed and unstable performance. With the increased acceleration factor, the image quality worsens. The reconstruction method based on deep learning avoids the difficulty of manual parameter adjustment. In the case of high acceleration, DLR can still perform well. After the network model is trained, the image can be reconstructed within seconds.

CNN FOR IMAGE RECONSTRUCTION

MRI

CNN has an excellent performance in image reconstruction[4]. In recent years, a large number of CNN-based abdominal image reconstruction methods have been proposed[26-36]. A major problem in abdominal imaging is the patient's motion, which blurs the image and produces severe artifacts. Breath holding while scanning can minimize these artifacts, but residual artifacts are persistent[37]. Self-gating techniques[38,39] can overcome this problem, but the reconstructed image at a low sampling rate causes additional streaking artifacts. In order to address the problem of free-breathing abdominal imaging under a high under-sampling rate, Lv et al[26] proposed a reconstruction algorithm based on a stacked convolutional autoencoder (SCAE). Experimental results showed that the SCAE method eliminates the streak artifacts caused by insufficient sampling. In order to realize high-resolution image reconstruction from radial under-sampled k-space data, Han et al[27] proposed a deep learning method with domain adaptation function. The network model was pre-trained with CT images, and then tuned for MRI with radial sampling. This method could be applied to limited training real-time data and multichannel reconstruction, which is in line with the clinical situation when multiple coils are used to acquire signals. Zhou et al[28] proposed a network combining parallel imaging (PI) and CNN for reconstruction. Real-time abdominal imaging was used to train and test the network; expected results were obtained.

In addition, CNN can also be applied to improve the quality of dynamic contrast-enhanced MRI. Tamada et al[29] proposed a multichannel CNN to reduce the artifacts and blur caused by the patient's motion. The detailed information on the MRI reconstruction methods mentioned above is described in Table 1.

Table 1 Abdominal image reconstruction algorithms based on a convolutional neural network.

Ref.	Task	Method	Images	Metric
Kang et al[30], 2017	Low-dose CT reconstruction	CNN	Abdominal CT images	PSNR: 34.55
Chen et al[31], 2017	Low-dose CT reconstruction	RED-CNN	Low-dose abdominal CT images	PSNR: 43.79 ± 2.01; SSIM: 0.98 ± 0.01; RMSE: 0.69 ± 0.07
Han et al[27], 2018	Accelerated projection-reconstruction MRI	U-netCNN	Low-dose abdominal CT images; synthetic radial abdominal MR images	PSNR: 31.55
Lv et al[26], 2018	Undersampled radial free-breathing 3D abdominal MRI	Auto-encoderCNN	3D golden angle-radial SOS liver MR images	P < 0.001
Ge et al[32], 2020	CT image reconstruction directly from a sinogram	Residual encoder-decoder + CNN	Low-dose abdominal CT images	PSNR: 43.15 ± 1.93; SSIM: 0.97 ± 0.01; NRMSE: 0.71 ± 0.16
MacDougall et al[33], 2019	Improving low-dose pediatric abdominal CT	CNN	Liver CT images;Spleen CT images	P < 0.001
Tamada et al[29], 2020	DCE MR imaging of the liver	CNN	T1-weighted liver MR images	SSIM: 0.91
Zhou et al[28], 2019	Applications in low-latency accelerated real-time imaging	PICNN	bSSFP cardiac MR images; bSSFP abdominal MR images	Abdominal: NRMSE: 0.08 ± 0.02; SSIM: 0.90 ± 0.02
Zhang et al[34], 2020	Reconstructing 3D liver vessel morphology from contrasted CT images	GNNCNN	Multi-phase contrasted liver CT images	F1 score: 0.8762 ± 0.0549
Zhou et al[35], 2020	Limited view tomographic reconstruction	Residual dense spatial-channel attention + CNN	Whole body CT images	LAR: PSNR: 35.82; SSIM: 0.97 SVR: PSNR: 41.98; SSIM: 0.97
Kazuo et al[36], 2021	Image reconstructionin low-dose and sparse-view CT	CS + CNN	Low-dose abdominal CT images; Sparse-view abdominal CT images	Low-Dose CT case: PSNR: 33.2; SSIM: 0.91 Sparse-View CT case: PSNR: 29.2; SSIM: 0.91

NRMSE (× 10^-2); RMSE (10^-2). MRI: Magnetic resonance imaging; CT: Computed tomography; CNN: Convolutional neural network; PSNR: Peak signal to noise ratio; SSIM: Structural similarity; RMSE: Root mean square error; NRMSE: Normalized root mean square error; RED: Residual encoder-decoder; DCE: Dynamic contrast-enhanced; PI: Parallel imaging; CS: Compressed sensing; LAR: Limited angle reconstruction; SVR: Sparse view reconstruction; GNN: Graph neural network; RNN: Recurrent neural network; SOS: Stack-of-stars.

CT imaging

In addition to the above application in abdominal MRI, CNN-based reconstruction methods show satisfactory results in CT images. Kang et al[30] used a deep CNN with residuals for LDCT imaging. The experimental results showed that this method reduces the noise level in the reconstructed image. Chen et al[31] proposed a residual encoder-decoder CNN by adding the autoencoder, deconvolution, and short jump connection to the residual encoder-decoder for LDCT imaging. This method had great advantages over the conventional method in terms of noise suppression, structure preservation, and lesion detection. Ge et al[32] proposed an ADAPTIVE-NET that directly reconstructs CT from sinograms. CNN can also be applied to pediatric LDCT images[33]. Zhang et al[34] proposed a graph attention neural network and CNN to reconstruct liver vessels.

Limited view tomographic reconstruction aimed to reconstruct images with a limited number of sinograms that could lead to high noise and artifacts. Zhou et al[35] proposed a novel residual dense reconstruction network architecture with spatial attention and channel attention to address this problem. The network used sinogram consistency layer interleaved to ensure that the output by the intermediate loop block was consistent with the sampled sinogram input. This method used the AAPM LDCT dataset[40] for validation and achieved the desired performance in both limited-angle and sparse-view reconstruction. In order to further improve the quality of sparse-view CT and low-dose CT reconstruction, Kazuo et al[36] proposed a reconstruction framework that combined CS and CNN. This method input a degraded filtered back projection image and multiplied CS reconstructed images obtained using various regularization items into a CNN. The detailed information on the abdominal CT reconstruction methods mentioned above is listed in Table 1.

GAN FOR IMAGE RECONSTRUCTION

MRI

GAN is optimized and learned through the game between generator G and discriminator D. This method is also suitable for abdominal image reconstruction. Mardani et al[41] used GAN for abdominal MRI reconstruction. This method also solves the problem of poor reconstruction performance of traditional CS-MRI[42,43] due to its slow iteration process and artifacts caused by noise. This method used least-squares GAN[44] and pixel-wise L1 as the cost function during training. The data showed that the reconstructed abdominal MR image was superior to that obtained using the traditional CS method with respect to image quality and reconstruction speed. Lv et al[45] compared the performance of GAN-based image reconstruction with DAGAN[46], ReconGAN[25], RefineGAN[25], and KIGAN[47]. Among these, the RefineGAN method was slightly better than DAGAN and KIGAN. In addition, Lv et al[48] combined PI and GAN for end-to-end reconstruction. The network added data fidelity items and regularization terms to the generator to obtain the information from multiple coils.

Most supervised learning methods require a large amount of fully sampled data for training. However, it is difficult or even impossible to obtain the full sampled data, and hence, unsupervised learning is necessary under the circumstances. Cole et al[49] proposed an unsupervised reconstruction method based on GAN. The detailed information on the reconstruction methods is described in Table 2.

Table 2 Abdominal image reconstruction based on generative adversarial network and recurrent neural network.

Ref.	Task	Method	Images	Metric
Mardani et al[41], 2017	Compressed sensing automates MRI reconstruction	GANCS	Abdominal MR images	SNR: 20.48; SSIM: 0.87
Yang et al[50], 2018	Low dose CT image denoising	WGAN	Abdominal CT images	PSNR: 23.39; SSIM: 0.79
Kuanar et al[52], 2019	Low-dose abdominal CT image reconstruction	Auto-encoderWGAN	Abdominal CT images	PSNR: 37.76; SSIM: 0.94; RMSE: 0.92
Lv et al[45], 2021	A comparative study of GAN-based fast MRI reconstruction	DAGANKIGANReconGANRefineGAN	T2-weighted liver images; 3D FSE CUBE knee images; T1-weighted brain images	Liver: PSNR: 36.25 ± 3.39; SSIM: 0.95 ± 0.02; RMSE: 2.12 ± 1.54; VIF: 0.93 ± 0.05; FID: 31.94
Zhang et al[53], 2020	3D reconstruction for super-resolution CT images	Conditional GAN	3D-IRCADb-01database liver CT images	Male: PSNR: 34.51; SSIM: 0.90Female: PSNR: 34.75; SSIM: 0.90
Cole et al[49], 2020	Unsupervised MRI reconstruction	UnsupervisedGAN	3D FSE CUBE knee images; DCE abdominal MR images	PSNR: 31.55; NRMSE: 0.23; SSIM: 0.83
Lv et al[48], 2021	Accelerated multichannel MRI reconstruction	PIGAN	3D FSE CUBE knee MR images; abdominal MR images	Abdominal: PSNR: 31.76 ± 3.04; SSIM: 0.86 ± 0.02; NMSE: 1.22 ± 0.97
Zhang et al[54], 2019	4D abdominal and in utero MR imaging	Self-supervised RNN	bSSFP uterus MR images; bSSFP kidney MR images	PSNR: 36.08 ± 1.13; SSIM: 0.96 ± 0.01

RMSE (× 10^-2); NMSE (× 10^-5). MRI: Magnetic resonance imaging; CT: Computed tomography; SNR: Signal-to-noise ratio; PSNR: Peak signal to noise ratio; SSIM: Structural similarity; RMSE: Root mean square error; NRMSE: Normalized root mean square error; VIF: Variance inflation factor; FID: Frechet inception distance; GAN: Generative adversarial network; RNN: Recurrent neural network; PI: Parallel imaging.

CT imaging

The usage of GAN can also improve the quality of abdominal LDCT images. Yang et al[50] used GAN combined with Wasserstein distance and perceptual loss for LDCT abdominal image denoising. Based on Wasserstein GAN (WGAN)[51], Kuanar et al[52] proposed an end-to-end RegNet-based autoencoder network model, in which GAN was used in the autoencoder. The loss function of this network was composed of RegNet perceptual loss[52] and WGAN adversarial loss[51]. The experimental results showed that this method improves the quality of the reconstructed image while reducing the noise.

Zhang et al[53] proposed the use of conditional GAN (CGAN) to reconstruct super-resolution CT images. The edge detection loss function was proposed in the CGAN to minimize the loss of the image edge. In addition, this study used appropriate bounding boxes to reduce the number of rays when performing 3D reconstruction. The reconstruction methods are described in Table 2.

RNN FOR IMAGE RECONSTRUCTION

RNN is suitable for processing data with sequence information. The dynamic abdominal images were collected from the currently collected frame and were similar to the previous and following frames. Unlike other networks, the nodes between the hidden layers of RNN are connected. Zhang et al[54] proposed a self-supervised RNN to estimate the breathing motion of the abdomen and in utero 4D MRI. The network used a self-supervised RNN to estimate breathing motion and then a 3D deconvolution network for super-resolution reconstruction. Compared to slice-to-volume registration, the experimental results of this method predicted the respiratory motion and reconstructed high-quality images accurately. The detailed information on the reconstruction method mentioned above is shown in Table 2.

APPLICATION OF DL IMAGE RECONSTRUCTION

Motion correction

Deep learning can also be applied to abdominal motion correction. Lv et al[55] proposed a CNN-based image registration algorithm to obtain images during the respiratory cycle. In addition, methods based on U-net and GAN can also be applied to abdominal motion correction. Jiang et al[56] proposed a densely connected U-net and GAN for abdominal MRI respiration correction. Küstner et al[57] combined non-rigid registration with 4D reconstruction networks for motion correction. The detailed information on the reconstruction methods mentioned above is summarized in Table 3.

Table 3 Applications of deep learning in abdominal reconstruction.

Ref.	Task	Method	Images	Metric
Lv et al[55], 2018	Respiratory motion correction for free-breathing 3D abdominal MRI	CNN	3D golden angle-radial SOS abdominal images	SNR: 207.42 ± 96.73
Jiang et al[56], 2019	Respiratory motion correction in abdominal MRI	U-NetGAN	T1-weighted abdominal images	FSE: 0.920; GRE: 0.910; Simulated motion: 0.928
Küstner et al[57], 2020	Motion-corrected image reconstruction in 4D MRI	U-netCNN	T1-weighted in-vivo 4D MR images	EPE: 0.17 ± 0.26; EAE: 7.9 ± 9.9; SSIM: 0.94 ± 0.04; NRMSE: 0.5 ± 0.1
Akagi et al[58], 2019	Improving image quality of abdominal U-HRCT using DLR method	DLR	U-HRCT abdominal CT images	P < 0.01
Nakamura et al[59], 2019	To evaluate the effect of a DLR method	DLR	Abdominal CT images	P < 0.001

NRMSE (× 10^-2). MRI: Magnetic resonance imaging; CT: Computed tomography; CNN: Convolutional neural network; GAN: Generative adversarial network; SNR: Signal-to-noise ratio; SSIM: Structural similarity; NRMSE: Normalized root mean square error; EPE: End-point error; EAE: End-angulation error; U-HRCT: Ultra-high-resolution computed tomography; DLR: Deep learning reconstruction; SOS: Stack-of-stars; FSE: Fast-spin echo; GRE: Gradient echo.

DLR

The DLR developed by Canon Medical Systems’ Advanced Intelligent Clear-IQ Engine is a commercial deep learning tool for image reconstruction. Some studies have confirmed the feasibility and effectiveness of this tool for abdominal image reconstruction. Akagi et al[58] used DLR for abdominal ultra-high-resolution computed tomography (U-HRCT) image reconstruction. The present study proved that DLR reconstruction has clinical applicability in U-HRCT. Compared to hybrid-IR and MBIR[10], DLR reduces the noise of abdominal U-HRCT and improves image quality. In addition, the DLR method is applicable to widely-used CT images. Nakamura et al[59] evaluated the effectiveness of the DLR method on hypovascular hepatic metastasis on abdominal CT images. The detailed information on the reconstruction methods mentioned is summarized in Table 3.

CURRENT CHALLENGES AND FUTURE DIRECTIONS

In summary, deep learning provides a powerful tool for abdominal image reconstruction. However, deep learning-based abdominal image reconstruction has several challenges. First, collecting a large amount of data for training the neural networks is rather challenging. Supervised learning means that a large amount of fully sampled data is required, which is time-consuming in clinical medicine. In addition, it is difficult or even impossible to obtain full sampling data in some specific applications[49]. Therefore, some semi-supervised learning is necessary. In addition, some researchers have proposed the use of self-supervised learning methods[54,60,61]. Self-supervised learning does not require training labels. It is suitable for image reconstruction problems when fully sampled data cannot be obtained easily. Therefore, self-supervised learning has great development potential and is one of the major research directions in the future. Second, deep learning is difficult to explain even if satisfactory reconstruction is achieved.

The current workflow of abdominal imaging starts from data acquisition to image reconstruction and then to diagnosis, deeming it possible to perform multiple tasks at the same time. For example, SegNetMRI[62] realizes image segmentation and image reconstruction simultaneously. Joint-FR-Net[63] can directly use k-space data for image segmentation. Thus, future studies could use the k-space data for lesion detection, classification, and other clinical applications directly.

CONCLUSION

We summarized the current deep learning-based abdominal image reconstruction methods in this review. The DLR methods can solve the issues of slow imaging speed in MRI and high-dose radiation in CT while maintaining high image quality. Deep learning has a wide range of clinical applications in current abdominal imaging. More advanced techniques are expected to be utilized in future studies.