Establishment and validation of a multimodal model using deep learning for the recognition of early colorectal cancer

Figure 1 Flowchart of this study. A: Dataset composition and stratified random split into training and validation sets (7:3 ratio); B: Univariate and multivariate logistic regression analyses of clinical data in the training set to identify independent risk factors for early colorectal cancer; C: Deep learning model training using VGG19, ResNet50, and InceptionV3 architectures on endoscopic images from the training set, followed by Gradient-weighted Class Activation Mapping visualization for model interpretability; D: Construction of the multimodal nomogram integrating clinical variables and ensemble deep learning prediction; E: Comprehensive evaluation of the multimodal model using receiver operating characteristic curves, calibration curves, decision curve analysis, and comparison with endoscopists of varying experience levels. ROC: Receiver operating characteristic; DCA: Decision curve analysis; Grad-CAM: Gradient-weighted Class Activation Mapping.

Figure 2 Confusion matrices (top) and receiver operating characteristic curves (bottom) for the VGG19, ResNet50, and InceptionV3 models in the training set. A: Confusion matrix of the VGG19 model; B: Confusion matrix of the ResNet50 model; C: Confusion matrix of the InceptionV3 model; D: Receiver operating characteristic (ROC) curve of the VGG19 model; E: ROC curve of the ResNet50 model; F: ROC curve of the InceptionV3 model. ROC: Receiver operating characteristic.

Figure 3 The Gradient-weighted Class Activation Mapping algorithms for three models. The right side shows the weight maps generated by the Gradient-weighted Class Activation Mapping algorithm for the VGG19, ResNet50, and InceptionV3 models, respectively. The color scale ranges from blue to red, with redder regions indicating greater importance to the model’s decision.

Figure 4 Nomogram of the multimodal model for distinguishing non-cancerous lesions from early colorectal cancers. Deep learning prediction represents the output of the majority-vote ensemble of VGG19, ResNet50, and InceptionV3 (early colorectal cancer if at least 1 model positive).

Figure 5 Receiver operating characteristic curves of each group of models in the validation set. ROC: Receiver operating characteristic.