Multimodal artificial intelligence integrates imaging, endoscopic, and omics data for intelligent decision-making in individualized gastrointestinal tumor treatment

Table 1 Application of multimodal data in gastrointestinal tumors

Data type	Core characteristics and key technologies	Main clinical application scenarios	AI empowerment and value
Imaging data	CT: High spatial resolution, rapid imaging, morphological analysis; MRI: Excellent soft tissue contrast (DWI, DCE), microenvironment assessment; PET: High metabolic sensitivity (SUV value), assessment of biological activity	Tumor localization, staging, efficacy evaluation, recurrence monitoring	AI application: Automatic segmentation based on CNN; radiomics feature mining. Value: Improves diagnostic consistency, predicts efficacy and metastasis risk
Endoscopic data	Provides HD real-time visualization of mucosal layer; chromo/electronic staining enhances contrast	Early screening and diagnosis (e.g., early gastric cancer, colorectal polyp detection)	AI Application: CNN models for automatic lesion identification, classification, and depth assessment. Value: Increases early detection rate, assists treatment decisions
Omics data	Genomics: Reveals driver mutations (e.g., HER2). Transcriptomics/proteomics/metabolomics: Reflects gene expression, protein function, metabolic status	Deciphering tumor heterogeneity, predicting treatment response and prognosis, facilitating personalized therapy	AI Application: Feature selection and dimension reduction; multimodal fusion (e.g., GNN model StereoMM, drug response prediction model DROEG). Value: Mines molecular mechanisms, enables precise typing, predicts drug sensitivity

AI: Artificial intelligence; CT: Computed tomography; MRI: Magnetic resonance imaging; PET: Positron emission tomography; DWI: Diffusion-weighted imaging; DCE: Dynamic contrast-enhanced; SUV: Standardized uptake value; CNN: Convolutional neural network; HER2: Human Epidermal growth factor Receptor 2; StereoMM Model: Stereoscopic Multi-Modal Mode; DROEG: Drug Response Omics Essential Gene Model; GNN: Graph neural networks.