Multimodal artificial intelligence integrates imaging, endoscopic, and omics data for intelligent decision-making in individualized gastrointestinal tumor treatment

Table 2 Core framework of multimodal data fusion technologies

Core stage	Key methods/technologies	Core challenges & solutions	Primary application value
Data preprocessing & standardization	Imaging data: N4 bias field correction, CLAHE, SMORE; Text data: Tokenization, word embedding, LLMs (e.g., BioBERT, GPT-4o); Standardization: Z-score, batch normalization, FHIR standard	Challenges: Data heterogeneity, missing values, noise, privacy. Solutions: Dedicated preprocessing, automated tools, unified standards (e.g., FHIR)	Improves data quality & consistency, lays foundation for fusion
Fusion strategy	Early fusion (data-level): Directly concatenates raw data. Middle fusion (feature-level): Multi-stream CNN, Attention Mechanism, GNNs. Late fusion (decision-level): Weighted averaging, voting, meta-learning	Challenges: Data heterogeneity, inter-modal relationships, information loss. Solutions: Select/combine strategies based on data traits and task goals (e.g., using attention to capture cross-modal dependencies)	Integrates multi-source complementary information, enhances model robustness & prediction accuracy
Model training & validation	Training techniques: Data augmentation, handling missing values, regularization, early stopping validation methods: K-fold cross-validation, external validation, multi-center validation evaluation metrics: ACC, AUC, sensitivity, specificity, f1-score	Challenges: Data imbalance, overfitting, generalization. Solutions: Employ rigorous internal/external validation, use explainable AI (e.g., SHAP) to enhance trust	Ensures model reliability, stability, and clinical applicability, promotes clinical translation

CLAHE: Contrast Limited Adaptive Histogram Equalization; SMORE: Super-Resolution for Medical Image Enhancement; BioBERT: Biomedical Bidirectional Encoder Representations from Transformers; GPT-4o: Generative Pre-trained Transformer 4 omni; LLM: Large Language Model; FHIR: Fast healthcare interoperability resources; CNN: Convolutional neural network; GNN: Graph neural network; AI: Artificial intelligence; AUC: Area under the ROC curve; SHAP: SHapley Additive exPlanations; ACC: Accuracy.