Multimodal artificial intelligence in capsule endoscopy: Integrating video and sensor data for advanced gastrointestinal diagnostics

Figure 2 Multimodal fusion paradigms in capsule endoscopy. Schematic representation of three primary strategies for integrating visual and sensor data in capsule endoscopy. A: Early fusion (e.g., Endo-VMFuseNet) combines visual and sensory inputs at the feature level using long short-term memory networks, achieving sub-millimeter localization accuracy without explicit calibration; B: Late fusion performs modality-specific analyses independently, then merges model outputs via weighted voting or averaging to produce a robust final prediction; C: Hybrid fusion (e.g., convolutional neural network-long short-term memory hybrid) integrates spatial features from video with temporal features from inertial measurement unit data, enabling accurate organ localization and transit-time estimation (> 95% accuracy). LSTM: Long short-term memory; IMU: Inertial measurement units. Created in BioRender.