Intelligent Transportation Systems (ITS) demand reliable vehicle identification and overload detection to enable proactive traffic management, enhancing infrastructure durability and road safety. Prior studies have employed unimodal approaches that primarily rely on visual imagery/video or sensor data, which often struggle in real-world environments due to challenges like occlusions, variable lighting, and sensor noise. To address such challenges, we propose VehicleSense, an adaptive, real-time, and multi-modal learning framework that fuses visual and inertial data for robust vehicle identification and overload detection. The framework integrates image-based (ImageSense) and sensor time-series-based (SensorSense) modalities using a trainable neural meta-classifier. We further introduce a late-fusion strategy to integrate predictions from both modalities, enabling dynamic weighting and conflict resolution to improve robustness. This fusion approach enhances classification reliability, ensuring adaptability across diverse environmental conditions and operational scenarios. Comprehensive evaluations on the collected real-world multi-modal dataset demonstrate that VehicleSense significantly outperforms unimodal baselines, achieving substantial gains. In addition, the framework is optimized for edge deployment, enabling scalable, real-time vehicle intelligence in resource-limited settings.