Recently, multimodal prompting, which introduces learnable missing-aware prompts for all missing modality cases, has exhibited impressive performance. However, it encounters two critical issues: 1) The number of prompts grows exponentially as the number of modalities increases; and 2) It lacks robustness in scenarios with different missing modality settings between training and inference. In this paper, we propose a simple yet effective prompt design to address these challenges. Instead of using missing-aware prompts, we utilize prompts as modality-specific tokens, enabling them to capture the unique characteristics of each modality. Furthermore, our prompt design leverages orthogonality between prompts as a key element to learn distinct information across different modalities and promote diversity in the learned representations. Extensive experiments demonstrate that our prompt design enhances both performance and robustness while reducing the number of prompts.

Multimodal learning is essential for understanding interactions between different input domains. However, dealing with various modalities often leads to a high number of network parameters and extended training time. To tackle these challenges, a recent approach called "missing modality aware prompting" enhances model robustness with minimal parameters by freezing the transformer-based backbone network and introducing missing modality aware prompts. In this paper, we propose a robust missing modality aware prompting approach with the same parameter numbers as the naive prompts by adding noise. Our experiments demonstrate that robust missing modality aware prompts outperform state-of-the-art missing modality prompt-based learning in various scenarios. Additionally, our ablation study verifies the effectiveness of robust missing modality aware prompts across different signal-to-noise ratios.

Road extraction from very high resolution (VHR) satellite images is one of the most important topics in the field of remote sensing. In this letter, we propose an efficient nonlocal LinkNet with nonlocal blocks (NLBs) that can grasp relations between global features. This enables each spatial feature point to refer to all other contextual information and results in more accurate road segmentation. In detail, our single model without any postprocessing like conditional random field (CRF) refinement performed better than any other published state-of-the-art ensemble model in the official DeepGlobe Challenge. Moreover, our nonlocal LinkNet (NL-LinkNet) beat the D-LinkNet, the winner of the DeepGlobe challenge (Demiret al., 2018), with 43% less parameters, less giga floating-point operations per seconds (GFLOPs), and shorter training convergence time. We also present empirical analyses on the proper usages of NLBs for the baseline model.

Driving scene understanding is an essential technique for realizing autonomous driving. Although many large-scale datasets for autonomous driving have enabled studies in terms of driving scene understanding, unpaved dirt roads have not been considered as their target driving environment. Drivable region identification is important in dirt roads to prevent damage to the vehicle and passengers. Semantic segmentation of camera images and lidar point clouds have been used to recognize the environment around an autonomous vehicle. The road and objects are recognized by classifying image pixels and point clouds into semantic classes. In this study, we introduce a perception method for drivable region identification on dirt roads through fusion of image and point cloud semantic segmentation. Our approach includes an image semantic segmentation algorithm and a point cloud semantic segmentation algorithm, which we combine into abird’s-eye-view grid mapformat. To transform the point-wise drivable region identification results into the area-wise information, we adopt the alphashape algorithm. Speed improvements are made by using small proportion of drivable points, and the accuracy degradation is compensated by accumulating the perception results along the time.

Recent studies of driving scene understanding based on image semantic segmentation have achieved dramatic advances in speed and accuracy. Large-scale public datasets for semantic segmentation of paved road driving scenes have led the advances, but there is no large-scale public dataset for unpaved road environments. Building a large-scale image semantic segmentation dataset for unpaved roads is very expensive, and domain gaps between geographically distributed locations and those of seasonal changes hinder building a training dataset that is adequate to train a convolutional neural network model. In this paper, to resolve the data insufficiency problem, we use an one-shot learning setting in unpaved road driving scene understanding. Our One-shot Summary Prototypical Network (OSPNet) is trained with paved road driving scenes, and it identifies drivable regions in unpaved roads given only a single support image and unpaved road mask data. The OSPNet improves previous two branch few-shot segmentation approaches by introducing the summary branch which enables channel-wise weighting for important features in the feature map of support and query branches. Our experiments show that our model quantitatively and qualitatively outperforms recent supervised and few-shot segmentation models.

Recent works on convolutional neural networks (CNN) have attempted to find the local optima with ensemble-based approaches. Fast Geometric Ensemble (FGE) showed that captured weight points at the end of training time circulate local optima. This led to the Stochastic Weight Averaging (SWA) approach, which averages multiple model weights to find the local optima. However, they are limited by their output of fully-parameterized models, including needless parameters, after the training procedure. To solve this problem, we propose a novel training procedure: Stochastic Weight Averaging by One-way Variational Pruning (SWA-OVP). SWA-OVP reduces the number of model parameters by variationally updating the mask of weights for pruning. SWA-OVP variationally generates a mask for pruned weights in each iteration while recent pruning approaches produce the mask at the end of each training. In addition, our SWA-OVP prunes the model in a one-way training procedure, while other recent approaches prune the model weights in iterative training or require additional computation. Our experiment shows that SWA-OVP using only a 0.5x% ~ 0.7x% parameter size achieves even higher accuracy than SWA and FGE on several networks, such as Pre-ResNet110, Pre-ResNet164 and WideResNet28x10 on CIFAR10 and CIFAR100 datasets. SWA-OVP also achieves better performance compared to state-of-the-art pruning approaches.

Understanding images by predicting pixel-level semantic classes is a fundamental task in computer vision and is one of the most important techniques for autonomous driving. Recent approaches based on deep convolutional neural networks have dramatically improved the speed and accuracy of semantic segmentation on paved road datasets, however, dirt roads have yet to be systematically studied. Dirt roads do not contain clear boundaries between drivable and non-drivable regions; and thus, this difficulty must be overcome for the realization of fully autonomous vehicles. The key idea of our approach is to apply lightweight non-local blocks to reinforce stage-wise long-range dependencies in encoder-decoder style backbone networks. Experiments on 4,687 images of a dirt road dataset show that our transitional asymmetric non-local neural networks present a higher accuracy with lower computational costs compared to state-of-the-art models.