This paper delves into the task of arbitrary modality salient object detection (AM SOD), aiming to detect salient objects from the images with arbitrary modality types or arbitrary modality numbers by using a single model trained once. Specifically, we develop a novel model, termed modality adaptive network (MAN), for AM SOD, which addresses two fundamental challenges in AM SOD: the diverse modality discrepancies arising from varying modality types and the dynamic fusion dilemma resulting from an unfixed number of modalities in the input data. Technically, MAN first introduces a novel Modality-Adaptive Feature Extractor (MAFE) to adaptively extract features from different input modalities based on their characteristics by utilizing a set of learnable modality prompts. Concurrently, a new modality translation contractive (MTC) loss is devised to facilitate the training of MAFE as well as modality prompts, thereby effectively addressing the inherent modality discrepancies and extracting more discriminative features from each modality image. Subsequently, MAN presents a hybrid dynamic fusion (HDF) strategy to effectively resolve the challenge of dynamic inputs in multi-modal feature fusion as well as enhance the exploitation of complementary information across different modalities. This is specially achieved by a Channel- wise Dynamic Fusion Module (CDFM) and a Spatial- wise Dynamic Fusion Module (SDFM). Experimental results show that by virtue of MAFE, MTC loss and HDF strategy, our proposed method achieves significant increasements over existing models on benchmark datasets.

Toward desirable saliency prediction, the types and numbers of inputs for a salient object detection (SOD) algorithm may dynamically change in many real-life applications. However, existing SOD algorithms are mainly designed or trained for one particular type of inputs, failing to be generalized to other types of inputs. Consequentially, more types of SOD algorithms need to be prepared in advance for handling different types of inputs, raising huge hardware and research costs. Differently, in this paper, we propose a new type of SOD task, termed Arbitrary Modality SOD (AM SOD). The most prominent characteristics of AM SOD are that the modality types and modality numbers will be arbitrary or dynamically changed. The former means that the inputs to the AM SOD algorithm may be arbitrary modalities such as RGB, depths, or even any combination of them. While, the latter indicates that the inputs may have arbitrary modality numbers as the input type is changed, e.g. single-modality RGB image, dual-modality RGB-Depth (RGB-D) images or triple-modality RGB-Depth-Thermal (RGB-D-T) images. Accordingly, a preliminary solution to the above challenges, i.e. a modality switch network (MSN), is proposed in this paper. In particular, a modality switch feature extractor (MSFE) is first designed to extract discriminative features from each modality effectively by introducing some modality indicators, which will generate some weights for modality switching. Subsequently, a dynamic fusion module (DFM) is proposed to adaptively fuse features from a variable number of modalities based on a novel Transformer structure. Finally, a new dataset, named AM-XD, is constructed to facilitate research on AM SOD. Extensive experiments demonstrate that our AM SOD method can effectively cope with changes in the type and number of input modalities for robust salient object detection. Our code and AM-XD dataset will be released on https://github.com/nexiakele/AMSODFirst.