Multimodal remote sensing object detection (MM-RSOD) holds great promise for around-the-clock applications. However, it faces challenges in effectively extracting complementary features due to the modality inconsistency and redundancy. Inconsistency can lead to semantic-spatial misalignment, while redundancy introduces uncertainty that is specific to each modality. To overcome these challenges and enhance complementarity exploration and exploitation, this article proposes a dual-dynamic cross-modal interaction network (DDCINet), a novel framework comprising two key modules: a dual-dynamic cross-modal interaction (DDCI) module and a dynamic feature fusion (DFF) module. The DDCI module simultaneously addresses both modality inconsistency and redundancy by employing a collaborative design of channel-gated spatial cross-attention (CSCA) and cross-modal dynamic filters (CMDFs) on evenly segmented multimodal features. The CSCA component enhances the semantic-spatial correlation between modalities by identifying the most relevant channel-spatial features through cross-attention, addressing modality inconsistency. In parallel, the CMDF component achieves cross-modal context interaction through static convolution and further generates dynamic spatial-variant kernels to filter out irrelevant information between modalities, addressing modality redundancy. Following the improved feature extraction, the DFF module dynamically adjusts interchannel dependencies guided by modal-specific global context to fuse features, achieving better complementarity exploitation. Extensive experiments conducted on three MM-RSOD datasets confirm the superiority and generalizability of the DDCINet framework. Notably, our DDCINet, based on the RoI Transformer benchmark and ResNet50 backbone, achieves 78.4% mAP50 on the DroneVehicle test set and outperforms state-of-the-art (SOTA) methods by large margins.

Transfer learning aims to learn robust classifiers for the target domain by leveraging knowledge from a source domain. Since the source and the target domains are usually from different distributions, existing methods mainly focus on adapting the cross-domain marginal or conditional distributions. However, in real applications, the marginal and conditional distributions usually have different contributions to the domain discrepancy. Existing methods fail to quantitatively evaluate the different importance of these two distributions, which will result in unsatisfactory transfer performance. In this article, we propose a novel concept called Dynamic Distribution Adaptation (DDA), which is capable of quantitatively evaluating the relative importance of each distribution. DDA can be easily incorporated into the framework of structural risk minimization to solve transfer learning problems. On the basis of DDA, we propose two novel learning algorithms: (1) Manifold Dynamic Distribution Adaptation (MDDA) for traditional transfer learning, and (2) Dynamic Distribution Adaptation Network (DDAN) for deep transfer learning. Extensive experiments demonstrate that MDDA and DDAN significantly improve the transfer learning performance and set up a strong baseline over the latest deep and adversarial methods on digits recognition, sentiment analysis, and image classification. More importantly, it is shown that marginal and conditional distributions have different contributions to the domain divergence, and our DDA is able to provide good quantitative evaluation of their relative importance, which leads to better performance. We believe this observation can be helpful for future research in transfer learning.

This paper introduces a novel deep learning based method, named bridge neural network (BNN) to dig the potential relationship between two given data sources task by task. The proposed approach employs two convolutional neural networks that project the two data sources into a feature space to learn the desired common representation required by the specific task. The training objective with artificial negative samples is introduced with the ability of mini-batch training and it’s asymptotically equivalent to maximizing the total correlation of the two data sources, which is verified by the theoretical analysis. The experiments on the tasks, including pair matching, canonical correlation analysis, transfer learning, and reconstruction demonstrate the state-of-the-art performance of BNN, which may provide new insights into the aspect of common representation learning.

The recent advances in deep transfer learning reveal that adversarial learning can be embedded into deep networks to learn more transferable features to reduce the distribution discrepancy between two domains. Existing adversarial domain adaptation methods either learn a single domain discriminator to align the global source and target distributions, or pay attention to align subdomains based on multiple discriminators. However, in real applications, the marginal (global) and conditional (local) distributions between domains are often contributing differently to the adaptation. There is currently no method to dynamically and quantitatively evaluate the relative importance of these two distributions for adversarial learning. In this paper, we propose a novel Dynamic Adversarial Adaptation Network (DAAN) to dynamically learn domain-invariant representations while quantitatively evaluate the relative importance of global and local domain distributions. To the best of our knowledge, DAAN is the first attempt to perform dynamic adversarial distribution adaptation for deep adversarial learning. DAAN is extremely easy to implement and train in real applications. We theoretically analyze the effectiveness of DAAN, and it can also be explained in an attention strategy. Extensive experiments demonstrate that DAAN achieves better classification accuracy compared to state-of-the-art deep and adversarial methods. Results also imply the necessity and effectiveness of the dynamic distribution adaptation in adversarial transfer learning.

Human depth recovery is an essential task for tele-immersive video interaction systems with limited bandwidth consumption. This paper is motivated by the first human depth recovery method, i.e., the weighted large margin nearest center (WLMNC) distance based method, which compresses each human depth map into several skeletal block structures by learning a WLMNC distance at the remote stage and then recovers it by a rough-to-fine approach at the local stage. Since the WLMNC distance is equivalent to employing a linear transformation on the human pixels, for human postures with complex self-occlusion, the depth recovery performance of WLMNC is limited. To address the problem, this paper proposes to learn a nonlinear WLMNC distance via a lightweight neural network. Different from traditional classification or clustering problems, the neural network is introduced for storing skeletal block structure information instead of predicting new information. The quantitative experimental results on the VGA-sized benchmark dataset demonstrates that such a slight modification over WLMNC can achieve an inspiring performance improvement: the average depth recovery MAD error is reduced from 3.56cm to 2.77cm and the total running cost is reduced from about 150 seconds to 43 seconds on a laptop with just a little loss on the sampling rate, from 64x to 51x.

Transfer learning aims at transferring knowledge from a well-labeled domain to a similar but different domain with limited or no labels. Unfortunately, existing learning-based methods often involve intensive model selection and hyperparameter tuning to obtain good results. Moreover, cross-validation is not possible for tuning hyperparameters since there are often no labels in the target domain. This would restrict wide applicability of transfer learning especially in computationally-constraint devices such as wearables. In this paper, we propose a practically Easy Transfer Learning (EasyTL) approach which requires no model selection and hyperparameter tuning, while achieving competitive performance. By exploiting intra-domain structures, EasyTL is able to learn both non-parametric transfer features and classifiers. Extensive experiments demonstrate that, compared to state-of-the-art traditional and deep methods, EasyTL satisfies the Occam's Razor principle: it is extremely easy to implement and use while achieving comparable or better performance in classification accuracy and much better computational efficiency. Additionally, it is shown that EasyTL can increase the performance of existing transfer feature learning methods.

Visual domain adaptation aims to learn robust classifiers for the target domain by leveraging knowledge from a source domain. Existing methods either attempt to align the cross-domain distributions, or perform manifold subspace learning. However, there are two significant challenges: (1) degenerated feature transformation, which means that distribution alignment is often performed in the original feature space, where feature distortions are hard to overcome. On the other hand, subspace learning is not sufficient to reduce the distribution divergence. (2) unevaluated distribution alignment, which means that existing distribution alignment methods only align the marginal and conditional distributions with equal importance, while they fail to evaluate the different importance of these two distributions in real applications. In this paper, we propose a Manifold Embedded Distribution Alignment (MEDA) approach to address these challenges. MEDA learns a domain-invariant classifier in Grassmann manifold with structural risk minimization, while performing dynamic distribution alignment to quantitatively account for the relative importance of marginal and conditional distributions. To the best of our knowledge, MEDA is the first attempt to perform dynamic distribution alignment for manifold domain adaptation. Extensive experiments demonstrate that MEDA shows significant improvements in classification accuracy compared to state-of-the-art traditional and deep methods.

This paper proposes a weighted large margin nearest center (WLMNC) distance-based human depth recovery method for tele-immersive video interaction systems with limited bandwidth consumption. In the remote stage, the proposed method highly compresses the depth data of the remote human into skeletal block structures by learning the WLMNC distance, which is equivalent to downsampling the human depth map at $64{\times}$ the sampling rate. In the local stage, the method first recovers a rough human depth map based on a WLMNC distance augmented clustering approach and then obtains a fine depth map based on a rough depth-guided autoregressive model to preserve the depth discontinuities and suppress texture copy artifacts. The proposed WLMNC distance is learned by the large margin clustering problem with a weighted hinge loss to balance the clustering accuracy and depth recovery accuracy and is verified to be able to preserve depth discontinuities between skeletal block structures with occlusion. A theoretical analysis is conducted to verify the effectiveness of using the weighted hinge loss. Furthermore, a novel data set containing various types of human postures with self-occlusion is built to benchmark the human depth recovery methods. The quantitative comparison with the state-of-the-art depth recovery methods on the introduced benchmark data set demonstrates the effectiveness of the proposed method for human depth recovery with such a high upsampling rate.

Cerebral small vessel disease (SVD) is common in the elderly and is associated with loss of functional independence, institutionalization, and death. In this paper, we propose a coarse-to-fine feature selection method for accurate SVD detection and timely implementation of interventions. The proposed method first uses an Iterative Random Forest based Feature Selection (IRFFS) method to obtain the most representative features from a feature set that includes gait, balance, and agility performance features extracted from 17 predefined clinical actions. The method then uses the Feature Incremental Extreme Learning Machine (FIELM) model to further verify the discriminant ability of each kind of selected features. Our results demonstrate that the proposed method can effectively select the most significant features for SVD detection, which include gait and agility performance features. Our method achieves up to 91.44% classification accuracy, outperforming other state-of-the-art feature selection methods. Our findings also verify clinical observations indicating that the fine motor pattern features of upper and lower limbs are helpful for high-accuracy SVD detection.

Moving-object segmentation is the key issue of Telepresence systems. With monocular camera--based segmentation methods, desirable segmentation results are hard to obtain in challenging scenes with ambiguous color, illumination changes, and shadows. Approaches based on depth sensors often cause holes inside the object and missegmentations on the object boundary due to inaccurate and unstable estimation of depth data. This work proposes an adaptive multi-cue decision fusion method based on Kinect (which integrates a depth sensor with an RGB camera). First, the algorithm obtains an initial foreground mask based on the depth cue. Second, the algorithm introduces a postprocessing framework to refine the segmentation results, which consists of two main steps: (1) automatically adjusting the weight of two weak decisions to identify foreground holes based on the color and contrast cue separately; and (2) refining the object boundary by integrating the motion probability weighted temporal prior, color likelihood, and smoothness constraint. The extensive experiments we conducted demonstrate that our method can segment moving objects accurately and robustly in various situations in real time.

This paper presents a lightweight Tele-immersive photograph system named TiPS. TiPS allows remote users to take a photo as a souvenir in a virtual space with natural social behavior, through adaptively merging two participants of remote video interaction into the same shared background. We address two key technologies in this paper, first, we propose an adaptive multi-cue decision fusion algorithm for accurate and robust foreground segmentation of live video. Second, we present a user behavioral intention driven video composition algorithm, aiming at obtaining life-like photographs in real time. Experimental results show TiPS achieves high performance in both foreground segmentation and video composition, which well satisfies remote users' requirement of taking photos together.

Time-varying property of practical wireless channels affects the performance of cooperative protocol in multi-relay networks. In this paper, we evaluate and compare four decode-and-forward (DF) retransmission strategies over time-correlated Rayleigh channels, which are the combinations of two relay selection (RS) schemes and two repeater switching patterns: RS can be based on either statistical or outdated channel state information (CSI); in the static switching pattern all forwardings are executed by the same relay that is selected at the beginning, while in the dynamic pattern relays that have not recovered the source message keep receiving from the forwarding relay to extend the decoding subset, from which the activated relay for next retransmission is reselected. Numerical simulations validate that statistical information is more beneficial than outdated CSI over weakly correlated channels. On the other hand, the dynamic switching pattern generally outperforms the static one especially with relatively large maximum transmission number, at the cost of higher complexity of decoding and signalling.

Adaptive Relay-Assisted/ Direct Transmission (ARDT) proposed in [13] is a simple and efficient protocol which adaptively utilizes the direct link between two sources with only channel state information at receiver (CSIR), and it validly enhances the spectrum efficiency of time division broadcast (TDBC) system. In this paper, we combine ARDT with joint power allocation and relay selection for multi-relay scenario. Each relay optimize its own forwarding power to maximize the minimum end to end signal to noise ratio (SNR) towards two sources, and the optimal relay is then selected to cooperate. System outage probability, average throughput and average bit error rate (BER) are analyzed with theoretical upper and lower bounds. Numerical simulations validate the rationality of bounds, and show that whether maximal ratio combining (MRC) is adopted for receiving makes little difference. The proposed relay selection scheme is verified to outperform that based on original TDBC in reference work.

In this paper, performance of three-time slot (3TS) physical layer network coding (PNC) is investigated in interference-limited two-way relaying system. Effective lower and upper bounds are derived for important performance metrics like outage probability and average bit error rate (BER) with different modulation modes. Asymptotic behavior is also proposed to intuitively exhibit the trend of performance in high signal-to-interference and noise ratio (SINR) regime. We differentiate practical applications with different relay power for the bounds. A particular scenario with positional grid structure of sources, relay and co-channel interferers is established for numerical analysis, and the effects of some critical parameters, such as relay's location, forwarding capability and power allocation factor, are compared and discussed. The theoretic solutions are finally verified by simulation results.

In this paper, we study the communication between terrestrial source and destination terminals via a geosynchronous (GEO) multi-beam satellite in satellite mobile communication (SMC) system. The end-to-end transmission is modeled by a relaying process, in which inter-beam interference plays a key role due to frequency reuse among adjacent beam cells and side-lobe leaking of practical beam antenna. Through theoretic analysis we obtain the closed-form expressions of end-to-end outage probability and system achievable throughput as the major performance metrics. Both theoretic and simulation results, which coincide well with each other under diverse beam frequency planning, suggest that reuse factor of 1/3 is an appropriate compromise between system throughput and reliability in the GEO SMC system.