Low-rank tensor completion (LRTC) restores missing elements in multidimensional visual data; the challenge is representing the inherent structures within this data. Typical methods either suffer from inefficiency owing to the combination of multiple regularizers or perform suboptimally using inappropriate priors. In this study, we further investigated LRTC using tensor singular value decomposition (t-SVD). Inspired by the tensor-tensor product (t-product), we proposed a unified transformed t-SVD method that employs an invertible linear transform with a unitary transform matrix. However, the t-SVD-based framework lacks the flexibility necessary to represent different inherent relations along the tensor modes. To address this issue, we propose a tensor represented by a series of multidimensional unfolding tensors to fully explore the hidden structure of the original data. Furthermore, the proposed model can be solved efficiently using the alternate-direction method of the multiplier (ADMM) approach. Extensive experimental results on multidimensional visual data (multispectral images, hyperspectral images, and videos) demonstrated the superiority of the proposed method over other state-of-the-art LRTC-related methods.

Object-goal navigation is a challenging task that requires guiding an agent to specific objects based on first-person visual observations. The ability of agent to comprehend its surroundings plays a crucial role in achieving successful object finding. However, existing knowledge-graph-based navigators often rely on discrete categorical one-hot vectors and vote counting strategy to construct graph representation of the scenes, which results in misalignment with visual images. To provide more accurate and coherent scene descriptions and address this misalignment issue, we propose the Aligning Knowledge Graph with Visual Perception (AKGVP) method for object-goal navigation. Technically, our approach introduces continuous modeling of the hierarchical scene architecture and leverages visual-language pre-training to align natural language description with visual perception. The integration of a continuous knowledge graph architecture and multimodal feature alignment empowers the navigator with a remarkable zero-shot navigation capability. We extensively evaluate our method using the AI2-THOR simulator and conduct a series of experiments to demonstrate the effectiveness and efficiency of our navigator.

Scene graph generation (SGG) methods have historically suffered from long-tail bias and slow inference speed. In this paper, we notice that humans can analyze relationships between objects relying solely on context descriptions, and this abstract cognitive process may be guided by experience. For example, given descriptions of cup and table with their spatial locations, humans can speculate possible relationshipsor. Even without visual appearance information, some impossible predicates like flying in and looking at can be empirically excluded. Accordingly, we propose a contextual scene graph generation (C-SGG) method without using visual information and introduce a context augmentation method. We propose that slight perturbations in the position and size of objects do not essentially affect the relationship between objects. Therefore, at the context level, we can produce diverse context descriptions by using a context augmentation method based on the original dataset. These diverse context descriptions can be used for unbiased training of C-SGG to alleviate long-tail bias. In addition, we also introduce a context guided visual scene graph generation (CV-SGG) method, which leverages the C-SGG experience to guide vision to focus on possible predicates. Through extensive experiments on the publicly available dataset, C-SGG alleviates long-tail bias and omits the huge computation of visual feature extraction to realize real-time SGG. CV-SGG achieves a great trade-off between common predicates and tail predicates.

The goal of temporal language grounding (TLG) task is to temporally localize the most semantically matched video segment with respect to a given sentence query in an untrimmed video. How to effectively incorporate the cross-modal interactions between video and language is the key to improve grounding performance. Previous approaches focus on learning correlations by computing the attention matrix between each frame-word pair, while ignoring the global semantics conditioned on one modality for better associating the complex video contents and sentence query of the target modality. In this paper, we propose a novel Cross-modal Hybrid Attention Network, which integrates two parallel attention fusion modules to exploit the semantics of each modality and interactions in cross modalities. One is Intra-Modal Attention Fusion, which utilizes gated self-attention to capture the frame-by-frame and word-by-word relations conditioned on the other modality. The other is Inter-Modal Attention Fusion, which utilizes query and key features derived from different modalities to calculate the co-attention weights and further promote inter-modal fusion. Experimental results show that our CHAN significantly outperforms several existing state-of-the-arts on three challenging datasets (ActivityNet Captions, Charades-STA and TACOS), demonstrating the effectiveness of our proposed method.

Despite the giant leap made in object 6D pose estimation and robotic grasping under structured scenarios, most approaches depend heavily on the exact CAD models of target objects beforehand, thereby limiting their wide applications. To address this, we propose a novel knowledge-guided network - KGNet to estimate the pose and size of category-level unseen objects. This network includes three primary innovations: knowledge-guided categorical model generation, pointwise deformation probability matrix and synergetic RGBD feature fusion, with the former two leveraging categorical object knowledge for unseen object reconstruction and the latter one facilitating pose-sensitive feature extraction. Exten-sive experiments on CAMERA25 and REAL275 verify their effectiveness, and KGNet achieves the SOTA performance on these two acknowledged benchmarks. Additionally, a real-world robotic grasping experiment is conducted, and its results further qualitatively prove the practicability and robustness of KGNet.

Human Action Recognition plays a driving engine of many human-computer interaction applications. Most current researches focus on improving the model generalization by integrating multiple homogeneous modalities, including RGB images, human poses, and optical flows. Furthermore, contextual interactions and out-of-context sign languages have been validated to depend on scene category and human per se. Those attempts to integrate appearance features and human poses have shown positive results. However, with human poses' spatial errors and temporal ambiguities, existing methods are subject to poor scalability, limited robustness, and sub-optimal models. In this paper, inspired by the assumption that different modalities may maintain temporal consistency and spatial complementarity, we present a novel Bi-directional Co-temporal and Cross-spatial Attention Fusion Model (B2C-AFM). Our model is characterized by the asynchronous fusion strategy of multi-modal features along temporal and spatial dimensions. Besides, the novel explicit motion-oriented pose representations called Limb Flow Fields (Lff) are explored to alleviate the temporal ambiguity regarding human poses. Experiments on publicly available datasets validate our contributions. Abundant ablation studies experimentally show that B2C-AFM achieves robust performance across seen and unseen human actions. The codes are available at https://github.com/gftww/B2C.git.

This paper explores how to boost the performance of current person re-identification (ReID) models by incorporating auxiliary information such as contour sketch. Most current ReID methods consider only RGB images as input, with little attention on extra yet important information contained in other modal images. We propose a dual-modal model (DMM), consisting of a main stream that inputs RGB images, and an auxiliary stream that inputs other modal images, to explore how the auxiliary information will help to promote the performance of existing ReID models. To fuse these two streams, a novel dual-modal attention (DMA) mechanism is proposed. Specifically, we apply spatial attention to auxiliary feature maps to take full advantage of the informative spatial locations contained in this stream. Then channel attention is applied to the spatially refined main feature maps, resulting in further refined representations. Moreover, we adopt DMA at multiple scales to exploit different semantics from low to high levels, which finally generates more discriminative feature representations. Comprehensive experiments on publicly available datasets, Market1501, DukeMTMC, MSMT17, and Black ReID, show that our proposal achieves SOTA results.

Gaze-Following is a complex task that needs to combine the gaze with the scene. Previous works performed well on predicting single-person gaze-following but expensive computations are impractical to the real-world project. Moreover, when there are multiple people appearing at the same time, previous works will excecute repeated scene feature extraction. In addition, obtaining gaze target point through the heatmap argmax method seems to be a convention for gaze-following while the quantization error of the heatmap is ignored. In this paper, a simple but efficient network structure is proposed to provide shared scene features for the multi-person gaze-following, and a numerical coordinate regression is firstly introduced to calculate the gaze target point and regression loss. Our experiments show that the accuracy of our method can achieve SOTA on both GazeFollow dataset and VideoAttentionTarget dataset. At the same time, by using the ghostnet, the FLOPs of our method is only about 1/18 of other methods with the same accuracy. Further, sharing scene features saves nearly 40% of inference time in multi-person gaze-following task when more than 6 people in the frame.