Previous research on sparse feature matching typically involves a staged optimization process of keypoint detection, description, and matching. While it allows the network to adapt to specific inputs, it may limit the network's expressive capability and the overall architectural flexibility. In this study, we rethink the matching framework and propose to directly match any given keypoints, optimizing the matching network in an approximately end-to-end manner. To achieve this, firstly, we dynamically sample random positions within the images as assumed keypoints during training, allowing the network to explore a broader matching space. Secondly, we replace specific descriptors with high-efficiency sparse embeddings at multi levels of the image, facilitating the direct learning of underlying textures. Thirdly, we propose a novel and promising architecture, called Proposal-Guided TRansformer (PGTR), which aggregates context information from neighboring match proposals instead of searching globally with local features. PGTR works especially well under our training approach, and attain a synergistic advantage in terms of performance and efficiency. The overall pipeline achieves outstanding performance on various keypoints without any retraining, and can be flexibly reused when new keypoints emerge, making it valuable for real-world applications. Code will be available.

Many complex systems prevalent in nature and society, from particle physics systems to social networks and team sports, can be viewed as dynamical interacting systems. Understanding the underlying interactions of agents in the system is the key task for predicting future behaviors of agents, which can be applied in various applications, e.g., autonomous vehicles and smart video surveillance. Since the interaction patterns between agents in the system can be dynamic and heterogeneous rather than fixed and homogeneous, it is very challenging to model interacting systems. In this paper, we design a novel graph structure called Signed Relation Graph (SRG) to model dynamical interacting systems. Since collective behaviors are very common in real-world scenes, our method is a group based model that takes heterogeneous relationships between agents into consideration, and achieves jointly modeling inter-group interactions and intra-group interactions. To assign signs on SRG, an unsupervised method called Relationship Reasoning Network is proposed. The relationship categories are reasoned explicitly, which makes handling multi-agent systems with multiple and dynamic interactions available. Further, Group Interaction Attention Graph Neural Network is proposed to aggregate information on SRG, which achieves not only reasoning the intensity of different interaction patterns but also modeling the trade-off between inter-group interactions and intra-group interactions. Our interacting systems modeling method can be used to predict multi-agent future trajectories in a variety of scenes with hard scenarios, including dense and drastic scenarios. Experimental results on three widely used human trajectory prediction datasets, including ETH and UCY in traffic scenes and NBA SportVU in sports scenes, demonstrate the effectiveness of our proposed model.

Finding correspondences between images is essential for many computer vision tasks and sparse matching pipelines have been popular for decades. However, matching noise within and between images, along with inconsistent key-point detection, frequently degrades the matching performance. We review these problems and thus propose: 1) a novel and unified Filtering and Calibrating (FC) approach that jointly rejects outliers and optimizes inliers, and 2) leveraging both the matching context and the underlying image texture to remove matching uncertainties. Under the guidance of the above innovations, we construct Filtering and Calibrating Graph Neural Network (FC-GNN), which follows the FC approach to recover reliable and accurate correspondences from various interferences. FC-GNN conducts an effectively combined inference of contextual and local information through careful embedding and multiple information aggregations, predicting confidence scores and calibration offsets for the input correspondences to jointly filter out outliers and improve pixel-level matching accuracy. Moreover, we exploit the local coherence of matches to perform inference on local graphs, thereby reducing computational complexity. Overall, FC-GNN operates at lightning speed and can greatly boost the performance of diverse matching pipelines across various tasks, showcasing the immense potential of such approaches to become standard and pivotal components of image matching. Code is avaiable at https://github.com/xuy123456/fcgnn.

Multimodal understanding aims at constructing semantic correlations among modalities of data while performing various downstream tasks. As one of the primary multimodal downstream tasks, image-text retrieval imposes a high demand on semantic alignment because of the independent expression paradigms of images and text. Existing methods mainly construct a joint embedding space at a single granularity level (either global or local). However, such single reasoning paradigms lack granularity interaction, resulting in semantic inconsistency and cross-domain catastrophes. To address these issues, we design a novel Joint Intra and Inter-grained Network (JIIGNet), focusing on not only intra- but also inter-grained interaction between modalities by combining scene information (global) with region-level (local) instances. Specifically, we simultaneously initiate three specific alignment modules, i.e., global-grained, local-grained, and cross-grained alignment modules, followed by Triplet Attention Refinement to better refine the fused embedding at the alignment-level with proper self and cross attention. For different scenarios, a Style Adaptation Head is further designed to smartly accommodate different samples. We validate JIIGNet through extensive experiments conducted on two widely used datasets: Flickr-30 K and MS-COCO, demonstrating the effectiveness of our proposed method.