Referring expression comprehension (REC) is a vision-language task to locate a target object in an image based on a language expression. Fully fine-tuning general-purpose pre-trained vision-language foundation models for REC yields impressive performance but becomes increasingly costly. Parameter-efficient transfer learning (PETL) methods have shown strong performance with fewer tunable parameters. However, directly applying PETL to REC faces two challenges: (1) insufficient multi-modal interaction between pre-trained vision-language foundation models, and (2) high GPU memory usage due to gradients passing through the heavy vision-language foundation models. To this end, we present M2IST: Multi-Modal Interactive Side-Tuning with M3ISAs: Mixture of Multi-Modal Interactive Side-Adapters. During fine-tuning, we fix the pre-trained uni-modal encoders and update M3ISAs to enable efficient vision-language alignment for REC. Empirical results reveal that M2IST achieves better performance-efficiency trade-off than full fine-tuning and other PETL methods, requiring only 2.11% tunable parameters, 39.61% GPU memory, and 63.46% training time while maintaining competitive performance. Our code is released at https://github.com/xuyang-liu16/M2IST.

We propose a non-cascaded and crosstalk-free multi-image encryption method based on optical scanning holography and 2D orthogonal compressive sensing. This approach enables the simultaneous recording and encryption of multiple plaintext images without mechanical scanning, while allows for independent retrieval of each image with exceptional quality and no crosstalk. Two features would bring about more substantial security and privacy. The one is that, by employing a sequence of pre-designed structural patterns as encryption keys at the pupil, multiple samplings can be achieved and ultimately the holographic cyphertext can be obtained. These patterns are generated using a measurement matrix processed with the generalized orthogonal one. As a result, one can accomplish the differentiation of images prior to the recording and thus neither need to pretreat the pending images nor to suppress the out-of-focus noise in the decrypted image. The other one is that, the non-cascaded architecture ensures that different plaintexts do not share sub-keys. Meanwhile, compared to 1D orthogonal compressive sensing, the 2D counterpart makes the proposed method to synchronously deal with multiple images of more complexity, while acquire significantly high-quality decrypted images and far greater encryption capacity. Further, the regularities of conversion between 1D and 2D orthogonal compressive sensing are identified, which may be instructive when to manufacture a practical multi-image cryptosystem or a single-pixel imaging equipment. A more general method or concept named synthesis pupil encoding is advanced. It may provide an effective way to combine multiple encryption methods together into a non-cascaded one. Our method possesses nonlinearity and it is also promising in multi-image asymmetric or public key cryptosystem as well as multi-user multiplexing.

Detection of moving objects in a video captured by a freely moving camera is a challenging problem in computer vision. Most existing methods often assume that the background (BG) can be approximated by dominant single plane/multiple planes or impose significant geometric constraints on BG, or utilize a complex BG/foreground probabilistic model. Instead, we propose a computationally efficient algorithm that is able to detect moving objects accurately and robustly in a general 3D scene. This problem is formulated as a coarse-to-fine thresholding scheme on the particle trajectories in the video sequence. First, a coarse foreground (CFG) region is extracted by performing reduced singular value decomposition on multiple matrices that are built from bundles of particle trajectories. Next, the BG motion of pixels in the CFG region is reconstructed by a fast inpainting method. After subtracting the BG motion, the fine foreground is segmented out by an adaptive thresholding method that is capable of solving multiple-moving-objects scenarios. Finally, the detected foreground is further refined by the mean-shift segmentation method. Extensive simulations and a comparison with the state-of-the-art methods verify the effectiveness of the proposed method.

This letter presents a novel approach to extract reliable dense and long-range motion trajectories of articulated human in a video sequence. Compared with existing approaches that emphasize temporal consistency of each tracked point, we also consider the spatial structure of tracked points on the articulated human. We treat points as a set of vertices, and build a triangle mesh to join them in image space. The problem of extracting long-range motion trajectories is changed to the issue of consistency of mesh evolution over time. First, self-occlusion is detected by a novel mesh-based method and an adaptive motion estimation method is proposed to initialize mesh between successive frames. Furthermore, we propose an iterative algorithm to efficiently adjust vertices of mesh for a physically plausible deformation, which can meet the local rigidity of mesh and silhouette constraints. Finally, we compare the proposed method with the state-of-the-art methods on a set of challenging sequences. Evaluations demonstrate that our method achieves favorable performance in terms of both accuracy and integrity of extracted trajectories.

Super-resolution (SR) from a single image plays an important role in many computer vision applications. It aims to estimate a high-resolution (HR) image from an input low- resolution (LR) image. To ensure a reliable and robust estimation of the HR image, we propose a novel single image SR method that exploits both the local geometric duality (GD) and the non-local similarity of images. The main principle is to formulate these two typically existing features of images as effective priors to constrain the super-resolved results. In consideration of this principle, the robust soft-decision interpolation method is generalized as an outstanding adaptive GD (AGD)-based local prior. To adaptively design weights for the AGD prior, a local non-smoothness detection method and a directional standard-deviation-based weights selection method are proposed. After that, the AGD prior is combined with a variational-framework-based non-local prior. Furthermore, the proposed algorithm is speeded up by a fast GD matrices construction method, which primarily relies on the selective pixel processing. The extensive experimental results verify the effectiveness of the proposed method compared with several state-of-the-art SR algorithms.

We propose a novel approach to align unsynchronized video sequences of the same dynamic scene that can be subframe accurate and is applicable for different frame rate problem. The proposed approach relies on matching motion trajectories and it is assumed that the object moves on a planar surface. By exploring the invariants of planar trajectories under projective transformation, the cross ratio as an invariant feature is computed for each point along the trajectories and the similarity between invariant features of different trajectories is measured with a distance that takes into account the statistical properties of the cross ratio. Then the smooth high frame rate trajectory is synthesized for searching subframe temporal displacement under our alignment framework. The experimental results with synthetic and real-world sequences show that our approach achieves fairly accuracy and efficiency in subframe temporal alignment of the multiple unsynchronized video sequences.