RGB-Thermal (RGB-T) tracking enhances visual tracking robustness by combining RGB and thermal infrared (TIR) modalities, addressing limitations of RGB-only trackers under challenging conditions such as low light and appearance variations. However, most existing RGB-T trackers rely on complex fusion modules or modality-specific architectures, sacrificing efficiency for performance. In this paper, we propose a novel Multi-level Self-Distillation (MSD) framework that adapts a one-stream RGB tracker to the RGB-T setting without modifying the network architecture or adding any extra parameters. RGB and TIR inputs are jointly processed through a shared backbone, and training is guided by a combination of self-supervised and supervised objectives to enhance cross-modal feature representation. The self-supervised component includes a contrastive loss that aligns semantically consistent regions across template-search pairs, as well as a modality-gap alignment loss that reduces discrepancies between RGB and TIR features. These internal signals complement task-driven supervision, including an intermediate focal loss that strengthens early localization by enhancing shallow and mid-level features, modality-specific losses that preserve distinctive cues under partial modality degradation, and a fused tracking loss that drives final bounding box prediction. Comprehensive evaluations on LasHeR, RGBT234, and GTOT benchmarks demonstrate that MSD achieves state-of-the-art tracking accuracy while maintaining the computational efficiency of the original RGB tracker. Our work establishes a new paradigm in multi-modal tracking by demonstrating that optimized training strategies can outperform complex architectural modifications, offering significant practical advantages for real-world deployment.

RGB-T object tracking leverages visible and thermal infrared modalities to enhance robustness in challenging environments. While deep learning-based RGB-T trackers predominantly use feature-level fusion, pixel-level fusion remains underexplored. This paper introduces the Hierarchical Feature Difference Auto-Encoder (HFDAE), a novel pixel-level fusion approach that refines the RGB modality before tracker input. HFDAE adaptively enhances RGB content using hierarchical TIR features, dynamically emphasizing object saliency. HFDAE consists of three key components: (1) a shallow RGB autoencoder that preserves structural and color information, (2) a TIR encoder with variable-depth decoders generating hierarchical TIR representations, and (3) a fusion module that integrates salient thermal features into the RGB image. Salient features are extracted by computing differences between hierarchical and base TIR images, which are then added to the base RGB image to generate the final fused output. Unlike conventional pixel-level fusion methods, HFDAE is optimized directly for tracking, learning fusion strategies without predefined modality assumptions. Extensive experiments on benchmark datasets demonstrate HFDAE’s superior tracking accuracy and robustness across diverse scenarios. The proposed approach improves the base tracker’s precision rate by approximately 10%. Code is available at https://github.com/mohamed-e-awad/HFDAE.

Abstract This paper introduces a novel steganographic technique for H.264 video that achieves an outstanding performance against state-of-the-art steganalysis techniques while maintaining real-time encoding performance constraints. The proposed technique embeds the secret message by altering the motion vectors (MVs) while preserving their local optimality and consistency feature to withstand the recently emerged steganalysis methods. Thanks to its macro-block (MB) basis architecture, the proposed technique satisfies the real-time constraints, eliminating the need to wait for the whole frame or group of pictures (GOP) and avoiding the need to perform any additional re-encoding step(s). Additionally, altering the MVs is performed in the motion estimation (ME) sub-pixel-refinement stage through a rule-based scheme that ensures each MB’s compatibility for embedding without detection by the aforementioned steganalysis methods. The proposed technique is integrated with the OpenH264 real-time video encoder and evaluated on widely used video sequences. The results prove that the proposed technique achieves a significant security performance against the steganalysis methods while maintaining an acceptable embedding rate, outperforming other state-of-the-art MV-based steganographic methods in real-time constrained environments. The proposed technique adds about $$1-2\%$$ 1 - 2 % overhead beyond the encoder running time. The source code is publicly available here: https://github.com/HassanMohamedGit/OpenH264-RealTime-steg.

Steganography is the art and science of hiding data within innocent-looking objects (cover objects). Multimedia objects such as images and videos are an attractive type of cover objects due to their high embedding rates. There exist many techniques for performing steganography in both the literature and the practical world. Meanwhile, the definition of the steganographic capacity for multimedia and how to be calculated has not taken full attention. In this paper, for multivariate quantized-Gaussian-distributed multimedia, we study the maximum achievable embedding rate with respect to the statistical properties of cover objects against the maximum achievable performance by any steganalytic detector. Toward this goal, we evaluate the maximum allowed entropy of the hidden message source subject to the maximum probability of error of the steganalytic detector which is bounded by the KL-divergence between the statistical distributions for the cover and the stego objects. We give the exact scaling constant that governs the relationship between the entropies of the hidden message and the cover object.

Near-infrared (NIR) band sensors capture achromatic images that contain complementary details of a scene which are diminished in visible (VS) band images when the scene is obscured by haze, mist, or fog. To exploit these complementary details, special cameras are produced that can capture both VS and NIR bands. Additionally, several sophisticated VS-NIR fusion approaches have been proposed to produce enhanced VS image with high quality. However, these approaches usually encounter heavy computations that are performed off cameras, limiting its suitableness for real-time applications. In this paper, we present an efficient design and FPGA implementation of VS-NIR fusion based image enhancement approach. To be suitable for camera hardware integration and to meet video real-time requirements, the proposed implementation is designed such that the consumed FPGA resources and the number of clock cycles required for producing the fused image are kept small. To achieve this, data and calculations are reused whenever possible in addition to the parallel and pipelined operations at both data and task levels. The proposed implementation is synthesized and validated on a low-end FPGA device. The proposed implementation produces a real time, indistinguishable, and 5 times faster fused image compared with optimized C++ implementation executed on a modern computer.

Vehicle tracking in Wide Area Motion Imagery (WAMI) relies on associating vehicle detections across multiple WAMI frames to form tracks corresponding to individual vehicles. The temporal window length, i.e., the number M of sequential frames, over which associations are collectively estimated poses a trade-off between accuracy and computational complexity. A larger M improves performance because the increased temporal context enables the use of motion models and allows occlusions and spurious detections to be handled better. The number of total hypotheses tracks, on the other hand, grows exponentially with increasing M, making larger values of M computationally challenging to tackle. In this paper, we introduce SPAAM an iterative approach that progressively grows M with each iteration to improve estimated tracks by exploiting the enlarged temporal context while keeping computation manageable through two novel approaches for pruning association hypotheses. First, guided by a road network, accurately co-registered to the WAMI frames, we disregard unlikely associations that do not agree with the road network. Second, as M is progressively enlarged at each iteration, the related increase in association hypotheses is limited by revisiting only the subset of association possibilities rendered open by stochastically determined dis-associations for the previous iteration. The stochastic disassociation at each iteration maintains each estimated association according to an estimated probability for confidence, obtained via a probabilistic model. Associations at each iteration are then estimated globally over the M frames by (approximately) solving a binary integer programming problem for selecting a set of compatible tracks. Vehicle tracking results obtained over test WAMI datasets indicate that our proposed approach provides significant performance improvements over state of the art alternatives.

We propose a practical combinatorial approach for addressing the challenging task of vehicle tracking in Wide area motion imagery (WAMI) by leveraging a pixel accurate co-registered vector road-map. Specifically, guided by the co-registered road network, we obtain a sparse trellis graph linking each vehicle detection (VD) in a WAMI frame with a road reachable VD in the next frame, which then allows us to enumerate all possible hypotheses tracks. The globally optimal selection of tracks over a K frame window is then formulated as a minimum cost K capacitated facility location problem, where each hypothesis track is allocated VDs in each frame and assigned a cost that models desirable properties of vehicle track. Computation of optimized combinations of jointly feasible tracks that minimize the total cost for the tracks becomes feasible in our formulation for moderate values of K by utilizing available solvers for the facility location problem. The approach automatically selects the optimal number of tracks and provides flexibility in defining costs for tracks globally across the K frames. Vehicle tracking results obtained over test WAMI datasets indicate that our proposed method provides significant better performance than two other state of the art alternatives.

We propose a novel framework for accurate 3D georegistration of wide area motion imagery (WAMI), which is a challenging problem because parametric transformations are insufficient for aligning WAMI image frames to a georeferenced coordinate system in urban areas containing tall buildings and 3D structures. Using structure from motion (SfM) we estimate a 3D point cloud for the scene. Independently, we also compute a precise alignment between the roads in the WAMI frames and a georeferenced vector roadmap by detecting locations of moving vehicles and aligning these locations with the roads in the vector roadmap via parametric chamfer matching. The aligned vector roadmap then identifies corresponding pixels in the WAMI frames, which can be triangulated using the SfM camera parameters to obtain a set of sparse but georeferenced points in the SfM 3D coordinate frame that directly enable georegistration of the complete 3D scene point cloud via a similarity transform. The proposed methodology enables 3D georegistration of a sequence of WAMI frames using only georeferenced vector roadmaps, which are readily available, and without requiring independent georeferenced lidar scans that have been used in prior work. Our framework is validated on WAMI dataset including high resolution WAMI frames for the downtown Rochester, NY region. Experimental results demonstrate that the proposed framework produces an accurate georeferenced point cloud representation for the scene.

Modern aerial imaging platforms provide wide-area motion imagery (WAMI) at high spatial and moderate temporal resolutions making feasible a range of new applications. We consider the dual tasks of registering WAMI frames to geo-referenced vector road-maps and tracking vehicles through the progression of WAMI frames. We present a novel algorithm that performs these tasks jointly and offers improvements in both by exploiting the synergy between the tasks. Tracking for the large number of vehicles seen in urban-area WAMI is improved by auxiliary information that registration to the vector road-map provides by localizing roads within the scene. Similarly, registration of the WAMI frames to the vector map is improved by formulating the registration as a chamfer minimization between the vehicular trajectories and the road network, an approach that resolves challenges for registration posed by the fundamentally different data modalities between the aerial images and the vector road maps. Results obtained over our test datasets show the effectiveness of the proposed joint methodology. For both road network alignment and vehicle tracking, the proposed method offers a very significant improvement over available alternatives: the proposed approach yields better numerical metrics for quantification of registration accuracy and fewer false identification switches for tracked vehicles.

This paper reports on a novel application of computer vision and image processing technologies to an interdisciplinary project in architectural history that seeks to help identify and visualize differences between homologous buildings constructed to a common template design. By identifying the mutations in homologous buildings, we assist humanists in giving voice to the contributions of the myriad additional “authors” for these buildings beyond their primary designers. We develop a framework for comparing 3D point cloud representations of homologous buildings captured using lidar: focusing on identifying similarities and differences, both among 3D scans of different buildings and between the 3D scans and the design specifications of architectural drawings. The framework addresses global and local alignment for highlighting gross differences as well as differences in individual structural elements and provides methods for readily highlighting the differences via suitable visualizations. The framework is demonstrated on pairs of homologous buildings selected from the Canadian and Ottoman rail networks. Results demonstrate the utility of the framework confirming differences already apparent to the humanist researchers and also revealing new differences that were not previously observed.

Image forensics using sensor photo-response nonuniformity (PRNU) provides a powerful method for associating an image with the camera that captured the image. To preserve privacy despite the availability of this powerful tool, we present a new framework for image anonymization. We formulate anonymization as a feasibility problem subject to multiple constraints that seek to ensure non-detectability of the PRNU fingerprint, visual fidelity to the original image, and compatibility to compression. A feasible anonymized image is then obtained via the method of projections onto convex sets using the inherent convexity of several constraints and convex approximations for the others. We demonstrate the effectiveness of our framework by benchmarking it over a publicly available dataset of images from multiple cameras and comparing against a recently presented alternative method. In the process we also highlight a key failing of several prior methods that fail to account for quantization in the compression process and suffer from catastrophic loss of anonymity when the anonymized image is stored in a compressed format, as would commonly be the case in realistic applications. We demonstrate specifically that the compression compatibility constraints we introduce help ensure that our method does not encounter this common pitfall.

To enrich large-scale visual analytics applications enabled by aerial wide area motion imagery (WAMI), we propose a novel methodology for accurately registering a geo-referenced vector roadmap to WAMI by using the locations of detected vehicles and determining a parametric transform that aligns these locations with the network of roads in the roadmap. Specifically, the problem is formulated in a probabilistic framework, explicitly allowing for spurious detections that do not correspond to on-road vehicles. The registration is estimated via the expectation-maximization (EM) algorithm as the planar homography that minimizes the sum of weighted squared distances between the homography-mapped detection locations and the corresponding closest point on the road network, where the weights are estimated posterior probabilities of detections being on-road vehicles. The weighted distance minimization is efficiently performed using the distance transform with the Levenberg-Marquardt nonlinear least-squares minimization procedure, and the fraction of spurious detections is estimated within the EM framework. The proposed method effectively sidesteps the challenges of feature correspondence estimation, applies directly to different imaging modalities, is robust to spurious detections, and is also more appropriate than feature matching for a planar homography. Results over three WAMI data sets captured by both visual and infrared sensors indicate the effectiveness of the proposed methodology: both visual comparison and numerical metrics for the registration accuracy are significantly better for the proposed method as compared with the existing alternatives.

For specular regions (SRs), the assumption of brightness (or other) constancy between images corresponding to multiple views of a scene breaks down. As a consequence, optical-flow (OF) based motion-estimation (ME) algorithms that rely on constancy assumptions fail for specular regions. At the same time estimation of SRs in an image is also prone to errors, particularly to false positives from bright regions in the scene. In this paper, motivated by the fact that specular regions are typically encountered in image regions corresponding to portions of relatively smooth 3D surfaces, we propose an algorithm for improving ME and SRs localization via joint processing. Initial estimates of OF and of the SRs are obtained by conventional methods. The estimate of the SRs is updated using inconsistency of the OF with respect to the neighboring region to reinforce true positives and to reject false positives. The OF is then re-computed with a modified energy functional that, in effect, emphasizes regularization in a spatially adaptive neighborhood of the SRs to improve the estimated OF. Experimental results on synthetic and real image pairs demonstrate that the proposed algorithm offers a significant improvement in both SRs localization and ME over recently proposed methods for tackling these problems.