This paper studies the problem of integrated optical camera communication and scene sensing. Due to the tight coupling between background images and stripe information in low signal-to-noise ratio (SNR) encoded images, existing methods cannot simultaneously achieve high-quality optical signal decoding for LED-to-camera communication and background image reconstruction for scene sensing. To address this challenge, this paper analyzes the adversarial characteristics between stripe information and background images, and proposes GANOCCAS, a generative adversarial learning framework tailored for integrated optical camera communication and scene sensing that effectively resolves mutual interference between stripes and background content. First, we design a generator using a CondConv-based 4-layer U-NET architecture with SimAM modules on the last three residual layers and CondConv+PixelShuffle combinations as upsampling layers. Second, we develop a discriminator that combines multi-scale convolutional networks, pooling layers, and residual networks to output stripe sequences for optical signal decoding. Third, by leveraging pixel loss, multi-scale structural similarity loss, and adversarial loss, we ensure that the generator outputs clean background images suitable for scene sensing while the discriminator decodes optical signals for communication in complex environments. Experiments on synthetic and real-world datasets demonstrate that GANOCCAS effectively reduces communication interference from background images and accurately reconstructs stripe-free background images across various SNR scenarios, outperforming current state-of-the-art methods in both reflected OCC and scene sensing tasks.

We study the problem of cross-modality person reidentification (ReID) and tracking with dual visible-infrared (VI) cameras, while most existing efforts on tracking-by-detection have been paid on single-modality visible ReID which is inapplicable for poor-light environments. The major difficulties for cross-modality (e.g., VI) ReID stem from the large modality gap between three-channel visible images and one-channel infrared images and such unknown environmental factors as background clutter, occlusions, etc. To tackle these issues, we propose to enrich the diversities of visible and infrared images for intra- and cross-modality matching by using both the channel-aware data augmentation (DA) techniques (e.g., channel exchanged augmentation and random occlusions) and standard DA techniques. On top of these DA techniques, we incorporate ResNet50 and vision transformer (ViT) into the feature extraction backbone network and apply the dynamic weight average (DWA) strategy for learning loss weights by regarding the minimization of identity loss and triplet loss as a multitask learning problem. We then apply the proposed ReID approach for person tracking in the field of interests. The experiments on two public data sets, i.e., RegDB and SYSU-MM01, show that our approach can improve the performance of state-of-the-art rank-1, mAP, and mINP for cross-modality matching. In addition, the experiments on our data set show that tracking by VI-ReID using dual VI cameras can achieve an accuracy of around 0.24 m.

In this article, we study the problem of passive human localization using an infrared (IR) thermal imaging camera which detects IR radiation emitted by human without carry-on devices and thereby generates a heat map of human body. Rather than directly using the heat map, we propose to exploit temperature of human body and design a lightweight approach for human localization using machine learning techniques. We observe that person-to-camera distance is closely related with the position and the size of a person’s head in the heat map, and several other features, such as variance, skewness, and kurtosis of temperatures in the head region are also good indicators of person-to-camera distance estimation. Accordingly, we propose a set of features and construct a model for inferring person-to-camera distance using machine learning techniques. With the estimated distance, we further compute human localization based on the relative position of the person in the heat map, the estimated person-to-camera distance, and the location and the DFoV of the IR thermal camera. Our experiments in real environments show that the proposed approach can accurately estimate person-to-camera distance and human localization with submeter errors.

In this paper, the computational spectral imaging system is reexamined, and the most time-consuming module, namely the two-step iterative shrinkage/thresholding (TwIST) reconstruction, is accelerated by GPU. The acceleration can be roughly divided into two level: (1) data parallelization and (2)operation parallelization. Data parallelization: by discovering that the observation data array in computational spectral imaging is independent with each other in the norm of the dispersion direction, we propose a strategy to decompose the large scale 2-D observation data array into several small scale overlapped stripes. Considering that the complexity for TwIST is highly non-linear, this divide-conquer strategy can significantly reduce the executing time of TwIST. (2)operation parallelization: we proposed a new 3-projection onto convex sets (POCs) structure for the GPU implementation of TwIST. All the operations are classified into 3 sets which only consist of independent matrix/vector operations. With the above two contributions, the proposed 3-POCs structure based parallel implementation achieves more than 100 times ratio than the CPU based version.