Multispectral filter array (MSFA) imaging enables low-cost, real-time, and compact spectral sensing, making it highly suitable for IoT applications such as smartphone imaging, smart agriculture, UAV-based remote sensing, and wearable diagnostics. However, due to the strong coupling among spectral response functions (SRFs), MSFA pattern, and reconstruction algorithms, achieving high-fidelity reconstruction remains a significant challenge. Conventional independent optimization of each module often leads to suboptimal results, highlighting the necessity of a co-optimized design method. To this end, we propose an end-to-end spectral reconstruction framework that jointly optimizes the SRF, MSFA pattern, and reconstruction network. In our method, the SRFs are parameterized as Gaussian functions defined by the center wavelength and full width at half maximum (FWHM). The MSFA pattern is designed using the optimal sphere-packing (OSP) method to enhance sampling uniformity and reduce artifacts. In addition, a dual-stage reconstruction pipeline is introduced: the first stage employs a Res2FFT-based U-Net with multi-scale gated attention (MGA) for spatial demosaicing, and the second stage uses an enhanced deep residual network (EDSR) for spectral super-resolution. All components are jointly trained under a hybrid loss with supervision from both multispectral and hyperspectral reconstruction. Experiments on three benchmark datasets demonstrate that our method achieves superior spatial and spectral fidelity compared to state-of-the-art methods. Furthermore, the model has been successfully deployed on an Nvidia Jetson-based IoT device, achieving real-time inference at 45 ms per frame, validating its potential for real-time spectral imaging in IoT systems.

Compared to conventional RGB images, hyperspectral images offer a more comprehensive range of spectral information, encompassing both visible and infrared bands. This enhanced spectral information facilitates trackers in effectively differentiating the target object from background clutter, realizing more robust recognition. However, hyperspectral video cameras exhibit variability in terms of spectral ranges and band numbers, producing distinct modalities. Developing specialized networks for each modality proves to be inefficient and time-consuming, while networks trained for one modality struggle to perform well on others, generating unsatisfactory outcomes. To confront these challenges encountered in various tracking scenarios, HyperTrack is proposed as a unified object tracking network tailored for hyperspectral videos. The proposed HyperTrack can be employed individually for single object tracking across three different modalities of hyperspectral videos: near infrared (NIR), visible (VIS), and red-to-near infrared (RedNIR). Specifically, since hyperspectral data have multiple bands, a band gate module is introduced into the network to enable it to select bands from hyperspectral images, thereby reducing the dimensionality for hyperspectral images. Furthermore, in order to effectively utilize the variability amongst the three different modal data, a computationally simple band embedding module is introduced to improve the object tracking performance of different modal hyperspectral videos. Additionally, a hybrid attention module is devised to efficiently extract and interact features between the template and search at each stage. As a unified network, HyperTrack achieves state-of-the-art comprehensive results across three different types of hyperspectral videos, particularly excelling with VIS and NIR type data. The code and models are publicly available at https://github.com/supertyd/HyperTrack.

Hyperspectral videos contain richer spectral and physical features than RGB videos and thus have greater potential for use in object tracking. The mainstream hyperspectral object tracking approach involves the integration of multiple RGB-based video tracking models. Although ensembles of multiple models can effectively utilize spectral information and improve tracker performance, this approach has high computational complexity, making it difficult to meet the real-time requirements of video object tracking. To bridge the gap, we propose a new hyperspectral object tracking framework (HotMoE) based on Mixture-of-Experts (MoE). HotMoE leverages a divide-and-conquer strategy, where only a subset of expert models is computed for each input, reducing computational complexity while maintaining performance. In this paper, we first design a splitter to group multiple spectral bands into multiple false-color images based on spectral correlations. Then, we design a hyperspectral MoE router that can adaptively learn to aggregate spectral image feature information and route it to suitable experts. Different experts can handle various scenarios, and HotMoE effectively utilizes the capabilities of different experts to obtain better overall performance. Compared with previous state-of-the-art hyperspectral object tracking networks, our model has significantly reduced inference time and performs well, with a processing speed of 43.7 FPS and an AUC of 0.704 with the HOT2022 dataset.

Single-sensor multispectral cameras generally employ a multispectral filter array (MSFA) to rapidly acquire spatial-spectral information. However, MSFA cameras capture only one spectral band’s information at each pixel location. Thus, to fully utilize the spatial-spectral information recorded by MSFA cameras, demosaicing methods are necessary. In this paper, we propose a novel two-stage demosaicing method that combines the traditional method and the deep learning method. We first use a weighted bilinear interpolation convolution filter to preliminarily interpolate the raw MSFA image. And then a multi-scale dense connections large kernel attention method (MDLKA) is proposed to further reconstruct the full-resolution spectral images. MDLKA can effectively increase the image receptive field and capture long-range dependence. Through dense connections, attention maps of different scales are fused to enrich image features. Finally, compared with current advanced demosaicing methods on the dataset provided by NTIRE 2022, we exceed traditional interpolation methods by 11–15 dB in PSNR. Compared with the mosaic convolution-attention network (MCAN) and Res2Unet network based on deep learning methods, we exceed them by 5.4 dB and 1.95 dB in PSNR, respectively.