In recent years, lossy compression algorithms such as H.264/AVC, H.265/HEVC, and H.266/VVC have been proposed and widely applied in image and video encoding. However, these compression algorithms inevitably introduce various complex types of compression artifacts, which severely degrade image quality. Although existing methods have attempted to remove artifacts through filter design or probabilistic prior modeling, they are often effective only for specific types of artifacts, lacking generalization and adaptability. To address this, we propose a novel image compression artifacts removal model: ARMoE, which combines multiple frequency domain transformations with the Mixture of Experts (MoE). Considering the frequency distribution and energy distribution differences of images, we introduce various frequency domain transformations as expert branches and use the Sparse Activation Strategy to adaptively select the optimal frequency domain expert to suppress compression artifacts, achieving an efficient artifacts removal method. Furthermore, we reencode and decode multiple original uncompressed high-quality datasets, including DF2K and Kodak24, using the VTM-20.0 codec under the H.266/VVC standard, constructing a more challenging artifacts dataset. We conducted rigorous comparative experiments with current state-of-the-art image restoration methods and the results demonstrate that ARMoE exhibits outstanding image restoration capability.

Intra coding plays a critical role in video compression, yet conventional directional intra prediction mainly relies on boundary pixels and fails to capture complex textures and long-range spatial correlations. Template Matching Prediction (TMP) alleviates this issue by searching similar templates in reconstructed regions without motion-vector signaling; however, its MSE/MAE-based matching and simple averaging fusion are sensitive to noise and often unreliable for complex content. This paper proposes E2ETMPN, an end-to-end template matching prediction network for intra coding. E2ETMPN jointly models template matching and multi-candidate fusion under a unified loss function. It consists of two components: (1) a CNN-based multiscale matching sub-network that searches candidate predictions from reconstructed regions at different spatial scales; and (2) a Transformer-based fusion sub-network that adaptively fuses multiple candidates with template information using attention mechanisms to generate the final prediction. Experimental results demonstrate that integrating E2ETMPN into the reference encoder achieves consistent BD-rate reductions over conventional TMP and standard intra prediction methods on standard test sequences. Notably, E2ETMPN yields larger gains on screen content and scenes with rich non-local repetitive structures, validating the effectiveness of end-to-end learning for template matching prediction in intra coding.

Video-Text Retrieval (VTR) aims to align video content with natural language descriptions and is a fundamental task in multi-modal understanding. Most existing methods model videos as uniformly sampled RGB frames, which overlooks rich temporal cues, especially motion dynamics encoded in videos. We propose Compressed Video Stream Learning for Video-Text Retrieval (CVSVTR), a framework that directly exploits information from compressed video streams to enhance retrieval performance without fully decoding videos. Specifically, CVSVTR decodes only the I-frames of each GOP and extracts appearance features using a CLIP-based encoder. Meanwhile, motion vectors and residuals are parsed from the compressed bitstream and processed by a lightweight P-frame Feature Generation (PFG) module to construct motion-aware representations for P-frames. A spatial-channel attention mechanism is further introduced to adaptively fuse appearance features with compressed-domain motion cues, compensating for temporal information missed by uniform frame sampling. Extensive experiments on the MSR-VTT and MSVD benchmarks demonstrate that CVSVTR consistently outperforms existing video-text retrieval methods across multiple evaluation metrics, validating the effectiveness of leveraging compressed video streams for efficient and accurate temporal modeling in VTR.

In today's digitally connected world, keyboards remain the primary interface for inputting sensitive information, making them a persistent target for eavesdropping attacks. While prior keystroke inference techniques have exploited side-channel signals such as acoustics and vibrations, they typically rely on conspicuous, short-range sensors and require victim-specific data for model training, limiting their practicality, scalability, and stealth. In this paper, we present RadKey, an RF backscatter system for covert, long-range, through-wall keystroke eavesdropping. RadKey comprises two components: a compact batteryless backscatter tag and an RF reader. The tag captures keystroke-induced vibrations and acoustic signals, modulating them onto the frequency shift of its backscattered RF signal using two magnetically-coupled LC resonators. This design also enables spectral separation between the excitation and backscatter signals, mitigating self-interference for the RF reader and thus extending eavesdropping range. The RF reader demodulates the backscattered RF signal to infer typed content. It employs a dedicated signal processing pipeline that extracts user- and keyboard-independent keystroke features across time and frequency domains, enabling strong generalizability. To further enhance adaptability, RadKey integrates an LLM for online adaptation, leveraging LLM outputs as pseudo ground-truth labels to refine the classifier during runtime. We have built a prototype of the full RadKey system and evaluated it through extensive over-the-air experiments. Results show that RadKey achieves accurate and robust keystroke inference across diverse users in real-world settings. A demo video is available at: https://radkey-submission.github.io/RadKey/

Eye motion tracking plays a vital role in many applications such as human-computer interaction (HCI), virtual reality, and disease detection.Camera-based eye tracking, albeit accurate and easy to use, may raise privacy concerns and appear to be unreliable in poor lighting conditions.In this paper, we present RadEye, a radar system capable of detecting fine-grained human eye motions from a distance.RadEye is realized through an integrated hardware and software design.It customizes a sub-6GHz FMCW radar so as to detect millimeter-level eye movement while extending its detection range using low frequency.It further employs a deep neural network (DNN) to refine the detection accuracy through camera-guided supervisory training.We have built a prototype of RadEye.Extensive experimental results show that it achieves 90% accuracy when detecting human eye rotation directions (up, down, left, and right) in various scenarios.

Technically, Augmented Reality Head-Up Display (AR-HUD) technology can augment multiple targets in the scene. However, existing literature predominantly focuses on scenarios involving single-target augmentation. The understanding of multi-target augmentation and studies exploring effective presentation methods under such conditions are notably scarce. This study evaluates the efficacy of integrating color-based warning priority design across multi-target scenarios. 45 Participants in different warning modes (Equivalent, Hierarchical, and Baseline) view AR-augmented driving videos and respond to risky targets. Their behavioral performance and eye-tracking data are compared. Findings indicate that the equivalent warning mode, lacking in priority design, adversely affects driver performance, prolongs reaction times, and elevates saccade counts, and gaze entropy. Conversely, the hierarchical warning mode significantly ameliorates driver reaction times and the time to first fixation, while also reducing saccade counts and gaze entropy, demonstrating the efficacy of the warning priority design. The findings provide insight into the design of AR-HUD with multi-target augmentation.

The JPEG standard is still widely-used in current applications. The optimization on the quantization table in JPEG standard not only can maintain the compatibility with the existing JPEG standard, and but also can improve the coding efficiency of JPEG compression. In this paper, JPEG quantization table optimization via multi-objective evolutionary algorithm based on decomposition is proposed. In the multi-objective evolutionary optimization framework, the fitness of each quantization table for JPEG compression is evaluated efficiently by the searching in a look-up table, which is constructed based on the statistics of each DCT band in a pre-defined manner. Then, the MOEA/D algorithm is utilized to minimize the rate and distortion simultaneously. After the multi-objective optimization, a set of non-dominated solutions (quantization tables) can be obtained. Therefore, our method can provide several optimal solutions to address the multiple-rate requirement in applications in one running time. The experimental results on the classic Kodak24 dataset demonstrate the superiority of our method in terms of coding efficiency and subjective quality.

Sub-6GHz radio sensing offers several compelling advantages, such as resilience to poor lighting conditions, privacy preservation, and the ability to see through walls. However, in indoor environments, the sub-6GHz ISM spectrum is heavily occupied by WiFi devices, leaving little available spectrum for sensing purposes. In this paper, we introduce SiWiS, a new approach to integrate radio sensing capabilities into individual WiFi devices for fine-grained human activity detection. SiWiS comprises two main components: (i) a new hardware component that can be easily installed on an off-the-shelf WiFi device, and (ii) a dual-branch deep neural network (DNN) optimized for concurrent human mask segmentation and pose estimation. We have built a prototype of SiWiS and installed it on a commercial WiFi router for evaluation. Extensive experimental results demonstrate a significant performance improvement over WiFi channel state information (CSI) based sensing methods. More importantly, zero-shot experiments confirm that SiWiS can be directly transferred to unseen real-world environments.

With the development of augmented reality (AR) technology, AR navigation tools have gradually received increasing attention. Few previous studies have comprehensively discussed the advantages and disadvantages of AR and 2D maps from a spatial cognition perspective. Through three experiments, this article investigated the effects of different AR navigation display methods on wayfinding performance and spatial knowledge acquisition. First, we compared participants’ performances in using AR and 2D map navigation aids for wayfinding and spatial knowledge tasks. Second, we developed a combined map and compared the usability of this map with that of an AR map. Then, we applied the focus + context technique to improve the combined map’s minimap and verified the improved map’s effectiveness by laboratory simulation. The AR map is superior to the 2D map in acquiring spatial knowledge. AR displays are crucial for helping individuals acquire landmark and route knowledge. The improved map expands the range of exocentric spatial information and highlights the landmark locations to provide comprehensive information about the environment; thus, the improved map can further assist in acquiring spatial knowledge.

Tiny machine learning (TinyML) becomes appealing as it enables machine learning on resource-constrained devices with ultra low energy and small form factor. In this paper, a model-specific end-to-end design methodology is presented for TinyML hardware design. First, we introduce an end-to-end system evaluation method using Roofline models, which considering both AI and other general-purpose computing to guide the architecture design choices. Second, to improve the efficiency of AI computation, we develop an enhanced design space exploration framework, TinyScale, to enable optimal low-voltage operation for energy-efficient TinyML. Finally, we present a use case driven design selection method to search the optimal hardware design across a set of application use cases. Our model-specific design methodology is evaluated on both TSMC 22nm and 55nm technology for MLPerf Tiny benchmark and a keyword spotting (KWS) SoC design. With the help of our end-to-end design methodology, an optimal TinyML hardware can be automatically explored with significant energy and EDP improvements for a diverse of TinyML use cases.

In this paper, we propose a novel low-rank based non-local image denoising method for HEVC video compression with the strategy of gathering non-local patches in the rectified domain. Owing to the irreversible quantization, image compression can be considered as adding noises into the original image, causing the distortion between the original image and the de-compressed image. Current non-local collaboration based image denoising methods collect K-nearest patches from image interior with the underlying translational motion model, and restores the corrupted content from the similar image patches. However, these methods are usually not suitable for image content with repeated patterns with different scales due to the viewing perspective. To address this problem, we derive the plane orientation utilizing vanishing points, which can be determined through line segment detection and clustering. Vanishing points are used to construct a specific homographic transformation to model patch correspondence in rectified domain. With this transformation, the regularity in image can be extracted based on SIFT features and mean-shift clustering in the rectified domain, and can provide useful guidance for patch gathering. Thus, our method is especially suitable for urban building scenes. Our experimental results show that the non-local denoising in rectified domain can further improve the average PSNR by up to 0.32dB comparing to those non-local methods on Urban100 dataset compressed by HEVC standard.

Evolutionary algorithms (EAs) have shown their competitiveness in solving the problem of feature selection (FS). However, in most of the existing EA-based FS methods, one bit in the individual only represents one feature, which means with the number of features increasing, the search space of these methods increases exponentially and makes them not suitable for the data classification with high dimensions. To tackle the issue, in this article, a variable granularity search-based multiobjective EA, termed as VGS-MOEA, is proposed for high-dimensional FS, where one bit in the individual representation denotes a group of features and results in the search space reducing greatly. To be specific, at the beginning, the search granularity of VGS-MOEA is coarse (a bit denotes a great number of features), which helps the proposed algorithm detect the potentially good feature subsets quickly. As the evolution continues, the search granularity is refined gradually, where a bit denotes a smaller number of features until it only represents one feature. With this decomposition of granularity, a more refined search is performed and leads to the VGS-MOEA obtaining feature subsets with higher quality. Experimental results on 12 high-dimensional data sets with different characteristics have shown that in comparison with the state of the arts, the proposed VGS-MOEA has demonstrated its superiority in terms of the classification accuracy, the number of selected features, and the running time.

Recently, evolutionary algorithms (EAs) have shown their promising performance in solving the hyperspectral endmember extraction (EE) task. Despite that, most of the existing EA-based EE algorithms mainly take advantage of the global search capability of evolutionary computation. A few of them focus on the hyperspectral EE task itself, which is a sparse large-scale problem with constraint. To fill the gap, in this article, a global-to-local EA (GL-EA) is proposed, where the global and local search is performed sequentially to extract the endmembers effectively. Specifically, in the first global search stage, two complementary solution generation strategies, including asymmetric flip-based solution generation and spectral angle distance (SAD)-based solution repair, are designed, with which the sparse large-scale search space of hyperspectral EE is fully explored and the endmembers that satisfy the constraint could be achieved. Then, in the second stage, a perturbation-based local search is suggested, which further enhances the quality of the obtained endmembers. In addition, an endmember repetition-based solution selection strategy is also developed for both global and local search stages, by using which good solutions can be selected effectively during the evolution. Experimental results on different hyperspectral datasets demonstrate that when compared with the state-of-the-art EE algorithms, the proposed GL-EA could extract the endmembers with higher quality.

Hyperspectral images (HSI) contain hundreds of bands, which provide a wealth of spectral information and enable better characterization of features. However, the excessive dimensions and redundant information also cause a dimensional disaster for subsequent processing. Band selection is a widely-used dimension reduction technique for hyperspectral images. Traditional methods mainly consider the hyperspectral band selection problem at the level of data, and maintain the information contained in the data, without considering the spatial structures inside hyperspectral images. To fill the gap, in this work, an unsupervised hyperspectral band selection method through structure-conserved and neighborhood-grouped evolutionary algorithm (SNEA) is proposed. Different from other evolutionary algorithms for hyperspectral band selection, firstly, two spatial-structure related optimization objectives are designed, including the locally spatial structure denoted by the pixel’s spatial consistency with its adjacent neighbors and the globally spatial structure denoted by the affinity graph among pixels. With the designed objectives, the hyperspectral band selection is formulated as the problem of conserving spatial structures. Moreover, a neighborhood-grouped pair-wise learning strategy is proposed to generate high-quality offsprings. In this novel strategy, a neighborhood grouping operation is developed to divide the band space into several groups. The population can be initialized efficiently and the offspring solutions can be generated pairwisely under the guidance of grouping. Compared with 9 state-of-the-art comparison algorithms, experimental results on 3 standard hyperspectral datasets demonstrate that the band subset obtained by our proposed SNEA has a better classification performance than the comparison algorithms.

Automatic segmentation of COVID-19 pneumonia lesions is critical for quantitative measurement for diagnosis and treatment management. For this task, deep learning is the state-of-the-art method while requires a large set of accurately annotated images for training, which is difficult to obtain due to limited access to experts and the time-consuming annotation process. To address this problem, we aim to train the segmentation network from imperfect annotations, where the training set consists of a small clean set of accurately annotated images by experts and a large noisy set of inaccurate annotations by non-experts. To avoid the labels with different qualities corrupting the segmentation model, we propose a new approach to train segmentation networks to deal with noisy labels. We introduce a dual-branch network to separately learn from the accurate and noisy annotations. To fully exploit the imperfect annotations as well as suppressing the noise, we design a Divergence-Aware Selective Training (DAST) strategy, where a divergence-aware noisiness score is used to identify severely noisy annotations and slightly noisy annotations. For severely noisy samples we use an regularization through dual-branch consistency between predictions from the two branches. We also refine slightly noisy samples and use them as supplementary data for the clean branch to avoid overfitting. Experimental results show that our method achieves a higher performance than standard training process for COVID-19 pneumonia lesion segmentation when learning from imperfect labels, and our framework outperforms the state-of-the-art noise-tolerate methods significantly with various clean label percentages.

In intra coding, template matching prediction is an effective method to reduce the non-local redundancy inside image content. However, the prediction indicated by the best template matching is not always the actually best prediction. To solve this problem, we propose a method, which merges multiple template matching predictions through a convolutional neural network with attention module. The convolutional neural network aims at exploring different combinations of the candidate template matching predictions, and the attention module focuses on determining the most significant prediction candidate. Besides, the spatial module in attention mechanism can be utilized to model the relationship between the original pixels in current block and the reconstructed pixels in adjacent regions (template). Compared to the directional intra prediction and traditional template matching prediction, our method can provide a unified framework to generate prediction with high accuracy. The experimental results show that, compared the averaging strategy, the BD-rate reductions can reach up to 4.7%, 5.5% and 18.3% on the classic standard sequences (classB-classF), SIQAD dataset (screen content), and Urban100 dataset (natural scenes) respectively, while the average bit rate saving are 0.5%, 2.7% and 1.8%, respectively.

Regarding the interior non-local correlations of image content, the intra predictions in image/video coding standards have still not exploited them efficiently. In this article, we propose an intra coding method that leverages the geometric information extracted from the image content to strengthen the coding performance. During the process of image capturing, if the imaging plane of the camera is not fronto-parallel to the surface of an object in reality, the repetitive patterns on the object's surface in real world will appear with a scale shift in the image domain. Thus, the corresponding potential non-local redundancy cannot be removed through the direct utilization of intra block copy and its variants. To address this problem, we begin with a theoretic analysis into the perspective transformation matrix and derive the underlying geometric information that causes the scale shift in the image domain; afterwards, we propose an intra coding framework based on the geometric information to alleviate the scale shift. It is essential for the coding framework to realize block matching in the rectified domain. Our framework mainly consists of two components, i.e., the intra block copy in the rectified domain through planar perspective transformation to obtain a more accurate prediction for the current coding block and the non-local post-processing filtering in the rectified domain after decompression to achieve a better quality of the reconstructed image. The experimental results show that our proposed method can achieve as high as 18% and an average of 5.0% bit-rate saving on the common dataset Urban100 compared to the HEVC reference software HM-SCC-extension with intra block copy and non-local post-processing filtering.

Mining frequent and high utility itemsets from a transactional database is a significant task in the field of data mining and has attracted increasing attention in the past several years. Recently, researchers focus on designing multiobjective evolutionary algorithms (MOEAs) for the task of mining frequent and high utility itemsets, which has shown promising performance. In this paper, we continue this research line by further exploring the potential of MOEAs for mining frequent and high utility itemsets. Tb be specific, we suggest a closed itemset property based multi-objective evolutionary approach, termed as CP-MOEA, where two individual updating strategies are designed for improving the quality of mining frequent and high utility itemsets. We find that if the superset of an itemset is closed, then this itemset must be dominated by its superset, termed as closed itemset property. The proposed two individual updating strategies exploit this property of closed itemset to guide the evolution of the population at certain times. The experimental results on six real datasets demonstrate the effectiveness of the proposed algorithm CP-MOEA comparing to the state-of-the-art baseline.

In this paper, we propose an enhanced intra block copy method for intra prediction in HEVC through planar perspective transformation for urban building scenes. Since the imaging plane of the camera is not always fronto-parallel to the facades of buildings (object plane) in the image, repetitive patterns on object plane in 3D space appear with scale shift in the image, and the corresponding redundancy cannot be removed through conventional intra block copy based on translational motion model. To solve this problem, we start with theoretic analysis to perspective transformation, and get that the two-parametered planar perspective transformation is the cause of scale shift. Therefore, intra block copy can be enhanced via rectification through planar perspective transformation, whose detailed parameters are derived through vanishing points detected in the image. To achieve the best performance, rate-distortion optimization is utilized to determine whether planar perspective transformation would be used in intra block copy for coding unit (CU) or not. In this way, we enable much more flexible prediction for both non-local translational and non-local perspective repeated image content. Experimental results show that our proposed method can achieve as the highest as 9.9% and averagely 2.9% bit-rate saving on the test images for urban building scenes compared to conventional intra block copy on HEVC platform.

In video compression, intracoding plays an important role in terms of coding efficiency and error resilience and has been an attractive research topic since the standardization of H.264/AVC. In this paper, we propose a high-performance intracoding scheme with the help of epitomic priors. Different from intracoding in H.264/AVC and other video standards, we construct image epitomes as coding priors and use them to generate predictions of intrablocks at the encoder. In addition, we losslessly code and transmit the image epitomes to the decoder. We perform compression-oriented video epitomic analysis and search for the best epitomic priors by using the expectation maximization algorithm. The resulting image epitomes for a video sequence can be viewed as the base layer in spatially scalable video coding. Experiments show that our proposed intracoding scheme improves the state of the art by an average of 0.53 dB in PSNR. Simulations under a packet loss environment also demonstrate that intrarefresh with epitomic priors outperforms random intrarefresh by up to 2 dB, leading to better subjective quality.

Scalable video coding (SVC) is considered as a potentially promising solution to enable the adaptability of video to heterogonous networks and various devices. In spatially scalable video encoder, how to resize the captured high-resolution video to get low-resolution video has great effect on the quality of experience (QoE) in the clients receiving low-resolution video. In this paper, we propose a new resizing algorithm called hybrid epitomic resizing (HER), which can make the resized image preserve the same ‘physical’ resolution with original image by the way of utilizing texture similarity inside image and highlight regions of interest while avoiding potential artifacts. For hybrid epitomic resizing, we also design two new inter-layer prediction methods to eliminate the redundancy between adjacent spatial layers instead of conventional inter-layer prediction. Experimental results show that HER can get resized images with perceptually much better quality and the performance of new inter-layer prediction are comparable to that of conventional inter-layer prediction in H.264 SVC.

In video coding, intra coding plays an important role in both coding efficiency and error resilience. In this paper, a new intra coding method based on video epitomic analysis is proposed. Image epitome suitable for coding is extracted through video epitomic analysis, and is used to generate the prediction for each block. The mapping between the original image and image epitome as well as image epitome should be coded and transmitted to the decoder side. Due to the independence of adjacent reconstructed blocks, the proposed intra coding method can also be applied to improve error resilience of intra refresh. Experimental results show that, the proposed intra coding method can significantly improve the performance of intra coding by 0.7dB in terms of PSNR. The simulation results under packet loss environment show that the proposed intra refresh can out-perform random intra refresh (RIR) by up to 1dB, and better subjective quality can also be obtained.

In the basis of photon texture data generated by parallelized algorithm, we propose the real-time rendering algorithm. This method stores pre-computed results of photon mapping as textures before rendering, and looks for adjacent photons by programming on GPU (graphics processing unit). The algorithm takes full advantage of the computing capacity of GPU to accelerate the process of searching photon map. The experiment has proved that this algorithm can enhance efficiency of rendering for real-time interaction, also ensure rendering effect.