Forecasting user-level multi-energy demand under data scarcity is challenging due to the strong spatial interdependencies among users and the complex coupling across heterogeneous energy carriers. To address the challenges, this paper proposes a few-shot forecasting approach for integrated electricity-gas-heat systems, enabling reliable prediction with limited historical data. First, a meta-learning graph neural network is developed to extract generalizable energy conversion features adaptively across user patterns, effectively alleviating data insufficiency. Second, to accurately capture system dynamics, a multi-relational spatiotemporal graph module is designed to hierarchically model intra-energy dynamics and inter-energy coupling, achieving adaptive fusion of heterogeneous spatiotemporal features. Furthermore, a knowledge-enhanced training mechanism is introduced by embedding physical energy conversion laws into meta-training via pseudo-tasks, enhancing physical consistency and preventing overfitting in few-shot adaptation. Extensive experiments on multi-energy datasets ultimately demonstrate that compared to state-of-the-art forecasting models, the proposed approach achieves significant improvements in average metrics 3.80% for MAE, 7.89% for RMSE, and 13.12% for MAPE).

Transparent and specular objects are frequently encountered in daily life, factories, and laboratories. However, due to the unique optical properties, the depth information on these objects is usually incomplete and inaccurate, which poses significant challenges for downstream robotics tasks. Therefore, it is crucial to accurately restore the depth information of transparent and specular objects. Previous depth completion methods for these objects usually generate structure-less or ambiguous depth predictions. To address these issues, we propose a geometry-aware assisted depth completion method for transparent and specular objects, which focuses on exploring the 3D structural cues of the scene. Specifically, besides extracting 2D features from RGB-D input, we back-project the input depth to a point cloud and build the 3D branch to extract hierarchical scene-level 3D structural features. To exploit 3D geometric information, we design several gated cross-modal fusion modules to effectively propagate multi-level 3D geometric features to the image branch. In addition, we propose an adaptive correlation aggregation strategy to appropriately assign 3D features to the corresponding 2D features. Extensive experiments on ClearGrasp, OOD, TransCG, and STD datasets show that our method outperforms other state-of-the-art methods. We further demonstrate that our method significantly enhances the performance of downstream robotic grasping tasks. The code will be available at: https://github.com/lyz3356/GAA-TSO.

Masked autoencoders have been extended beyond images, but random masking often fails to capture non-uniform motion regions in inherently disordered and irregular data like point cloud videos. In this paper, we propose a motion-guided disentanglement method (MGD) to improve masked autoencoders for point cloud video representation learning. Specifically, we begin by estimating motion intensity using an optimal transport approach, which guides the separate masking of dynamic and static regions. This motion-guided masking ensures balanced coverage, addressing the limitations of random masking in capturing non-uniformly distributed motion regions. Furthermore, we disentangle the prediction tasks into motion prediction for high-motion point tubes and appearance reconstruction for low-motion ones. This disentanglement enables the model to more effectively capture both motion and appearance in point cloud videos. We conducted experiments on four widely used point cloud video datasets—NTU RGB+D, MSR-Action3D, NvGesture, and SHREC’17—which demonstrate that our approach consistently improves masked autoencoders for point cloud video representation learning, achieving new state-of-the-art results. Code will be publicly available on GitHub.

Prohibited item detection in X-Ray baggage images plays a crucial role for preventing the social security and stability. Well annotated X-Ray prohibited item training samples show necessity in achieving high detection performance for X-Ray inspection system. While collection of massive samples is extremely laborious and costly, especially for those X-Ray images, which need professional inspection machine. Synthesizing X-Ray images through Threat Image Projection (TIP) is a promising solution to overcome the data insufficient limitation in prohibited item detection. However, TIP based methods rarely consider the contextual style coherence between the foreground prohibited items and background images, resulting in generating low realistic X-Ray security images. For improving image quality and diversity, we propose a Contextual Style Coherence Network for X-Ray Prohibited item Image Synthesis. Specifically, we first propose a style fusion module to guarantee the style coherence and consistency between the foreground prohibited items and background images. We transfer the threat image projection from image space to feature space, and an affine transformation matrix is applied to uniformly sample the location, ratio and scale of the prohibited items to improve the sample diversity. We further normalize the features of the foreground prohibited item by implementing the style transfer through Gram matrix. Then, a mask partial convolution is designed for inpainting the non-object regions of the foreground prohibited items to achieve a better style transition, especially for the boundary parts. The whole network follows the adversarial training pipeline in an unsupervised manner guided by the incorporation of adversarial loss and total variation regularization. We evaluate the synthetic images generated by our method from different evaluating metrics including image quality and object detection performance on various prohibited item detection datasets. The results verify that our method can effectively generate realistic X-Ray prohibited item images and improve the detection performance.

Recently, with the development of the Vision-Language Model (VLM), adopting such VLM (e.g., CLIP) into object detection framework has gradually become a promising and attractive research direction, and the resulted open vocabulary object detection methods can effectively alleviate the limitations in those close-set ones, making the detectors perceive the unseen world. The core issue in open vocabulary object detection is to design an effective and efficient alignment between the visual (e.g., image) and textual (e.g., caption) features in the semantic space, so that the detectors can capture more information around the open-set scene. Current approaches deploy extra uncurated image-text pairs to pre-train a detector for obtaining a better visual-textual alignment in the feature space. Besides, knowledge distillation technology is also adopted to design an appropriate information transferring flow for aligning the visual-textual knowledge. However, large-scale image-text pairs are not always available to obtain, and the pre-training process will inevitable introduce much more computation overhead. While knowledge distillation methods focus on aligning between the local region visual feature in RoI and the textual features of VLM, neglecting the global information alignment between the image and text. For addressing the dilemmas in these alignment manners, we propose a Global and Local Visual-Textual Alignment for Open Vocabulary Object Detection in this paper. Specifically, our proposed method integrates global image-caption and local region-prompt alignments into a unified learning paradigm. The global alignment takes the whole image and caption as the visual and textual inputs, respectively, and matches the image and caption representations from the detector and the text encoder in CLIP by contrastive learning from the overall perspective. Different from global alignment, the local one concentrates on the accordance between regions and prompts from the aspect of portion description. It extracts and aligns the embeddings for the visual patch RoIs from the image encoder in CLIP and discriminating textual token prompts from the text encoder. Moreover, we also design a prompt tuning strategy, which contains global and local components corresponding to the alignment procedure, for better adapting CLIP to downstream task object detection in a parameter-efficient learning manner. By implementation on Faster R-CNN, we conduct experiments on open vocabulary benchmarks OV-COCO and OV-LVIS, respectively. The results verify that our proposed method can achieve clear improvement over counterparts on novel categories, while performing favorably against state-of-the-arts.

Recently, based on multi-view stereo (MVS) methods, utilizing stereo prior information to guide novel-view synthesis has become an important approach to addressing the generalization problem of neural radiance fields (NeRF). However, this approach faces challenges in handling certain difficult scenarios, such as weak texture regions, where it struggles to effectively extract geometric features information. As a result, it encounters limitations in stereo image feature representation and inaccuracies in prior depth estimation. To tackle these problems, we propose the first generalizable novel-view synthesis method that jointly optimizes structured light and NeRF. Considering that active stereo methods based on structured light optimization can enhance geometric feature extraction capability in weak texture regions through the addition of a texture layer, we integrate active stereo vision into novel-view synthesis. To this end, we propose a novel framework, dubbed SLNeRF. First, we design a structured light generation scheme based on Fourier transform and establish a differentiable imaging model using geometric optics and the Lambertian model to generate active stereo images. Then, we obtain stereo image features, as well as prior depth information through the feature extractor, which are used to construct 3D feature volumes. Finally, we accomplish the novel-view synthesis task through a neural renderer. Compared to state-of-the-art generalizable NeRF methods, our method reports encouraging results on public datasets as well as in real-world scenarios.

Most indoor depth completion tasks rely on convolutional auto-encoders to reconstruct depth images, especially in areas with significant missing values. While traditional convolution treats valid and missing pixels equally, Partial Convolution (PConv) has mitigated this limitation. However, PConv fails to distinguish the varying degree of invalidity across different missing areas, which highlights the need for a more refined strategy. To solve this problem, we propose a novel system for indoor depth completion tasks that leverages Mask-adaptive Gated Convolution (MagaConv). MagaConv utilizes gated signals to selectively apply convolution kernels based on the characteristics of missing depth data. These gating signals are generated using shared convolution kernels that jointly process depth features and corresponding masks, ensuring coherent weight optimization. Additionally, the mask undergoes iterative updates according to predefined rules. To improve the fusion of depth and color information, we introduce a Bi-directional Aligning Projection (Bid-AP) module, which utilizes a bi-directional projection scheme with global spatial-channel attention mechanisms to filter out depth-irrelevant features from other modalities. Extensive experiments on popular benchmarks, including NYU-Depth V2, DIML, and SUN RGB-D, demonstrate that our model outperforms state-of-the-art methods in both accuracy and efficiency.

Open-set domain generalization (OSDG) aims to recognize known categories in unseen target domains without fine-tuning, while rejecting unknown categories. Existing methods show limited practicality for the requirement of extra generated or collected unknown samples, or for the assumption that the known categories appear in all source domains. Additionally, they need to determine an optimal threshold for distinguishing between known and unknown samples during testing, which is impractical in OSDG, as the target domain is unavailable during training. Besides, they are usually studied with conventional CNNs and shows unsatisfactory generalizability. To address these issues, we harness the transferable property of the pre-trained vision-language model CLIP, and propose domain-agnostic unknown-aware prompt learning (DUPL) framework to achieve threshold-free OSDG. Specifically, we train unknown tokens (UT) to enable threshold-free unknown rejection through unknown-aware prompt learning (UAPL) with only known data, and then introduce a Fourier-based data augmentation (FDA) strategy to obtain domain-agnostic prompts via domain-agnostic semantic consistency (DASC) regularization. Extensive experiments show that our method achieves state-of-the-art OSDG performance. Code is available at https://github.com/X-funbean/DUPL.

Image captioning aims to create natural language descriptions of images. Recent advancements in image captioning have explored text-only training methods that eliminate the need for image annotations. However, these methods are prone to generate descriptions that include objects that do not actually appear in the image, but are instead drawn from the retrieval texts or hard prompts-resulting in object hallucinations. To address this issue, we propose synergistic prompting mechanism called NASCap (Noise-Aware Decoding with Salient Region Enhancing for Zero-Shot Image Captioning). Our method further improves the accuracy of generated captions by designing a fusion model with mask attention that isolate integrates retrieved captions with input features. The hard prompt mixed with negative entities is designed to further improve model's robust to wrong information. Additionally, we introduce a training-free multi-granularity fusion strategy that dynamically perceive and enhance salient regions into global representation. Extensive experiments demonstrate that NASCap sets a new state-of-the art cross-domain (transferable) captioning and performs Through extensive experiments, our straightforward yet powerful approach has demonstrated its efficacy, outperforming the state-of-the-art methods by a significant margin in image captioning compared to zero-shot captioning based on text-only training.

Prohibited item detection based on X-ray images is one of the most effective security inspection methods. However, the foreground-background feature coupling caused by the overlapping phenomenon specific to X-ray images makes general detectors designed for natural images perform poorly. To address this issue, we propose a Category Semantic Prior Contrastive Learning (CSPCL) mechanism, which aligns the class prototypes perceived by the classifier with the content queries to correct and supplement the missing semantic information responsible for classification, thereby enhancing the model sensitivity to foreground features. To achieve this alignment, we design a specific contrastive loss, CSP loss, which comprises the Intra-Class Truncated Attraction (ITA) loss and the Inter-Class Adaptive Repulsion (IAR) loss, and outperforms classic contrastive losses. Specifically, the ITA loss leverages class prototypes to attract intra-class content queries and preserves essential intra-class diversity via a gradient truncation function. The IAR loss employs class prototypes to adaptively repel inter-class content queries, with the repulsion strength scaled by prototype-prototype similarity, thereby improving inter-class discriminability, especially among similar categories. CSPCL is general and can be easily integrated into Deformable DETR-based models. Extensive experiments on the PIXray, OPIXray, PIDray, and CLCXray datasets demonstrate that CSPCL significantly enhances the performance of various state-of-the-art models without increasing inference complexity. The code is publicly available at https://github.com/Limingyuan001/CSPCL.

Capturing dense visual perception in cluttered environments using depth sensors is crucial for downstream robotics tasks. However, occlusions between objects and unreliable depth data make it very challenging for robots to perceive comprehensive and accurate scene information. To address these issues, we propose a novel robot active scene perception method called RASP, which is composed of two parts. First, we introduce a temporal attention-based view planning algorithm, which actively plans the minimum feasible viewpoint sequence based on latent dependencies in all previous observations to maximize the information perception of the cluttered environments. Subsequently, for the unreliable depth data obtained by the depth sensor, especially caused by transparent and specular objects in the scene, we design a geometry-guided depth completion network that fully utilizes the 3D scene information during the progressive perception process. Specifically, multi-level scene geometric features are extracted and projected into the image space, combining with the image features to guide the depth completion step. These two parts are learned jointly to achieve consistent and accurate results. Extensive experiments demonstrate that our method outperforms state-of-the-art methods. Furthermore, we show that our method significantly improves the performance of downstream grasping tasks.

Threat Image Projection (TIP) is a convenient and effective means to expand X-ray baggage images, which is essential for training both security personnel and computer-aided screening systems. Existing methods are primarily divided into two categories: X-ray imaging principle-based methods and GAN-based generative methods. The former cast prohibited items acquisition and projection as two individual steps and rarely consider the style consistency between the source prohibited items and target X-ray images from different datasets, making them less flexible and reliable for practical applications. Although GAN-based methods can directly generate visually consistent prohibited items on target images, they suffer from unstable training and lack of interpretability, which significantly impact the quality of the generated items. To overcome these limitations, we present a conceptually simple, flexible and unsupervised end-to-end TIP framework, termed as Meta-TIP, which superimposes the prohibited item distilled from the source image onto the target image in a style-adaptive manner. Specifically, Meta-TIP mainly applies three innovations: 1) reconstruct a pure prohibited item from a cluttered source image with a novel foreground-background contrastive loss; 2) a material-aware style-adaptive projection module learns two modulation parameters pertinently based on the style of similar material objects in the target image to control the appearance of prohibited items; 3) a novel logarithmic form loss is well-designed based on the principle of TIP to optimize synthetic results in an unsupervised manner. We comprehensively verify the authenticity and training effect of the synthetic X-ray images on four public datasets, i.e., SIXray, OPIXray, PIXray, and PIDray dataset, and the results confirm that our framework can flexibly generate very realistic synthetic images without any limitations.

Fine-grained domain generalization (FGDG) is a more challenging task than traditional DG tasks due to its small inter-class variations and relatively large intra-class disparities. When domain distribution changes, the vulnerability of subtle features leads to a severe deterioration in model performance. Nevertheless, humans inherently demonstrate the capacity for generalizing to out-of-distribution data, leveraging structured multi-granularity knowledge that emerges from discerning the commonality and specificity within categories. Likewise, we propose a Feature Structuralized Domain Generalization (FSDG) model, wherein features experience structuralization into common, specific, and confounding segments, harmoniously aligned with their relevant semantic concepts, to elevate performance in FGDG. Specifically, feature structuralization (FS) is accomplished through joint optimization of five constraints: a decorrelation function applied to disentangled segments, three constraints ensuring common feature consistency and specific feature distinctiveness, and a prediction calibration term. By imposing these stipulations, FSDG is prompted to disentangle and align features based on multi-granularity knowledge, facilitating robust subtle distinctions among categories. Extensive experimentation on three benchmarks consistently validates the superiority of FSDG over state-of-the-art counterparts, with an average improvement of 6.2% in FGDG performance. Beyond that, the explainability analysis on explicit concept matching intensity between the shared concepts among categories and the model channels, along with experiments on various mainstream model architectures, substantiates the validity of FS.

Prohibited item detection in X-ray images is one of the most essential and highly effective methods widely employed in various security inspection scenarios. Considering the significant overlapping phenomenon in X-ray prohibited item images, we propose an anti-overlapping detection transformer (AO-DETR) based on one of the state-of-the-art (SOTA) general object detectors, DETR with improved denoising anchor boxes (DINO). Specifically, to address the feature coupling issue caused by overlapping phenomena, we introduce the category-specific one-to-one assignment (CSA) strategy to constrain category-specific object queries in predicting prohibited items of fixed categories, which can enhance their ability to extract features specific to prohibited items of a particular category from the overlapping foreground-background features. To address the edge blurring problem caused by overlapping phenomena, we propose the look forward densely (LFD) scheme, which improves the localization accuracy of reference boxes in mid-to-high-level decoder layers and enhances the ability to locate blurry edges of the final layer. Similar to DINO, our AO-DETR provides two different versions with distinct backbones, tailored to meet diverse application requirements. Extensive experiments on the PIXray, OPIXray, and HIXray datasets demonstrate that the proposed method surpasses the SOTA object detectors, indicating its potential applications in the field of prohibited item detection. The source code will be available at: https://github.com/Limingyuan001/AO-DETR.

Personalized Human Activity Recognition (PHAR) based on wearable sensors is crucial in the medical, sports, industrial and other fields. PHAR faces challenges of privacy leakage and a shortage of labeled data. Therefore, we propose a framework called self-supervised federated learning for personalized human activity recognition (SSF-HAR) to implement private PHAR. To protect user privacy, our framework integrates federated learning (FL) to achieve the transmission of only model parameters between the cloud and clients, rather than user data. Besides, we propose a strategy of weighted aggregation to update the cloud model with the client models. To overcome the lack of labeled data, our framework introduces self-supervised learning (SSL) tasks to pretrain a feature extractor in the cloud. The proxy task of SSL transforms data and provides pseudo-labels in three forms. We test the performance on the benchmark datasets MotionSense and WIDSM. The experiments show that SSF-HAR outperforms other FL frameworks for PHAR.

Dual-view baggage inspection has been widely applied in real-world scenarios, where orthogonal viewpoints are deployed to capture diverse and complementary information. Compared with single-view, it can effectively improve the identification performance when rotation and overlay hinder the viewability of the objects. However, this topic has not been rigorously explored due to the scarcity of datasets. To overcome this limitation, we contribute the first fully public large-scale Dual-view X-ray dataset. Our dataset, named DvXray, contains 16,000 pairs, 32,000 X-ray images, in which 15 common classes of 5,496 prohibited items are manually labeled. Besides, we propose an approach named Adaptive Hierarchical Cross Refinement (AHCR) to establish a strong baseline for prohibited item discovery in dual-view X-ray images. AHCR hypothesizes that each input pair is sampled from one mixture distribution, hence gathering the non-overlapping and position-aware cues along the shared axis and complementarily delivering to the other in a hierarchical structure to enrich the feature discriminability of the objects of interest from background overlaps. Upon this structure, we propose an adaptive control strategy and a confidence-weighted view fusion term to make it robust to difficult samples. Extensive experiments on DvXray show that AHCR not only brings significant classification gains over various backbones, such as recent Swin Transformer and ConvNeXt, but also exhibits an impressively better ability to localize objects. In addition, AHCR performs favorably against the counterparts and some recent multi-view learning approaches, moving a step closer towards potential application in practice. Dataset and code are available at https://github.com/Mbwslib/DvXray.

Prohibited item detection in X-Ray baggage images can efficiently prevent the social security and stability. With the development of deep learning, applying specific methods in computer vision tasks to prohibited item detection has shown promising perspectives, and the resulted intelligent security inspection system can effectively address the limitations of human inspection. Despite several deep learning based methods have been proposed to flourish this researching field, there are still two issues which have not been fully explored. First, most of them suffer from the shortage of dataset and collection of massive and well-annotated samples is extremely laborious and costly. Second, the designation of backbone modules for current prohibited item detection methods shares similar idea with the ones in object detection. While little consideration has been paid to the properties of prohibited item, where most of them are over-lapped and cluttered. To overcome limitations, this paper proposes a cluttered prohibited item detection method for security inspection system. Specifically, our method first generates synthetic X-Ray images through cut-and-paste strategy from the training samples in each training mini-batch, where the strategy effectively and efficiently augments the dataset and quality of the synthetic samples can be guaranteed. Then, a high-order dilated convolution module is developed for enriching the representation ability of the feature, further promoting the localization ability for over-lapped and cluttered prohibited items. Experiments show that our proposed method can be well generalized to various datasets, and achieve clear improvement over state-of-the-art methods.

Sensor-based Human Activity Recognition (HAR) is widely used in daily life and is the basic-level bridge to virtual healthcare in the metaverse. The current challenge is the low recognition accuracy for personalized users on smart wearable devices. The limited resource cannot support large deep learning models updated locally. Besides, integrating and transmitting sensor data to the cloud would reduce the efficiency. Considering the tradeoff between performance and complexity, we propose a Lightweight Human Activity Recognition (LHAR) framework. In LHAR, we combine the cross-people HAR task with the lightweight model task. LHAR framework is designed on the teacher-student architecture and the student network consists of multiple depthwise separable convolution layers to achieve fewer parameters. The dark knowledge distilled from the complex teacher model enhances the generalization ability of LHAR. To achieve effective knowledge distillation, we propose two optimization methods. Firstly, we train the teacher model by ensemble learning to promote teacher performance. Secondly, a multi-channel data augmentation method is proposed for the diversity of the dataset, which is a plug-in operation for the ensemble teacher model. In the experiments, we compare LHAR with state-of-art models in comparison evaluation, ablation study and the hyperparameter analysis, which proves the better performance of LHAR in efficiency and effectiveness.

Machine learning models often suffer from severe performance degradation due to distributional shifts between testing and training data. To address this issue, researchers have focused on domain generalization (DG), which aims to generalize a model trained on source domains to arbitrary unseen target domains. Recently, ensemble learning has emerged as a popular strategy for addressing the DG problem, and domain-specific experts are typically involved. However, the existing methods do not sufficiently consider the generalizability of individual experts or leverage the consistency and diversity among them, thus limiting the generalizability of the constructed models. In this paper, we propose a consistent-but-diverse mixture of experts (CBDMoE) algorithm, which is an improved MoE framework that effectively harnesses ensemble learning for solving the DG problem. Specifically, we introduce individual expert learning (IEL), which incorporates a novel domain-class-balanced subset division (DCBSD)-based sampling strategy to facilitate a generalizable expert learning process. Additionally, we present consistent-but-diverse learning (CBDL), which employs two regularizing losses to encourage consistency and diversity in the predictions of the experts. Our proposed strategy significantly enhances the generalizability of the MoE framework. Extensive experiments conducted on three popular DG benchmark datasets demonstrate that our method outperforms the state-of-the-art approaches.

Recently, stereo vision based on lightweight RGBD cameras has been widely used in various fields. However, limited by the imaging principles, the commonly used RGB-D cameras based on TOF, structured light, or binocular vision acquire some invalid data inevitably, such as weak reflection, boundary shadows, and artifacts, which may bring adverse impacts to the follow-up work. In this paper, we propose a new model for depth image completion based on the Attention Guided Gated-convolutional Network (AGG-Net), through which more accurate and reliable depth images can be obtained from the raw depth maps and the corresponding RGB images. Our model employs a UNet-like architecture which consists of two parallel branches of depth and color features. In the encoding stage, an Attention Guided Gated-Convolution (AG-GConv) module is proposed to realize the fusion of depth and color features at different scales, which can effectively reduce the negative impacts of invalid depth data on the reconstruction. In the decoding stage, an Attention Guided Skip Connection (AG-SC) module is presented to avoid introducing too many depth-irrelevant features to the reconstruction. The experimental results demonstrate that our method outperforms the state-of-the-art methods on the popular benchmarks NYU-Depth V2, DIML, and SUN RGB-D. https://github.com/htx0601/AGG-Net

Personalized human activity recognition (HAR) on wearable sensor data is crucial for healthcare, industry and sport. Personalized HAR has two challenges: very few high-quality labeled personalized data and limited terminal computing resource. Most previous domain adaptation models ignore the transfer efficiency on the terminal, which is also one criterion of Artificial Intelligence of Things. Therefore, we propose a fast transfer HAR, FastTrans, to improve the efficiency and get a trade-off with recognition effectiveness. A fusion feature extraction module is designed to learn multi-scale features on the improved hybrid loss. The proposed heuristic parameter estimation method learns the approximate solutions of the classification weights in FastTrans by scanning the adaptation data only once. Besides, an efficient time series data augmentation is proposed as a plugin for dataset variety. The results on benchmark datasets show the dramatically competitive performance of FastTrans on transferring efficiency with close accuracy to other models.

Video Anomaly Detection (VAD) involves detecting anomalous events in videos, presenting a significant and intricate task within intelligent video surveillance. Existing studies often concentrate solely on features acquired from limited normal data, disregarding the latent prior knowledge present in extensive natural image datasets. To address this constraint, we propose a Prior Knowledge Guided Network(PKG-Net) for the VAD task. First, an auto-encoder network is incorporated into a teacher-student architecture to learn two designated proxy tasks: future frame prediction and teacher network imitation, which can provide better generalization ability on unknown samples. Second, knowledge distillation on proper feature blocks is also proposed to increase the multi-scale detection ability of the model. In addition, prediction error and teacher-student feature inconsistency are combined to evaluate anomaly scores of inference samples more comprehensively. Experimental results on three public benchmarks validate the effectiveness and accuracy of our method, which surpasses recent state-of-the-arts.

Prohibited item segmentation has a wide range of applications in the security check field, such as computer-aided screening, threat image projection and material discrimination. However, the severe object overlapping in X-ray baggage images restricts the performance of common CNN-based segmentation methods greatly. Worse, no public dataset can be used to promote research in this challenging and promising area. In this paper, to cope with these problems, we present the first Prohibited Item X-ray segmentation dataset named PIXray. PIXray comprises 5,046 X-ray images, in which 15 classes of 15,201 prohibited items are annotated as instance-level masks. Besides, we contribute a dense de-overlap attention snake (DDoAS) in the context of deep learning for automated and real-time prohibited item segmentation. DDoAS mainly includes a dense de-overlap module (DDoM) and an attention deforming module (ADM). Specifically, DDoM is designed to infer prohibited item information accurately from extreme background overlaps through dense reversed connections. ADM aims to improve the low learning efficiency introduced by large variations in shapes and sizes among different prohibited items. Comprehensive evaluation on the PIXray shows the effectiveness and superiority of DDoM and ADM. DDoM excels at recognizing prohibited items from complex backgrounds than other in-domain methods and achieves consistent performance gain over various network backbones, extending the idea of tackling overlapping images data. ADM can ease the model training and further refine the mask quality. Furthermore, out-of-domain experiments prove that DDoAS can also be applied to natural images and achieves comparable performance to the state-of-the-art methods, which implies its potential applications in other fields. The dataset and source code are available athttps://github.com/Mbwslib/DDoAS.

Skeleton-based human action recognition has received extensive attention due to its efficiency and robustness to complex backgrounds. Though the human skeleton can accurately capture the dynamics of human poses, it fails to recognize human actions induced by the interaction between human and objects, making it is of great importance to further explore the interaction between the human and objects for human action recognition. In this paper, we devise the multi-stream interaction networks (MSIN), to simultaneously explore the dynamics of human skeleton, objects, and the interaction between human and objects. Specifically, apart from the traditional human skeleton stream, 1) the second stream explores the dynamics of object appearance from the objects surrounding the human body joints; and 2) the third stream captures the dynamics of object position in regard to the distance between the object and different human body joints. Experimental results on three popular skeleton-based human action recognition datasets, NTU RGB + D, NTU RGB + D 120, and SYSU, demonstrate the effectiveness of the proposed method, especially for recognizing the human actions with human-object interactions.

Saliency detection is one of the key issues in simulating visual attention selection. Most attention models adopt the competitive structure to simulate the human visual system. Although these models provide remarkable results and convincing biological plausibility, they are still confronted with many difficulties in practical applications because of their extreme time cost and parameter sensitivity. Recently, a new saliency detection approach based on Fourier transform, as represented by spectral residual (SR) and phase Fourier transform (PFT), has been attracting much attention for its excellent accuracy and computational speed. All these models can be unified into one framework called amplitude spectrum modulation (ASM). The aim of this paper is to explore the intrinsic mechanism of ASM and develop an advanced ASM model. After analyzing SR and PFT, we give a mathematical description for the fundamental idea and the inherent limitations of the existing ASM models. A new saliency detective model, based on the scale-invariant ASM, scene and context-based modulation, and competitive structure, is also proposed breaking through the limitations of the traditional ASM models. Simulation results suggest that the proposed model is more accurate in predicting human eye fixation and is more robust against different types of stimulus when compared with competing models.

There have been many computational models mimicking the visual cortex that are based on spatial adaptations of unsupervised neural networks. In this paper, we present a new model called neuronal cluster which includes spatial as well as temporal weights in its unified adaptation scheme. The "in-place" nature of the model is based on two biologically plausible learning rules, Hebbian rule and lateral inhibition. We present the mathematical demonstration that the temporal weights are derived from the delay in lateral inhibition. By training with the natural videos, this model can develop spatio-temporal features such as orientation selective cells, motion sensitive cells, and spatio-temporal complex cells. The unified nature of the adaptation scheme allows us to construct a multilayered and task-independent attention selection network which uses the same learning rule for edge, motion, and color detection, and we can use this network to engage in attention selection in both static and dynamic scenes.