In recent years, humanoid robots have gradually emerged as crucial “smart terminals” within the IoT, significantly expanding the application scenarios of IoT. This paper proposes a visual enhancement based robust 3D Gaussian SLAM (3DGS-SLAM) to improve the environmental perception ability of humanoid robots. First, the dynamic mask provided by YOLOv11 is optimized using composite morphological operations to obtain high-confidence static features, which are then employed for accurate estimation of the robot’s pose. Subsequently, a keyframe selection strategy combining scaled interval Kalman filtering (SIKF) with the motion model of a floating base humanoid robot is designed. It quantifies the uncertainty of pose by dynamically dividing the motion interval of the robot, and introduces a scaling factor to adaptively adjust the interval width, thereby filtering out high-quality keyframes to ensure stable pose tracking and efficient mapping. Further, a mapping strategy that combines Gaussian-Laplacian pyramid and Gaussian ellipsoid adaptive density control is developed by combining 3D Gaussian splashing. This method can not only identify Gaussian ellipsoids requiring segmentation by preserving high-frequency details in keyframes, but also control their splitting direction through gradient modulation, effectively reducing redundancy among Gaussian ellipsoids while enhancing mapping quality. The experimental results show that our method improves the average absolute trajectory error(ATE) on the BONN and TUM datasets by an average of 90.8% and 93.1% compared to ORB-SLAM3, significantly improving the localization accuracy and mapping consistency of humanoid robots in dynamic crowded scenes.

Deep learning applications in fault diagnosis face two critical challenges. First, a significant distribution gap between source domain training data and target domain samples with unknown noise patterns. Second, labeled fault data remain scarce in practice. These issues hinder the practical deployment. This paper presents a Cross-View Consistency Network (CVC-Net) to tackle these problems through noise-generalization capabilities. The method learns robust features from limited source domain data. It maintains diagnostic accuracy with unknown noise data, without prior knowledge of target noise characteristics. CVC-Net processes temporal waveforms and Gramian Angular Field representations through specialized encoders, exploiting their asymmetric noise sensitivities. A cross-view consistency mechanism extracts fault patterns across modalities. The method integrates fault-aware prototype learning for enhanced discrimination with limited labels and employs adaptive fusion that weights view contributions based on cross-view prediction. Experimental validation shows that CVC-Net is effective in challenging scenarios. When tested on target domain with unknown noise types, CVC-Net maintains reliable performance, effectively handling noise patterns not present during source domain training. Under limited-label conditions, it outperforms existing methods in diagnostic performance.

With the popularity of social media, users are able to express their opinions in multiple forms, such as text, audio, and video. Traditional unimodal sentiment analysis methods can no longer meet the processing requirements of such multisource heterogeneous data, which makes multimodal sentiment analysis a research hotspot. However, most existing methods rely on simple feature splicing or weighted fusion, neglecting the differences in the reliability of different modalities and failing to fully explore the intermodality consistency and difference information. In this article, we propose a multimodal competitive fusion mechanism and construct multimodal competitive fusion model (MCFM). The model first dynamically evaluates the reliability of each modality through the competition mechanism and adaptively assigns weights accordingly. Then it decomposed the modality representations into similar and dissimilar features through modality feature decomposition, supplemented by the overlap of orthogonal traffic channel attention constraints, to achieve the collaborative learning of consistency and dissimilarity features. We evaluate the proposed model on several datasets. In our experiments, we used textual modality data from the dataset with audio modality data for the experiments. The experimental results show that MCFM has 2%–3% higher binary accuracy (ACC2) than the baseline model on the sentiment classification task (with 2% higher binary accuracy under the negative/nonnegative metrics and 3% higher binary accuracy under the positive/negative metrics), and that on the regression task, MCFM’s mean absolute error on the test dataset is 3% lower than that of the baseline model.

Given a query sentence, text-to-image person retrieval aims to identify matched pedestrian images from a large gallery. Most of the existing methods are designed for the unified domain setting, which is operated under the assumption that the training and test data are drawn from the same distribution. However, this assumption is difficult to guarantee in real application scenes, as data is often collected from various surveillance scenarios. To this end, in this paper, we introduce the concept of single-source domain generalization into the context of text-to-image person retrieval and propose a novel task called single-source domain generalizable text-to-image person retrieval (SSDG-TIPR). This task is applicable in real-world scenarios but poses significant challenges due to the limitation of accessible training data. Intuitively, a trained model is the most familiar with the domain on which it was trained, that is, the source domain. Therefore, to handle this SSDG-TIPR task, we propose a new method to infinitely close astray features from unseen target domains to the source domain, namely, to take it home (TIME), allowing the model to handle the features in a familiar manner. The proposed TIME method comprises three main modules: the Domain Astray Leading (DAL) module, the Domain Invariant Feature Extract (DIFE) module and the Domain Home Taking (DoT) module. We evaluated TIME on 3 benchmark datasets, namely CUHK-PEDES, ICFG-PEDES and RSTPReid, and demonstrated its superior performance on 10 SSDG-TIPR sub-tasks as well as on 3 conventional TIPR sub-tasks, establishing a new state-of-the-art in both settings.

Existing weakly supervised temporal action localization (WTAL) methods typically follow a decoupled classification-localization pipeline: segment-level classifiers are trained first, and their predictions are then aggregated to score proposals at inference. Under this training-inference discrepancy, proposal scoring at inference relies on an additional aggregation step, which can accumulate errors from noisy segment responses and thus undermine score reliability. Moreover, proposal scores are often directly used as confidence without explicit score-quality modeling or quality-aware evaluation, further contributing to pronounced score-quality misalignment and thus widening the classification-localization gap. To address proposal score-quality misalignment, we propose ACL-Net, a framework for proposal score calibration. At its core is a dual-axis Proposal-level Action Consistency Learning (PACL) paradigm, implemented through two complementary modules: (i) a Semantic Consistency Module (SCM) that refines proposal representations by maintaining fused class centers to enforce compact and robust same-class features; within SCM, a cross-modal consistency-driven Classification Enhancement Module (CEM) denoises the fused class centers to mitigate error accumulation under weak supervision; and (ii) a Process Consistency Module (PCM) that derives geometry-aware reference scores from relative temporal relations among overlapping proposals, guiding the model to assess proposal quality in terms of relative process completeness and improve score-quality alignment. By jointly modeling semantic and process consistency to calibrate proposal scores, ACL-Net markedly improves localization accuracy. On THUMOS14 and ActivityNet1.3, it achieves state-of-the-art performance with uniform and substantial gains across multiple established baselines, while markedly lowering the expected calibration error (ECE).

In human pose estimation, formulating keypoint localization as a classification task over discretized coordinate grids has proven effective. Essentially, the 2D features of the keypoints are reduced to 1D coordinate representations. This process leads to the loss of spatial constraints among keypoints and increases the difficulty for the model to capture their structural relationships. To address this issue, we propose an enhanced query attention mechanism constrained by bidirectional graphs. The core idea is to establish the topological constraints on the 1D coordinate representations. First, two fundamental connection directions of the skeleton are defined and encoded as a pair of adjacency matrices to enhance the feature interaction capability of the graph convolutional network (GCN). Second, a GCN-guided multi-scale feature fusion framework is designed to effectively combine multi-scale visual features with structural priors, thereby enhancing the representation of keypoint spatial distributions. Finally, a dual-gate module is incorporated into a GCN-guided attention unit to construct a structured query matrix constrained by the bidirectional skeleton graphs, which helps filter out spurious joint interactions and emphasize plausible ones. Extensive experiments on Tai Chi Chuan-Pose, Animal-Pose, AP-10K, MPII, COCO, and COCO-WholeBody datasets demonstrate that the proposed method outperforms existing methods in terms of both accuracy and robustness, particularly in balancing precise local keypoint localization with global pose consistency.

Video anomaly detection (VAD) has been extensively researched due to its potential for intelligent video systems. However, most existing methods based on CNNs and transformers still suffer from substantial computational burdens and have room for improvement in learning spatial-temporal normality. Recently, Mamba has shown great potential for modeling long-range dependencies with linear complexity, providing an effective solution to the above dilemma. To this end, we propose a lightweight and effective Mamba-based network named STNMamba, which incorporates carefully designed Mamba modules to enhance the learning of spatial-temporal normality. Firstly, we develop a dual-encoder architecture, where the spatial encoder equipped with Multi-Scale Vision Space State Blocks (MS-VSSB) extracts multi-scale appearance features, and the temporal encoder employs Channel-Aware Vision Space State Blocks (CA-VSSB) to capture significant motion patterns. Secondly, a Spatial-Temporal Interaction Module (STIM) is introduced to integrate spatial and temporal information across multiple levels, enabling effective modeling of intrinsic spatial-temporal consistency. Within this module, the Spatial-Temporal Fusion Block (STFB) is proposed to fuse the spatial and temporal features into a unified feature space, and the memory bank is utilized to store spatial-temporal prototypes of normal patterns, restricting the model's ability to represent anomalies. Extensive experiments on three benchmark datasets demonstrate that our STNMamba achieves competitive performance with fewer parameters and lower computational costs than existing methods.

Abstract Hydraulic-supports alignment is to keep the coal mining face in line and is heavily influenced by the various geological states. The experiences produced by the moving process are unbalanced, which leads to the agent not learning important knowledge from the rare samples. This paper is the first to introduce the reinforcement learning to the hydraulic-supports alignment, and establish the Markov optimal decision model by TD3 algorithm. Aiming at the imbalance issue of the experience, this paper proposes a segmented experience pool and three sampling replay mechanisms according to the characteristics of the moving process with various geological states. Experimental results show that the improved TD3, utilizing a segmented experience pool with three different replay mechanisms, could effectively identify the optimal moving policy and achieve significant convergence in cases involving both normal movement and insufficient movement of hydraulic-supports. In contrast, the TD3 performs inadequately and struggles to find the optimal policy.

Robotic arm target grasping by vision support is a commonly used method in grasping tasks and is usually used for multi-target complex scenes. Where vision support is generally used to identify the targets and to get their positions, categories and sizes. Most robotic arm grasping tasks using target recognition methods as visual inspection ignore the relationship between target objects such as the occlusion problem between objects. This limits the targets to be grasped and makes the crawling task inefficient. We propose Grasping with Occlusion-Aware aLly (GOAL) method based on binocular stereo-vision. Firstly, occlusion relationships in the view are directly inferred and targets are segmented as well as localized. Subsequently, multi-target grasping pose estimation is performed to obtain effective grasping positions. Ultimately, validation is conducted on a high-resolution dataset using the EPSON robotic arm. Note to Practitioners—This research significantly advances the field by addressing occlusion challenges in robotic grasping, offering effective methods, a valuable dataset, and practical insights. The proposed Grasping with Occlusion-Aware aLly (GOAL) method was validated on a high-resolution dataset using the EPSON robotic arm, showcasing its applicability and efficiency in real-world scenarios. This work provides valuable contributions to practitioners in the field of robotic manipulation and grasping tasks.

This paper aims to the remote synchronous control for complex networks with amplify-and-forward (AF) relays. Firstly, the AF relay dynamic with random noise is integrated into the control signal transmission. Secondly, the complex networks incorporating an AF relay is converted into the sliding control mode (SMC), which robustly addresses unknown disturbances among the nodes to achieve synchronous states. Thirdly, an adaptive gain mechanism based on the equivalent value of sign function of SMC is proposed to estimate the disturbance amplitude, hence mitigating the chattering of SMC. Finally, the illustrative simulation demonstrated the effectiveness of proposed methodology.

Noncontrastive self-supervised learning methods offer an effective alternative to contrastive approaches by avoiding the need for negative samples to avoid representation collapse. Noncontrastive learning methods explicitly or implicitly optimize the representation space, yet they often require large representation dimensions, leading to dimensional inefficiency. To provide negative samples, contrastive learning methods often require large batch sizes, thus regarded as sample inefficient, while noncontrastive learning methods require large representation dimensions, thus regarded as dimension inefficient. Although we have some understanding of the noncontrastive learning method, theoretical analysis of such phenomenon still remains largely unexplored. We present a theoretical analysis of the dimensional need for noncontrastive learning. We investigate the transfer between upstream representation learning and downstream tasks' performance, demonstrating how noncontrastive methods implicitly increase interclass distances within the representation space and how the distance affects the model performance of evaluation performance. We prove that the performance of noncontrastive methods is affected by the output dimension and the number of latent classes, and illustrate why performance degrades significantly when the output dimension is substantially smaller than the number of latent classes. We demonstrate our findings through experiments on image classification experiments, and enrich the verification in audio, graph and text modalities. We also perform empirical evaluation for image models on extensive detection and segmentation tasks beyond classification that show satisfactory correspondence to our theorem.

Segmenting objects from cluttered backgrounds in single-channel images, such as marine radar echoes, medical images, and remote sensing images, poses significant challenges due to limited texture, color information, and diverse target types. This paper proposes a novel solution: the Onet, an O-shaped assembly of twin U-Net deep neural networks, designed for unsupervised binary semantic segmentation. The Onet, trained with an intensity-complementary image pair and without the need for annotated labels, maximizes the Jensen-Shannon divergence (JSD) between the densely localized features and the class probability maps. By leveraging the symmetry of U-Net, Onet subtly strengthens the dependence between dense local features, global features, and class probability maps during the training process. The design of the complementary input pair aligns with the theoretical requirement that optimizing JSD needs the class probability of negative samples to accurately estimate the marginal distribution. Compared to the current leading unsupervised segmentation methods, the Onet demonstrates superior performance in target segmentation in marine radar frames and cloud segmentation in remote sensing images. Notably, we found that Onet's foreground prediction significantly enhances the signal-to-noise ratio (SNR) of targets amidst marine radar clutter. Onet's source code is publicly accessible at https://github.com/joeyee/Onet.

Text-based person retrieval is the process of searching a massive visual resource library for images of a particular pedestrian, based on a textual query. Existing approaches often suffer from a problem of color (CLR) over-reliance, which can result in a suboptimal person retrieval performance by distracting the model from other important visual cues such as texture and structure information. To handle this problem, we propose a novel framework to Excavate All-round Information Beyond Color for the task of text-based person retrieval, which is therefore termed EAIBC. The EAIBC architecture includes four branches, namely an RGB branch, a grayscale (GRS) branch, a high-frequency (HFQ) branch, and a CLR branch. Furthermore, we introduce a mutual learning (ML) mechanism to facilitate communication and learning among the branches, enabling them to take full advantage of all-round information in an effective and balanced manner. We evaluate the proposed method on three benchmark datasets, including CUHK-PEDES, ICFG-PEDES, and RSTPReid. The experimental results demonstrate that EAIBC significantly outperforms existing methods and achieves state-of-the-art (SOTA) performance in supervised, weakly supervised, and cross-domain settings.

Industrial sensors capture critical information for intelligent manufacturing maintenance. To promote equipment upgrading and manufacturing processes, intelligent decisions, and information learning play an important role. Although deep learning methods historically obtain excellent results, there is always a tradeoff between fine-tuning existing networks or designing models from scratch for sensor data processing. In this article, we propose the multihead attention self-supervised (MAS) representation model, which is a self-supervised learning-based sensor feature extraction network. To the best of our knowledge, this is the first time a self-supervised contrastive learning method using positive samples that represent multidimensional industry sensor data is being used for anomaly detection. We review alternative data augmentation methods proposed for better-representing sensor sequence data. We use this insight to design a new structure that adapts to the temporal characteristics of the application. We apply our method to a real-world water circulation system that uses a variety of industrial sensors. The effectiveness of the proposed MAS methods is demonstrated.

Pre-trained vision-language models have inspired much research on few-shot learning. However, with only a few training images, there exist two crucial problems: (1) the visual feature distributions are easily distracted by class-irrelevant information in images, and (2) the alignment between the visual and language feature distributions is difficult. To deal with the distraction problem, we propose a Selective Attack module, which consists of trainable adapters that generate spatial attention maps of images to guide the attacks on class-irrelevant image areas. By messing up these areas, the critical features are captured and the visual distributions of image features are calibrated. To better align the visual and language feature distributions that describe the same object class, we propose a cross-modal distribution alignment module, in which we introduce a vision-language prototype for each class to align the distributions, and adopt the Earth Mover's Distance (EMD) to optimize the prototypes. For efficient computation, the upper bound of EMD is derived. In addition, we propose an augmentation strategy to increase the diversity of the images and the text prompts, which can reduce overfitting to the few-shot training images. Extensive experiments on 11 datasets demonstrate that our method consistently outperforms prior arts in few-shot learning. The implementation code will be available at https://gitee.com/mindspore/models/tree/master/research/cv/SADA.

Recently, Transformer-based architecture has been introduced into face super-resolution task due to its advantage in capturing long-range dependencies. However, these approaches tend to integrate global information in a large searching region, which neglect to focus on the most relevant information and induce blurry effect by the irrelevant textures. Some improved methods simply constrain self-attention in a local window to suppress the useless information. But it also limits the capability of recovering high-frequency details when flat areas dominate the local searching window. To improve the above issues, we propose a novel self-refinement mechanism which could adaptively achieve texture-aware reconstruction in a coarse-to-fine procedure. Generally, the primary self-attention is first conducted to reconstruct the coarse-grained textures and detect the fine-grained regions required further compensation. Then, region selection attention is performed to refine the textures on these key regions. Since self-attention considers the channel information on tokens equally, we employ a dual-branch feature integration module to privilege the important channels in feature extraction. Furthermore, we design the wavelet fusion module which integrate shallow-layer structure and deep-layer detailed feature to recover realistic face images in frequency domain. Extensive experiments demonstrate the effectiveness on a variety of datasets.

Modeling the spatiotemporal relationship (STR) of traffic data is important yet challenging for existing graph networks. These methods usually capture features separately in temporal and spatial dimensions or represent the spatiotemporal data by adopting multiple local spatial-temporal graphs. The first kind of method mentioned above is difficult to capture potential temporal-spatial relationships, while the other is limited for long-term feature extraction due to its local receptive field. To handle these issues, the Synchronous Spatio-Temporal grAph Transformer (S2TAT) network is proposed for efficiently modeling the traffic data. The contributions of our method include the following: 1) the nonlocal STR can be synchronously modeled by our integrated attention mechanism and graph convolution in the proposed S2TAT block; 2) the timewise graph convolution and multihead mechanism designed can handle the heterogeneity of data; and 3) we introduce a novel attention-based strategy in the output module, being able to capture more valuable historical information to overcome the shortcoming of conventional average aggregation. Extensive experiments are conducted on PeMS datasets that demonstrate the efficacy of the S2TAT by achieving a top-one accuracy but less computational cost by comparing with the state of the art.

Batch normalization (BN) is a milestone technique in deep learning. It normalizes the activation using mini-batch statistics during training but the estimated population statistics during inference. This paper focuses on investigating the estimation of population statistics. We define the estimation shift magnitude of BN to quantitatively measure the difference between its estimated population statistics and expected ones. Our primary observation is that the estimation shift can be accumulated due to the stack of BN in a network, which has detriment effects for the test performance. We further find a batch-free normalization (BFN) can block such an accumulation of estimation shift. These observations motivate our design of XBNBlock that replace one BN with BFN in the bottleneck block of residual-style networks. Experiments on the ImageNet and COCO benchmarks show that XBNBlock consistently improves the performance of different architectures, including ResNet and ResNeXt, by a significant margin and seems to be more robust to distribution shift.

Energy-based latent variable models (EBLVMs) are more expressive than conventional energy-based models. However, its potential on visual tasks are limited by its training process based on maximum likelihood estimate that requires sampling from two intractable distributions. In this paper, we propose Bi-level doubly variational learning (BiDVL), which is based on a new bi-level optimization framework and two tractable variational distributions to facilitate learning EBLVMs. Particularly, we lead a decoupled EBLVM consisting of a marginal energy-based distribution and a structural posterior to handle the difficulties when learning deep EBLVMs on images. By choosing a symmetric KL divergence in the lower level of our framework, a compact BiDVL for visual tasks can be obtained. Our model achieves impressive image generation performance over related works. It also demonstrates the significant capacity of testing image reconstruction and out-of-distribution detection.

The core problem of text-based person retrieval is how to bridge the heterogeneous gap between multi-modal data. Many previous approaches contrive to learning a latent common manifold mapping paradigm following a cross-modal distribution consensus prediction (CDCP) manner. When mapping features from distribution of one certain modality into the common manifold, feature distribution of the opposite modality is completely invisible. That is to say, how to achieve a cross-modal distribution consensus so as to embed and align the multi-modal features in a constructed cross-modal common manifold all depends on the experience of the model itself, instead of the actual situation. With such methods, it is inevitable that the multi-modal data can not be well aligned in the common manifold, which finally leads to a sub-optimal retrieval performance. To overcome this CDCP dilemma, we propose a novel algorithm termed LBUL to learn a Consistent Cross-modal Common Manifold (C3 M) for text-based person retrieval. The core idea of our method, just as a Chinese saying goes, is to 'san si er hou xing', namely, to Look Before yoU Leap (LBUL). The common manifold mapping mechanism of LBUL contains a looking step and a leaping step. Compared to CDCP-based methods, LBUL considers distribution characteristics of both the visual and textual modalities before embedding data from one certain modality into C3 M to achieve a more solid cross-modal distribution consensus, and hence achieve a superior retrieval accuracy. We evaluate our proposed method on two text-based person retrieval datasets CUHK-PEDES and RSTPReid. Experimental results demonstrate that the proposed LBUL outperforms previous methods and achieves the state-of-the-art performance.

Given a natural language description, text-based person retrieval aims to identify images of a target person from a large-scale person image database. Existing methods generally face a color over-reliance problem, which means that the models rely heavily on color information when matching cross-modal data. Indeed, color information is an important decision-making accordance for retrieval, but the over-reliance on color would distract the model from other key clues (e.g. texture information, structural information, etc.), and thereby lead to a sub-optimal retrieval performance. To solve this problem, in this paper, we propose to Capture All-round Information Beyond Color (CAIBC) via a jointly optimized multi-branch architecture for text-based person retrieval. CAIBC contains three branches including an RGB branch, a grayscale (GRS) branch and a color (CLR) branch. Besides, with the aim of making full use of all-round information in a balanced and effective way, a mutual learning mechanism is employed to enable the three branches which attend to varied aspects of information to communicate with and learn from each other. Extensive experimental analysis is carried out to evaluate our proposed CAIBC method on the CUHK-PEDES and RSTPReid datasets in both supervised and weakly supervised text-based person retrieval settings, which demonstrates that CAIBC significantly outperforms existing methods and achieves the state-of-the-art performance on all the three tasks.

The automated conveyor system, as the core component in the modern manufacturing world, has gained lots of attention from researchers. To optimize the operation of the conveyor system, range-inspection control (RIC) has been considered an efficient strategy to bring this conventional technology to an intelligent level. Various algorithms have been put into use to achieve optimal control. However, the current methodologies are only focusing on control optimization, not scaled into the smart manufacturing framework. The schema of alignment and corporation between the physical and virtual spaces for the system remains an important problem. Therefore, the work in this article aims for an effective framework of implementation between the physical and virtual stations in an automated conveyor system. Since increasingly more application scenarios rely on the digital twin (DT) technology to realize the integration of physical and virtual systems, we proposed the DT automated conveyor system (DT-ACS) that constructs the road map to implement the RIC-based conveyor system under the background of a smart factory. Besides, profit-sharing-based deep Q-networks (PDQNs) have been proposed to cope with the RIC optimization problem. The robustness and efficiency of the proposed PDQN were evaluated via sets of experiments. The discussion and conclusion are presented at last accordingly.Note to Practitioners—This article aims to propose a strategy of control optimization for conveyor-based manufacturing systems under the digital twin (DT) framework. The conveyor system can be flexible to control the running flows to avoid overloading workstations. Due to the complex environment in the production line, the range that is able to be inspected and the capacity of the reserve area can be considerably diverse among the workstations. To maximally evaluate our framework, we set a comparatively complex environment for the experiments. Nevertheless, to obtain practically ideal performance under other circumstances, the parameters should be precisely tested and fine-tuned with simulation in advance.

Crowd counting is a valuable technology for extremely dense scenes in the transportation. Existing methods generally have higher-order inconsistencies between ground truth density maps and generated density maps. To address this issue, we incorporate an attentional discriminator to take charge of checking the density map between the generator and the ground truth. Thus, a Cascaded Attentional Generative Adversarial Network (CACrowdGAN) is proposed that enables the attentional-driven discriminator to distinguish implausible density maps and simultaneously to guide the generator to deliver fine-grained high quality density maps. The proposed CACrowdGAN consists of two components: an attentional generator and a cascaded attentional discriminator. The attentional generator has an attention module and a density module. The attention module is developed for the generator to focus on the crowd regions of the input images, while the density module is used to provide the attentional input of the discriminator. In addition, a cascaded attentional discriminator is proposed to synthesize attentional-driven fine-grained details at different crowd regions of the input image and compute a per-pixel fine-grained loss for training generator. The proposed CACrowdGAN achieves the state-of-the-art performance on five popular crowd counting datasets (ShanghaiTech, WorldEXPO’10, UCSD, UCF_CC_50 and UCF_QNRF), which demonstrates the effectiveness and robustness of the proposed approach in the complex scenes.

Differentiable Architecture Search (DARTS) improves the efficiency of architecture search by learning the architecture and network parameters end-to-end. However, the intrinsic relationship between the architecture’s parameters is neglected, leading to a sub-optimal optimization process. The reason lies in the fact that the gradient descent method used in DARTS ignores the coupling relationship of the parameters and therefore degrades the optimization. In this paper, we address this issue by formulating DARTS as a bi-linear optimization problem and introducing an Interactive Differentiable Architecture Search (IDARTS). We first develop a backtracking backpropagation process, which can decouple the relationships of different kinds of parameters and train them in the same framework. The backtracking method coordinates the training of different parameters that fully explore their interaction and optimize training. We present experiments on the CIFAR10 and ImageNet datasets that demonstrate the efficacy of the IDARTS approach by achieving a top-1 accuracy of 76.52% on ImageNet without additional search cost vs. 75.8% with the state-of-the-art PC-DARTS.

Many previous methods on text-based person retrieval tasks are devoted to learning a latent common space mapping, with the purpose of extracting modality-invariant features from both visual and textual modality. Nevertheless, due to the complexity of high-dimensional data, the unconstrained mapping paradigms are not able to properly catch discriminative clues about the corresponding person while drop the misaligned information. Intuitively, the information contained in visual data can be divided into person information (PI) and surroundings information (SI), which are mutually exclusive from each other. To this end, we propose a novel Deep Surroundings-person Separation Learning (DSSL) model in this paper to effectively extract and match person information, and hence achieve a superior retrieval accuracy. A surroundings-person separation and fusion mechanism plays the key role to realize an accurate and effective surroundings-person separation under a mutually exclusion constraint. In order to adequately utilize multi-modal and multi-granular information for a higher retrieval accuracy, five diverse alignment paradigms are adopted. Extensive experiments are carried out to evaluate the proposed DSSL on CUHK-PEDES, which is currently the only accessible dataset for text-base person retrieval task. DSSL achieves the state-of-the-art performance on CUHK-PEDES. To properly evaluate our proposed DSSL in the real scenarios, a Real Scenarios Text-based Person Reidentification (RSTPReid) dataset is constructed to benefit future research on text-based person retrieval, which will be publicly available.

Summary At present, human action recognition is a challenging and complex task in the field of computer vision. The combination of CNN and RNN is a common and effective network structure for this task. Especially, we use 3DCNN in CNN part and ConvLSTM in RNN part. We divide the video into multiple temporal segments by average and compress each segment into one feature map by pooling layer. Adding the pooling layer, dropout layer, and batch normalization layer into ConvLSTM is our groundbreaking work. We test our model on KTH, UCF‐11, and HMDB51 datasets and achieve a high accuracy of action recognition.

Generating action proposals in untrimmed videos is a challenging task, since video sequences usually contain lots of irrelevant contents and the duration of an action instance is arbitrary. The quality of action proposals is key to action detection performance. The previous methods mainly rely on sliding windows or anchor boxes to cover all ground-truth actions, but this is infeasible and computationally inefficient. To this end, this article proposes a RecapNet-a novel framework for generating action proposal, by mimicking the human cognitive process of understanding video content. Specifically, this RecapNet includes a residual causal convolution module to build a short memory of the past events, based on which the joint probability actionness density ranking mechanism is designed to retrieve the action proposals. The RecapNet can handle videos with arbitrary length and more important, a video sequence will need to be processed only in one single pass in order to generate all action proposals. The experiments show that the proposed RecapNet outperforms the state of the art under all metrics on the benchmark THUMOS14 and ActivityNet-1.3 datasets. The code is available publicly at https://github.com/tianwangbuaa/RecapNet.

Matching manually cropped pedestrian images between queries and candidates, termed as person re-identification, has achieved significant progress with deep convolutional neural networks. Recently, a topic called ‘person search’ is proposed for the end-to-end application of re-identification technologies. It integrates object detection and person re-identification and aims to both locate and match pedestrians on a gallery of raw images. However, the design and implementation of such kind of hybrid network are difficult and computationally consuming in real practical situations. In order to fasten the design and ease the implementation, this paper proposes a deep-frozen transfer learning framework, named FT-MDnet, to extract re-identification features from a pre-trained detection network in two steps. First, using a channel-wise attention mechanism, a network called adaptive transfer learning network (ATLnet) is used to convert the sharing data of the underlying detection network to a re-identification feature map. Then, a multi-branch feature representation network called multiple descriptor network (MDnet) is proposed to extract re-identification features from the re-identification feature map. Our proposed solution has been verified on different types of mainstream detection networks, including YOLOv3, YOLOv4, Mask RCNN, and CenterNet. The experimental results show that our solution outperforms all other person search solutions by a large margin. It proves that the feature representations of detection networks are highly compatible with re-identification, and the proposed framework effectively extracts these features out. To encourage further research, we have made our framework open source.

Soft computing is facing a rapid evolution thanks to the development of artificial intelligence especially the deep learning. With video surveillance technologies of soft computing, such as image processing, computer vision, and pattern recognition combined with cloud computing, the construction of smart cities could be maintained and greatly enhanced. In this article, we focus on the online detection of action start task in video understanding and analysis, which is critical to the multimedia security in smart cities. We propose a novel model to tackle this problem and achieves state-of-the-art results on the benchmark THUMOS14 data set.

Security surveillance is critical to social harmony and people's peaceful life. It has a great impact on strengthening social stability and life safeguarding. Detecting anomaly timely, effectively and efficiently in video surveillance remains challenging. This paper proposes a new approach, called S2-VAE, for anomaly detection from video data. The S2-VAE consists of two proposed neural networks: a Stacked Fully Connected Variational AutoEncoder (SF-VAE) and a Skip Convolutional VAE (SC-VAE). The SF-VAE is a shallow generative network to obtain a model like Gaussian mixture to fit the distribution of the actual data. The SC-VAE, as a key component of S2-VAE, is a deep generative network to take advantages of CNN, VAE and skip connections. Both SF-VAE and SC-VAE are efficient and effective generative networks and they can achieve better performance for detecting both local abnormal events and global abnormal events. The proposed S2-VAE is evaluated using four public datasets. The experimental results show that the S2-VAE outperforms the state-of-the-art algorithms. The code is available publicly at https://github.com/tianwangbuaa/.

Spectral graph analysis based on Laplace operators has been ubiquitously applied in graph signal processing. Extending these tools to a generalization of graphs is helpful for several applications, such as network analysis, a sensor network coverage problem, and other fields, where the relationships between two or more vertices should be modeled. Such mathematical theories already exist, but the associated algorithms are still in their infancy. In this letter, we propose a new algorithm to compute harmonic forms, i.e., solutions of the Laplace equation, whose implementation is simple enough to be distributed among networks without central control center.

The aim of this paper is to detect abnormal events in video streams, a challenging but important subject in video surveillance. We propose a novel algorithm to address this problem. The algorithm is based on an image descriptor and a nonlinear classification method. We introduce a histogram of optical flow orientation as a descriptor encoding the moving information of each video frame. The nonlinear one-class support vector machine classification algorithm, following a learning period characterizing the normal behavior of training frames, detects abnormal events in the current frame. Further, a fast version of the detection algorithm is designed by fusing the optical flow computation with a background subtraction step. We finally apply the method to detect abnormal events on several benchmark data sets, and show promising results.

In this paper, we propose an algorithm to detect abnormal events based on video streams. The algorithm is based on histograms of the orientation of optical flow descriptor and one-class SVM classifier. We introduce grids of Histograms of the Orientation of Optical Flow (HOFs) as the descriptors for motion information of the monolithic video frame. The one-class SVM, after a learning period characterizing normal behaviors, detects the abnormal events in the current frame. Extensive testing on benchmark dataset corroborates the effectiveness of the proposed detection method.