Abstract Compared with common urban landscape semantic segmentation, unmanned aerial vehicle (UAV) image semantic segmentation is more challenging because small targets have very low pixel percentages and multi‐scale features due to the influence of flight altitude. Yet, the commonly used successive grid downsampling strategy in the current transformer‐based methods omits some important features of small targets. Furthermore, due to the complex background interference, it can lead to even worse results. In reaction to this, existing strategies aim to maintain superior resolution. Nevertheless, the application of this method incurs considerable computational costs, which brings challenges for the practical applications of UAVs. So it is significant to design a novel framework to balance retaining more pixels representing small objects during downsampling and reducing computational costs. For this, the Channel Selection and the Local Attention Transformer Model (CSLFormer) are proposed. During the overlap patch embedding process of feature maps, the model allocates half of the important channels to global attention and local attention. These two types of attention focus on different aspects: one learns the relationships and importance among various patches, while the other emphasizes the features of individual patches. The method shows superior performance on two public datasets: AeroScapes and Vaihingen, achieving mean intersection over union (mIoU) of 75.57% and 78.93%, respectively. The proposed CSLFormer has been released on GitHub: https://github.com/leoda1/CSLFormer .

Accurate monitoring of the state of health (SOH) and remaining useful life (RUL) of lithium batteries is a key technology to realize their stable and economic operation. The current data-driven methods have problems such as local prediction errors caused by capacity self-recovery effect, model drift phenomenon, and inaccurate trend prediction caused by complex degradation process. In this article, we propose a joint SOH and RUL forecasting based on the fusion of kolmogorov-Arnold network (KAN), deep bidirectional long short-term memory network (DBLSTM) and adaptive mechanism method (KAN-HDBLSTM-AM). The method innovatively introduces an adaptive mechanism that considers the capacity degradation trend for different prediction steps, suppresses the accumulation of errors in the model prediction process, and further improves the prediction accuracy by combining the nonlinear approximation capability of KAN and the mastery of data temporal characteristics of DBLSTM. Experimental validation shows that the proposed method can effectively overcome the prediction errors caused by capacity self-recovery and model drift, and can retain sufficient accuracy under long prediction steps, with the prediction trend being more in line with the actual change trend and the prediction error being smaller compared with other methods.

Incomplete feature extraction and underutilization of unlabeled target data exist in the actual situation of rotating machinery fault diagnosis. To this end, a class-aware adversarial multiwavelet convolutional neural network (CAMCNN) is developed for cross-domain fault diagnosis of rotating machinery. Specifically, a class-aware classification mechanism (CCM) is first designed to autonomously sense the prediction effect of the target samples and improve the discrimination of classifiers. Thereafter, a class-aware adversarial domain adaptation approach based on CCM is proposed to precisely align the domain features at the class level, with the help of the developed reinforced conditional distribution alignment strategy. Finally, multiple wavelet convolutional kernels are introduced to replace the conventional convolutional kernel, and a multiwavelet convolutional neural network is constructed as the feature extractor to distill the implied feature information. Numerous results verify the validity of CAMCNN, while comparison results with mainstream methods demonstrate the superiority of CAMCNN.

This study presents an innovative reinforcement-learning-based control algorithm for a vertical take-off and landing (VTOL) aircraft under wind disturbances. In our approach, the tracking control problem of the VTOL aircraft is formulated as a Markov decision process, and the appropriate system state, reward function, and soft update method are presented. To improve the control accuracy under wind disturbances, three kinds of wind fields were added in the learning environment to expand the exploration space and simulate the effect of wind disturbances on the flight control. Moreover, to ensure the tracking accuracy and the practical implementation, a quantum-inspired experience replay strategy was developed based on quantum computation theory. In this strategy, the preparation operation scheme was designed to encourage the exploration and speed up the convergence. The depreciation operation method was developed to enrich the sample diversity, which increased the robustness of the controller and allowed the control strategy learned in the numerical simulations to be directly transferred into real-world environments. Numerical simulations, hardware-in-the-loop experiments, and real-world flight experiments were conducted to evaluate the performance and merits of the proposed method. The results demonstrated high accuracy and effectiveness and good robustness of the proposed control algorithm in terms of standoff target tracking control and flight stability.

Effective 3-D local features are significant elements for 3-D shape analysis. Existing hand-crafted 3-D local descriptors are effective but usually involve intensive human intervention and prior knowledge, which burdens the subsequent processing procedures. An alternative resorts to the unsupervised learning of features from raw 3-D representations via popular deep learning models. However, this alternative suffers from several significant unresolved issues, such as irregular vertex topology, arbitrary mesh resolution, orientation ambiguity on the 3-D surface, and rigid and slightly nonrigid transformation invariance. To tackle these issues, we propose an unsupervised 3-D local feature learning framework based on a novel permutation voxelization strategy to learn high-level and hierarchical 3-D local features from raw 3-D voxels. Specifically, the proposed strategy first applies a novel voxelization which discretizes each 3-D local region with irregular vertex topology and arbitrary mesh resolution into regular voxels, and then, a novel permutation is applied to permute the voxels to simultaneously eliminate the effect of rotation transformation and orientation ambiguity on the surface. Based on the proposed strategy, the permuted voxels can fully encode the geometry and structure of each local region in regular, sparse, and binary vectors. These voxel vectors are highly suitable for the learning of hierarchical common surface patterns by stacked sparse autoencoder with hierarchical abstraction and sparse constraint. Experiments are conducted on three aspects for evaluating the learned local features: 1) global shape retrieval; 2) partial shape retrieval; and 3) shape correspondence. The experimental results show that the learned local features outperform the other state-of-the-art 3-D shape descriptors.

Learning 3D global features by aggregating multiple views is important. Pooling is widely used to aggregate views in deep learning models. However, pooling disregards a lot of content information within views and the spatial relationship among the views, which limits the discriminability of learned features. To resolve this issue, 3D to Sequential Views (3D2SeqViews) is proposed to more effectively aggregate the sequential views using convolutional neural networks with a novel hierarchical attention aggregation. Specifically, the content information within each view is first encoded. Then, the encoded view content information and the sequential spatiality among the views are simultaneously aggregated by the hierarchical attention aggregation, where view-level attention and class-level attention are proposed to hierarchically weight sequential views and shape classes. View-level attention is learned to indicate how much attention is paid to each view by each shape class, which subsequently weights sequential views through a novel recursive view integration. Recursive view integration learns the semantic meaning of view sequence, which is robust to the first view position. Furthermore, class-level attention is introduced to describe how much attention is paid to each shape class, which innovatively employs the discriminative ability of the fine-tuned network. 3D2SeqViews learns more discriminative features than the state-of-the-art, which leads to the outperforming results in shape classification and retrieval under three large-scale benchmarks.

Learning 3D global features by aggregating multiple views has been introduced as a successful strategy for 3D shape analysis. In recent deep learning models with end-to-end training, pooling is a widely adopted procedure for view aggregation. However, pooling merely retains the max or mean value over all views, which disregards the content information of almost all views and also the spatial information among the views. To resolve these issues, we propose Sequential Views To Sequential Labels (SeqViews2SeqLabels) as a novel deep learning model with an encoder-decoder structure based on recurrent neural networks (RNNs) with attention. SeqViews2SeqLabels consists of two connected parts, an encoder-RNN followed by a decoder-RNN, that aim to learn the global features by aggregating sequential views and then performing shape classification from the learned global features, respectively. Specifically, the encoder-RNN learns the global features by simultaneously encoding the spatial and content information of sequential views, which captures the semantics of the view sequence. With the proposed prediction of sequential labels, the decoder-RNN performs more accurate classification using the learned global features by predicting sequential labels step by step. Learning to predict sequential labels provides more and finer discriminative information among shape classes to learn, which alleviates the overfitting problem inherent in training using a limited number of 3D shapes. Moreover, we introduce an attention mechanism to further improve the discriminative ability of SeqViews2SeqLabels. This mechanism increases the weight of views that are distinctive to each shape class, and it dramatically reduces the effect of selecting the first view position. Shape classification and retrieval results under three large-scale benchmarks verify that SeqViews2SeqLabels learns more discriminative global features by more effectively aggregating sequential views than state-of-the-art methods.

The discriminability of Bag-of-Words representations can be increased via encoding the spatial relationship among virtual words on 3D shapes. However, this encoding task involves several issues, including arbitrary mesh resolutions, irregular vertex topology, orientation ambiguity on 3D surface, invariance to rigid and non-rigid shape transformations. To address these issues, a novel unsupervised spatial learning framework based on deep neural network, deep spatiality (DS), is proposed. Specifically, DS employs two novel components: spatial context extractor and deep context learner. Spatial context extractor extracts the spatial relationship among virtual words in a local region into a raw spatial representation. Along a consistent circular direction, a directed circular graph is constructed to encode relative positions between pairwise virtual words in each face ring into a relative spatial matrix. By decomposing each relative spatial matrix using SVD, the raw spatial representation is formed, from which deep context learner conducts unsupervised learning of global and local features. Deep context learner is a deep neural network with a novel model structure to adapt the proposed coupled softmax layer, which encodes not only the discriminative information among local regions but also the one among global shapes. Experimental results show that DS outperforms state-of-the-art methods.

The advent of low-cost depth cameras, such as the Microsoft Kinect in the consumer market, has made many indoor applications and games based on motion tracking available to the everyday user. However, it is a large challenge to track human motion via such a camera because of its low-quality images, missing depth values, and noise. In this paper, we propose a novel human motion capture method based on a cooperative structure of multiple low-cost RGBD cameras, which can effectively avoid these problems. This structure can also manage the problem of body occlusions that appears when a single camera is used. Moreover, the whole process does not require training data, which makes this approach easily deployed and reduces operation time. We use the color image, depth image, and point cloud acquired in each view as the data source, and an initial pose is extracted in our optimization framework by aligning multiple point clouds from different cameras. The pose is dynamically updated by combining a filtering approach with a Markov model to estimate new poses in video streams. To verify the efficiency and robustness of our approach, we capture a wide variety of human actions via three cameras in indoor scenes and compare the tracking results of the proposed method to those of the current state-of-the-art methods. Moreover, our system is tested on more complex situations, in which multiple humans move within a scene, possibly occluding each other to some extent. The actions of multiple humans are tracked simultaneously, which would assist group behavior analysis.

Full human body shape scans provide valuable data for a variety of applications including anthropometric surveying, clothing design, human-factors engineering, health, and entertainment. However, the high price, large volume, and difficulty of operating professional 3-D scanners preclude their use in home entertainment. Recently, portable low-cost red green blue-depth cameras such as the Kinect have become popular for computer vision tasks. However, the infrared mechanism of this type of camera leads to noisy and incomplete depth images. We construct a stereo full-body scanning environment composed of multiple depth cameras and propose a novel registration algorithm. Our algorithm determines a segment constrained correspondence for two neighboring views, integrating them using rigid transformation. Furthermore, it aligns all of the views based on uniform error distribution. The generated 3-D mesh model is typically sparse, noisy, and even with holes, which makes it lose surface details. To address this, we introduce a geometric and topological fitting prior in the form of a professionally designed high-resolution template model. We formulate a template deformation optimization problem to fit the high-resolution model to the low-quality scan. Its solution overcomes the obstacles posed by different poses, varying body details, and surface noise. The entire process is free of body and template markers, fully automatic, and achieves satisfactory reconstruction results.

Fault detection and isolation (FDI) is very difficult for electronics-rich analog systems due to its sophisticated mechanism and variable operational conditions. Traditionally, FDI in such systems is done through the monitoring of deviation of output signals in voltage or current at system level, which commonly arises from the degradation of one or more critical components. Therefore, FDI can be transformed to a multiclass classification task given the extracted features of the output signals in voltage or current of the circuit. Traditional feature extraction on the circuit output is mostly based on time-domain, frequency-domain, or time-frequency signal processing, which collapse high-dimensional raw signals into a lower dimensional feature set. Such low-dimensional feature set usually suffers from information loss so as to affect the accuracy of the later fault diagnosis. In order to retain as much information as possible, deep learning is proposed which employs a hierarchical structure to capture the different levels of semantic representations of the signals. In this paper, a novel fault diagnostic application of Gaussian-Bernoulli deep belief network (GB-DBN) for electronics-rich analog systems is developed which can more effectively capture the high-order semantic features within the raw output signals. The novel fault diagnosis is validated experimentally on two typical analog filter circuits. Experimental results show the fault diagnosis based on GB-DBN is with superior diagnostic performance than the traditional feature extraction methods.

Highly discriminative 3D shape representations can be formed by encoding the spatial relationship among virtual words into the Bag of Words (BoW) method. To achieve this challenging task, several unresolved issues in the encoding procedure must be overcome for 3D shapes, including: 1) arbitrary mesh resolution; 2) irregular vertex topology; 3) orientation ambiguity on the 3D surface; and 4) invariance to rigid and non-rigid shape transformations. In this paper, a novel spatially enhanced 3D shape representation called bag of spatial context correlations (BoSCCs) is proposed to address all these issues. Adopting a novel local perspective, BoSCC is able to describe a 3D shape by an occurrence frequency histogram of spatial context correlation patterns, which makes BoSCC become more compact and discriminative than previous global perspective-based methods. Specifically, the spatial context correlation is proposed to simultaneously encode the geometric and spatial information of a 3D local region by the correlation among spatial contexts of vertices in that region, which effectively resolves the aforementioned issues. The spatial context of each vertex is modeled by Markov chains in a multi-scale manner, which thoroughly captures the spatial relationship by the transition probabilities of intra-virtual words and the ones of inter-virtual words. The high discriminability and compactness of BoSCC are effective for classification and retrieval, especially in the scenarios of limited samples and partial shape retrieval. Experimental results show that BoSCC outperforms the state-of-the-art spatially enhanced BoW methods in three common applications: global shape retrieval, shape classification, and partial shape retrieval.

Discriminative features of 3-D meshes are significant to many 3-D shape analysis tasks. However, handcrafted descriptors and traditional unsupervised 3-D feature learning methods suffer from several significant weaknesses: 1) the extensive human intervention is involved; 2) the local and global structure information of 3-D meshes cannot be preserved, which is in fact an important source of discriminability; 3) the irregular vertex topology and arbitrary resolution of 3-D meshes do not allow the direct application of the popular deep learning models; 4) the orientation is ambiguous on the mesh surface; and 5) the effect of rigid and nonrigid transformations on 3-D meshes cannot be eliminated. As a remedy, we propose a deep learning model with a novel irregular model structure, called mesh convolutional restricted Boltzmann machines (MCRBMs). MCRBM aims to simultaneously learn structure-preserving local and global features from a novel raw representation, local function energy distribution. In addition, multiple MCRBMs can be stacked into a deeper model, called mesh convolutional deep belief networks (MCDBNs). MCDBN employs a novel local structure preserving convolution (LSPC) strategy to convolve the geometry and the local structure learned by the lower MCRBM to the upper MCRBM. LSPC facilitates resolving the challenging issue of the orientation ambiguity on the mesh surface in MCDBN. Experiments using the proposed MCRBM and MCDBN were conducted on three common aspects: global shape retrieval, partial shape retrieval, and shape correspondence. Results show that the features learned by the proposed methods outperform the other state-of-the-art 3-D shape features.

In this paper we present a real-time approach to stitch large-scale aerial images incrementally. A monocular SLAM system is used to estimate camera position and attitude, and meanwhile 3D point cloud map is generated. When GPS information is available, the estimated trajectory is transformed to WGS84 coordinates after time synchronized automatically. Therefore, the output orthoimage retains global coordinates without ground control points. The final image is fused and visualized instantaneously with a proposed adaptive weighted multiband algorithm. To evaluate the effectiveness of the proposed method, we create a publicly available aerial image dataset with sequences of different environments. The experimental results demonstrate that our system is able to achieve high efficiency and quality compared to state-of-the-art methods. In addition, we share the code on the website with detailed introduction and results.

Extracting local features from 3D shapes is an important and challenging task that usually requires carefully designed 3D shape descriptors. However, these descriptors are hand-crafted and require intensive human intervention with prior knowledge. To tackle this issue, we propose a novel deep learning model, namely circle convolutional restricted Boltzmann machine (CCRBM), for unsupervised 3D local feature learning. CCRBM is specially designed to learn from raw 3D representations. It effectively overcomes obstacles such as irregular vertex topology, orientation ambiguity on the 3D surface, and rigid or slightly non-rigid transformation invariance in the hierarchical learning of 3D data that cannot be resolved by the existing deep learning models. Specifically, by introducing the novel circle convolution, CCRBM holds a novel ring-like multi-layer structure to learn 3D local features in a structure preserving manner. Circle convolution convolves across 3D local regions via rotating a novel circular sector convolution window in a consistent circular direction. In the process of circle convolution, extra points are sampled in each 3D local region and projected onto the tangent plane of the center of the region. In this way, the projection distances in each sector window are employed to constitute a novel local raw 3D representation called projection distance distribution (PDD). In addition, to eliminate the initial location ambiguity of a sector window, the Fourier transform modulus is used to transform the PDD into the Fourier domain, which is then conveyed to CCRBM. Experiments using the learned local features are conducted on three aspects: global shape retrieval, partial shape retrieval, and shape correspondence. The experimental results show that the learned local features outperform other state-of-the-art 3D shape descriptors.

We introduce a new concept for detecting the saliency of 3-D shapes, that is, human-centered saliency (HCS) detection on the surface of shapes, whereby a given shape is analyzed not based on geometric or topological features directly obtained from the shape itself, but by studying how a human uses the object. Using virtual agents to simulate the ways in which humans interact with objects helps to understand shapes and detect their salient parts in relation to their functions. HCS detection is less affected by inconsistencies between the geometry or topology of the analyzed 3-D shapes. The potential benefit of the proposed method is that it is adaptable to variable shapes with the same semantics, as well as being robust against a geometrical and topological noise. Given a 3-D shape, its salient part is detected by automatically selecting a corresponding agent and making them interact with each other. Their adaption and alignment depend on an optimization framework and a training process. We demonstrate the detected salient parts for different types of objects together with the stability thereof. The salient parts can be used for important vision tasks, such as 3-D shape retrieval.

In this study, fireworks algorithm (FWA) for the multi-satellite control resource scheduling problem (MSCRSP) is presented. FWA is a meta-heuristic method and widely used in continuous problems while MSCRSP is a constrained and large scale combinatorial problem. The key points of FWA are to define a suitable neighborhood structure for launching the local search procedure and to find a metric for quantifying the disparity between solutions. Three kinds of neighborhood structures are presented and the best fitted one is picked. Due to the speciality of this problem, each solution is transformed into a binary vector, and Hamming distance is adopted for defining disparity metric. The experimental results demonstrate the proposed FWA is more competitive than those commonly used methods.

In this letter, we develop a novel framework of leveraging weakly supervised learning techniques to efficiently detect targets from remote sensing images, which enables us to reduce the tedious manual annotation for collecting training data while maintaining the detection accuracy to large extent. The proposed framework consists of a weakly supervised training procedure to yield the detectors and an effective scheme to detect targets from testing images. Comprehensive evaluations on three benchmarks which have different spatial resolutions and contain different types of targets as well as the comparisons with traditional supervised learning schemes demonstrate the efficiency and effectiveness of the proposed framework.

Land-use classification using remote sensing images covers a wide range of applications. With more detailed spatial and textural information provided in very high resolution (VHR) remote sensing images, a greater range of objects and spatial patterns can be observed than ever before. This offers us a new opportunity for advancing the performance of land-use classification. In this paper, we first introduce an effective midlevel visual elementsoriented land-use classification method based on “partlets,” which are a library of pretrained part detectors used for midlevel visual elements discovery. Taking advantage of midlevel visual elements rather than low-level image features, a partlets-based method represents images by computing their responses to a large number of part detectors. As the number of part detectors grows, a main obstacle to the broader application of this method is its computational cost. To address this problem, we next propose a novel framework to train coarse-to-fine shared intermediate representations, which are termed “sparselets,” from a large number of pretrained part detectors. This is achieved by building a single-hidden-layer autoencoder and a single-hidden-layer neural network with an L0-norm sparsity constraint, respectively. Comprehensive evaluations on a publicly available 21-class VHR landuse data set and comparisons with state-of-the-art approaches demonstrate the effectiveness and superiority of this paper.

Deep learning has emerged as a powerful technique to extract high-level features from low-level information, which shows that hierarchical representation can be easily achieved. However, applying deep learning into 3D shape is still a challenge. In this paper, we propose a novel high-level feature learning method for 3D shape retrieval based on deep learning. In this framework, the low-level 3D shape descriptors are first encoded into visual bag-of-words, and then highlevel shape features are generated via deep belief network, which facilitates a good semantic preserving ability for the tasks of shape classification and retrieval. Experiments on 3D shape recognition and retrieval demonstrate the superior performance of the proposed method in comparison to the state-of-the-art methods.

In this paper, we propose a 3D finger detecting and tracking algorithm (3dFDT) based on a sliding window for RGB-D sequences. Microsoft Kinect is utilized as a 3D depth camera, which provides RGB images and depth data simultaneously. However the depth data have large noises when the background is white. We are focusing on tracking the up or down movements of the fingertips in order to apply in some applications such as playing the piano on the table or " in the air". First, the skin statistical ellipse model is applied to detect the hand skin region, and the pseudo-contours are removed to get the interested hand region. Then, the region center is considered as the center of the palm. Next, the fingertips are located precisely using the convex defect detection based on the refined hand contours. Furthermore, the depth information is aligned to the fingertips in the RGB channels. Finally, the sliding window strategy is employed to stabilize the depth information due to large noises from the depth space especially when dealing with the white surface as backgrounds. The experimental results show that our proposed method is effective, and it can be applied in the real-time applications for non-contact interactions.

Abstract Since indoor scenes are frequently changed in daily life, such as re‐layout of furniture, the 3D reconstructions for them should be flexible and easy to update. We present an automatic 3D scene update algorithm to indoor scenes by capturing scene variation with RGBD cameras. We assume an initial scene has been reconstructed in advance in manual or other semi‐automatic way before the change, and automatically update the reconstruction according to the newly captured RGBD images of the real scene update. It starts with an automatic segmentation process without manual interaction, which benefits from accurate labeling training from the initial 3D scene. After the segmentation, objects captured by RGBD camera are extracted to form a local updated scene. We formulate an optimization problem to compare to the initial scene to locate moved objects. The moved objects are then integrated with static objects in the initial scene to generate a new 3D scene. We demonstrate the efficiency and robustness of our approach by updating the 3D scene of several real‐world scenes.

3-D shape analysis has attracted extensive research efforts in recent years, where the major challenge lies in designing an effective high-level 3-D shape feature. In this paper, we propose a multi-level 3-D shape feature extraction framework by using deep learning. The low-level 3-D shape descriptors are first encoded into geometric bag-of-words, from which middle-level patterns are discovered to explore geometric relationships among words. After that, high-level shape features are learned via deep belief networks, which are more discriminative for the tasks of shape classification and retrieval. Experiments on 3-D shape recognition and retrieval demonstrate the superior performance of the proposed method in comparison to the state-of-the-art methods.

This paper presents a novel structural model based scene recognition method. In order to resolve regular grid image division methods which cause low content discriminability for scene recognition in previous methods, we partition an image into a pre-defined set of regions by superpixel segmentation. And then classification is modelled by introducing a structural model which has the capability of organizing unordered features of image patches. In the implementation, CENTRIST which is robust to scene recognition is used as original image feature, and bag-of-words representation is used to capture the local appearances of an image. In addition, we incorporate adjacent superpixel's differences as edge features. Our models are trained using structural SVM. Two state-of-the-art scene datasets are adopted to evaluate the proposed method. The experiment results show that the recognition accuracy is significantly improved by the proposed method.

By utilizing the dataset provided by 3DLife/Huawei Challenge of ACM Multimedia, we propose a refreshing application that automatically evaluates player's energy consumption in gaming scenarios by a model with tracked skeleton, which may help users to know their exercise effects and even diet or reduce their weights. We develop a program to compute the energy consumption in real time by analyzing data captured from Microsoft Kinect, and also give a cue in the dynamic interaction. We model 3D human skeleton by joining different body parts with 15 nodes, and decompose player action into rigid body motions of these parts. Amount of energy consumed in the action is calculated as the sum of powers required to overcome gravity of each part. Experimental results show that instantaneous and total energy consumption of different dancers can be stably calculated. The hardware system is based on low-price Kinect, and easily accepted by users. The proposed application also provides a quantitative approach which help users to control their dining and exercise intensity.