ABSTRACT Recently, significant progress has been made in completing static knowledge graphs. However, knowledge tends to evolve with time, and static knowledge graph completion (KGC) methods struggle to capture the changes. Therefore, temporal knowledge graph (TKG) reasoning has become a focus of research. Most existing TKG methods incorporate temporal information into triplets and transform them into KGC tasks, ignoring the important influence of time information and implicit relationships between entities. In this paper, we propose a new method called TD‐RKG, which addresses the challenges of temporal variability and implicit entity correlations based on a dynamic fusion representation learning approach. The method consists of four modules: dynamic local recurrent encoding layer, dynamic implicit encoding layer, dynamic global information attention layer and decoding layer. Experimental results on three benchmark datasets demonstrate substantial improvements in TD‐RKG across multiple evaluation metrics.

In recent years, sequential recommendation has received widespread attention for its role in enhancing user experience and driving personalized content recommendations. However, it also encounters challenges, including the limitations of modeling information and the variability of user preferences. A novel time-aware Long-Short Term Transformer (TLSTSRec) for sequential recommendation is introduced in this paper to address these challenges. TLSTSRec has two major innovative features. (1) Accurate modeling of users is achieved by fully leveraging temporal information. Time information is modeled by creating a trainable timestamp matrix from both the perspectives of time duration and time spectrum. (2) A novel time-aware Transformer model is proposed. To address the inherent variability of user preferences over time, the model combines long-term and short-term temporal information and adjusts the personalized trade-offs between long-term and short-term sequences using adaptive fusion layers. Subsequently, newly designed encoders and decoders are employed to model timestamps and interaction items. Finally, extensive experiments substantiate the effectiveness of TLSTSRec relative to various state-of-the-art sequential recommendation models based on MC/RNN/GNN/SA across a spectrum of widely used metrics. Furthermore, experiments are conducted to validate the rationality of the TLSTSRec structure.

Due to the rapid development of industrial manufacturing technology, modern mechanical equipment involves complex operating conditions and structural characteristics of hardware systems. Therefore, the state of components directly affects the stable operation of mechanical parts. To ensure engineering reliability improvement and economic benefits, bearing diagnosis has always been a concern in the field of mechanical engineering. Therefore, this article studies an effective machine learning method to extract useful fault feature information from actual bearing vibration signals and identify bearing faults. Firstly, variational mode decomposition decomposes the source signal into several intrinsic mode functions according to the actual situation. The vibration signal of the bearing is decomposed and reconstructed. By iteratively solving the variational model, the optimal modulus function can be obtained, which can better describe the characteristics of the original signal. Then, the feature subset is efficiently searched using the wrapper method of feature selection and the improved binary salp swarm algorithm (IBSSA) to effectively reduce redundant feature vectors, thereby accurately extracting fault feature frequency signals. Finally, support vector machines are used to classify and identify fault types, and the advantages of support vector machines are verified through extensive experiments, improving the ability of global search potential solutions. The experimental findings demonstrate the superior fault recognition performance of the IBSSA algorithm, with a highest recognition accuracy of 97.5%. By comparing different recognition methods, it is concluded that this method can accurately identify bearing failure.

Pose registration is critical in vision and robotics. This article focuses on the challenging task of initialization-free pose registration up to 7DoF for homogeneous and heterogeneous measurements. While recent learning-based methods show promise using differentiable solvers, they either rely on heuristically defined correspondences or require initialization. Phase correlation seeks solutions in the spectral domain and is correspondence-free and initialization-free. Following this, we propose a differentiable solver and combine it with simple feature extraction networks, namely DPCN++. It can perform registration for homo/hetero inputs and generalizes well on unseen objects. Specifically, the feature extraction networks first learn dense feature grids from a pair of homogeneous/heterogeneous measurements. These feature grids are then transformed into a translation and scale invariant spectrum representation based on Fourier transform and spherical radial aggregation, decoupling translation and scale from rotation. Next, the rotation, scale, and translation are independently and efficiently estimated in the spectrum step-by-step. The entire pipeline is differentiable and trained end-to-end. We evaluate DCPN++ on a wide range of tasks taking different input modalities, including 2D bird's-eye view images, 3D object and scene measurements, and medical images. Experimental results demonstrate that DCPN++ outperforms both classical and learning-based baselines, especially on partially observed and heterogeneous measurements.

Multiview learning aims to learn beneficial patterns from heterogeneous data sources and has captured growing attention in recent years. Most of the previous research studies focused on searching for an effective feature embedding of downstream tasks using diverse optimization algorithms, however, very limited work has been conducted to explore the connection between multiview learning and deep neural networks of structure sharing hidden layers. In this article, we propose a multiview deep matrix factorization model to learn a shared compact representation from multiview data. First, the proposed model constructs a multiview auto-encoder architecture with one shared encoder and multiple decoders, where each view corresponds to a factorization and the shared encoder leads to a common hidden layer. Accordingly, matrix factorizations from multiview data share the last hidden layer for a high-level semantic representation. Second, the nonnegativity constraint of the learned representation is transformed to the projection operation, which can be easily achieved by activating weights of the shared encoder network. Third, this network is trained with a joint loss of the reconstruction error and the compactness loss. By employing the clustering layer, the proposed method serves as an end-to-end multiview clustering method. Finally, comprehensive experiments on nine real-world datasets demonstrate the superiority of the proposed method against state-of-the-art multiview clustering methods.

Detecting changes in real-time using remote sensing data is of paramount importance in areas such as crop health monitoring, weed detection, and disaster management. However, real-time change detection using remote sensing imagery faces several challenges: a) it requires real-time data extraction which is a challenge for traditional satellite imagery sources such as MODIS and LANDSAT due to the latency associated with collecting and processing the data. Due to the advances made in the past decade in drone technology, Unmanned Aerial Vehicles (UAVs) can be used for real-time data collection. However, a large percentage of this data will be unlabeled which limits the use of well-known supervised machine learning methods; b) from an infrastructure perspective, the cloud-edge solution of processing the data collected from UAVs (edge) only on the cloud is also constrained by latency and bandwidth-related issues. Due to these limitations, transferring large amounts of data between cloud and edge, or storing large amounts of information regarding past time periods on an edge device is infeasible. We can limit the amount of data transferred between the cloud and edge by performing analyses on-the-fly at the edge using low-power devices (edge devices) that can be connected to UAVs. However, edge devices have computational and memory bottlenecks, which would limit the usage of complex machine learning algorithms. In this paper, we demonstrate how an unsupervised GMM-based real-time change detection method at the edge can be used to identify weeds in real-time. We evaluate the scalability of our method on edge computing and traditional devices such as NVIDIA Jetson TX2, RTX 2080, and traditional Intel CPUs. We perform a case study for weed detection on images collected from UAVs. Our results demonstrate both the efficacy and computational efficiency of our method.

One of the challenges in vision-based driving trajectory generation is dealing with out-of-distribution scenarios. In this paper, we propose a domain generalization method for vision-based driving trajectory generation for autonomous vehicles in urban environments, which can be seen as a solution to extend the Invariant Risk Minimization (IRM) method in complex problems. We leverage an adversarial learning approach to train a trajectory generator as the decoder. Based on the pre-trained decoder, we infer the latent variables corresponding to the trajectories, and pre-train the encoder by regressing the inferred latent variable. Finally, we fix the decoder but fine-tune the encoder with the final trajectory loss. We compare our proposed method with the state-of-the-art trajectory generation method and some recent domain generalization methods on both datasets and simulation, demonstrating that our method has better generalization ability. Our project is available at https://sites.google.com/view/dg-traj-gen.

LiDAR-based global localization is a fundamental problem for mobile robots. It consists of two stages, place recognition and pose estimation, which yields the current orientation and translation, using only the current scan as query and a database of map scans. Inspired by the definition of a recognized place, we consider that a good global localization solution should keep the pose estimation accuracy with a lower place density. Following this idea, we propose a novel framework towards sparse place-based global localization, which utilizes a unified and learning-free representation, Radon sinogram (RING), for all sub-tasks. Based on the theoretical derivation, a translation invariant descriptor and an orientation invariant metric are proposed for place recognition, achieving certifiable robustness against arbitrary orientation and large translation between query and map scan. In addition, we also utilize the property of RING to propose a global convergent solver for both orientation and translation estimation, arriving at global localization. Evaluation of the proposed RING based framework validates the feasibility and demonstrates a superior performance even under a lower place density.

As an emulous alternative to deep neural networks, Deep Forest emerges with features like low complexity, fewer hyper-parameters, and good robustness, which are predominantly desired in distributed computing applications and ecosystems. Recently, an efficient distributed Deep Forest system, named ForestLayer, was proposed, designing a fine-grained sub-Forest-based task-parallel algorithm to improve the parallel computing efficiency of Deep Forest. However, the sub-Forest splitting of ForestLayer is static and one-off without adaptability to the computing environment, nevertheless, the size of splitting granularity has a significant impact on the system performance. To further improve the computing efficiency and scalability of the distributed Deep Forest, in this paper, we propose a novel distributed Deep Forest algorithm, named BLB-gcForest (Bag of Little Bootstraps-gcForest), which augments the gcForest (multi-Grained Cascade Forest) approach for constructing Deep Forest. BLB-gcForest carries out parallel computation for each tree in sub-Forests at a finer parallel granularity and integrates with the Bag of Little Bootstraps (BLB) mechanism to reduce massive transmitted feature instances for Cascade Forest Layers, utterly improving both computation efficiency and communication efficiency. Moreover, to solve the problem of the forest splitting granularity, we further design an adaptive sub-Forest splitting algorithm to ensure the maximum resource utilization for parallel computation of each sub-Forest. Experimental results on four well-known large-scale datasets, namely YEAST, LETTER, MNIST, CIFAR10, show that the training efficiency of BLB-gcForest achieves up to 20.3x and 1.64x speedups compared with the state-of-the-art gcForest and ForestLayer, respectively while guaranteeing higher accuracy and better robustness

Recent studies on deep-learning-based small defection segmentation approaches are trained in specific settings and tend to be limited by fixed context. Throughout the training, the network inevitably learns the representation of the background of the training data before figuring out the defection. They underperform in the inference stage once the context changed and can only be solved by training in every new settings. This eventually leads to the limitation in practical robotic applications where contexts keep varying. To cope with this, instead of training a network context by context and hoping it to generalize, why not stop misleading it with any limited context and start training it with pure simulation? In this paper, we propose the network SSDS that learns a way of distinguishing small defections between two images regardless of the context, so that the network can be trained once for all. A small defection detection layer utilizing the pose sensitivity of phase correlation between images is introduced and is followed by an outlier masking layer. The network is trained on randomly generated simulated data with simple shapes and is generalized across the real world. Finally, SSDS is validated on real-world collected data and demonstrates the ability that even when trained in cheap simulation, SSDS can still find small defections in the real world showing the effectiveness and its potential for practical applications. Code is available here

In-flight objects capture is extremely challenging. The robot is required to complete trajectory prediction, interception position calculation and motion planning within tens of milliseconds. As in-flight uneven objects are affected by various kinds of forces, which leads to the time-varying acceleration, motion prediction for them is difficult. In order to compensate the system’s non-linearity, we propose using a recurrent neural network model, which we call the Neural Acceleration Estimator (NAE), to estimate the varying acceleration by observing a small fragment of previous deflected trajectory without any prior information. Moreover, end-to-end training with Differantiable Filter (NAE-DF) gives a supervision for measurement uncertainty and further improves the prediction accuracy. Experimental results show that motion prediction with NAE and NAE-DF is superior to other methods and has a good generalization performance on unseen objects. We test our methods on a robot, performing velocity control in real world and respectively achieve 83.3% and 86.7% success rate on a ploy urethane banana and a gourd. We also release an object in-flight dataset containing 1,500 trajectorys for uneven objects, which can be found on the project website:https://sites.google.com/view/neural-motion-prediction.

Assembly orders of components have direct influence on feasibility and efficiency of assembly process in manufacturing and are usually defined by experienced operators. To automate the assembly sequence generation process, we present a method using the idea of case-based reasoning, which can take advantage of experience of a reference assembly to generate the assembly sequence of a new assembly. First, a novel assembly representation method named assembly graph is present in which nodes indicating components’ 3D shape information and edges indicating the geometry constraints. Second, a similar components retrieve process is conducted based on assembly graph representation. Then, the assembly sequence is generated by applying the assembly order of retrieved components to the new ones. Next, the generated sequence is revised to satisfy the inherent constraints in the new assembly which is formulated as a contact graph. Finally, the revised sequence is stored into a case library with corresponding assembly model. We apply the proposed method to generate assembly sequences for chair assemblies and experimental results show its effectiveness and flexibility.

The Mean Teacher (MT) model of Tarvainen and Valpola has shown good performance on several semi-supervised benchmark datasets. MT maintains a teacher model's weights as the exponential moving average of a student model's weights and minimizes the divergence between their probability predictions under diverse perturbations of the inputs. However, MT is known to suffer from confirmation bias, that is, reinforcing incorrect teacher model predictions. In this work, we propose a simple yet effective method called Local Clustering (LC) to mitigate the effect of confirmation bias. In MT, each data point is considered independent of other points during training; however, data points are likely to be close to each other in feature space if they share similar features. Motivated by this, we cluster data points locally by minimizing the pairwise distance between neighboring data points in feature space. Combined with a standard classification cross-entropy objective on labeled data points, the misclassified unlabeled data points are pulled towards high-density regions of their correct class with the help of their neighbors, thus improving model performance. We demonstrate on semi-supervised benchmark datasets SVHN and CIFAR-10 that adding our LC loss to MT yields significant improvements compared to MT and performance comparable to the state of the art in semi-supervised learning11The code is available at: https://github.com/jay1204/local_clustering_with_mt_for_ssl.

The Food and Agriculture Organization (FAO) of the United Nations predicts that in order to meet the needs of the expected 3 billion population growth by 2050, food production has to increase by 60%. Therefore, monitoring and mapping crops accurately is essential for estimating food production during each crop growing season across the globe. Traditionally, multispectral remote sensing imagery has been widely used for mapping crops worldwide. However, single date imagery does not capture temporal characteristics (phenology) of growing crops, leading to imprecise crop maps and food estimates. On the other hand, purely temporal classification approaches also produce inaccurate crop maps as they do not account for spatial autocorrelations. In this paper, we present a multimodal deep learning solution that jointly exploits spatial-spectral and phenological properties to identify major crop types. Using a two stream architecture, spatial characteristics are captured via a spatial stream consisting of very high resolution images (single date, 1m, 3-spectral bands, USDA NAIP) with a CNN and the phenological characteristics via a temporal stream images (biweekly, 250m, MODIS NDVI) with an LSTM. Experimental results show that the proposed multimodal solution reduces prediction error by 60%.

Terrain classification is an important application of polarimetric SAR (PolSAR) data. Traditional classification methods need to extract the feature and then classify by classifiers. Besides, it should consider the influence of speckle noise. As a new method for image processing, convolutional neural network (CNN) has attracted more and more attention because of its good performance in image processing. It can deal with the original image directly with a higher classification accuracy without considering the impact of speckle noise. Moreover, three-dimensional convolutional neural network (3D CNN) has stronger feature extraction capability compared with traditional two-dimensional convolutional neural network (2D CNN). In this paper, the application of 3D CNN in terrain classification is studied, in which a new convolutional neural network architecture is designed and the elements of polarimetric coherency matrix are used as the input data of this network. The experiments of two real PolSAR data are conducted to verify the performance of the proposed network.

In this paper, we present a nearest neighbor based hierarchical change detection methodology for analyzing multi-temporal remote sensing imagery. A key contribution of this work is to define change as hierarchical rather than boolean. Based on this definition of change pattern, we developed a novel time series similarity based change detection framework for identifying inter-annual changes by exploiting phenological properties of growing crops from satellite time series imagery. The proposed framework consists of four components: hierarchical clustering tree construction, nearest neighbor based classification, relaxation labeling, and change detection using similarity hierarchy. Though the proposed approach is unsupervised, we present evaluation using manually induced change regions embedded in the real dataset. We compare our method with the widely used K-Means clustering and evaluation shows that K-Means over-detects changes in comparison to our proposed method.

Monitoring biomass over large geographic regions for changes in vegetation and cropping patterns is important for many applications. Changes in vegetation happen due to reasons ranging from climate change and damages to new government policies and regulations. Remote sensing imagery (multi-spectral and multi-temporal) is widely used in change pattern mapping studies. Existing bi-temporal change detection techniques are better suited for multi-spectral images and time series based techniques are more suited for analyzing multi-temporal images. A key contribution of this work is to define change as hierarchical rather than boolean. Based on this definition of change pattern, we developed a novel time series similarity based change detection framework for identifying inter-annual changes by exploiting phenological properties of growing crops from satellite time series imagery. The proposed framework consists of three components: hierarchical clustering tree construction, nearest neighbor based classification, and change detection using similarity hierarchy. Though the proposed approach is unsupervised, we present evaluation using manually induced change regions embedded in the real dataset. We compare our method with the widely used K-Means clustering and evaluation shows that K-Means over-detects changes in comparison to our proposed method.