During the service process of active phased array antenna (APAA), factors such as array heating and wind loading cause significant deformation, which will seriously affect the electromagnetic (EM) performance. Obtaining the geometric parameters of the array through sensors and predicting the real-time EM performance based on the digital twin (DT) technique are crucial to guarantee the service quality of the array antenna (e.g., the detection accuracy of a target). Traditional offline simulation methods can calculate the real EM performance of APAA with geometric errors. However, it requires a large number of computational resources and computation time, which is difficult to meet the demand for real-time prediction in DT. In this article, a generalized DT framework based on the reduced-order model for APAA is proposed. First, we introduce a dynamic-static attention-enhanced convolutional network for real-time computation of key EM indicators. Then, a super-resolution generating network is proposed, which realizes the mapping of array geometrical errors to the 3-D far-field pattern, and provides support for comprehensive performance evaluation. The framework proposed in this article constructs a twin model of APAA, realizes the virtual-reality mirroring of the EM performance, and is deployed and applied in an APAA DT platform.

Trajectory planning is a fundamental step for robotic spraying. This article proposes a semantic segmentation based spraying trajectory planning method for complex products. In order to divide the complex product surface into different painting regions, a geometric feature guided segmentation network (GFGS-Net) is developed. In the GFGS-Net, a geometric local encoding module is constructed to capture edge points based on the proposed local geometric feature, then encode the geometric feature and semantic feature of spraying surface. An attentive feature augmentation module is further designed to adaptively reweight and concatenate the encoded geometric feature and encoded semantic feature. With the aid of such feature representation, GFGS-Net can better describe the industrial product surface and perform accurate product surface segmentation. Afterwards, different spraying patterns are designed for each region, and key spraying parameters spraying overlap distance and spray gun speed are optimized considering the paint thickness and spraying time. Based on these two aspects, the spraying trajectory can be generated through point cloud slicing followed by alpha-shapes and intercept reduction. Alpha-shapes is used to obtain the contour lines of complex product surface, and the contour lines is further smoothed by intercept reduction to make the painting robot work more smoothly. The experiment results demonstrate that the proposed method can significantly improve painting quality and efficiency for complex products.

Dependency-based models are widely used to extract semantic relations in text. Most existing dependency-based models establish stacked structures to merge contextual and dependency information, which encode the contextual information first and then encode the dependency information. However, this unidirectional information flow weakens the representation of words in the sentence, which further restricts the performance of existing models. To establish bidirectional information flow, a dual attention graph convolutional network (DAGCN) with a parallel structure is proposed. Most importantly, DAGCN can build multi-turn interactions between contextual and dependency information to imitate the multi-turn looking-back actions of human beings. In addition, multi-layer adjacency matrix-aware multi-head attention (AMAtt), including context-to-dependency attention and dependency-to-context attention, is carefully designed as a merge mechanism in the parallel structure to preserve the structural information of sentences and dependency trees during interactions. Furthermore, DAGCN is evaluated on the popular PubMed dataset, TACRED dataset and SemEval 2010 Task 8 dataset to demonstrate its validity. Experimental results show that our model outperforms the existing dependency-based models.

Accurate 3D object segmentation in point clouds is a basis for industrial robot applications, such as robot manipulation and digital twin, which require an understanding of the 3D environment. However, the unstructured and disordered nature of point clouds makes it challenging, especially for the incomplete 3D data under a single view in the real-world scenario. To this end, this article proposes a novel 3D object segmentation framework (3DT-Seg) based on Cross-Window Point Transformer (CP-Former). CP-Former captures the long-range dependencies between local windows and latent semantic boundaries to enhance the point-wise features extracted from irregular point clouds via a bidirectional cross-attention mechanism. In addition, a contrastive learning loss and an adaptive dual aggregation strategy are introduced on semantic transition regions during the semantic supervising and instance clustering process, respectively. In this way, the latent boundary information is further utilized to improve the overall segmentation performance. Experiments on the popular benchmark dataset (S3DIS) show the state-of-the-art performance of the proposed approach in terms of semantic and instance segmentation. Furthermore, a real-world point cloud dataset (IP-Cloud) for the robotic grasping task is presented to fully validate the effectiveness of our method in practice, where it also achieves remarkable performance.

Nowadays, most high-precision products are still assembled manually, which leads to a low one-time pass rate of products. Workers need to adjust unqualified products repeatedly based on experience, resulting in inefficiency and poor quality consistency. In this work, we propose an explore-exploit reinforcement learning (EERL) framework to suggest the assembly parameters and quantity that workers need to adjust at each step. Jointed with the pretrained product performance prediction model, EERL can output a sequential decision to guide workers hand in hand to adjust unqualified products. EERL includes a two-phase learning process: 1) exploration; and 2) exploitation. In exploration phase, agents are encouraged by the curiosity to fully explore the qualified assembly states in the assembly-adjustment feature space. The regulated difference of random network distillation is used as a measure of curiosity. During exploitation, the agent is trained to learn an assembly-adjustment guidance policy of moving from any unqualified initial assembly state to the corresponding qualified state while satisfying the assembly-adjustment constraints. The curriculum learning mechanism is introduced to learn effectively in the complex environment with sparse reward and adjustment constraints. The proposed approach is validated on an benchmark optimization function and a case study of the gyroscope. The experimental results demonstrate that the proposed approach outperforms other existing approaches.

Quality prediction is committed to predicting the key quality-related variables to obtain real-time feedback information for process control. To achieve the robust and transferable quality prediction of products processed in complex and uncertain manufacturing processes, deep learning methods have been developed. However, the training process of deep learning methods requires a large amount of annotated data to avoid overfitting, and the labeling process of quality-related variables is often time-consuming and labor-intensive. Therefore, few-shot quality prediction in a multistage manufacturing process is formalized to address the lack of annotated data and deal with previously unseen tasks without an additional training process. In addition, a novel contrastive decoder generator (CDG) is proposed to enable few-shot quality prediction, which consists of a machine feature encoder, a contrastive stage, and task feature generator, and an instance-specific decoder generator. Experiments are conducted on a public quality prediction dataset collected from an actual production line. The CDG achieves state-of-the-art results on this dataset for few-shot quality prediction settings, which proves the effectiveness of the CDG. Additionally, detailed experiments are performed to evaluate the roles of different modules in the CDG.

For a complex electromechanical product that is a cyber–physical system (CPS), its dynamic behaviors are embodied in the closed-loop control between the logic process in its cyber component and actual actuators/sensors in its physical component, and thus, a well-defined model of the control is important to create a digital twin that acts as much like the real machine as possible. This article proposes a Petri nets (PNs)-based modeling solution that employs hybrid PNs (HPNs) for physics and system of sequential systems with shared resources (S4R) nets for logic in building a hierarchical control model. We also present PNs technologies for implementing a smooth transition and bidirectional mapping from the virtual prototype to the real machine. These technologies involve a PNs integration of a reinforcement learning (RL) method for generating a workflow scheduling agent in design, an extension of PNs definitions that is compatible with the microcontroller for easy deployment in manufacturing, and an architecture of PNs execution recording for data-driven monitoring in service. A software kit is provided for the solution that includes an integrated development environment of PNs, tools for quickly building a virtual prototype, and a monitor server for remote data-driven monitoring. This solution is successfully applied in the development of a typical cyber–physical product case, namely, the chemiluminescence immunoassay (CLIA) analyzer.

Mapping human hand motion to robotic hands has great significance in a wide range of applications, such as teleoperation and imitation learning. The ultimate goal is to develop a device-independent control solution based on human hand synergies. Over the past twenty years, a considerable number of mapping methods have been proposed, but most of the mapping methods use intrusive devices, such as the CyberGlove data gloves, to capture human hand motion. Until recently, a very small number of mapping methods have been proposed based on vision-based human hand pose estimation. Traditionally, mapping methods and vision-based human hand pose estimation have been studied independently. To the best of our knowledge, no review has been conducted to summarize the achievements on haptic mapping methods or explore the feasibility of applying off-the-shelf human hand pose estimation algorithms to teleoperation. To address this literature gap, we present the first survey on mapping human hand motion to robotic hands from a kinematic and algorithmic perspective. We discuss the realistic challenges, intuitively summarize recent mapping methods, analyze the theoretical solutions, and provide a teleoperation-oriented human hand pose estimation overview. As a preliminary exploration, a vision-based human hand pose estimation algorithm is introduced for robotic hand teleoperation.

The class imbalance problem has a huge impact on the performance of diagnostic models. When it occurs, the minority samples are easily ignored by classification models. Besides, the distribution of class imbalanced data differs from the actual data distribution, which makes it difficult for classifiers to learn an accurate decision boundary. To tackle the above issues, this article proposes a novel imbalanced data classification method based on weakly supervised learning. First, Bagging algorithm is employed to sample majority data randomly to generate several relatively balanced subsets, which are then used to train several support vector machine (SVM) classifiers. Next, these trained SVM classifiers are adopted to predict the labels of those unlabeled data, and samples that are predicted as minority class are added to the original dataset to reduce the imbalance ratio. The critical idea of this article is to introduce real-world samples into the imbalanced dataset by virtue of weakly supervised learning. In addition, bidirectional gated recurrent units are used to construct a diagnostic model for fault diagnosis, and a new weighted cross-entropy function is proposed as the loss function to reduce the impact of noise. Besides, it also increases the model's attention to the original minority samples. Furthermore, experimental evaluations of the proposed method are conducted on two datasets, i.e., Prognostics and Health Management challenge 2008 and 2010 datasets, and the experimental results demonstrate the effectiveness and superiority of the proposed method.

For metal surface defect inspection, deep-learning-based methods have largely improved the inspection accuracy. However, insufficient data and the diversity of defects usually pose challenges for these methods. To solve these problems, traditional data augmentation methods often augment data by applying image-level geometric variations, usually without introducing new features of unknown defects, which yields limited improvements in defect inspection. Given such circumstances, in this article, a new data augmentation algorithm named Mask2Defect is proposed. Via prior knowledge-based data infusing, this method is able to generate defects with varied features. A large volume of defects with different shapes, severities, scales, rotation angles, spatial locations, and part numbers can be generated in a controllable manner. These generated defects will work as teacher samples to fine-tune the inspection model and automatically adapt it to a wider range of defects. To be specific, we first encode the prior knowledge into the teacher mask via the industrial prior knowledge encoder and render the defect details according to the mask with the mask-to-defect construction network. Then, the fake-to-real domain transformation GAN is used to transform the rendered samples from the fake domain into the real defect domain. Experiments reveal that the synthesized image quality of our method outperforms the state-of-the-art generative methods, and the performance of the inspection model in defect classification and localization has also been improved by fine-tuning with the generated samples.

Quality prediction, as the basis of quality control, is dedicated to predicting quality indices of the manufacturing process. In recent years, data-driven deep learning methods have received a lot of attention due to their accuracy, robustness, and convenience for the prediction of quality indices. However, the existing studies mainly focus on the quality prediction of a single machine, while ignoring dependency relationships among multiple machines in multistage manufacturing process. To tackle the above issues, a novel path enhanced bidirectional graph attention network (PGAT) is proposed in this article. PGAT models the dependencies among machines into directed graphs and introduces graph attention network to encode the dependencies. Nonetheless, it is difficult for graph neural networks to encode long-distance dependencies. Hence, dependency path information is introduced into the features of machines. Moreover, in order to solve the label noise problem that often occurs in actual industrial dataset, a masked loss function is devised. With its help, batch training with noisy labels can be achieved, which improves the training efficiency. Furthermore, experiments are conducted on a public quality prediction dataset collected from an actual production line. PGAT achieves the state-of-the-art results on this dataset, which confirms the effectiveness of PGAT. Additionally, the experimental results demonstrate the critical role of modeling dependency relationships among machines.

While it is risky considering spacecraft constraints and unknown environment on asteroid, surface sampling is an important technique for asteroid exploration. One of the sample return missions is to seek an optimal landing site, which may be in hazardous terrain. Since autonomous landing is particularly challenging, it is necessary to simulate the effectiveness of this process and prove the onboard optical hazard avoidance is robust to various uncertainties. This paper aims to generate realistic surface images of asteroids for simulations of asteroid exploration. A SinGAN-based method is proposed, which only needs a single input image for training a pyramid of multi-scale patch generators. Various images with high fidelity can be generated, and manipulations such as shape variation, illumination direction variation, super resolution generation are well achieved. The method's applicability is validated by extensive experimental results and evaluations. At last, the proposed method has been used to help set up a test environment for landing site selection simulation.

Deep learning plays an increasingly important role in industrial applications, such as the remaining useful life (RUL) prediction of machines. However, when dealing with multifeature data, most deep learning approaches do not have effective mechanisms to weigh the input features adaptively. In this article, a novel feature-attention-based end-to-end approach is proposed for RUL prediction. First, the proposed feature-attention mechanism is directly applied to the input data, which gives greater attention weights to more important features dynamically in the training process. This helps the model focus more on those critical inputs, and the prediction performance is therefore improved. Next, bidirectional gated recurrent units (BGRU) are used to extract long-term dependencies from the weighted input data, and convolutional neural networks are employed to capture local features from the output sequences of BGRU. Finally, fully connected networks are used to learn the above-mentioned abstract representations to predict the RUL. The proposed approach is validated in a case study of turbofan engines. The experimental results demonstrate that the proposed approach outperforms other latest existing approaches.

A novel encoder-decoder model based on deep neural networks is proposed for the prediction of remaining useful life (RUL) in this work. The proposed model consists of an encoder and a decoder. In the encoder, the Bi-directional Long Short-Term Memory Networks (Bi-LSTM) and Convolutional Neural Networks (CNN) are used to capture the long-term temporal dependencies and important local features from the sequential data, respectively. Besides, single 1*1 convolution filter in the last convolutional layer is used for dimensionality reduction. In the decoder, the fully connected networks are employed to decode the feature information to predict RUL. In addition, the proposed data-driven method can achieve end-to-end prediction, which does not need feature engineering. To evaluate the proposed model, experimental verification is carried out on a commonly used aero-engine C-MAPSS dataset. Compared with other state-of-the-art approaches on the same dataset, the effectiveness and superiority of the proposed framework are demonstrated. For example, the scoring function value of the second subset is reduced by up to 64.99% compared with the best existing result.

Product assembly simulation is considered as one of the key technologies in the process of complex product design and manufacturing. Virtual assembly realizes the assembly process design, verification, and optimization of complex products in the virtual environment, which plays an active and effective role in improving the assembly quality and efficiency of complex products. In recent years, augmented reality (AR) and digital twin (DT) technology have brought new opportunities and challenges to the digital assembly of complex products owing to their characteristics of virtual reality fusion and interactive control. This paper expounds the concept and connotation of AR, enumerates a typical AR assembly system structure, analyzes the key technologies and applications of AR in digital assembly, and notes that DT technology is the future development trend of intelligent assembly research.

Currently, human motion analysis using three‐dimensional (3D) data creates closer awareness in computer vision with the introduction of cost‐effective Kinect or other depth cameras. This study attempts to segment a continuous 3D skeletal sequence into several disjointed sub‐sequences, each of which is corresponding to a complete action. To address this issue, the authors propose a supervised time‐series segmentation algorithm. A bidirectional propagation search scheme is employed to reach a solution. Specifically, a human skeleton is formulated as a point in multidimensional space, and a motion trajectory is further represented as a sequence. Each training action sequence serves as an atom in a dictionary, which is adopted by an l 2 ‐ regularised collaborative representation classifier. Based on the fact that the reconstruction error of the collaborative representation measures the similarity between a test sub‐sequence and training sequences, they utilise its variation over time to capture action transition. Cut point detection and sub‐sequence recognition are simultaneously achieved. Experiments on the authors’ recorded 3D skeletal sequences demonstrate that the proposed algorithm outperforms existing human motion segmentation techniques. Their algorithm is capable of extending to segment various dimensional sequences. This extensibility is validated by synthetic signal segmentation experiments.

Assembly variation analysis of parts that have flexible curved surfaces is much more difficult than that of solid bodies, because of structural deformations in the assembly process. Most of the current variation analysis methods either neglect the relationships among feature points on part surfaces or regard the distribution of all feature points as the same. In this study, the problem of flexible curved surface assembly is simplified to the matching of side lines. A methodology based on Bézier curves is proposed to represent the side lines of surfaces. It solves the variation analysis problem of flexible curved surface assembly when considering surface continuity through the relations between control points and data points. The deviations of feature points on side lines are obtained through control point distribution and are then regarded as inputs in commercial finite element analysis software to calculate the final product deformations. Finally, the proposed method is illustrated in two cases of antenna surface assembly.

This paper studies the linear quadratic Gaussian (LQG) control problem for wireless networked control systems in which control inputs are randomly dropped without the packet acknowledgment The packet acknowledgment is based on a signal scheme between the actuator and the estimator, which make the estimator know whether control packets are dropped or not. For such systems, the calculation of the optimal LQG controller has been shown to be computationally prohibitive and impractical, and sub-optimal solutions are usually obtained under an optimality criteria which is different from that for the optimal LQG problem. An approximate optimal LQG controller is obtained by introducing two approximations in computing the optimal estimation and control. Illustrative examples are provided to verify effectiveness of the proposed design scheme.

In this paper, an automatic visual inspection scheme with phase identification of microdrill bits in printed circuit board (PCB) production is proposed. Different from conventional methods in which the geometric quantities of microdrill bits are measured to compare with the prior standards, the proposed method adopts a strategy of machine learning. Thus, it lowers the requirement for the enlargement of lens and the resolution of charge-coupled device; therefore, the cost of inspecting instrument can be relatively reduced. Our method mainly includes two procedures: First, the statistical shape models of microdrill bit are built to get the shape subspace, and then the phase identification is performed in the shape subspace using some pattern recognition techniques. In this paper, we compared the performance of two statistical model methods, principal component analysis (PCA) and linear discriminate analysis, together with three classifiers, support vector machines (SVMs), neural networks, andk-nearest neighbors, respectively, for phase identification of microdrill bits. The experimental results demonstrate that using low enlargement and resolution microdrill bit images the proposed method can measure up to high inspection accuracy, and provide a conclusion that the highest identification rates are obtained by PCA-SVMs, which are higher than that of the conventional method.

Virtual assembly and tolerance analysis are effective tools to avoid assembly conflicts between components designed by different CAD teams during collaborative product design. This paper develops a virtual assembly and tolerance analysis system supporting collaborative product design and puts forward a component model expression method based on assembly ports as well as giving approaches to interactive assembly and tolerance analysis in virtual environment, which include recognition of assembly relation based on assembly ports, partial disassembly in assembly procedure, assembly movement navigation, recording of assembly procedure and tolerance information visualization. Besides, the paper also realizes the reusing of virtual assembly procedure to support modification and refinement of components' structure in collaborative product design.

The paper proposes a semantics knowledge modeling approach for product assembly, which is suitable for virtual reality interaction. Assembly semantics knowledge is used to express design conception, constrain the relationship between parts, and encapsulate design knowledge in semantics knowledge modeling. Using assembly semantics, the designer is able to express design intention in an engineer's language, which is natural and convenient for designer. In addition, the intelligence of the system can be promoted through the inference of implicit knowledge of semantics. These methods are implemented in the development of VIRDAS (Virtual Reality Design and Assembly System) and several assembly examples are also given.