This paper analyses the digital twins' current status and future interests in the intelligent motion control domain based on the web survey conducted within a large European research project named IMOCO4.E. We aim to understand how the companies and institutions developing and utilizing digital twin technologies see the main benefits, challenges and next steps in their domain. We conclude, based on the survey, that digital twins are used, primarily in the design phase, to speed up the development of complex systems where utilizing physical prototypes is not feasible or even possible. The operational phase utilization is not yet seen as important, but it has great potential, especially when the development of digital twins has become more mature in relation to real-time interfacing, exact representation of real physical objects, and maintainability of implementation.

Accurate object positioning is critical in many industrial manufacturing applications. The execution time and precision of the object positioning task have a significant impact on the overall performance and throughput, especially in cost-sensitive industries such as semiconductor manufacturing. In addition, the object positioning algorithm must adapt to changes in object size, features, and environmental conditions in real-time. While traditional sensors struggle to cope with dynamic conditions, vision-based perception is more adaptable and robust. Vision-based perception can capture and analyze visual information by using cameras and image processing algorithms, providing a robust way to locate objects in dynamic environments. However, classical perception algorithms based on vision cannot handle objects with different characteristics, and modern object detectors that rely on deep neural networks struggle to adapt to image sizes, resulting in unnecessary computations. To address these challenges, this paper proposes an approach for designing a branched multi-input deep neural network (DNN) that considers variations in input image sizes to adapt the input branches. In essence, the proposed DNN reduces the computation time for images with lower dimensions. To validate the proposed approach, an IC dataset is created that represents the variations in object sizes as seen in semiconductor manufacturing machines. Depending on the choice of input branches, the average inference time is reduced by over 30% with a slight gain in detection accuracy.

With the growing demand for semiconductor products, the semiconductor manufacturing industries are trying to increase their production capacities. Additional requirements and constraints are also enforced on semiconductor manufacturing equipment, particularly on robustness for visual inspections and vision-based alignment. Deep neural networks (DNNs) are prominently used for vision-based tasks to improve robustness. The challenge, however, is that semiconductor manufacturing industries still use brownfield systems and equipment with legacy hardware and software. The legacy systems introduce challenging requirements and constraints on the DNN deployment and the traditional approach to inference optimization results in poor inference performance. This paper presents a structured approach to optimize the inference of DNNs for vision-based tasks for industrial brownfield architectures with existing legacy hardware, software, and the associated requirements and constraints. Four directions in the machine learning operations (MLOps) pipeline are explored in this approach - DNN architecture selection, DNN model optimization, target deployment platform, and inference engine - while adhering to the legacy systems’ requirements and constraints. We present our approach using the case study from the semiconductor manufacturing industry that deploys DNNs for vision-based position detection in their legacy equipment. The results of the optimized DNN deployment are compared with a baseline implementation, and up to 44% improvement in inference timing performance is achieved without compromising on inference accuracy.

Intelligent motion control is integral to modern cyber-physical systems. However, smart integration of intelligent motion control with commercial and industrial systems requires domain expertise, industrial ‘know-how’ of the production processes, and resilient adaptation for the various engineering phases. The challenge is amplified with the adoption of advanced digital twin approaches, big data and artificial intelligence in the various industrial domains. This paper proposes the IMOCO4.E reference framework for the smart integration of intelligent motion control with commercial platforms (e.g. from SMEs) and industrial systems. The IMOCO4.E reference framework brings together the architecture, data management, artificial intelligence and digital twin viewpoints from the industrial users of the large-scale ‘Intelligent Motion Control under Industry4.E’ (IMOCO4.E) consortium. The framework envisions a generic platform for designing, developing, and implementing novice and complex motion-controlled industrial systems. Refinements and instantiations of the framework for the IMOCO4.E industrial cases validate the framework’s applicability for various industrial domains throughout the engineering phases and under different constraints imposed on the industrial cases.

Industrial applications and processes such as quality inspections, pick and place operations, and semiconductor manufacturing require accurate positioning control for achieving the high throughput of the assembly machines. Vision-based sensing is considered to be a potential means to achieve robust positioning control which is referred to as a vision-in-the-loop (VIL) system. In such motion systems, the point-of-control and the point-of-interest are often different due to several physical factors. In this case, validation of a system is done only when a machine prototype is available. A physical prototype is often expensive and infeasible in real-life. This paper proposes an evaluation framework for VIL systems targeting a predictable multi-core embedded platform. The presented framework offers model-in-the-loop (MIL), software-in-the-loop (SIL), and processor-in-the-loop (PIL) simulation features for evaluating the closed-loop performance of industrial motion control systems. As a deployment platform, we consider a predictable embedded platform CompSOC. The predictable nature of the CompSOC platform guarantees periodic and deterministic execution of the control applications and allows verification of the timing properties and performance of the VIL system. Additionally, the framework offers automatic code generation feature targeting the CompSOC platform. Closed-loop simulation setup models the system dynamics and camera position in the CoppeliaSim physics simulation engine and simulates the system software in C and MATLAB. CoppeliaSim runs as a server and MATLAB as a client in synchronous mode. We show the effectiveness of our framework using a vision-based motion control example.

While vision is an attractive alternative to many sensors targeting closed-loop controllers, it comes with high time-varying workload and robustness issues when targeted to edge devices with limited energy, memory and computing resources. Replacing classical vision processing pipelines, e.g., lane detection using Sobel filter, with deep learning algorithms is a way to deal with the robustness issues while hardware-efficient implementation is crucial for their adaptation for safe closed-loop systems. However, while implemented on an embedded edge device, the performance of these algorithms highly depends on their mapping on the target hardware and situation encountered by the system. That is, first, the timing performance numbers (e.g., latency, throughput) depends on the algorithm schedule, i.e., what part of the AI workload runs where (e.g., GPU, CPU) and their invocation frequency (e.g., how frequently we run a classifier). Second, the perception performance (e.g., detection accuracy) is heavily influenced by the situation - e.g., snowy and sunny weather condition provides very different lane detection accuracy. These factors directly influence the closed-loop performance, for example, the lane-following accuracy in a lane-keep assist system (LKAS). We propose a hardware- and situation-aware design of AI perception where the idea is to define the situations by a set of relevant environmental factors (e.g., weather, road etc. in an LKAS). We design the learning algorithms and parameters, overall hardware mapping and its schedule taking the situation into account. We show the effectiveness of our approach considering a realistic LKAS case-study on heterogeneous NVIDIA AGX Xavier platform in a hardware-in-the-loop framework. Our approach provides robust LKAS designs with 32% better performance compared to traditional approaches.

Image-based control (IBC) systems use camera sensor(s) to perceive the environment. The inherent compute-heavy nature of image processing causes long processing delay that negatively influences the performance of the IBC systems. Our idea is to reduce the long delay using coarse-grained approximation of the image signal processing pipeline without affecting the functionality and performance of the IBC system. The question is: how is the degree of approximation related to the closed-loop quality-of-control (QoC), memory utilization and energy consumption? We present a software-in-the-loop (SiL) evaluation framework for the above approximation-in-the-loop system. We identify the error resilient stages and the corresponding coarse-grained approximation settings for the IBC system. We perform trade off analysis between the QoC, memory utilisation and energy consumption for varying degrees of coarse-grained approximation. We demonstrate the effectiveness of our approach using a concrete case study of a lane keeping assist system (LKAS). We obtain energy and memory reduction of upto 84% and 29% respectively, for 28% QoC improvements.

Image-Based Control (IBC) systems have a long sample period. Sensing in these systems consists of compute-intensive image processing algorithms whose response times are dependent on image workload. IBC systems are typically designed for the worst-case workload that results in a long sample period and hence suboptimal quality-of-control (QoC). This worst-case based design is further considered for mapping of controller tasks and allocating platform resources, resulting in significant resource over-provisioning. Our design philosophy is to sample as fast as possible to optimise QoC for a given platform allocation, and for this, we present a structured design flow. Workload variations determine how fast we can sample and we model this dynamic behaviour using the concept of workload scenarios. Our choice of scenario-aware dataflow as the formal model for our application enables us to: i) model dynamic behaviour, analyse timing, and optimally map application tasks to the platform for maximising the effective utilisation of allocated resources, ii) relate throughput of the dataflow graph to the sample period, and thus combine dataflow analysis and mapping with control design parameters and QoC to identify system scenarios, and iii) to efficiently implement a run-time mechanism that manages necessary dynamic reconfiguration between system scenarios. Our results show that our design approach outperforms the worst-case based design with respect to optimising QoC and maximising effective resource utilisation.

Automotive software applications implement a variety of control algorithms, with many of them being safety-critical in nature. A typical design flow starts with modeling these control algorithms using tools like MATLAB/Simulink. However, at this stage, a number of assumptions, like negligible sensor-to-actuator delay and instantaneous computation of the controller software, are often made. In particular, the details of the software implementation and the computing platform, both eventually defining the timing properties of the applications, are not accounted for. Such idealistic assumptions can cause a significant deviation of the control performance compared to what was proven at the modeling stage. This is usually addressed with multiple design iterations, which are costly and may lead to over-provisioned and thus poorly designed systems. In this paper we attempt to address this problem by proposing a design-and tool flow that integrates software-and platform-level timing information into the high-level modeling stage. We outline our proposed flow using concrete, industry-strength design tools.