Assembly-picking requires sequence planning, object detection, pose estimation, and trajectory planning. Given a general CAD model, the existing ASAP (Automated Sequence Planning for complex Robotic Assembly with physical Feasibility) method generates a physically feasible assembly sequence. We have further optimised ASAP to address common structural challenges in LEGO assemblies. Then, we address cluttered-bin detection challenges using a voting-based strategy that reduces false positives across varying point cloud scales. Our image processing method combines best-fit and iterative closest point (ICP) techniques, enabling robust millimetre-level 3D localisation. Gripping points are identified for collision-free trajectories using the k-dimensional Tree (K-d Tree) and rapidly-exploring random trees (RRT) algorithms, and verified via a digital twin before execution. The system is validated in a Lego assembly case and benchmarked against industrial tools.

In cluttered, reconfigurable environments with varied objects, materials, and sizes—where obstacles can be added, removed, or moved—planning robot arm actions between grasps is a key challenge for traditional bin picking. We propose a trajectory planner that quickly determines how to grasp and precisely assemble customized products in reconfigurable environment. Inspired by voting and global registration methods, our strategy reduces false positives and enhances object identification accuracy, even with vastly different point cloud scales. We present an image processing technique combining best-fit and iterative closest point methods to improve system robustness, adaptability, and stability for objects of varying shapes, sizes, and materials, achieving millimeter-level 3D localization with millions of scene points. An enhanced tree method manages uneven 3D space distribution, enabling fast collision-free planning. A space configuration algorithm minimizes computational load, supporting complex tasks like liquid handling. After digital twin verification of collision-free paths, the robotic arm executes assembly tasks. This approach is validated through Lego assembly and other industrial use cases.

Redundant robots offer significant potential for optimization strategies aimed at enhancing their performance. This paper conducts a kinematic analysis of a 7-DoF manipulator, the KUKA LBR iiwa, addressing the inverse kinematic problem and discussing a method for optimizing the robot configuration to improve the manipulability of the robot and therefore increase the accuracy of the estimated force based on measurements from the joint-integrated torque sensors. Furthermore, the condition number of the Jacobian matrix is introduced as a performance index within the context of this work. Finally, the results are validated through a sensitive robot application, where the accuracy of the estimated force serves as a performance indicator for the manipulator.

Achieving seamless human-robot collaboration in tasks with uncertainty requires anticipating human intentions. In order to overcome the limitations of classical techniques for inferring human intentions, which often struggle due to delayed articulation and lack of knowledge about spatio-temporal dependencies, this study proposes an approach centered around real-time prediction of human intention using Long-Short Term Memories (LSTMs). To achieve this goal, the human hands and assembly product components were detected in each frame, using a novel dataset, the former is then used as the primary input to our LSTM model. Finally, we demonstrate and validate the effectiveness of this framework in a real industrial assembly scenario, where a robotic agent utilizes the predicted intention to efficiently assist the human in successfully completing the assembly of a product. The results indicate, with high confidence, a response time about three times faster using our approach, on average, compared to waiting for the object detection module to recognize that a component was removed from a work bin.

In rework areas operators pick faulty products to be repaired (jobs) randomly from a storage. This leads to inefficient stock reduction. To improve the stock reduction the developed system presented in this paper assigns jobs to the operator, so that overall the stock reduction improves. In the first step the system checks the operator's capabilities and matches them to the job's requirements using simulated annealing as a meta-heuristics. Then it is checked which matching is even possible considering the layout of the storage area. Overall a stock reduction is achieved.

Digital assistance systems are used in manual assembly to support workers in assembling complex products. Moreover, they ensure the product quality by verifying and checking the committed mechanical assembly process. For example, checking resources like cameras can be associated with an assembly process, in order to verify it against the right data. Depending on the detected product state the digital assistance system switches automatically to the worker's instruction. Hence, the assistance system's digital product state needs to be in sync with the physical product state. However, usually various ways exist to assemble a product which is especially the case in rework areas where faulty products are repaired. Due to the limited computational resources and lack of reliability of certain sensors, some product states may not be captured correctly leading to asynchrony between the digital product state and real product state. To overcome the described problem, this paper presents a method to ensure the assembly assistance system's digital product state is in sync with the real product state by using neural networks. This approach applies the Sequence to One (Seq2One) Model in Recurrent Neural Networks (RNN) in order to predict the committed assembly processes. At first, the workers movement (Sequence) is captured by position trackers. Every process can have positional characteristics depending on where it is performed and which resources are being used. Then, the sequential data is preprocessed and fed into the RNN. The movements are matched to a predefined set of processes (One) that are within the scope of the workplace. Finally, the trained model can be used by feeding in live capture-motion (sub-processes) to predict and verify the assembly process or hint the worker to correct his actions.

In production companies, manual assembly tasks are still indispensable when complex products are assembled in a dynamic environment. Since the tasks are becoming more complex and more versatile, the employees need more support during the execution of their work. Cognitive assistance systems are used to support the employee in decision-making. The humans rely on predefined process descriptions and the systems provide the user (employee) with the right information for the current process. Due to their limitations, most of the cognitive assistance systems are not suitable for the use in dynamic environments such as the rework. This article presents a solution to introduce a cognitive assistance system in dynamic environments such as the rework area, in order to assure manual assembly processes through employee centered assistance.

Computer-based recording of live presentations has become a widespread method of producing learning materials in both higher education and companies. The success of such contents depends on their acceptance by the end-users working with the materials. From their point of view, the support of certain features and, thus, the choice of formats used for network delivery and replay are crucial issues. We present a list of key factors from the end-user's perspective and report on a user study carried out to evaluate a number of formats used for delivering recorded presentations to learners.

In presentation recording special effort is usually put into the automation of the production process, that is in automatically creating high quality data files without much or any need for manual recording and post-editing [5]. With the advent of such systems and their usage in classroom teaching, at conferences, etc., there is an increasing need for techniques and abilities which enable users to search in those documents and to localize some specific information. In this paper we describe how we integrated information retrieval techniques into the Authoring on the Fly (AOF) system, an approach for automatic presentation recording. We have chosen the AOF system for two reasons. On the one hand, it is a well-established way for presentation recording, used by various universities and institutions1. On the other hand it is general enough to illustrate typical problems and challenges a developer is facing when designing a system for information retrieval from multimedia data streams which occur in the presentation recording scenario.

In order to improve the acceptance of recorded presentations, we introduce a new open document type covering a wide range of different media classes typically appearing in this scenario. Instances of this document type can be replayed using our time-based synchronization model. Random access in combination with the realized stream/media-layered synchronization mechanism results in essential features such as Random Visible Scrolling and Unrestricted Cross-Referencing. The proposed synchronization model has been implemented and is routinely used by a variety of institutions. Furthermore we show how the properties of the considered documents and the functionality of the proposed synchronization model affect cross-modal retrieval techniques.