Robotic manipulation in small and medium-sized enterprises (SMEs) is challenged by high product variability, small lot sizes, and frequent task reconfigurations. Under these conditions, traditional data-driven grasp planning methods are often infeasible due to their reliance on large, annotated datasets and stable object libraries. This paper presents an architecture for ontology-based semantic grasp planning that enables efficient and intuitive manual grasp pose specification without requiring extensive expertise in geometric programming or machine learning. By leveraging semantic representations of object geometries, gripper capabilities, and task-specific constraints, the system supports context-aware grasp candidate generation and prioritization. While the interpretation of the semantic models identifies suitable grasp types and involved geometric faces of objects and grippers, an optimization step generates suitable 6D poses that meet given constraints. As a result, the complexity of manually instructing robot systems for manipulation tasks is reduced and can be efficiently carried out by non-expert users.

CAD models using a boundary representation (BREP) are widely used across industrial applications. However, extracting and interpreting information from these models typically requires expert knowledge. In this paper, we present a GraphRAG-based system that lowers this barrier by enabling natural language access to detailed geometric and topological information. The approach builds upon an ontological representation of geometries, which is accessible via the semantic query language SPARQL. Large language models are used to translate user questions into formal queries and generate natural language replies, allowing for flexible, domain-aware interaction with the geometric data. The effectiveness of the system is demonstrated through a systematic evaluation using diverse CAD models and a range of natural language queries. Our system empowers domain experts to explore and analyze complex CAD models without requiring expertise in query languages or ontology design. By reducing the complexity of geometric information retrieval, the GraphRAG-based system enables more accessible, automated, and intelligent use of CAD data in industrial applications.

The potential benefits of digital transformation for manufacturing companies include reduced costs, increased interconnectedness, and improved adaptability. Semantic Web technologies such as IRIs, RDF graphs, OWL ontologies, and SPARQL requests are a well-known and actively researched ap-proach for supporting these transformation efforts. One challenge with this concept of knowledge augmentation is identifying where and how to integrate such semantic technologies into a manu-facturing system, as it could require frequent translations into other non-semantic representations, which may entail a loss of expressivity and other disadvantages. Therefore, this work aims to use semantic technologies in a knowledge-augmented robotic manufacturing platform as directly and natively as possible. This approach includes the semantic modeling of manufacturing processes (similarly to flow charts) and context knowledge such as generalized mechanisms of how to apply them. All of this semantic knowledge is instantiated and persistently stored in a Robot Knowledge Base application, which implements mecha-nisms to automatically derive the next robot skill invocations and their parameter values during process execution. These semantic description models and the Robot Knowledge Base were tested in simulation as well as integrated into a physical mobile robot system with an articulated arm tackling an industrial use case.

Manufacturing companies are exposed to increasingly complex products and shorter product engineering cycles. Unstructured data hinders the integration of knowledge over the different product engineering stages and complicates structured product development. However, combining an integrated view on relevant data sources following the Advanced Product Quality Planning (APQP) approach provides guidance for product engineers. In this paper, a semantic Knowledge Base (KB), a Process Execution System (PES), and a Computer Vision System (CVS) are introduced, which, in their interaction, compose a Socio-Technical Assistance System (STAS). We combine semantic models of production knowledge, APQP-guided product development, and ontology-based geometric representations of products and manufacturing resources. The PES coordinates the interaction with the user and other system components. The CVS tracks used tools and parts during the assembly and, therefore, enables traceability features and creates confidence in the quality of the assembly. As a result, the developed STAS prototype offers support from customer inquiry through product design and development to manufacturing and assembly, as well as after-sales support. The assistance system enables handling of complex products efficiently in order to reduce required times and costs.

Robot programming is still an elaborate process in industrial assembly, especially in cases requiring a specific force profile for successfully executing an assembly step. In this work, we present the Haptic Knob, a low-cost haptic device (around 100 EUR on hardware costs) for (currently) one-dimensional robot movement programming on both position and force profiles. The Haptic Knob is equipped with a Field-Oriented Controlled (FOC) motor and a positional encoder that reads the human input position and generates feedback forces to the operator. The Haptic Knob is not equipped with an external Force/Torque sensor to measure the human input force, but works together with the impedance controller of the robot arm, which allows a dynamic force teach-in. A control interface is implemented to map the positional signal from the Haptic Knob to the movement of different actuators, as well as the force/torque signals from various devices to the Haptic Knob for force feedback. We showcase and validate the proposed hardware and control interface on the industrial use case of automotive fuse box assembly.

This work aims at facilitating the integration of industrial robots and other devices such as their gripper tools at small and medium-sized enterprises (SMEs). For this purpose, an intuitive user interface for the skill-based instruction of robot systems is combined with standardized opc UA-based skill interfaces that support various hardware and software resources from different manufacturers. Special emphasis is laid on supporting different user groups with varying levels of expertise. Production system engineers are provided with a detailed graphical user interface (GUI) for hierarchically defining new skills by combining preexisting ones. System operators receive a simplified view with limited complexity for process instruction and changing high-level task parameterizations. The skills and relevant semantic context knowledge about products, processes, and resources (PPR) are formally represented in OWL ontologies to enable hardware-agnostic process descriptions that can be deployed to different production environments, while automatically deriving parameterizations for skill invocations. The proposed concept has been qualitatively evaluated in two real-world robot workcells based on a smartphone accessory packaging use case.

As robots become smarter and more ubiquitous, optimizing the power consumption of intelligent compute becomes imperative towards ensuring the sustainability of technological advancements. Neuromorphic computing hardware makes use of biologically inspired neural architectures to achieve energy and latency improvements compared to conventional von Neumann computing architecture. Applying these benefits to robots has been demonstrated in several works in the field of neurorobotics, typically on relatively simple control tasks. Here, we introduce an example of neuromorphic computing applied to the real-world industrial task of object insertion. We trained a spiking neural network (SNN) to perform force-torque feedback control using a reinforcement learning approach in simulation. We then ported the SNN to the Intel neuromorphic research chip Loihi interfaced with a KUKA robotic arm. At inference time we show latency competitive with current CPU/GPU architectures, and one order of magnitude less energy usage in comparison to traditional low-energy edge-hardware. We offer this example as a proof of concept implementation of a neuromoprhic controller in real-world robotic setting, highlighting the benefits of neuromorphic hardware for the development of intelligent controllers for robots.

In this paper, we introduce a knowledge-based Programming by Demonstration (kb-PbD) paradigm to facilitate robot programming in small and medium-sized enterprises (SMEs). PbD in production scenarios requires the recognition of product-specific actions but faces challenges in the lack of suitable and comprehensive datasets, due to the large variety of involved hand actions across different production scenarios. To address this issue, we utilize standardized grasp types as the fundamental feature to recognize basic hand movements, where a Long Short-Term Memory (LSTM) network is employed to recognize grasp types from hand landmarks. The product-specific actions, aggregated from the basic hand movements, are formally modeled in a semantic description language based on the Web Ontology Language (OWL). Description Logic (DL) is used to define the actions with their characteristic properties, which enables the efficient classification of new action instances by an OWL reasoner.The semantic models of hand actions, robot tasks, and work-cell resources are interconnected and stored in a Knowledge Base (KB), which enables the efficient pair-wise translation between hand actions and robot tasks. For the reproduction of human assembly processes, actions are converted to robot tasks via skill descriptions, while reusing the action parameters of involved objects to ensure product integrity. We showcase and evaluate our method in an industrial production setting for control cabinet assembly. Demonstration video available at: https://kb-pbd.github.io/.

Digital tools for handling the whole product engineering phase are getting more and more important in the context of Industry 4.0 and an increasing product variety. However, especially in small and medium-sized enterprises, a lot of information about product development and production is stored in different documents or isolated data silos. A promising way to arrive at a solution is to model data and knowledge with ontologies and enrich it with context information. This paper presents a concept and a showcase implementation of a company-internal and personalized assistance system for an end-to-end digital product engineering process. We combine a generic and cost-efficient human assistance solution focusing on social aspects and a company-wide knowledge graph to create a seamless and highly integrated data structure that assists many stakeholders in the product engineering process, from product designers to assembly workers. As a result, more complex products can be handled and the product engineering process can be accelerated.

Inferring a complete 3D geometry given an in-complete point cloud is essential in many vision and robotics applications. Previous work mainly relies on a global feature extracted by a Multi-layer Perceptron (MLP) for predicting the shape geometry. This suffers from a loss of structural details, as its point generator fails to capture the detailed topology and structure of point clouds using only the global features. The irregular nature of point clouds makes this task more challenging. This paper presents a novel method for shape completion to address this problem. The Transformer structure is currently a standard approach for natural language processing tasks and its inherent nature of permutation invariance makes it well suited for learning point clouds. Furthermore, the Transformer’s attention mechanism can effectively capture the local context within a point cloud and efficiently exploit its incomplete local structure details. A morphing-atlas-based point generation network further fully utilizes the extracted point Transformer feature to predict the missing region using charts defined on the shape. Shape completion is achieved via the concatenation of all predicting charts on the surface. Extensive experiments on the Completion3D and KITTI data sets demonstrate that the proposed PCTMA-Net outperforms the state-of-the-art shape completion approaches and has a 10% relative improvement over the next best-performing method.

Digital transformation efforts in manufacturing companies bear the potential to reduce product costs and increase the flexibility of production systems. The semantic integration of data and information along the value chain enables the automated interpretation of interrelations between its different aspects such as product design, production process and manufacturing resources. These interrelations can be used to automatically generate semantic process descriptions and execute corresponding robot motions. An initial one-time effort to model the required knowledge of a particular application domain can make the manufacturing of high-variant products in small batches or even lot size one production more efficient.This paper introduces a knowledge-based digital engineering concept to automate engineering and production activities without human involvement. The concept was integrated and evaluated in a physical robot workcell where automotive fuse boxes are autonomously fitted with different fuse configurations.

This paper presents an approach to describe abilities of manufacturing resources by a formal description of capabilities using Semantic Web technologies. A hierarchical ontology architecture is proposed to represent, publish, and extend knowledge on capabilities for different application domains and use cases. Furthermore, the paper describes patterns of how the underlying formal logic can be used in taxonomy modeling and the inference of implicit capability facts. The usability and performance of the approach was validated by formalizing capability knowledge of related work and evaluated in benchmarking a prototypical implemented tool for managing and querying catalogs of resources and their capabilities. The proposed concept is intended to be used as a foundation for a future multi-layered feasibility checking, which evaluates the compatibility of resources and their offered skills with the requirements of manufacturing tasks at symbolic and subsymbolic levels. Extended evaluations might be based on parameters, analytics, simulation, and other means.

The effectiveness of cognitive manufacturing systems in agile production environments heavily depend on the automatic assessment of various levels of interoperability between manufacturing resources. For taking informed decisions, a semantically rich representation of all resources in a workcell or production line is required. OPC UA provides means for communication and information exchange in such distributed settings.This paper proposes a semantic representation of a resource's properties, in which we use OWL ontologies to encode the information models that can be found in OPC UA NodeSet specifications. We further combine these models with an OWL-based description of the resource's geometry and - if applicable - its kinematic model. This leads to a comprehensive semantic representation of hardware and software features of a manufacturing resource, which we call semantic digital twin. Among other things, it reduces costs through virtual prototyping and enables the automatic deployment of manufacturing tasks in production lines. As a result, small-batch assemblies become financially viable.In order to minimize the effort of creating OWL-based UA NodeSet descriptions, we provide a software tool for the automatic transformation of XML-based NodeSet specifications that adhere to the OPC Foundation's NodeSet2 XML schema.

In this paper, we enhance our knowledge-based and constraint-based approach of robot programming with the concept of Semantic Mates. They describe intended mechanical connections between parts of an assembly. This allows deriving appropriate assembly poses from the type of connection and the geometric properties of the involved parts. The paper presents an ontology-based representation of Semantic Mates that is used to augment object models with additional information regarding their potential use in an assembly. Such semantically annotated object models can be used in our instruction framework to program a robot to perform assembly tasks through simple drag-and-drop operations in a graphical user interface. We conducted a user study with 21 participants in order to evaluate the efficiency and usability of the Semantic Mates concept based on a use-case from the domain of mechanical assembly. Across different experience levels in robotics, the participants achieved a significantly faster workflow and improved perceived usability compared to the manual specification of constraint-based assembly operations.

In this paper, we introduce a novel robot programming paradigm. It focuses on reducing the required expertise in robotics to a level that allows shop floor workers to use robots in their application domain without the need of extensive training. Our approach is user-centric and can interpret underspecified robot tasks, enabling communication on an abstract level. Such high-level task descriptions make the system amenable for users that are experts in a particular domain, but have limited knowledge about robotics and are thus not able to specify low-level details and instructions. Semantic models for all involved entities, i.e., processes, workpieces, and workcells, enable automatic reasoning about underspecified tasks and missing pieces of information. We showcase and evaluate this methodology on two industrial use cases from the domains of assembly and woodworking, comparing it to state-of-the-art solutions provided by robot manufacturers.

In this paper, we propose a framework for prioritized constraint-based specification of robot tasks. This framework is integrated with a cognitive robotic system based on semantic models of processes, objects, and workcells. The target is to enable intuitive (re-)programming of robot tasks, in a way that is suitable for non-expert users typically found in SMEs. Using CAD semantics, robot tasks are specified as geometric inter-relational constraints. During execution, these are combined with constraints from the environment and the workcell, and solved in real-time. Our constraint model and solving approach supports a variety of constraint functions that can be non-linear and also include bounds in the form of inequalities, e.g., geometric inter-relations, distance, collision avoidance and posture constraints. It is a hierarchical approach where priority levels can be specified for the constraints, and the nullspace of higher priority constraints is exploited to optimize the lower priority constraints. The presented approach has been applied to several typical industrial robotic use-cases to highlight its advantages compared to other state-of-the-art approaches.

We present an approach for monitoring and interpreting human activities based on a novel multimodal vision-based interface, aiming at improving the efficiency of human-robot interaction (HRI) in industrial environments. Multi-modality is an important concept in this design, where we combine inputs from several state-of-the-art sensors to provide a variety of information, e.g. skeleton and fingertip poses. Based on typical industrial workflows, we derived multiple levels of human activity labels, including large-scale activities (e.g. assembly) and simpler sub-activities (e.g. hand gestures), creating a duration- and complexity-based hierarchy. We train supervised generative classifiers for each activity level and combine the output of this stage with a trained Hierarchical Hidden Markov Model (HHMM), which models not only the temporal aspects between the activities on the same level, but also the hierarchical relationships between the levels.

In this paper, we introduce an approach for leveraging CAD description to a semantic level, in order to link additional knowledge to CAD models and to exploit resulting synergy effects. This has been achieved by designing a description language, based on the Web Ontology Language (OWL), that is used to define boundary representations (BREP) of objects. This involves representing geometric entities in a semantic meaningful way, e.g., a circle is defined by a coordinate frame and a radius instead of a set of polygons. Furthermore, the scope of this semantic description language also covers geometric constraints between multiple objects. Constraints can be specified not only on the object level, but down to single edges or faces of an object. This semantic representation is used to improve a variety of applications, ranging from shape-based object recognition to constraint-based robot task descriptions. Results from a quantitative evaluation are presented to assess the practicability of this approach.

Intuitive programming of industrial robots is especially important for small and medium-sized enterprises. We evaluated four different input modalities (touch, gesture, speech, 3D tracking device) regarding their preference, usability, and intuitiveness for robot programming.

In this paper, we propose a framework for intuitive task-based programming of robots using geometric inter-relational constraints. The intended applications of this framework are robot programming interfaces that use semantically rich task descriptions, allow intuitive (re-)programming, and are suitable for non-expert users typically found in SMEs. A key concept in this work is the use of CAD semantics to represent geometric entities in the robotic workcell. The robot tasks are then represented as a set of geometrical inter-relational constraints, which are solved in real-time to be executed on the robot. Since these constraints often specify the target pose only partially, the robot can be controlled to move in the constraints' null space in order to handle external disturbances or further optimize the robot's pose during runtime. Geometrical inter-relational constraints are easy to understand and can be intuitively specified using CAD software. A number of applications common in industrial robotic scenarios have been chosen to highlight the advantages of the presented approach vis-à-vis the state-of-the-art approaches.

The community-based generation of content has been tremendously successful in the World-Wide Web-people help each other by providing information that could be useful to others.We are trying to transfer this approach to robotics in order to help robots acquire the vast amounts of knowledge needed to competently perform everyday tasks.ROBOEARTH is intended to be a web community by robots for robots to autonomously share descriptions of tasks they have learned, object models they have created, and environments they have explored.In this paper, we report on the formal language we developed for encoding this information and present our approaches to solve the inference problems related to finding information, to determining if information is usable by a robot, and to grounding it on the robot platform.Note to Practitioners-In this paper, we report on a formal language for knowledge representation that is used in the ROBOEARTH system, a web-based knowledge base intended to be like a "Wikipedia for robots."The objective is to enable robots to share information about how to perform actions, how to recognize and interact with objects, and where to find objects in an environment.The developed language allows to store such information in a format that supports logical inference, so that robots can for example autonomously decide if they have all prerequisites needed for performing a described action.In laboratory experiments, the system has been applied to the exchange of pick-and-place style activities between two mobile manipulation robots.We are currently extending the representation towards more fine-grained action specifications.

This paper presented an approach to create 3D object models for robotic and vision applications in a fast and inexpensive way compared to established approaches. By using the RoboEarth system for storing the created object models users have world-wide access to the data and can immediately reuse a model as soon as it was created and uploaded. The approach shows general applicability for different kinds of cameras. In this work this was shown by two example implementations for the recognition process of objects. The quality of the recognition can be verified in the video. Combined with the knowledge saved in the RoboEarth database the objects can also be properly classified.

The community-based generation of content has been tremendously successful in the World Wide Web - people help each other by providing information that could be useful to others. We are trying to transfer this approach to robotics in order to help robots acquire the vast amounts of knowledge needed to competently perform everyday tasks. RoboEarth is intended to be a web community by robots for robots to autonomously share descriptions of tasks they have learned, object models they have created, and environments they have explored. In this paper, we report on the formal language we developed for encoding this information and present our approaches to solve the inference problems related to finding information, to determining if information is usable by a robot, and to grounding it on the robot platform.