This paper explores the integration of augmented reality into robotic manufacturing systems, emphasizing the enhancement of connectivity, real-time data processing, and interactive visual interfaces. Utilizing Hololens 2 headsets equipped with OPC UA clients, the system establishes a direct interface between operators and the digital and physical components of the manufacturing environment. The architecture employs augmented reality to facilitate sophisticated operational control and visualization, ranging from robotic additive manufacturing to complex tasks like robotic sanding and screwing with integrated camera systems. This integration showcases significant advancements in manufacturing processes, allowing operators to engage interactively with the systems, optimize workflows, and improve overall precision and efficiency in a smart factory setting.

This study aims to integrate and evaluate digital twin technology with robotic multi-axis additive manufacturing to bridge the gap between virtual planning and physical execution, enhancing the precision and efficiency of manufacturing processes. A laser tracker is employed to calibrate the machine workspace and tools and introduce a comprehensive system involving an industrial robot with a filament extruder head and a rotary-tilt positioner. The digital twin not only encapsulates the geometric and operational intricacies of the multi-axis additive manufacturing process but also automates code generation for robotic operations. The application of this methodology is demonstrated through the production of two complex parts, where their dimensional fidelity is assessed to evaluate the precision limits and potential of digital twin technology in multi-axis additive manufacturing. The findings reveal significant advancements in the alignment of virtual models and physical objects, confirming the potential of digital twins to minimize deviations common in traditional manufacturing setups.

Human-Robot collaboration in home and industrial workspaces is on the rise. However, the communication between robots and humans is a bottleneck. Although people use a combination of different types of gestures to complement speech, only a few robotic systems utilize gestures for communication. In this paper, we propose a gesture pseudo-language and show how multiple types of gestures can be combined to express human intent to a robot (i.e., expressing both the desired action and its parameters - e.g., pointing to an object and showing that the object should be emptied into a bowl). The demonstrated gestures and the perceived tabletop scene (object poses detected by CosyPose) are processed in real-time) to extract the human's intent. We utilize behavior trees to generate reactive robot behavior that handles various possible states of the world (e.g., a drawer has to be opened before an object is placed into it) and recovers from errors (e.g., when the scene changes). Furthermore, our system enables switching between direct teleoperation of the end-effector and high-level operation using the proposed gesture sentences. The system is evaluated on increasingly complex tasks using a real 7-DoF Franka Emika Panda manipulator. Controlling the robot via action gestures lowered the execution time by up to 60%, compared to direct teleoperation.

With the advent of different electronic skins sensitive to touch and robots composed of soft materials, tactile or haptic human-robot interaction is gaining importance. We designed a highly realistic artificial hand aiming to reproduce human-to-human physical contact through a special morphology imitating flesh and bones and a heating system imitating human body temperature. The mechanical response properties of different finger designs were analyzed and the most mimetic one came very close to a human finger. We designed three experiments with participants using haptic exploration to evaluate the human-likeness of: (1) finger morphologies; (2) complete hands: real human vs. soft and warm artificial hand vs. rubber hand (3) the hand mounted on a manipulator with fixed vs. passive compliant wrist in a handshake scenario. First, participants find the mimetic finger morphology most humanlike. Second, people can reliably distinguish the real human hand, the artificial one, and a rubber hand. In terms of humanlikeness (Anthropomorphism, Animacy, and Likeability), the human hand scores better than the artificial hand which in turn clearly outperforms the rubber hand. The temperature, or "warmth", was rated as the most human-like feature of the artificial hand.

The estimation of the fabric material property during the folding is presented. The available techniques for the accurate garment folding rely on known material properties. Currently, the properties are estimated by an operator in advance of folding. We propose an iterative strategy, which updates the property while the garment is folded. The estimation is formulated as an optimisation task. It is based on measurements from a laser range finder. The proposed algorithm improves the estimation iteratively and prevents the garment from slipping at the same time. We demonstrate the estimation procedure for 10 fabric strips of different materials.

The presented work deals with classification of garment categories including pants, shorts, shirts, T-shirts and towels. The knowledge of the garment category is crucial for its robotic manipulation. Our work focuses particularly on garments being held in a hanging state by a robotic arm. The input of our method is a set of depth maps taken from different viewpoints around the garment. The depths are fused into a single 3D point cloud. The cloud is fed into a convolutional neural network that transforms it into a single global feature vector. The network utilizes a generalized convolution operation defined over the local neighborhood of a point. It can deal with permutations of the input points. It was trained on a large dataset of common 3D objects. The extracted feature vector is classified with SVM trained on smaller datasets of garments. The proposed method was evaluated on publicly available data and compared to the original methods, achieving competitive performance and better generalization capability.

The proposed work deals with robotic unfolding of a garment that has been placed flat on a table and folded over a certain axis. The algorithm combines image and depth data to detect the bottom and top (folded) layer of the garment. The detection is formulated as a labeling of the garment surface and solved in an energy minimization framework. Once the garment pose is known, several candidate folding axes are generated and used to unfold the garment virtually. The correct folding axis is selected from these candidate axes. The method does not set any constraints on the garment shape; thus it can deal with various types of garments including jackets, pants, shorts, skirts or T-shirts of any sleeve lengths. The garment is unfolded by the dual-arm robot. One arm grasps boundary of the top layer and brings it over the estimated folding axis, while the second arm is holding the bottom layer to prevent the garment from slipping. The perception procedure was tested on the annotated dataset that we are making publicly available. The experimental evaluation of the robotic manipulation is also provided.

We experiment with off-line recognition of handwritten flowcharts based on strokes reconstruction and our state-of-the-art on-line diagram recognizer. A simple baseline algorithm for strokes reconstruction is presented and necessary modifications of the original recognizer are identified. We achieve very promising results on a flowcharts database created as an extension of our previously published on-line database.

The ability to perform an accurate robotic fold is essential to obtain the properly folded garment. Available solutions rely on a rough folding surface or on a comprehensive simulation, both preventing the garment from slipping on the table during folding. This paper proposes a new algorithm for a folding path design respecting the garment material properties and preventing the garment slipping. The folding path is derived based on the equilibrium of forces under the simplifying assumptions of a rectangular and homogeneous garment. This approach allows folding the rectangular garment on a low friction table surface as we demonstrated in the experiments performed by a dual-arm robotic testbed.

This work presents a complete pipeline for folding a pile of clothes using a dual-armed robot. This is a challenging task both from the viewpoint of machine vision and robotic manipulation. The presented pipeline is comprised of the following parts: isolating and picking up a single garment from a pile of crumpled garments, recognizing its category, unfolding the garment using a series of manipulations performed in the air, placing the garment roughly flat on a work table, spreading it, and, finally, folding it in several steps. The pile is segmented into separate garments using color and texture information, and the ideal grasping point is selected based on the features computed from a depth map. The recognition and unfolding of the hanging garment are performed in an active manner, utilizing the framework of active random forests to detect grasp points, while optimizing the robot actions. The spreading procedure is based on the detection of deformations of the garment's contour. The perception for folding employs fitting of polygonal models to the contour of the observed garment, both spread and already partially folded. We have conducted several experiments on the full pipeline producing very promising results. To our knowledge, this is the first work addressing the complete unfolding and folding pipeline on a variety of garments, including T-shirts, towels, and shorts.

Controlling mobile robots with complex articulated parts and hence many degrees of freedom generates high cognitive load on the operator, especially under demanding conditions such as in Urban Search & Rescue missions. We propose a solution based on reinforcement learning in order to accommodate the robot morphology automatically to the terrain and the obstacles it traverses. In this paper, we concentrate on the crucial issue of predicting rewards from incomplete or missing data. For this purpose we exploit the Gaussian processes as a predictor combined with decision trees. We demonstrate our achievements in a series of experiments on real data.

This paper deals with recognition of arrows in online sketched diagrams. Arrows have varying appearance and thus it is a difficult task to recognize them directly. It is beneficial to detect arrows after other symbols (easier to detect) are already found. We proposed [4] an arrow detector which searches for arrows as arbitrarily shaped connectors between already found symbols. The detection is done two steps: a) a search for a shaft of the arrow, b) a search for its head. The first step is relatively easy. However, it might be quite difficult to find the head reliably. This paper brings two contributions. The first contribution is a design of an arrow recognizer where the head is detected using relative strokes positioning. We embedded this recognizer into the diagram recognition pipeline proposed earlier [4] and increased the overall accuracy. The second contribution is an introduction of a new approach to evaluate the relative position of two given strokes with neural networks (LSTM). This approach is an alternative to the fuzzy relative positioning proposed by Bout ruche et al. [2]. We made a comparison between the two methods through experiments performed on two datasets for two different tasks. First, we used a benchmark database of hand-drawn finite automata to evaluate detection of arrows. Second, we used a database presented in the paper by Bout ruche et al. containing pairs of reference and argument strokes, where argument strokes are classified into 18 classes. Our method gave significantly better results for the first task and comparable results for the second task.

We present our recent model of a diagram recognition engine. It extends our previous work which approaches the structural recognition as an optimization problem of choosing the best subset of symbol candidates. The main improvement is the integration of our own text separator into the pipeline to deal with text blocks occurring in diagrams. Second improvement is splitting the symbol candidates detection into two stages: uniform symbols detection and arrows detection. Text recognition is left for post processing when the diagram structure is already known. Training and testing of the engine was done on a freely available benchmark database of flowcharts. We correctly segmented and recognized 93.0% of the symbols having 55.1% of the diagrams recognized without any error. Considering correct stroke labeling, we achieved the precision of 95.7%. This result is superior to the state-of-the-art method with the precision of 92.4%. Additionally, we demonstrate the generality of the proposed method by adapting the system to finite automata domain and evaluating it on own database of such diagrams.

In this paper we introduce the concept of Adaptive Traversability (AT), which we define as means of autonomous motion control adapting the robot morphology - configuration of articulated parts and their compliance - to traverse unknown complex terrain with obstacles in an optimal way. We verify this concept by proposing a reinforcement learning based AT algorithm for mobile robots operating in such conditions. We demonstrate the functionality by training the AT algorithm under lab conditions on simple EUR-pallet obstacles and then testing it successfully on natural obstacles in a forest. For quantitative evaluation we define a metrics based on comparison with expert operator. Exploiting the proposed AT algorithm significantly decreases the cognitive load of the operator.

The work addresses the problem of clothing perception and manipulation by a two armed industrial robot aiming at a real-time automated folding of a piece of garment spread out on a flat surface. A complete solution combining vision sensing, garment segmentation and understanding, planning of the manipulation and its real execution on a robot is proposed. A new polygonal model of a garment is introduced. Fitting the model into a segmented garment contour is used to detect garment landmark points. It is shown how folded variants of the unfolded model can be derived automatically. Universality and usefulness of the model is demonstrated by its favorable performance within the whole folding procedure which is applicable to a variety of garments categories (towel, pants, shirt, etc.) and evaluated experimentally using the two armed robot. The principal novelty with respect to the state of the art is in the new garment polygonal model and its manipulation planning algorithm which leads to the speed up by two orders of magnitude.

This work deals with the on-line recognition of hand-drawn graphical sketches with structure. We present a novel approach, in which the search for a suitable interpretation of the input is formulated as a combinatorial optimization task - the max-sum problem. The recognition pipeline consists of two main stages. First, groups of strokes possibly representing symbols of a sketch (symbol candidates) are segmented and relations between them are detected. Second, a combination of symbol candidates best fitting the input is chosen by solving the optimization problem. We focused on flowchart recognition. Training and testing of our method was done on a freely available benchmark database. We correctly segmented and recognized 82.7% of the symbols having 31.5% of the diagrams recognized without any error. It indicates that our approach has promising potential and can compete with the state-of-the-art methods.

This paper announces a ground truthed database of on-line handwritten mathematical formulae. It have recently been collected in our group in connection with the research on methods for structural pattern recognition. Unlike the availability of handwritten characters or texts, collections of structural objects are rather scarce, thus we would like to provide them to the community. We also present the methodology and tools used for data acquisition. Finally, we report on our experiment with the automatic generation of additional samples. The process utilizes the dataset to extract statistical descriptions of symbols alignments and relative sizes.

We consider a simple statistical model of the image, in which the image is represented as a sum of two parts: one part is explained by an i.i.d. color Gaussian mixture and the other part by a (piecewise-) smooth gray scale shading function. The smoothness is ensured by a quadratic (Tikhonov) or total variation regularization. We derive an EM algorithm to estimate simultaneously the parameters of the mixture model and the shading. Our algorithms for both kinds of the regularization solve for shading and mean parameters of the mixture model jointly.

The proposed method serves for detecting and analysing translating and non-translating periodic motion with small oscillation from video data. Periodic motion in a video sequence is detected using principles of motion accumulation. Non-translating periodic motion is described as change of the point brightness over the time. The trajectory of the motion is obtained by tracking the translating object with periodic motion. Following frequency analysis is done by the means of Fourier transform. The developed method is applied for observing patients suffering from Parkinson disease.

This paper presents a method for recognizing human actions based on pose primitives. In learning mode, the parameters representing poses and activities are estimated from videos. In run mode, the method can be used both for videos or still images. For recognizing pose primitives, we extend a Histogram of Oriented Gradient (HOG) based descriptor to better cope with articulated poses and cluttered background. Action classes are represented by histograms of poses primitives. For sequences, we incorporate the local temporal context by means of n-gram expressions. Action recognition is based on a simple histogram comparison. Unlike the mainstream video surveillance approaches, the proposed method does not rely on background subtraction or dynamic features and thus allows for action recognition in still images.

A new unsupervised filter-based feature selection method is introduced. Its principle consists in merging similar features into clusters using a distance measure derived from the correlation coefficient. Subsequently, only one representative feature is selected from each cluster. In experiments with real-world data, we show that the proposed method is benefical as a pre-filtering step for more sophisticated feature selection techniques.

We propose a novel MRF-based model for deformable image matching. Given two images, the task is to estimate a mapping from one image to the other maximizing the quality of the match. We consider mappings defined by a discrete deformation field constrained to preserve 2D continuity. We pose the task as finding MAP configurations of a pairwise MRF. We propose a more compact MRF representation of the problem which leads to a weaker, though computationally more tractable, linear programming relaxation - the approximation technique we choose to apply. The number of dual LP variables grows linearly with the search window side, rather than quadratically as in previous approaches. To solve the relaxed problem (suboptimally), we apply TRW-S (Sequential Tree-Reweighted Message passing) algorithm [13, 5]. Using our representation and the chosen optimization scheme, we are able to match much wider deformations than was considered previously in global optimization framework. We further elaborate on continuity and data terms to achieve more appropriate description of smooth deformations. The performance of our technique is demonstrated on both synthetic and real-world experiments.

We present a method for off-line mathematical formulae recognition based on the structural construction paradigm and two-dimensional grammars. In general, this approach can be successfully used in the analysis of images containing objects that exhibit rich structural relations. An important benefit of the structural construction is in treating the symbol segmentation in the image and its structural analysis as a single intertwined process. This allows the system to avoid errors usually appearing during the segmentation phase. We have developed and tested a pilot study proving that the method is computationally efficient, practical and able to cope with noise.

In this paper we propose rectification procedure for binocular stereoscopic vision that minimizes the loss of local image neighbourhood discriminability in rectified images. The optimality of the rectification is thus influenced by image contents. Such rectification helps seek for precise dense correspondences.

This text is related to the invited talk at the British Machine Vision Conference in Cardiff in September 2002 and highlights several recent contributions of our group to computer vision research. The aim is to hint the reader to several quite self-contained topics which we developed: omni-directional vision, scene reconstruction from images, and computational stereo. Pointer to results and references to appropriate publications are provided.

We present a technique for camera calibration and Euclidean reconstruction from multiple images of the same scene. Unlike standard Tsai's camera calibration from a known scene, we exploited controlled known motions of the camera to obtain its calibration and Euclidean reconstruction without any knowledge about the scene. We consider three linearly independent translations of an uncalibrated camera mounted on a robot arm that provides us with four views of the scene. The translations of the robot arm are measured in a robot coordinate system. This special, but still realistic, arrangement allowed us to find a linear algorithm for recovering all intrinsic camera calibration parameters, the rotation of the camera with respect to the robot coordinate system, and proper scaling factors for all points allowing their Euclidean reconstruction. The experiments showed that an efficient and robust algorithm was obtained by exploiting Total Least Squares in combination with careful normalization of image coordinates.

This work describes a method for synthesizing correct virtual images of a real scene, described by a set of uncalibrated reference images. The approach used is rendering a projective 3D model. Projective model reconstruction and positioning a projective virtual camera are not addressed. First, the algorithm transfers the vertices of triangles in the triangulated model and then warps interiors of triangles. Hidden faces are removed by z-buffering. The algorithm is more efficient than the ray-tracing-like algorithm for virtual view synthesis by Laveau and Faugeras (1996).

Recently, much attention has been devoted to image-based scene representations. They allow one to construct an arbitrary view of a 3D scene by the interpolation (transfer) from a sparse set of real 2D (reference) images, rather than by rendering an explicit 3D model. While many authors address mainly the purely geometrical aspect of the task, we focus on the problem of how to select the optimal set of reference views. Selection of reference views from a dense set of real primary views is posed as a selection and fitting of parametric models. The selected set must minimize a weighted sum of the number of reference views and the total fit error. We propose two different algorithms for solving this optimization problem. The experimental results on synthetic and real data indicate the feasibility of the approach for 1-DOF camera movement. We discuss the possibility to extend one of the algorithms for more general case.

Presents a new approach to rendering arbitrary views of real-world 3D objects of complex shapes. We propose to represent an object by a sparse set of corresponding 2D views, and to construct any other view as a combination of these reference views. We show that this combination can be linear, assuming proximity of the views, and we suggest how the visibility of constructed points can be determined. Our approach makes it possible to avoid difficult 3D reconstruction, assuming only rendering is required. Moreover, almost no calibration of views is needed. We present preliminary results on real objects, indicating that the approach is feasible.>

The improvement of computing the the curvature of the digitized curves is presented. The standard scheme, i.e. computing curvature using convolution with the truncated Gaussian kernel, was studied. First, we show that systematic bias caused by curvature smoothing can be removed. Second, we demonstrate that large portion of the error has roots in other phenomena (i.e. anisotropy of the raster, limited size of the Gaussian, numerical integration of the convolution, and discretization).

The authors present a formulation of discontinuity detection in range images. The emphasis is on roof edge detection which is evaluated as C/sup 1/ discontinuity strength at every point in the range image. Contrary to other local approaches, it is possible to detect all shapes of C/sup 1/ discontinuity from one polynomial approximation even when the viewpoint changes. First, the range data are locally approximated by a second-order bivariate polynomial. Second, the 2-D problem of surface discontinuity strength estimation is converted to a 1-D probem along the direction of maximal normal curvature at each point on the surface. Third, C/sup 1/ is computed by comparing the polynomial approximation of data and the polynomial approximation of the discretized model of the discontinuity.>