Inertial human motion capture (IHMC) has become a robust tool to estimate human kinematics in the wild such as industrial facilities. In contrast to optical motion capture, where occlusions might take place, the kinematics of a worker can be continuously provided. This is for instance a prerequisite for an ergonomic assessments of the workers. State-of-the-art IHMC solutions require inertial sensors to be tightly attached to body segments. This requires an additional setup time and lowers the practicability and ease of use when it comes to an industrial application. In contrast, sensors integrated into loose clothing such as a working jacket, may yield corrupted kinematics estimates due to the additional motion of loose clothing. In this work we present a study of orientations deviations obtained from kinematics estimates using tightly attached inertial sensors and into a working jacket integrated ones. We performed a quantitative analysis using data from the two hardware setups worn by 19 subjects performing different industry related tasks and measures of their body shapes. Using this data we approximated probability distributions of the deviation angles for each person and body segment. Applying different statistical measures we could gain insights to questions like, how severe orientation deviations are, if there is an influence of body shapes on the distribution and how probability distributions of the deviation angles can indicate physical motion limitations of a sensor attached to a segment.

The reliability assessment of a machine learning model’s prediction is an important quantity for the deployment in safety critical applications. Not only can it be used to detect novel sceneries, either as out-of-distribution or anomaly sample, but it also helps to determine deficiencies in the training data distribution. A lot of promising research directions have either proposed traditional methods like Gaussian processes or extended deep learning based approaches, for example, by interpreting them from a Bayesian point of view. In this work we propose a novel approach for uncertainty estimation based on autoencoder models: The recursive application of a previously trained autoencoder model can be interpreted as a dynamical system storing training examples as attractors. While input images close to known samples will converge to the same or similar attractor, input samples containing unknown features are unstable and converge to different training samples by potentially removing or changing characteristic features. The use of dropout during training and inference leads to a family of similar dynamical systems, each one being robust on samples close to the training distribution but unstable on new features. Either the model reliably removes these features or the resulting instability can be exploited to detect problematic input samples. We evaluate our approach on several dataset combinations as well as on an industrial application for occupant classification in the vehicle interior for which we additionally release a new synthetic dataset.

Learning on synthetic data and transferring the resulting properties to their real counterparts is an important challenge for reducing costs and increasing safety in machine learning. In this work, we focus on autoencoder architectures and aim at learning latent space representations that are invariant to inductive biases caused by the domain shift between simulated and real images showing the same scenario. We train on synthetic images only, present approaches to increase generalizability and improve the preservation of the semantics to real datasets of increasing visual complexity. We show that pre-trained feature extractors (e.g. VGG) can be sufficient for generalization on images of lower complexity, but additional improvements are required for visually more complex scenes. To this end, we demonstrate a new sampling technique, which matches semantically important parts of the image, while randomizing the other parts, leads to salient feature extraction and a neglection of unimportant parts. This helps the generalization to real data and we further show that our approach outperforms fine-tuned classification models.

Inertial motion capture has become an attractive alternative to optical motion capture for human joint angle estimation outside the laboratory. Usually inertial sensors are assumed to be tightly fixed to the body segments, which can be cumbersome regarding setup-time and ease-of-use. However, integrating the sensors directly into loose clothing, usually, results in additional clothing motion relative to the motion of the underlying bones that should be captured. In this work we propose the Difference Mapping distributions approach that corrects the segment orientations of a given inertial motion capture system that assumes tightly coupled sensors. The approach allows to reduce the joint angle errors due to clothing artefacts by at least 77.2% for people with similar morphology performing a similar task as seen in the training data, including an ergonomic assessments scenario at work places with ten participants. Moreover, we show that the uncertainty of the distribution can be used to measure the reliability of the predicted map if e.g. the motion is further away from the training data to allow for an artefact aware inertial motion tracking approach. The experimental data for this study is available online under [1].

Common domain shift problem formulations consider the integration of multiple source domains, or the target domain during training. Regarding the generalization of machine learning models between different car interiors, we formulate the criterion of training in a single vehicle: without access to the target distribution of the vehicle the model would be deployed to, neither with access to multiple vehicles during training. We performed an investigation on the SVIRO dataset for occupant classification on the rear bench and propose an autoencoder based approach to improve the transferability. The autoencoder is on par with commonly used classification models when trained from scratch and sometimes out-performs models pre-trained on a large amount of data. Moreover, the autoencoder can transform images from unknown vehicles into the vehicle it was trained on. These results are corroborated by an evaluation on real infrared images from two vehicle interiors.

Images recorded during the lifetime of computer vision based systems undergo a wide range of illumination and environmental conditions affecting the reliability of previously trained machine learning models. Image normalization is hence a valuable preprocessing component to enhance the models' robustness. To this end, we introduce a new strategy for the cost function formulation of encoder-decoder networks to average out all the unimportant information in the input images (e.g. environmental features and illumination changes) to focus on the reconstruction of the salient features (e.g. class instances). Our method exploits the availability of identical sceneries under different illumination and environmental conditions for which we formulate a partially impossible reconstruction target: the input image will not convey enough information to reconstruct the target in its entirety. Its applicability is assessed on three publicly available datasets. We combine the triplet loss as a regular- izer in the latent space representation and a nearest neighbour search to improve the generalization to unseen illuminations and class instances. The importance of the aforementioned post-processing is highlighted on an automotive application. To this end, we release a synthetic dataset of sceneries from three different passenger compartments where each scenery is rendered under ten different illumination and environmental conditions: https://sviro.kl.dfki.de.

Inertial measurement units (IMUs) enable the capture of human motion in-field. This can be used in various analysis applications ranging from the medical domain over sports to daily activities. Manual data labelling for classification or regression tasks is often time-consuming and cumbersome, in particular when it comes to larger datasets. Active learning algorithms try to reduce the labelling cost, for instance via suggesting samples with high prediction uncertainty that should be explicitly labelled. In this work, we apply two probabilistic deep learning approaches on different state-of-the-art deep neural network structures for timeseries data, with uncertainty based measures to actively query sample labels. This is applied to gait phase classification using IMU data as inputs. We demonstrate the performance on a newly captured dataset, where we obtained high accuracy (up to 96%) with up to 43% fewer samples as compared to random sampling in an online setting. In an offline setting, we could extract heel strike and toe-off foot events with an accuracy of 99.9% using active learning strategies with up to 58% fewer samples.

The ability to accurately capture locomotion is relevant in various use cases, in particular in the sports and health area. With the major goal of providing a measurement system that can deliver different types of relevant information (3D body segment kinematics, spatiotemporal locomotion parameters, and locomotion patterns) in-field and in real-time, we propose a novel probabilistic (single-plane) ground contact estimation method, using four contact points defined through a biomechanical foot model, and integrate this into an existing inertial motion capturing method. The resulting method is quantitatively evaluated on simulated and real IMU data in comparison to an optical motion capture system on walking, running, and jumping sequences. The results show its ability to maintain a good average 3D kinematics estimation error on low- and high-acceleration locomotion, whereas many previous accuracy studies restrict themselves to movements with low to moderate global accelerations, such as upper body activities or slow locomotion. Moreover, a qualitative evaluation of the estimated ground contact probabilities demonstrates the method's ability to also provide consistent information also for deriving spatiotemporal locomotion parameters as well as locomotion patterns (e.g., over-pronation/-supination) simultaneously with the 3D kinematics.

We present an elegant solution to joint pre-alignment and rigid point set registration, given prior matches. Instead of performing pre-alignment and the actual registration in the separate steps, prior matches explicitly influence the registration procedure in our approach. This results in several advantages. Firstly, our approach solves the pre-alignment task - an approximate resolving of rotation and translation - with an insufficient number of prior correspondences, when other methods fail. Secondly, it produces more accurate rigid registrations of noisy point sets than the state of the art Coherent Point Drift method. Combined with application specific methods for correspondence establishment, we demonstrate superiority of our approach in several synthetic and real-world scenarios.

A new dataset for hand-pose is introduced. The dataset includes the top view images of the palm by Time of Flight (ToF) camera. It is recorded in an experimental setting with twelve participants for six hand-poses. An evaluation on the dataset is carried out with a dedicated Convolutional Neural Network (CNN) architecture for Hand Pose Recognition (HPR). This architecture uses a model-layer. The small size model layer creates a funnel shape network which adds a priori knowledge and constrains the network by modelling the degree of freedom of the palm, such that it learns palm features. It is demonstrated that this network performs better than a similar network without the prior added. A two-phase learning scheme which allows training the model on full dataset even when the classification problem is confined to a subset of the classes is described. The best model performs at an accuracy of 92%. Finally, we show the feature transfer capability of the network and compare the extracted features from various networks and discuss usefulness for various applications.

The problem of dense point set registration, given a sparse set of prior correspondences, often arises in computer vision tasks. Unlike in the rigid case, integrating prior knowledge into a registration algorithm is especially demanding in the non-rigid case due to the high variability of motion and deformation. In this paper we present the Extended Coherent Point Drift registration algorithm. It enables, on the one hand, to couple correspondence priors into the dense registration procedure in a closed form and, on the other hand, to process large point sets in reasonable time through adopting an optimal coarse-to-fine strategy. Combined with a suitable keypoint extractor during the preprocessing step, our method allows for non-rigid registrations with increased accuracy for point sets with structured outliers. We demonstrate advantages of our approach against other non-rigid point set registration methods in synthetic and real-world scenarios.

This paper addresses the problem of video registration for dense non-rigid structure from motion under suboptimal conditions, such as noise, self-occlusions, considerable external occlusions or specularities, i.e. the computation of optical flow between the reference image and each of the subsequent images in a video sequence when the camera observes a highly deformable object. We tackle this challenging task by improving previously proposed variational optimization techniques for multi-frame optical flow (MFOF) through detection, tracking and handling of uncertain flow field estimates. This is based on a novel Bayesian inference approach incorporated into the MFOF. At the same time, computational costs are significantly reduced through iterative pre-computation of the flow fields. As shown through experiments, the resulting method performs superior to other state-of-the-art (MF)OF methods on video sequences showing a highly non-rigidly deforming object with considerable occlusions.

Modern large displacement optical flow algorithms usually use an initialization by either sparse descriptor matching techniques or dense approximate nearest neighbor fields. While the latter have the advantage of being dense, they have the major disadvantage of being very outlier-prone as they are not designed to find the optical flow, but the visually most similar correspondence. In this article we present a dense correspondence field approach that is much less outlier-prone and thus much better suited for optical flow estimation than approximate nearest neighbor fields. Our approach does not require explicit regularization, smoothing (like median filtering) or a new data term. Instead we solely rely on patch matching techniques and a novel multi-scale matching strategy. We also present enhancements for outlier filtering. We show that our approach is better suited for large displacement optical flow estimation than modern descriptor matching techniques. We do so by initializing EpicFlow with our approach instead of their originally used state-of-the-art descriptor matching technique. We significantly outperform the original EpicFlow on MPI-Sintel, KITTI 2012, KITTI 2015 and Middlebury. In this extended article of our former conference publication we further improve our approach in matching accuracy as well as runtime and present more experiments and insights.