Abstract Most real‐time animation techniques for digital humans are limited to deforming the outer skin surface. Geometric skinning methods are highly efficient but struggle with artifacts such as collapsing joints or self‐intersections when animating inner anatomy along with the outer skin. Volumetric physics‐based simulations, on the other hand, naturally resolve these issues by coordinating bones, muscles, and skin, but are far too slow for interactive use. We solve this problem by training a neural network to predict deformation gradients. Learning deformation gradients instead of vertex displacements makes our method naturally robust to artifacts such as element inversion or volume deviation. Our model, trained on high‐quality finite element simulations, generalizes well across diverse body shapes and poses. This enables anatomically consistent and physically grounded animation of bones, muscles, and skin at interactive frame rates.

As passengers spend more time in vehicles, the demand for non-driving related tasks (NDRTs) increases. In-car Augmented Reality (AR) has the potential to enhance passenger experiences by enabling interaction with the environment through NDRTs using world-fixed Points of Interest (POIs). However, the effectiveness of existing interaction techniques and visualization methods for in-car AR remains unclear. Based on a survey (N=110) and a pre-study (N=10), we developed an interactive in-car AR system using a video see-through head-mounted display to engage with POIs via eye-gaze and pinch. Users could explore passed and upcoming POIs using three visualization techniques: List, Timeline, and Minimap. We evaluated the system's feasibility in a field study (N=21). Our findings indicate general acceptance of the system, with the List visualization being the preferred method for exploring POIs. Additionally, the study highlights limitations of current AR hardware, particularly the impact of vehicle movement on 3D interaction.

With the transition to fully autonomous vehicles, non-driving related tasks (NDRTs) become increasingly important, allowing passengers to use their driving time more efficiently. In-car Augmented Reality (AR) gives the possibility to engage in NDRTs while also allowing passengers to engage with their surroundings, for example, by displaying world-fixed points of interest (POIs). This can lead to new discoveries, provide information about the environment, and improve locational awareness. To explore the optimal visualization of POIs using in-car AR, we conducted a field study (N = 38) examining six parameters: positioning, scaling, rotation, render distance, information density, and appearance. We also asked for intention of use, preferred seat positions and preferred automation level for the AR function in a post-study questionnaire. Our findings reveal user preferences and general acceptance of the AR functionality. Based on these results, we derived UX-guidelines for the visual appearance and behavior of location-based POIs in in-car AR.

Recent developments in smartphone-based avatar reconstruction have made the creation of personalized and realistic avatars significantly more accessible. However, relying on one smartphone camera leads to capturing images sequentially, which introduces new challenges; particularly longer capture times increase the susceptibility to subject motion, which results in degraded reconstructions.

Abstract Due to the increasing use of virtual avatars, the animation of head‐hand interactions has recently gained attention. To this end, we present a novel volumetric and physics‐based interaction simulation. In contrast to previous work, our simulation incorporates temporal effects such as collision paths, respects anatomical constraints, and can detect and simulate skin pulling. As a result, we can achieve more natural‐looking interaction animations and take a step towards greater realism. However, like most complex and computationally expensive simulations, ours is not real‐time capable even on high‐end machines. Therefore, we train small and efficient neural networks as accurate approximations that achieve about 200 FPS on consumer GPUs, about 50 FPS on CPUs, and are learned in less than four hours for one person. In general, our focus is not to generalize the approximation networks to low‐resolution head models but to adapt them to more detailed personalized avatars. Nevertheless, we show that these networks can learn to approximate our head‐hand interaction model for multiple identities while maintaining computational efficiency. Since the quality of the simulations can only be judged subjectively, we conducted a comprehensive user study which confirms the improved realism of our approach. In addition, we provide extensive visual results and inspect the neural approximations quantitatively. All data used in this work has been recorded with a multi–view camera rig. Code and data are available at https://gitlab.cs.hs‐rm.de/cvmr_releases/HeadHand .

As the presence of flying robots continues to grow in both commercial and private sectors, it necessitates an understanding of appropriate methods for nonverbal interaction with humans. While visual cues, such as gestures incorporated into trajectories, are more apparent and thoroughly researched, acoustic cues have remained unexplored, despite their potential to enhance human-drone interaction. Given that additional audiovisual and sensory equipment is not always desired or practicable, and flight noise often masks potential acoustic communication in rotary-wing drones, such as through a loudspeaker, the rotors themselves offer potential for nonverbal communication. In this paper, the consequential sound during the flight of a quadrotor is utilized and modified to carry acoustic information while maintaining the visually perceived flight characteristics. A user study (N=192) demonstrates that acoustically augmenting the trajectories of two aerial gestures with the proposed approach makes them more easily distinguishable. This enhancement contributes to human-drone interaction through onboard means, particularly in situations where the human cannot see or look at the drone.

Facial animation on computationally limited systems still heavily relies on linear blendshape models. Nonetheless, these models exhibit common issues like volume loss, self-collisions, and inaccuracies in soft tissue elasticity. Furthermore, personalizing blendshapes models demands significant effort, but there are limited options for simulating or manipulating physical and anatomical characteristics afterwards. Also, second-order dynamics can only be partially represented. For many years, physics-based facial simulations have been explored as an alternative to linear blendshapes, however, those remain cumbersome to implement and result in a high computational burden. We present a novel deep learning approach that offers the advantages of physics-based facial animations while being effortless and fast to use on top of linear blendshapes. For this, we design an innovative hypernetwork that efficiently approximates a physics-based facial simulation while generalizing over the extensive DECA model of human identities, facial expressions, and a wide range of material properties that can be locally adjusted without re-training. In addition to our previous work, we also demonstrate how the hypernetwork can be applied to facial animation from a sparse set of tracked landmarks. Unlike before, we no longer require linear blendshapes as the foundation of our system but directly operate on neutral head representations. This application is also used to complement an existing framework for commodity smartphones that already implements high resolution scanning of neutral faces and expression tracking.

Offline facial retargeting, i.e., transferring facial expressions from a source to a target character, is a common production task that still regularly leads to considerable algorithmic challenges. This task can be roughly dissected into the transfer of sequential facial animations and non-sequential blendshape personalization. Both problems are typically solved by data-driven methods that require an extensive corpus of costly target examples. Other than that, geometrically motivated approaches do not require intensive data collection but cannot account for character-specific deformations and are known to cause manifold visual artifacts. We present AnaConDaR, a novel method for offline facial retargeting, as a hybrid of data-driven and geometry-driven methods that incorporates anatomical constraints through a physics-based simulation. As a result, our approach combines the advantages of both paradigms while balancing out the respective disadvantages. In contrast to other recent concepts, AnaConDaR achieves substantially individualized results even when only a handful of target examples are available. At the same time, we do not make the common assumption that for each target example a matching source expression must be known. Instead, AnaConDaR establishes correspondences between the source and the target character by a data-driven embedding of the target examples in the source domain. We evaluate our offline facial retargeting algorithm visually, quantitatively, and in two user studies.

Abstract Human shape spaces have been extensively studied, as they are a core element of human shape and pose inference tasks. Classic methods for creating a human shape model register a surface template mesh to a database of 3D scans and use dimensionality reduction techniques, such as Principal Component Analysis, to learn a compact representation. While these shape models enable global shape modifications by correlating anthropometric measurements with the learned subspace, they only provide limitedlocalizedshape control. We instead register a volumetric anatomical template, consisting of skeleton bones and soft tissue, to the surface scans of the CAESAR database. We further enlarge our training data to the full Cartesian product of all skeletons and all soft tissues using physically plausible volumetric deformation transfer. This data is then used to learn an anatomically constrained volumetric human shape model in a self‐supervised fashion. The resultingTailorMemodel enables shape sampling, localized shape manipulation, and fast inference from given surface scans.

Nowadays, cars offer many possibilities to explore the world around you by providing location-based information displayed on a 2D-Map. However, this information is often only available to front-seat passengers while being restricted to in-car displays. To propose a more natural way of interacting with the environment, we implemented an augmented reality head-mounted display to overlay points of interest onto the real world. We aim to compare multiple selection techniques for digital objects located outside a moving car by investigating head gaze with dwell time, head gaze with hardware button, eye gaze with hardware button, and hand pointing with gesture confirmation. Our study was conducted in a moving car under real-world conditions (N=22), with significant results indicating that hand pointing usage led to slower and less precise content selection while eye gaze was preferred by participants and performed on par with the other techniques.

Facial animation on computationally weak systems is still mostly dependent on linear blendshape models. However, these models suffer from typical artifacts such as loss of volume, self-collisions, or erroneous soft tissue elasticity. In addition, while extensive effort is required to personalize blendshapes, there are limited options to simulate or manipulate physical and anatomical properties once a model has been crafted. Finally, second-order dynamics can only be represented to a limited extent.

We address contextualized code retrieval, the search for code snippets helpful to fill gaps in a partial input program. Our approach facilitates a large-scale self-supervised contrastive training by splitting source code randomly into contexts and targets. To combat leakage between the two, we suggest a novel approach based on mutual identifier masking, dedentation, and the selection of syntax-aligned targets. Our second contribution is a new dataset for direct evaluation of contextualized code retrieval, based on a dataset of manually aligned subpassages of code clones. Our experiments demonstrate that the proposed approach improves retrieval substantially, and yields new state-of-the-art results for code clone and defect detection.

We suggest two approaches to incorporate syntactic information into transformer models encoding trees (e.g. abstract syntax trees) and generating sequences. First, we use self-attention with relative position representations to consider structural relationships between nodes using a representation that encodes movements between any pair of nodes in the tree, and demonstrate how those movements can be computed efficiently on the fly. Second, we suggest an auxiliary loss enforcing the network to predict the lowest common ancestor of node pairs. We apply both methods to source code summarization tasks, where we outperform the state-of-the-art by up to 6 % F1. On natural language machine translation, our models yield competitive results. We also consistently outperform sequence-based transformers, and demonstrate that our method yields representations that are more closely aligned with the AST structure.

The increasing autonomy and presence of Unmanned Aerial Vehicles (UAVs), especially quadrotors, in everyday applications requires in-depth studies of proxemics in Human-Drone Interaction (HDI) and novel methods of user interaction suitable for different distances. This paper presents a user study (N=32) that evaluates proxemics with a miniature quadrotor (92 mm wheelbase) from four directions (front, back, left, right) in a seated setting investigating preferred approach directions and distances in future home or workplace scenarios. The goal of this study is to determine if humans are willing to allow flying robots of that size and mechanical appearance to approach close enough to enable tactile interaction. Moreover, the participants' inclination to physically interact with the quadrotor is examined. Studies evaluating proxemics in HDI are highly dependent on repeatable results and actually flying robots. In most comparable studies, the quadrotors used did not fly freely or at all, but were moved, manually controlled, or flew barely repeatable trajectories due to unstable onboard navigation. Only few studies have used pose estimation systems that ensure smooth and reproducible trajectories and thus reliable findings of the studies. For this reason, in addition to the presented study and its results, an insight into the used testbed is provided, that also integrates full skeleton pose estimation rather than tracking participants with only a single marker.

Aerial robots such as quadrotors enjoy ever-increasing popularity and emerge in everyday applications that require user interaction. At immediate proximity, physical control of the quadrotor by touch may be desired or even necessary. In this paper we present a tactile 3D touch interaction scenario with a quadrotor by introducing virtual buttons whose operation is detected in the accelerometer data of the built-in Inertial Measurement Unit (IMU) of the quadrotor. By dispensing with additional sensors, we are able to keep the size of the used quadrotor to a minimum and thus address the problem of users being discouraged from interaction with quadrotors at immediate proximity. As an example for the proposed interaction scenario, we introduce MetroDrone, a quadrotor responding to repeated user taps to virtually defined buttons by flying trajectories according to the beat and operated button. This introduces a minimalist interaction technique that requires no intermediary devices and strengthens human-robot connections through shared musical experience.

We present a novel extension to embedding-based knowledge graph completion models which enables them to perform open-world link prediction, i.e. to predict facts for entities unseen in training based on their textual description. Our model combines a regular link prediction model learned from a knowledge graph with word embeddings learned from a textual corpus. After training both independently, we learn a transformation to map the embeddings of an entity’s name and description to the graph-based embedding space.In experiments on several datasets including FB20k, DBPedia50k and our new dataset FB15k-237-OWE, we demonstrate competitive results. Particularly, our approach exploits the full knowledge graph structure even when textual descriptions are scarce, does not require a joint training on graph and text, and can be applied to any embedding-based link prediction model, such as TransE, ComplEx and DistMult.

We propose an algorithm for real-time 6DOF pose tracking of rigid 3D objects using a monocular RGB camera. The key idea is to derive a region-based cost function using temporally consistent local color histograms. While such region-based cost functions are commonly optimized using first-order gradient descent techniques, we systematically derive a Gauss-Newton optimization scheme which gives rise to drastically faster convergence and highly accurate and robust tracking performance. We furthermore propose a novel complex dataset dedicated for the task of monocular object pose tracking and make it publicly available to the community. To our knowledge, it is the first to address the common and important scenario in which both the camera as well as the objects are moving simultaneously in cluttered scenes. In numerous experiments-including our own proposed dataset-we demonstrate that the proposed Gauss-Newton approach outperforms existing approaches, in particular in the presence of cluttered backgrounds, heterogeneous objects and partial occlusions.

We present a novel approach to 6DOF pose estimation and segmentation of rigid 3D objects using a single monocular RGB camera based on temporally consistent, local color histograms. We show that this approach outperforms previous methods in cases of cluttered backgrounds, heterogenous objects, and occlusions. The proposed histograms can be used as statistical object descriptors within a template matching strategy for pose recovery after temporary tracking loss e.g. caused by massive occlusion or if the object leaves the camera's field of view. The descriptors can be trained online within a couple of seconds moving a handheld object in front of a camera. During the training stage, our approach is already capable to recover from accidental tracking loss. We demonstrate the performance of our method in comparison to the state of the art in different challenging experiments including a popular public data set.

research-article Share on Cross-modal Image-Graphics Retrieval by Neural Transfer Learning Authors: Fabian Junkert RheinMain University of Applied Sciences, Wiesbaden, Germany RheinMain University of Applied Sciences, Wiesbaden, GermanyView Profile , Markus Eberts RheinMain University of Applied Sciences, Wiesbaden, Germany RheinMain University of Applied Sciences, Wiesbaden, GermanyView Profile , Adrian Ulges RheinMain University of Applied Sciences, Wiesbaden, Germany RheinMain University of Applied Sciences, Wiesbaden, GermanyView Profile , Ulrich Schwanecke RheinMain University of Applied Sciences, Wiesbaden, Germany RheinMain University of Applied Sciences, Wiesbaden, GermanyView Profile Authors Info & Claims ICMR '17: Proceedings of the 2017 ACM on International Conference on Multimedia RetrievalJune 2017 Pages 330–337https://doi.org/10.1145/3078971.3078994Published:06 June 2017Publication History 1citation212DownloadsMetricsTotal Citations1Total Downloads212Last 12 Months3Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

We present AMIGO, an automatic indexer for video presentations which - given an e-lecture and supplementary slides - localizes the exact time and position of each slide displayed in the video footage. This offers richer access to viewers, including a slide-accurate navigation and a text-based interaction with the video. AMIGO is based on a matching of local features between video frames and presentation slides. Our key contribution, however, is the combination of local feature matching with two temporal models (a Hidden Markov Model (HMM) and a simple heuristic filter), exploiting the alignment of the presentation with the reading order of its supplementary material. We demonstrate the effectiveness of our approach in quantitative experiments on a dataset of e-lectures and screencasts, which show - with an average accuracy of over 95% - that the approach works under occlusion and camera motion.

In this paper, we present a new variant of the well-known Random Sample Consensus (RANSAC) algorithm for robust estimation of model parameters. The idea of our method is based on a kind of volatile memory which is similar to the pheromone evaporation in the ant colony optimization algorithm. Therefore, we call our improved RANSAC like algorithm ANTSAC. We describe our new approach and the influence of its relevant parameters to the achieved performance in detail. ANTSAC is computationally efficient and convincingly easy to implement. It turns out that ANTSAC significantly outperforms RANSAC regarding the number of inliers after a given number of iterations. Further, we show that the advantage of ANTSAC increases with the complexity of the problem, i.e., with the number of model parameters, as well as with the relative number of outliers. ANTSAC is entirely generic, such that no further domain knowledge is required, as it is for many other RANSAC extensions. Nevertheless, we show that it is competitive to state-of-the-art methods even in domain specific scenarios.

Enhancing desk-based computer environments with virtual reality technology requires natural interaction support, in particular head and hand tracking. Todays motion capture systems instrument users with intrusive hardware like optical markers or data gloves which limit the perceived realism of interactions with a virtual environment and constrain the free moving space of operators. Our work therefore focuses on the development of fault-tolerant techniques for fast, non-contact 3D hand motion capture, targeted to the application in standard office environments. This paper presents a table-top setup which utilizes vision based volumetric reconstruction and tracking of skin colored objects to integrate the users hands and face into virtual environments. The system is based on off-the-shelf hardware components and satisfies real-time constraints.

Figure 1: We present a new technique to extract high quality triangle meshes from volume representations of geometric objects.The two main contributions are an enhanced distance field representation and an extended Marching Cubes algorithm.The above figures show reconstructions of the well-known "fandisk" dataset from its distance field representation.The distance field has been sampled on a uniform 65×65×65 grid.The far left image shows the standard Marching Cubes reconstruction, center left is the reconstruction by the same algorithm but applied to the enhanced distance field with the same resolution.Center right shows the result of our new extended Marching Cubes algorithm applied to the original volume data, and finally on the far right we show the reconstruction by our new algorithm applied to the enhanced distance field.The approximation error to the original polygonal model is below 0.25 %.