Abstract Symmetric objects are common in daily life and industry, yet their inherent orientation ambiguities that impede the training of deep learning networks for pose estimation are rarely discussed in the literature. To cope with these ambiguities, existing solutions typically require the design of specific loss functions and network architectures or resort to symmetry-invariant evaluation metrics. In contrast, we focus on the numeric representation of the rotation itself, modifying trigonometric identities with the degrees of symmetry derived from the objects’ shapes. We use our representation, SARR, to obtain canonic (symmetry-resolved) poses for the symmetric objects in two popular 6D pose estimation datasets, T-LESS and ITODD, where SARR is unique and continuous w.r.t. the visual appearance. This allows us to use a standard CNN for 3D orientation estimation whose performance is evaluated with the symmetry-sensitive cosine distance AR C . Our networks outperform the state of the art using AR C and achieve satisfactory performance when using conventional symmetry-invariant measures. Our method does not require any 3D models but only depth, or, as part of an additional experiment, texture-less RGB/grayscale images as input. We also show that networks trained on SARR outperform the same networks trained on rotation matrices, Euler angles, quaternions, standard trigonometrics or the recently popular 6d representation – even in inference scenarios where no prior knowledge of the objects’ symmetry properties is available. Code and a visualization toolkit are available at https://github.com/akriegler/SARR .

Recently, transformers have gained prominence in video action recognition due to their ability to capture spatio-temporal dependencies. Despite their effectiveness, the interpretability of their self-attention mechanisms remains limited, posing obstacles in understanding model decisions, impacting transparency and bias identification. Additionally, the computational demands of transformer architectures, particularly the self-attention mechanism, present practical difficulties. To tackle both challenges, we adapt existing interpretability techniques and introduce a layer pruning method guided by importance metrics. In the context of driver action recognition, our findings highlight the efficacy of the applied head importance metrics in pinpointing crucial attention heads and identifying key visual cues essential for recognizing driver behavior. Experimental results, conducted on three mainstream video transformers, demonstrate the effectiveness of the proposed pruning technique with significantly reduced computational costs and only slight performance degradation by removing low-relevance layers. Specifically, on our DriverActionInsight (DAI) dataset, we achieve a 23.5% FLOPs saving in compressing Video Swin with less than a 1 % decrease in Top-1 accuracy.

This review article provides an overview of papers that have studied body movement mirroring and synchrony within the field of human-robot interaction. The papers included in this review cover system studies, which focus on evaluating the technical aspects of mirroring and synchrony robotic systems, and user studies, which focus on measuring particular interaction outcomes or attitudes towards robots expressing mirroring and synchrony behaviors. We review the papers in terms of the employed robotic platforms and the focus on parts of the body, the techniques used to sense and react to human motion, the evaluation methods, the intended applications of the human-robot interaction systems and the scenarios utilized in user studies. Finally, challenges and possible future directions are considered and discussed.

The training of computer vision models for human pose estimation requires large amounts of data. Since labelling image data with pose keypoints is very time consuming and costly, we aim to alleviate this requirement by using synthetic data during pre-training and thus relax the need for large amounts of real data samples during fine-tuning. To this end, we investigate the impact of synthetic data on the performance of a 2D keypoint detection model in the context of driver body pose estimation. We present our approach for synthetic data generation to automatically provide large amounts of in-cabin views as training data. The utilization of the generated synthetic data is evaluated in different learning schemes. We achieve a notable performance gain of +30.5% by pre-training with our in-cabin synthetic data when only 1% of real training data from the DriPE dataset is available. The proposed approach also outperforms pre-training with PeopleSansPeople by +8.3% when the reduced DriPE dataset is used for fine-tuning.

This paper explores the effects of selected exaggerated nonverbal cues on the perception of the Pepper robot in a story-telling scenario. We conduct a video-based user study based on an online survey containing the Godspeed questionnaire and additional interviews. The results of the study indicate that using exaggeration as a method to design nonverbal cues can improve the perception of the robot in both robot-talking and robot-listening cases. We find that Pepper is perceived as animate and safe in both cases, and additionally as anthropomorphic for the case when the robot is talking.

In this paper, a human-humanoid imitation system is proposed, with a focus on the kinematic model used for translating end effector positions to joint angles. The overall system comprises the humanoid robot Pepper and a Kinect v2 camera for capturing human 3D joint positions. The presented kinematic model is based on analytical solutions of Pepper’s inverse kinematics and also uses the forward kinematics. The aim of the paper is to provide insights into deriving the kinematics of robotic chains for the purpose of pose matching imitation, as well as accuracy evaluation of the derived forward and inverse kinematic solutions. The solutions of the inverse kinematics provide results with a mean error of approximately 0.2°for the angle solutions of the head joints, 0.7°for the arm joints, and 4° for the torso (leg) joints. The evaluated speed lies within a range of 0.002 to 0.08 ms. These results indicate that the presented kinematic model is an effective method for translating end effector positions to joint angles for our pose imitation application in real-time or close to it. Finally, we show preliminary results of the proposed imitation system and discuss future work.

We describe the development of an intelligent film assistant system that supports the creation of professional video content on mobile devices such as smart phones by amateur users. Cinematographic expert knowledge on scene composition and camera motion is provided to the user in the form of story boards that are tailored to specific use cases. We give an overview of the project concept and some selected components including algorithms that are required for video stabilization and shot classification. The goal is to translate the scene characteristics detected by the vision algorithms into real-time feedback to the user during video recording and to support post-production. A major focus of the project is the incorporation of human-centered design principles with usability studies and expert interviews into the whole research process.

In this work we investigate an infrared structured light prototype which is intended for 3D reconstruction in resource-restricted mobile applications. We explore the constraints on working range and pattern resolution that are imposed by the low-light property of our single-shot set-up. While focusing on the most light-sensitive steps of the decoding workflow, we suggest adaptations of image rectification and pattern generation and segmentation algorithms that are tailored to the specific spatial and radiometric requirements of our system.

In this paper we propose an optimized semiautomatic approach for efficient 2D-to-3D video conversion. It is based on a conversion algorithm that leverages segmentation and filtering techniques to propagate sparse depth information that was provided by a user. Our GPU acceleration of in the work of Brosch et al. (2011) significantly reduces the computation time of the original algorithm. Since the limited capacity of the CPU's onboard memory hinders the parallel execution of large data such as videos, we additionally propose a temporally coherent clip-based 2D-to-3D conversion approach for long videos. Evaluations show that the proposed, optimized conversion approach is capable of generating high-quality results, while significantly reducing the execution time compared to the original, un-optimized approach.

We tackle the problem of jointly increasing the spatial resolution and apparent measurement accuracy of an input low-resolution, noisy, and perhaps heavily quantized depth map. In stark contrast to earlier work, we make no use of ancillary data like a color image at the target resolution, multiple aligned depth maps, or a database of high-resolution depth exemplars. Instead, we proceed by identifying and merging patch correspondences within the input depth map itself, exploiting patch wise scene self-similarity across depth such as repetition of geometric primitives or object symmetry. While the notion of 'single-image' super resolution has successfully been applied in the context of color and intensity images, we are to our knowledge the first to present a tailored analogue for depth images. Rather than reason in terms of patches of 2D pixels as others have before us, our key contribution is to proceed by reasoning in terms of patches of 3D points, with matched patch pairs related by a respective 6 DoF rigid body motion in 3D. In support of obtaining a dense correspondence field in reasonable time, we introduce a new 3D variant of Patch Match. A third contribution is a simple, yet effective patch up scaling and merging technique, which predicts sharp object boundaries at the target resolution. We show that our results are highly competitive with those of alternative techniques leveraging even a color image at the target resolution or a database of high-resolution depth exemplars.

Many computer vision tasks can be formulated as labeling problems. The desired solution is often a spatially smooth labeling where label transitions are aligned with color edges of the input image. We show that such solutions can be efficiently achieved by smoothing the label costs with a very fast edge-preserving filter. In this paper, we propose a generic and simple framework comprising three steps: 1) constructing a cost volume, 2) fast cost volume filtering, and 3) Winner-Takes-All label selection. Our main contribution is to show that with such a simple framework state-of-the-art results can be achieved for several computer vision applications. In particular, we achieve 1) disparity maps in real time whose quality exceeds those of all other fast (local) approaches on the Middlebury stereo benchmark, and 2) optical flow fields which contain very fine structures as well as large displacements. To demonstrate robustness, the few parameters of our framework are set to nearly identical values for both applications. Also, competitive results for interactive image segmentation are presented. With this work, we hope to inspire other researchers to leverage this framework to other application areas.

Many computer vision tasks can be formulated as labeling problems. The desired solution is often a spatially smooth labeling where label transitions are aligned with color edges of the input image. We show that such solutions can be efficiently achieved by smoothing the label costs with a very fast edge preserving filter. In this paper we propose a generic and simple framework comprising three steps: (i) constructing a cost volume (ii) fast cost volume filtering and (iii) winner-take-all label selection. Our main contribution is to show that with such a simple framework state-of-the-art results can be achieved for several computer vision applications. In particular, we achieve (i) disparity maps in real-time, whose quality exceeds those of all other fast (local) approaches on the Middlebury stereo benchmark, and (ii) optical flow fields with very fine structures as well as large displacements. To demonstrate robustness, the few parameters of our framework are set to nearly identical values for both applications. Also, competitive results for interactive image segmentation are presented. With this work, we hope to inspire other researchers to leverage this framework to other application areas.

Adaptive support weight algorithms represent the state-of the-art in local stereo matching. Their limitation is a high computational demand, which makes them unattractive for many (real-time) applications. To our knowledge, the algorithm proposed in this paper is the first local method which is both fast (real-time) and produces results comparable to global algorithms. A key insight is that the aggregation step of adaptive support weight algorithms is equivalent to smoothing the stereo cost volume with an edge-preserving filter. From this perspective, the original adaptive support weight algorithm [1] applies bilateral filtering on cost volume slices, and the reason for its poor computational behavior is that bilateral filtering is a relatively slow process. We suggest to use the recently proposed guided filter [2] to overcome this limitation. Analogously to the bilateral filter, this filter has edge preserving properties, but can be implemented in a very fast way, which makes our stereo algorithm independent of the size of the match window. The GPU implementation of our stereo algorithm can process stereo images with a resolution of 640 × 480 pixels and a disparity range of 26 pixels at 25 fps. According to the Middlebury on-line ranking, our algorithm achieves rank 14 out of over 100 submissions and is not only the best performing local stereo matching method, but also the best performing real-time method.

In this paper we considerably improve on a state-of-the-art alpha matting approach by incorporating a new prior which is based on the image formation process. In particular, we model the prior probability of an alpha matte as the convolution of a high-resolution binary segmentation with the spatially varying point spread function (PSF) of the camera. Our main contribution is a new and efficient de-convolution approach that recovers the prior model, given an approximate alpha matte. By assuming that the PSF is a kernel with a single peak, we are able to recover the binary segmentation with an MRF-based approach, which exploits flux and a new way of enforcing connectivity. The spatially varying PSF is obtained via a partitioning of the image into regions of similar defocus. Incorporating our new prior model into a state-of-the-art matting technique produces results that outperform all competitors, which we confirm using a publicly available benchmark.

We propose an algorithm that simultaneously extracts disparities and alpha matting information given a stereo image pair. Our method divides the reference image into a set of overlapping, partially transparent color segments. Each segment pixel is assigned an alpha value and color. The disparity inside the segment is modeled via a plane. The goodness of alphas, colors and disparity planes is measured by a new energy function. Its basic idea is to use the three parameters for generating artificial views representing the left and right images. If alphas, colors and disparity planes are correct, these artificial images should be very similar to the real ones. For generating the artificial right view, we warp all pixels of the left into the geometry of the right image using the disparity planes. We introduce the assumption of constant solidity in order to correctly model how pixels' alpha values are affected by the warping operation. Experimental results on the Middlebury set show that our algorithm gives good results in comparison to the state-of-the-art in stereo matching.

The availability of quantitative online benchmarks for low-level vision tasks such as stereo and optical flow has led to significant progress in the respective fields. This paper introduces such a benchmark for image matting. There are three key factors for a successful benchmarking system: (a) a challenging, high-quality ground truth test set; (b) an online evaluation repository that is dynamically updated with new results; (c) perceptually motivated error functions. Our new benchmark strives to meet all three criteria. We evaluated several matting methods with our benchmark and show that their performance varies depending on the error function. Also, our challenging test set reveals problems of existing algorithms, not reflected in previously reported results. We hope that our effort will lead to considerable progress in the field of image matting, and welcome the reader to visit our benchmark at www.aIphamatting.com.

We present a simple and fast method for performing unsupervised segmentation. Our method works by centering a square window on each pixel of the input image. Each pixel is then assigned to a new color which is computed by averaging the pixel colors inside the window. The idea is that if this averaging operation is repeated a few times then we should obtain an image in which pixels of the same color surface are assigned to the same (or at least to very similar) color values. Consequently, the desired color segments are formed by groups of spatially neighboring pixels that share the same color in the convolved image. Obviously, our method would deliver poor performance if the averaging operation is applied in a naive manner, as pixel colors of different segments would be mixed. To overcome this problem, we propose to compute a geodesic weight mask that regulates a pixel's influence in the averaging operation. A pixel's weight in the window is determined by computing the geodesic distance to the center pixel. In other words, we enforce that a pixel obtains high influence only if there exists a path to the center pixel along which the color does not change significantly (connectivity). The proposed method is evaluated on some widely used test images. Our method seems to produce accurate segmentation results and to capture object outlines correctly. We show by quantitative evaluation that our segmentation algorithm outperforms two competing segmentation methods.

Local stereo matching has recently experienced large progress by the introduction of new support aggregation schemes. These approaches estimate a pixel's support region via color segmentation. Our contribution lies in an improved method for accomplishing this segmentation. Inside a square support window, we compute the geodesic distance from all pixels to the window's center pixel. Pixels of low geodesic distance are given high support weights and therefore large influence in the matching process. In contrast to previous work, we enforce connectivity by using the geodesic distance transform. For obtaining a high support weight, a pixel must have a path to the center point along which the color does not change significantly. This connectivity property leads to improved segmentation results and consequently to improved disparity maps. The success of our geodesic approach is demonstrated on the Middlebury images. According to the Middlebury benchmark, the proposed algorithm is the top performer among local stereo methods at the current state-of-the-art.

In this paper, we introduce a high-level simulation methodology for the modeling of multicore video processing architectures. This method allows design space explorations of parallel video processing applications (VPAs). It is used to test the performance of running a VPA on arbitrary virtual hardware and software configurations. The method represents an alternative to performing a “complete” decoder implementation on a field-programmable gate array or an application-specific integrated circuit. The use of our method, therefore, yields the advantage of being considerably more time, labor, and cost efficient. As an application, we use our method for designing a parallel H.264 decoder targeting 720 p 25 resolution at bit-rates up to 50 Mb/s. Starting from a single-core decoder implementation, we use our simulator for estimating the performance gain when using a multicore architecture. We then detect the major performance bottlenecks in this multicore system and perform additional decoder splittings accordingly until we reach the targeted requirements. The use of functional splitting (i.e., pipelining) and data-parallel processing is demonstrated. The final H.264 decoder architecture is capable of fulfilling our performance requirements.

This paper addresses the problem of extracting an alpha matte from a single photograph given a user-defined trimap. A crucial part of this task is the color modeling step where for each pixel the optimal alpha value, together with its confidence, is estimated individually. This forms the data term of the objective function. It comprises of three steps: (i) Collecting a candidate set of potential fore- and background colors; (ii) Selecting high confidence samples from the candidate set; (iii) Estimating a sparsity prior to remove blurry artifacts. We introduce novel ideas for each of these steps and show that our approach considerably improves over state-of-the-art techniques by evaluating it on a large database of 54 images with known high-quality ground truth. 1

The high computational demands of the H.264 decoding process pose serious challenges on current processor architectures. A natural way to tackle this problem is the use of multi-core systems. The contribution of this paper lies in a systematic overview and performance evaluation of parallel video decoding approaches. Our study investigates six methods for accomplishing data-parallel splitting in strongly resource-restricted environments inherent to mobile devices. These methods are compared against each other in terms of run-time complexity, core usage, inter-communication and bus transfers. We present benchmark results using different numbers of processor cores. Our results shall aid in finding a splitting strategy that is best suited for the targeted hardware-architecture.

We present a method to generate stylized stereo imagery that effectively communicates shape and distance of the depicted scene objects. We use computer vision techniques to analyze real stereo image pairs. In particular, a region based stereo matching algorithm with symmetrical treatment of occlusions is used to extract a disparity map and successively the depth information of the scene. The reference image is color segmented for the purpose of color stylization and an algorithm combining intensity image edges and depth discontinuities is applied to depict dominant object contours in the image. We use disparity information to propagate stylized color segments to the second view together with the object outlining contours. The stylized image pairs are consistent across the two views and can be easily fused for stereoscopic viewing. The stereoscopic image fusion provides an extra dimension of depth that is absent on the individual images.

We propose a new stereo algorithm which uses colour segmentation to allow the handling of large untextured regions and precise localization of depth boundaries. Each segment is modelled as a plane. Robustness of the depth representation is achieved by the use of a layered model. Layers are extracted by mean-shift-based clustering of depth planes. For layer assignment a global cost function is defined. The quality of the disparity map is measured by warping the reference image to the second view and compares it with the real image. Z-buffering enforces visibility and allows the explicit detection of occlusions. An efficient greedy algorithm searches for a local minimum of the cost function. Layer extraction and assignment are alternately applied. Results obtained for benchmark and self-recorded images indicate that the proposed algorithm can compete with the state-of-the-art.

In this case study, we utilize X3D (eXtensible 3D) for the geoscientific visualization of an interferometric (InSAR) data set of Kilauea volcano, Hawaii. We describe the modeling and animation of the geospatial data using X3D features and tools and demonstrate the Web-based interaction of the user with the displayed 3D content.

The authors present a method for improving relief reconstruction from SAR stereo images, relying on preprocessing of input data with an optimal filter. This filter strongly reduces speckle noise and enhances relief structures, leading to better matching results. The method was tested using X-SAR data of a mountainous site whose DEM was available for control.