We introduce a novel urban visualization system that integrates 3D urban model (CityGML) and 360$^{\circ }$∘ walkthrough videos. By aligning the videos with the model and dynamically projecting relevant video frames onto the geometries, our system creates photorealistic urban visualizations, allowing users to intuitively interpret geospatial data from a pedestrian view.

In response to rising societal awareness of privacy concerns, face anonymization techniques have advanced, including the emergence of face-swapping methods that replace one identity with another. Achieving a balance between anonymity and naturalness in face swapping requires careful selection of identities: overly similar faces compromise anonymity, while dissimilar ones reduce naturalness. Existing models, however, focus on binary identity classification "the same person or not", making it difficult to measure nuanced similarities such as "completely different" versus "highly similar but different." This paper proposes a human-perception-based face similarity metric, creating a dataset of 6,400 triplet annotations and metric learning to predict the similarity. Experimental results demonstrate significant improvements in both face similarity prediction and attribute-based face classification tasks over existing methods. Our dataset is available at https://github.com/kumanotanin/PerFace.

In this work, we investigate facial anonymization techniques in 360° videos and assess their influence on the perceived realism, anonymization effect, and presence of participants. In comparison to traditional footage, 360° videos can convey engaging, immersive experiences that accurately represent the atmosphere of real-world locations. As the entire environment is captured simultaneously, it is necessary to anonymize the faces of bystanders in recordings of public spaces. Since this alters the video content, the perceived realism and immersion could be reduced. To understand these effects, we compare non-anonymized and anonymized 360° videos using blurring, black boxes, and face-swapping shown either on a regular screen or in a head-mounted display (HMD). Our results indicate significant differences in the perception of the anonymization techniques. We find that face-swapping is the most realistic and least disruptive; however, participants raised concerns regarding the effectiveness of the anonymization. Furthermore, we observe that presence is affected by facial anonymization in HMD condition. Overall, the results underscore the need for facial anonymization techniques that balance both photo-realism and a sense of privacy.

We propose a system to generate 360° realistic virtual worlds (360RVW) for the interactive spatial exploration of omnidirectional street-view videos. Our 360RVW enables users to explore photorealistic scenes with digital avatars, and interact with others. To create the virtual worlds our system only requires 360° videos with annotations of the start and end camera coordinate as input. We first detect street intersections to divide the input videos and remove the camera operator from the recordings using a video completion technique. Next, we analyze the 3D structure of the scene using semantic segmentation to define walkable areas. Finally, we render the environment using an ellipsoid projection surface to achieve a more realistic integration of the avatar into real-world 360° videos. The whole process is largely automated, enabling users to produce realistic and interactive virtual worlds without specialized skills or time-consuming manual interventions.

Abstract In this paper, we tackle the challenging problem of rendering real‐world 360° panorama videos that support full 6 degrees‐of‐freedom (DoF) head motion from a prerecorded omnidirectional stereo (ODS) video. In contrast to recent approaches that create novel views for individual panorama frames, we introduce a video‐specific temporally‐consistent multi‐sphere image (MSI) scene representation. Given a conventional ODS video, we first extract information by estimating framewise descriptive feature maps. Then, we optimize the global MSI model using theory from recent research on neural radiance fields. Instead of a continuous scene function, this multi‐sphere image (MSI) representation depicts colour and density information only for a discrete set of concentric spheres. To further improve the temporal consistency of our results, we apply an ancillary refinement step which optimizes the temporal coherency between successive video frames. Direct comparisons to recent baseline approaches show that our global MSI optimization yields superior performance in terms of visual quality. Our code and data will be made publicly available.

In this paper, we investigate the perceived personalities of face swaps and how they relate to the personalities of the real people used to create the synthetic individuals' appearance and movements. Given that face swaps have become nearly indistinguishable from real humans, they offer a promising direction for the fast creation of realistic avatars. To investigate the usability of face swaps as avatars, we perform an experiment assessing their personality on the Five-Factor Model, their eeriness and appeal, as well as effects due to familiarity with the original individuals. Our results indicate that face swaps are perceived similarly to real humans and are affected by familiarity. Furthermore, we find a stronger influence of the body and movements on the perceived synthetic personality, especially for their extroversion and conscientiousness.

We present a novel framework for the evaluation of eye tracking data in portrait videos including the automatic generation of customized areas of interest (AOIs) based on facial landmarks. In contrast to previous work, our framework allows the user to flexibly create AOIs by grouping the detected landmarks. Moreover, their shape and size can be modified to better fit both the research question and the precision of the eye tracker. The framework can be used as an integrated solution to not only generate AOIs but also to evaluate viewing behavior like the overall fixation times, the similarity of scanpaths, and the number of saccades between AOIs. Other functionalities include the visualization of gaze paths and the creation of heatmaps. We demonstrate the benefits of our framework and user-defined AOI layouts via an exemplary application, i.e., the investigation of face swapping artifacts.

In this paper, we report on perceptual experiments indicating that there are distinct and quantitatively measurable differences in the way we visually perceive genuine versus face-swapped videos.

These days convolutional neural networks (CNNs) are one of the most popular techniques for optical flow estimation yielding exceptional results on many benchmarks. However, the size of their perceptive fields are fixed and displacements not present in the training set cannot be estimated reliably. Additionally, these CNNs need to learn numerous parameters which increases memory consumption. Consequently, their application to high resolution images is impractical, as displacement and memory consumption increase prohibitively. To overcome these limitations, we present an iterative flow refinement approach that can adapt any flow estimation CNN to arbitrary resolution images. In a pyramidal approach, we perform flow estimation for image patches at multiple increasing image resolutions while matching the patches based on the flow of previous iterations. We evaluate our approach using different baselines, displacements and resolutions. The results show that flow estimators can be adapted to high resolution and even panorama images while preserving fine details and reliably handling large displacements without retraining.