Abstract Reconstructing outdoor environments from “inside-out” captures, where a camera moves within a restricted area but looks outward, remains challenging due to the presence of both well-textured nearby regions and low-detail distant backgrounds. We introduce a two-stage Gaussian Splatting framework that explicitly separates and optimizes these regions, yielding higher-fidelity novel view synthesis and allowing the replacement of the distant part with a high-quality, inpainted environment map to speedup the rendering process. In stage one, background primitives are initialized within a spherical shell and optimized using a loss that combines a background-only photometric term with two geometric regularizers: one constraining Gaussians to remain inside the shell, and another one aligning them with local tangential planes. In stage two, foreground Gaussians are initialized from a Structure-from-Motion reconstruction, added and refined using the standard rendering loss, while the background set remains fixed but contributes to the final image formation. Background Gaussians can be rendered to an object-free environment map that is inpainted to fill missing parts and can replace the Gaussian-based background for faster rendering. Experiments on diverse outdoor datasets show that our method reduces background artifacts and improves perceptual quality of novel view renderings compared to state-of-the-art baselines, including the removal of floaters in the navigation region.

Reflectance Transformation Imaging (RTI) is very popular for its ability to visually analyze surfaces by enhancing surface details through interactive relighting, starting from only a few tens of photographs taken with a fixed camera and variable illumination. Traditional methods like Polynomial Texture Maps (PTM) and Hemispherical Harmonics (HSH) are compact and fast, but struggle to accurately capture complex reflectance fields using few per-pixel coefficients and fixed bases, leading to artifacts, especially in highly reflective or shadowed areas. The NeuralRTI approach, which exploits a neural autoencoder to learn a compact function that better approximates the local reflectance as a function of light directions, has been shown to produce superior quality at comparable storage cost. However, as it performs interactive relighting with custom decoder networks with many parameters, the rendering step is computationally expensive and not feasible at full resolution for large images on limited hardware. Earlier attempts to reduce costs by directly training smaller networks have failed to produce valid results. For this reason, we propose to reduce its computational cost through a novel solution based on Knowledge Distillation (DISK-NeuralRTI). Starting from a teacher network that can be one of the original Neural RTI methods or a more complex solution, DISK-NeuralRTI can create a student architecture with a simplified decoder network that preserves image quality and has computational cost compatible with real-time web-based visualization of large surfaces. Experimental results show that we can obtain a student prediction that is on par or more accurate than the existing NeuralRTI solutions with up to 80% parameter reduction. Using a novel benchmark of high-resolution Multi-Light image collections (RealRTIHR), we also tested the usability of a web-based visualization tool based on our simplified decoder for realistic surface inspection tasks. The results show that the solution reaches interactive frame rates without the necessity of using progressive rendering with image quality loss. • Previous neural methods improve classic RTI in quality but are much slower to render. • Knowledge Distillation achieves 80% neural parameter reductions at same quality. • First neural RTI method for full-res real-time web exploration on > 4K displays. • New RealRTIHR dataset introduced for high-res training and benchmarking. • Code and data made public for reproducibility and further research.

In this paper, we review the existing benchmarks for continuous gesture recognition, e.g., the online analysis of hand movements over time to detect and recognize meaningful gestures from a specific dictionary. Focusing on human–computer interaction scenarios, we classify these benchmarks based on input data types, gesture dictionaries, and evaluation metrics. Specific metrics for the continuous recognition task are crucial for understanding how effectively gestures are spotted in real time within input streams. We also discuss the most effective detection and classification methods proposed for these benchmarks. Our findings indicate that the number and quality of publicly available datasets remain limited, and evaluation methodologies for continuous recognition are not yet standardized. These issues highlight the need for new benchmarks that reflect real-world usage conditions and can support the development of best practices in gesture-based interface design.

In recent years, Augmented Reality (AR) technology has permeated various domains. This paper focuses on the critical aspect of enhancing customer interaction within AR e-commerce environments by investigating the impact of virtual product size manipulation on usability, user experience, and shopping satisfaction. We tested two manipulation modalities: an unconstrained scaling modality, enabling users to manually adjust product dimensions, and an assisted modality providing automatic 1:1 scaling. Using the Microsoft HoloLens 2 AR headset, we engaged 40 participants with small and large virtual products in shopping scenarios using these two manipulation modalities. Results show that users found the automatic manipulation modality to provide a superior user experience, being more effective, easy, useful, and pleasant when interacting with large virtual products. For small virtual products, they expressed a preference for free manipulation. Customer satisfaction with the shopping experience is positive, however, product size and manipulation modality affect the repatronage intention. The findings offer insights into designing AR e-commerce interfaces, highlighting that providing different manipulation modalities depending on the size of the products allows for enriching the shopping experience and improving the AR market potential.

In this demo we present an application featuring the integration of Virtual Reality (VR) technologies with the demonstration laboratory (ICELab) built around Industry 4.0/5.0 concepts. In particular, we showcase a digital twin of the real laboratory that allows the user to explore its environment in VR and interact with the different machinery to obtain several data and information.

Gesture recognition is a tool to enable novel interactions with different techniques and applications, like Mixed Reality and Virtual Reality environments. With all the recent advancements in gesture recognition from skeletal data, it is still unclear how well state-of-the-art techniques perform in a scenario using precise motions with two hands. This paper presents the results of the SHREC 2024 contest organized to evaluate methods for their recognition of highly similar hand motions using the skeletal spatial coordinate data of both hands. The task is the recognition of 7 motion classes given their spatial coordinates in a frame-by-frame motion. The skeletal data has been captured using a Vicon system and pre-processed into a coordinate system using Blender and Vicon Shogun Post. We created a small, novel dataset with a high variety of durations in frames. This paper shows the results of the contest, showing the techniques created by the 5 research groups on this challenging task and comparing them to our baseline method.

In this paper, we present an example of complete integration of eXtended Reality technologies within a demonstration laboratory showcasing Industry 4.0/5.0 compliant machinery in realistic scenarios of use. We describe the design choices and the implementation of the augmented and virtual reality applications developed and potentially usable to support different real-world tasks, featuring advanced gesture-based interaction modes. We also describe the optimized communication architecture used to synchronize data between the cyber-physical factory environment with all its components, its industrial digital twin, and the augmented and virtual replica of the factory. Example tasks supported with the tools in public demonstrations allow users wearing Microsoft HoloLens 2 or Meta Quest 2 headsets to monitor the status of the prototype production line and operate on it, locally or remotely. An example video showing the applications is available in the supplementary material.

Many real-world hand-crafted objects are decorated with elements that are packed onto the object’s surface and deformed to cover it as much as possible. Examples are artisanal ceramics and metal jewelry. Inspired by these objects, we present a method to enrich surfaces with packed volumetric decorations. Our algorithm works by first determining the locations in which to add the decorative elements and then removing the non-physical overlap between them while preserving the decoration volume. For the placement, we support several strategies depending on the desired overall motif. To remove the overlap, we use an approach based on implicit deformable models creating the qualitative effect of plastic warping while avoiding expensive and hard-to-control physical simulations. Our decorative elements can be used to enhance virtual surfaces, as well as 3D-printed pieces, by assembling the decorations onto real surfaces to obtain tangible reproductions.

Currently, both understanding and developing interactive visual interfaces become ever more challenging, since different visual solutions exist that involve large interdisciplinary teams and deal with massive amounts of data, a wide range of interaction techniques, and domain-specific aspects. At the same time, traditional design methods become obsolete with more and more resources and knowledge that needs to be acquired. The proposed workshop provides a forum for discussing experimental and theoretical techniques, frameworks, and prototyping methods to design visual interfaces in different domains such as data visualization, tangible and embedded interaction, extended and mixed reality, multi-modal interfaces, and others.

One of the most challenging aspects of the implementation of Virtual/Mixed reality training systems is the effective simulation of real-world manipulation of the physical devices included in control interfaces like buttons, sliders, levers, knobs, etc. In this paper we describe a mockup airplane cockpit (XR-Cockpit), featuring interactive components of this kind that demonstrate the feasibility of effective simulations of device manipulation using low cost hand tracking technology and gesture recognition. Based on this system, we performed a user study to compare the effectiveness of the interaction with virtual tools using different visualization solutions: immersive VR, optical and video see-through based MR. In our study, we also checked how well it is possible to perform manipulation of real objects wearing the two video see-through solutions. The analysis of the experimental results provides useful guidelines for the design of Virtual and Mixed Reality training systems involving virtual and physical actions on manipulation devices.

We present SIMCO, a completely agnostic multiclass object counting approach. SIMCO starts by detecting foreground objects through a novel Mask RCNN-based architecture trained beforehand (just once) on a brand-new synthetic 2D shape dataset, InShape; the idea is to highlight every object resembling a primitive 2D shape (circle, square, rectangle, etc.), Each object detected is described by a low-dimensional embedding, obtained from a novel similarity-based head branch; this latter implements a triplet loss, encouraging similar objects (same 2D shape + color and scale) to map close. Subsequently, SIMCO uses this embedding for clustering, so that different “classes” of similar objects can emerge and be counted, making SIMCO the very first multi-class unsupervised counter. The only required assumption is that repeated objects are present in the image. Experiments show that SIMCO provides state-of-the-art scores on counting benchmarks and that it can also help in many challenging image understanding tasks.

Abstract Reflectance transformation imaging (RTI) is a computational photography technique widely used in the cultural heritage and material science domains to characterize relieved surfaces. It basically consists of capturing multiple images from a fixed viewpoint with varying lights. Handling the potentially huge amount of information stored in an RTI acquisition that consists typically of 50–100 RGB values per pixel, allowing data exchange, interactive visualization, and material analysis, is not easy. The solution used in practical applications consists of creating “relightable images” by approximating the pixel information with a function of the light direction, encoded with a small number of parameters. This encoding allows the estimation of images relighted from novel, arbitrary lights, with a quality that, however, is not always satisfactory. In this paper, we present NeuralRTI, a framework for pixel-based encoding and relighting of RTI data. Using a simple autoencoder architecture, we show that it is possible to obtain a highly compressed representation that better preserves the original information and provides increased quality of virtual images relighted from novel directions, especially in the case of challenging glossy materials. We also address the problem of validating the relight quality on different surfaces, proposing a specific benchmark, SynthRTI, including image collections synthetically created with physical-based rendering and featuring objects with different materials and geometric complexity. On this dataset and as well on a collection of real acquisitions performed on heterogeneous surfaces, we demonstrate the advantages of the proposed relightable image encoding.

Abstract Interactions within virtual environments often require manipulating 3D virtual objects. To this end, researchers have endeavoured to find efficient solutions using either traditional input devices or focusing on different input modalities, such as touch and mid‐air gestures. Different virtual environments and diverse input modalities present specific issues to control object position, orientation and scaling: traditional mouse input, for example, presents non‐trivial challenges because of the need to map between 2D input and 3D actions. While interactive surfaces enable more natural approaches, they still require smart mappings. Mid‐air gestures can be exploited to offer natural manipulations mimicking interactions with physical objects. However, these approaches often lack precision and control. All these issues and many others have been addressed in a large body of work. In this article, we survey the state‐of‐the‐art in 3D object manipulation, ranging from traditional desktop approaches to touch and mid‐air interfaces, to interact in diverse virtual environments. We propose a new taxonomy to better classify manipulation properties. Using our taxonomy, we discuss the techniques presented in the surveyed literature, highlighting trends, guidelines and open challenges, that can be useful both to future research and to developers of 3D user interfaces.

Abstract Multi‐Light Image Collections (MLICs), i.e., stacks of photos of a scene acquired with a fixed viewpoint and a varying surface illumination, provide large amounts of visual and geometric information. In this survey, we provide an up‐to‐date integrative view of MLICs as a mean to gain insight on objects through the analysis and visualization of the acquired data. After a general overview of MLICs capturing and storage, we focus on the main approaches to produce representations usable for visualization and analysis. In this context, we first discuss methods for direct exploration of the raw data. We then summarize approaches that strive to emphasize shape and material details by fusing all acquisitions in a single enhanced image. Subsequently, we focus on approaches that produce relightable images through intermediate representations. This can be done both by fitting various analytic forms of the light transform function, or by locally estimating the parameters of physically plausible models of shape and reflectance and using them for visualization and analysis. We finally review techniques that improve object understanding by using illustrative approaches to enhance relightable models, or by extracting features and derived maps. We also review how these methods are applied in several, main application domains, and what are the available tools to perform MLIC visualization and analysis. We finally point out relevant research issues, analyze research trends, and offer guidelines for practical applications.

We present a novel framework to directly estimate body fat percentage from depth images of human subjects and to visually evaluate salient points of the body shape related to the fat distribution. For this purpose, we created a novel, publicly available dataset including front and back depth images of a set of subjects with specific features (active young men or professional sportsmen) with associated ground truth fat values estimated with dual-energy x-ray absorptiometry (DXA) scanning. These depth images were obtained with depth rendering of an available dataset of whole body scans, simulating low-cost depth sensor acquisitions. We customized a ResNet-50 regressor to estimate fat percentage values directly from the front/back scans, achieving promising accuracy (standard errors of estimate SEE less than 2.1 on the depth renderings and 2.5 on a small set of real depth scans). We also demonstrate that, using a custom perturbation-based procedure for analyzing deep networks, it is possible to highlight, on subjects' depth images, the specific body areas related to fat accumulation (typically neck, shoulders, hip, and abdomen) and those characterizing skinny subjects (chest and abdomen).

The choice of a suitable method for object manipulation is one of the most critical aspects of virtual environment design. It has been shown that different environments or applications might benefit from direct manipulation approaches, while others might be more usable with indirect ones, exploiting, for example, three dimensional virtual widgets. When it comes to mid-air interactions, the success of a manipulation technique is not only defined by the kind of application but also by the hardware setup, especially when specific restrictions exist. In this paper we present an experimental evaluation of different techniques and hardware for mid-air object manipulation in immersive virtual environments (IVE). We compared task performances using both deviceless and device-based tracking solutions, combined with direct and widget-based approaches. We also tested, in the case of freehand manipulation, the effects of different visual feedback, comparing the use of a realistic virtual hand rendering with a simple cursor-like visualization.

Abstract In this paper, we address the problem of characterizing relief patterns over surface meshes independently on the underlying shape. We propose to tackle the problem by estimating local invariant features and encoding them using the Improved Fisher Vector technique, testing both features estimated on 3D meshes and local descriptors estimated on raster images created by encoding local surface properties (e.g. mean curvature) over a surface parametrization. We compare the robustness of the obtained descriptors against noise and surface bending and evaluate retrieval performances on a specific benchmark proposed in a track of the Eurographics Shape REtrieval Contest 2017. Results show that, with the proposed framework, it is possible to obtain retrieval results largely improving the state of the art and that the image‐based approach is still effective when the underlying surface is heavily deformed.

3D models of humans are commonly used within computer graphics and vision, and so the ability to distinguish between body shapes is an important shape retrieval problem. We extend our recent paper which provided a benchmark for testing non-rigid 3D shape retrieval algorithms on 3D human models. This benchmark provided a far stricter challenge than previous shape benchmarks. We have added 145 new models for use as a separate training set, in order to standardise the training data used and provide a fairer comparison. We have also included experiments with the FAUST dataset of human scans. All participants of the previous benchmark study have taken part in the new tests reported here, many providing updated results using the new data. In addition, further participants have also taken part, and we provide extra analysis of the retrieval results. A total of 25 different shape retrieval methods are compared.

In this paper we introduce a novel framework for 3D object retrieval that relies on tree-based shape representations (TreeSha) derived from the analysis of the scale-space of the Auto Diffusion Function (ADF) and on specialized graph kernels designed for their comparison. By coupling maxima of the Auto Diffusion Function with the related basins of attraction, we can link the information at different scales encoding spatial relationships in a graph description that is isometry invariant and can easily incorporate texture and additional geometrical information as node and edge features. Using custom graph kernels it is then possible to estimate shape dissimilarities adapted to different specific tasks and on different categories of models, making the procedure a powerful and flexible tool for shape recognition and retrieval. Experimental results demonstrate that the method can provide retrieval scores similar or better than state-of-the-art on textured and non textured shape retrieval benchmarks and give interesting insights on effectiveness of different shape descriptors and graph kernels.

In this paper, we present an automatic tool for estimating geometrical parameters from 3-D human scans independent on pose and robustly against the topological noise. It is based on an automatic segmentation of body parts exploiting curve skeleton processing and ad hoc heuristics able to remove problems due to different acquisition poses and body types. The software is able to locate body trunk and limbs, detect their directions, and compute parameters like volumes, areas, girths, and lengths. Experimental results demonstrate that measurements provided by our system on 3-D body scans of normal and overweight subjects acquired in different poses are highly correlated with the body fat estimates obtained on the same subjects with dual-energy X-rays absorptiometry (DXA) scanning. In particular, maximal lengths and girths, not requiring precise localization of anatomical landmarks, demonstrate a good correlation (up to 96%) with the body fat and trunk fat. Regression models based on our automatic measurements can be used to predict body fat values reasonably well.

In this paper we present an analysis of image features used to discriminate arteries and veins in digital fundus images. Methods proposed in the literature to analyze the vasculature of the retina and compute diagnostic indicators like the Arteriolar to Venular ratio (AVR), use, in fact, different approaches for this classification task, extracting different color features and exploiting different additional information. We concentrate our analysis on finding optimal features for the vessel classification, considering not only simple color features, but also spatial location and vessel size and testing different supervised labeling approaches. The results obtained show that best results are obtained mixing features related with color values and contrast inside and outside the vessels and positional information. Furthermore, the discriminative power of the features changes with the image resolution and best results are not obtained at the finest one. Our experiments demonstrate that using a good set of descriptors it is possible to achieve very good classification performances even without using vascular connectivity information.

Abstract We present a novel method to characterize 3D surfaces through the computation of a function called (multiscale) area projection transform, measuring the likelihood of points in the 3D space to be center of radial symmetry at selected scales (radii). The function is derived through a simple geometric framework based on parallel surfaces and can be easily computed on triangulated meshes. It measures locally the area of the surface well approximated by a sphere of radius R centered in the point and can be normalized in order to obtain a scale invariant radial symmetry enhancement transform. This transform can therefore be used to detect and characterize salient regions like approximately spherical and approximately cylindrical surface parts and, being robust against holes and missing parts, it is suitable for real world applications e.g. anatomical features detection. Furthermore, its histograms can be effectively used to build a global shape descriptor that provides very good results in shape retrieval experiments.

With regard to the above titled paper (ibid., vol. 20, no. 10, pp. 2760-2768, Oct. 2011), the authors have corrected some formulas that are affected by small errors (shifted indices, etc.). These errors had not been detected earlier, because the descriptions made the right ones rather evident; however, it is necessary to correct them. The corrections are presented here.

In this paper we present a novel method for the automatic location and segmentation of the optical disk in fundus images. It is based on the decoupling of vessel and background information obtained with morphological segmentation and inpainting. A multiresolution optimization scheme finding elliptic contours optimally adapted to a brightness model is then applied on the inpainted brightness image, under the constraint that a reasonable amount of vasculature must be present inside the disc. An effective objective function and a multiresolution scheme allow a deterministic optimizer to converge on the OD contour with good accuracy, while the vessel-based constraint limits wrong contour detections in anomalous cases. Our initial experiments (30 DRIVE images, ground truth from two doctors) suggest that the method can provide accurate results both in term of optic disc location and contour segmentation accuracy.

The problem of creating artifact-free upscaled images appearing sharp and natural to the human observer is probably more interesting and less trivial than it may appear. The solution to the problem, often referred to also as "single-image super-resolution," is related both to the statistical relationship between low-resolution and high-resolution image sampling and to the human perception of image quality. In many practical applications, simple linear or cubic interpolation algorithms are applied for this task, but the results obtained are not really satisfactory, being affected by relevant artifacts like blurring and jaggies. Several methods have been proposed to obtain better results, involving simple heuristics, edge modeling, or statistical learning. The most powerful ones, however, present a high computational complexity and are not suitable for real-time applications, while fast methods, even if edge adaptive, are not able to provide artifacts-free images. In this paper, we describe a new upscaling method (iterative curvature-based interpolation) based on a two-step grid filling and an iterative correction of the interpolated pixels obtained by minimizing an objective function depending on the second-order directional derivatives of the image intensity. We show that the constraints used to derive the function are related with those applied in another well-known interpolation method, providing good results but computationally heavy (i.e., new edge-directed interpolation (NEDI). The high quality of the images enlarged with the new method is demonstrated with objective and subjective tests, while the computation time is reduced of one to two orders of magnitude with respect to NEDI so that we were able, using a graphics processing unit implementation based on the nVidia Compute Unified Device Architecture technology, to obtain real-time performances.

In this paper we present a new image up scaling (single image super resolution) algorithm. It is based on the refinement of a simple pixel decimation followed by an optimization step maximizing the smoothness of the second order derivatives of the image intensity while keeping the sum of the brightness values of each subdivided pixel (i.e. the estimated irradiance on the area) constant. The method is physically grounded and creates images that appear very sharp and with reduced artifacts. Subjective and objective tests demonstrate the high quality of the results obtained.

In this paper we describe a novel general purpose image interpolation method based on the combination of two different procedures. First, an adaptive algorithm is applied interpolating locally pixel values along the direction where second order image derivative is lower. Then interpolated values are modified using an iterative refinement minimizing differences in second order image derivatives, maximizing second order derivative values and smoothing isolevel curves. The first algorithm itself provides edge preserving images that are measurably better than those obtained with similarly fast methods presented in the literature. The full method provides interpolated images with a ”natural ” appearance that do not present the artifacts affecting linear and nonlinear methods. Objective and subjective tests on a wide series of natural images clearly show the advantages of the proposed technique over existing approaches. 1

Bone dissection is an important component of many surgical procedures. In this paper we discuss adaptive techniques for providing real-time haptic and visual feedback during a virtual bone dissection simulation. The simulator is being developed as a component of a training system for temporal bone surgery. We harness the difference in complexity and frequency requirements of the visual and haptic simulations by modeling the system as a collection of loosely coupled concurrent components. The haptic component exploits a multi-resolution representation of the first two moments of the bone characteristic function to rapidly compute contact forces and determine bone erosion. The visual component uses a time-critical particle system evolution method to simulate secondary visual effects, such as bone debris accumulation, blooding, irrigation, and suction.

Bone dissection is an important component of many surgical procedures. In this paper, we discuss a haptic and visual implementation of a bone-cutting burr that is being developed as a component of a training system for temporal bone surgery. We use a physically motivated model to describe the burr-bone interaction, which includes haptic force evaluation, the bone erosion process and the resulting debris. The current implementation, directly operating on a voxel discretization of patient-specific 3D CT and MRI data, is efficient enough to provide real-time feedback on a low-end multiprocessing PC platform.

The aim of the virtual vascular project (ViVa) is to develop tools for the modern hemodynamicist and cardiovascular surgeon to study and interpret the constantly increasing amount of information being produced by noninvasive imaging equipment. In particular, we are developing a system able to process and visualize three-dimensional (3-D) medical data, reconstruct the geometry of arteries of specific patients, and simulate blood flow in them. The initial applications of the system will be for clinical research and training purposes. In a later stage, we will explore the application of the system to surgical planning. ViVa is based on an integrated set of tools, each dedicated to a specific aspect of the data processing and simulation pipeline: image processing and segmentation; real-time 3-D volume visualization; 3-D geometry reconstruction; 3-D mesh generation; and blood flow simulation and visualization.

This paper describes procedures for obtaining a reliable and dense optical flow from image sequences taken by a TV camera mounted on a car moving in usual outdoor scenarios. By using correlation based techniques and by correcting the optical flows for shocks and vibrations, useful sequences of optical flows can be obtained. When the car is moving along a flat road and the optical axis of the TV camera is parallel to the ground, the motion field is expected to be almost quadratic and have a specific structure. As a consequence the egomotion can be estimated from this optical flow and information on the speed and the angular velocity of the moving vehicle are obtained. By analyzing the optical flow it is possible to recover also a coarse segmentation of the flow, in which objects moving with a different speed are identified. By combining information from intensity edges a better localization of motion boundaries are obtained. These results suggest that the optical flow can be successfully used by a vision system for assisting a driver in a vehicle moving in usual streets and motorways.

When a plane undergoes a deformation that can be represented by a planar linear vector field, the projected vector field on the image plane of an optical device is at most quadratic. This 2D motion field has one singular point, with eigenvalues identical to those of the singular point describing the deformation. As a consequence, the nature of the singular point of the deformation is a projective invariant. When the plane moves and experiences a linear deformation at the same time, the associated 2D motion field is still quadratic with at most 3 singular points. In the case of a normal rototranslation, i.e. when the angular velocity is normal to the plane, and of a linear deformation, the 2D motion field has at most one singular point and substantial information on the rigid motion and on the deformation can be recovered from it. Experiments with simulated deformations and real deformable objects show that the proposed analysis can provide accurate results and information on more general 3D deformations.>