In recent years, Virtual Reality systems moved towards network-based applications. This implies that the network transmission performance directly impacts on users’ experience, thus potentially impairing users’ immersion, engagement, and effectiveness. In this work, we present a work-in-progress study involving an experimental campaign aimed at testing the impact of network latency on users’ Quality of Experience during Virtual Reality gaming. We present the designed experimental protocol and the preliminary results.

Assessing the quality of 3D reconstructed models remains a key challenge in multimedia applications, especially in the context of cultural heritage, where visual fidelity and perceptual realism are equally crucial. This study investigates how reconstruction parameters, as well as existing objective quality metrics, align with human perception. In addition, we analyze how perceived quality and user interaction are related. A dataset of 3D models was generated by varying the number of input images, mesh complexity, and texture resolution. Results from a subjective study show that texture resolution significantly affects perceived quality, whereas variations in number of images and mesh complexity have a limited impact. Furthermore, interaction behavior was found to vary with perceived quality, with participants spending more time and exploring larger viewing angles for models receiving higher scores. These findings highlight the need for perceptually grounded, interaction-aware evaluation methodologies and provide guidelines for future perceptual optimization of 3D reconstruction pipelines.

In this work, an emotion recognition system for enhancing social XR applications is presented. Although several techniques for emotion recognition have been proposed in the literature, they either require invasive and advanced equipment or exploit facial expressions, speech excerpts, physiological data, and text. In this contribution, on the contrary, an approach for markerless emotion classification through body language is designed. More specifically, human movements are analyzed over time by extracting the skeleton joints in videos acquired by consumer cameras. A normalization procedure has been introduced to provide a depth-independent skeleton representation without distorting the skeleton shape. The performance of the proposed method have been assessed using a dataset of videos recorded from multiple points of view. An ad-hoc learning-based emotion classifier has been trained to recognize four emotions (happiness, boredom, interest, and disgust) achieving an average accuracy of 72.5%. The pre-processed dataset, code, and demo with pre-trained models are available at https://github.com/michaelneri/emotion-recognition-human-movements . • We define a human skeleton representation that does not change with the distance between the user and the camera. • We introduce a new approach for classifying emotions based exclusively on body movements. • We extend an existing dataset employing multiple points of view for classifying four emotional states: happiness, interest, boredom, and disgust.

The recent success of immersive applications is pushing the research community to define new approaches to process 360° images and videos and optimize their transmission. Among these, saliency estimation provides a powerful tool that can be used to identify visually relevant areas and, consequently, adapt processing algorithms. Although saliency estimation has been widely investigated for 2D content, very few algorithms have been proposed for 360° saliency estimation. Towards this goal, we introduce Sphere-GAN, a saliency detection model for 360° videos that leverages a Generative Adversarial Network with spherical convolutions. Extensive experiments were conducted using a public 360° video saliency dataset, and the results demonstrate that Sphere-GAN outperforms state-of-the-art models in accurately predicting saliency maps.

Extended reality technologies are increasingly used in cultural heritage for preserving and accessing sites and artworks, where 3D model acquisition and rendering are key. Despite progress in reconstruction methodologies, a standardized approach to quality assessment is still missing. This study aims to evaluate objective quality metrics —both image-based and model-based, Full Reference and No Reference— applied to 3D models generated using the Structure from Motion algorithm. By varying parameters such as the number of images, number of triangles, and texture resolution, we examine the impact of these factors on metric outcomes, aiming to assess their reliability in cultural heritage applications.

With the unprecedented diffusion of virtual reality, the number of application scenarios is continuously growing. As commercial and gaming applications become pervasive, the need for the secure and convenient identification of users, often overlooked by the research in immersive media, is becoming more and more pressing. Networked scenarios such as Cloud gaming or cooperative virtual training and teleoperation require both a user-friendly and streamlined experience and user privacy and security. In this work, we investigate the possibility of identifying users from their movement patterns and data traffic traces while playing four commercial games, using a publicly available dataset. If, on the one hand, this paves the way for easy identification and automatic customization of the virtual reality content, it also represents a serious threat to users’ privacy due to network analysis-based fingerprinting. Based on this, we analyze the threats and opportunities for virtual reality users’ security and privacy.

The constant increase of the number of connected devices, as well as of their heterogeneity, has greatly expanded the security threat landscape. For this reason, the prompt and effective detection of network traffic anomalies has become critical. In this work, we propose a new network traffic representation that aims at providing a compact and constantly updated summary of the current network condition. In addition, we propose an anomaly detection method based on the Principal Component Analysis of the aforementioned network representation. The proposed method exploits one-second time windows of network traffic, thus allowing an immediate reaction to anomalies. It is completely unsupervised, thus enabling the detection of zero-day attacks, and it has a low computational complexity, thus reducing the required capabilities of the monitoring nodes. The performance analysis showed that the proposed approach achieves comparable results with respect to state-of-the-art methods. • A new traffic representation providing updated and compact summaries of the network. • An unsupervised, low-complexity, and prompt anomaly detector based on PCA. • An in-depth comparison between the proposed approach and state-of-the-art methods.

The rapid development of Virtual Reality (VR) technology has led the industry and research community to look at its major challenges with increased interest. The main challenge in ensuring a high Quality of Experience (QoE) for users is represented by cybersickness, a phenomenon similar to motion sickness experienced by many VR users, while at the same time, the high data rates needed by VR require the definition of traffic models for network optimization. These two problems are intertwined, but have never been studied jointly before due to the lack of suitable datasets. In this paper, we present Questset, the first dataset designed for this purpose. Questset contains over 40 hours of VR traces from 70 users playing commercially available video games, and includes both traffic data for network optimization, and movement and user experience data for cybersickness analysis. Therefore, Questset represents an enabler to jointly address the main VR challenges in the near future.

This work aims to investigate the existence of a leading sensory cue in a mulsemedia Virtual Reality application involving three senses: vision, hearing, and touch. On the one hand this study can help in gaining insights into how the different senses contribute to mulsemedia applications in Virtual Reality. On the other, the identification of the leading cue could drive the optimization of Virtual Reality applications in terms of Quality of Experience, stimuli definition, and transmission requirements. In this paper, we present a mulsemedia experimental protocol for a subjective test aimed at identifying the material of a virtual object. We present the encountered challenges and describe and discuss the obtained results.

This demo presents an interactive communication system based on immersive media for analyzing the users’ Quality of Experience in collaborative tasks. The interaction between users is studied in an asymmetric scenario where a peer-to-peer communication has been set up between a PC and a Virtual Reality headset. Two application scenarios have been considered: a Block Building task and a Treasure Hunt game. The two users will cooperate to perform the two tasks. The goal is to study the relation between the type of transmitted information (i.e., audio and video or audio only) and the quality and quantity of interaction. During the demo, participants will have the opportunity to try one of the designed applications.

Measuring the complexity of visual content is crucial in various applications, such as selecting sources to test processing algorithms, designing subjective studies, and efficiently determining the appropriate encoding parameters and bandwidth allocation for streaming. While spatial and temporal complexity measures exist for 2D videos, a geometric complexity measure for 3D content is still lacking. In this paper, we present the first study to characterize the geometric complexity of 3D point clouds. Inspired by existing complexity measures, we propose several compression-based definitions of geometric complexity derived from the rate-distortion curves obtained by compressing a dataset of point clouds using G-PCC. Additionally, we introduce density-based and geometry-based descriptors to predict complexity. Our initial results show that even simple density measures can accurately predict the geometric complexity of point clouds.

Abstract In virtual reality applications, head-mounted displays allow users to explore virtual surroundings, thus creating a high sense of immersion. However, due to the novelty of the technology and the possibility of freely enjoying a $$360^\circ $$ 360 ∘ virtual world, users can get distracted and divert their attention from the content of the application. In this work, we define a set of guidelines for the design of virtual reality applications for enhancing the users’ attention. To the best of our knowledge, this is one of the first attempts to provide general guidelines for virtual application design based on visual attention. More specifically, we analyze the different categories of factors that contribute to the user’s responsiveness and define a set of experiments for measuring the user’s promptness with respect to visual stimuli with different features and in the presence of audio/visual distractions. Experimental tests have been carried out with 36 volunteers. The users’ reaction time has been recorded and the performed analysis allowed the definition of a set of guidelines based on individual, operational, and technological factors for the design of virtual reality applications optimized in terms of user attention. In particular, statistical tests demonstrated that the presence of distractions leads to significantly different reaction times with respect to the case of no distractions, and that users belonging to different age intervals have significantly different behaviors. Moreover, the optimal placement of objects has been identified and the impact of cybersickness has been analyzed.

Augmented reality is one of the enabling technologies of the upcoming future. Its usage in working and learning scenarios may lead to a better quality of work and training by helping the operators during the most crucial stages of processes. Therefore, the automatic detection of stress during augmented reality experiences can be a valuable support to prevent consequences on people's health and foster the spreading of this technology. In this work, we present the design of a non-invasive stress assessment approach. The proposed system is based on the analysis of the head movements of people wearing a Head Mounted Display while performing stress-inducing tasks. First, we designed a subjective experiment consisting of two stress-related tests for data acquisition. Then, a statistical analysis of head movements has been performed to determine which features are representative of the presence of stress. Finally, a stress classifier based on a combination of Support Vector Machines has been designed and trained. The proposed approach achieved promising performances thus paving the way for further studies in this research direction.

Traditional image and video enhancement techniques do not consider the subjective preferences of the user, who might be interested in modifying or highlighting specific elements of a scene. In fact, most image or video post-processing techniques are applied to the entire image or frame. In this work, a framework for selective video enhancement is presented. It is based on an adaptive multi-resolution edge enhancement technique performed in the Laguerre–Gauss complex wavelet domain. In the proposed scheme, edges are selectively enhanced or attenuated according to the inputs of the user, taking into account the characteristics of the content, the response of the human visual system, and the masking effects induced by textured background. Two scenarios have been implemented and tested: (i) an interactive video editing system, where the user selects the objects to enhance through a graphical interface and the system automatically propagates the selection through the following video frames belonging to the same shot, (ii) a perceptually-driven technique to improve the quality of RGB plus Depth videos, which uses depth maps as additional features to guide the enhancement.

Assessing the quality of images is a challenging task. To achieve this goal, images must be evaluated by a pool of subjects following a well-defined protocol or an objective quality metric must be defined. In this work, an objective quality metric based on deep neural network is proposed. The metric takes into account the human vision system by computing the saliency map and natural scene statistics features of the image under test. The neural network is composed by two modules: the convolutional layers and the regression units. The first one is trained by using preprocessed distorted images. The feature weights of the first module are smoothed by exploiting the estimated saliency map. The latter module is fit with the ground truth quality scores of the input image and the scaled feature weights obtained from first module by using visual sensitivity factor of image obtained using natural scene statistics features. The performances of the proposed metric have been evaluated by using four datasets: LIVEIQA, TID2013, CSIQ, and KADID10K. The achieved results show the effectiveness of the proposed system in closely matching the predicted quality scores with the ground truth ones.

Recently an impressive development in immersive technologies, such as Augmented Reality (AR), Virtual Reality (VR) and 360${^\circ }$video, has been witnessed. However, methods for quality assessment have not been keeping up. This paper studies quality assessment of 360${^\circ }$video from the cross-lab tests (involving ten laboratories and more than 300 participants) carried out by the Immersive Media Group (IMG) of the Video Quality Experts Group (VQEG). These tests were addressed to assess and validate subjective evaluation methodologies for 360${^\circ }$video. Audiovisual quality, simulator sickness symptoms, and exploration behavior were evaluated with short (from 10 seconds to 30 seconds) 360${^\circ }$sequences. The following factors’ influences were also analyzed: assessment methodology, sequence duration, Head-Mounted Display (HMD) device, uniform and non-uniform coding degradations, and simulator sickness assessment methods. The obtained results have demonstrated the validity of Absolute Category Rating (ACR) and Degradation Category Rating (DCR) for subjective tests with 360${^\circ }$videos, the possibility of using 10-second videos (with or without audio) when addressing quality evaluation of coding artifacts, as well as any commercial HMD (satisfying minimum requirements). Also, more efficient methods than the long Simulator Sickness Questionnaire (SSQ) have been proposed to evaluate related symptoms with 360${^\circ }$videos. These results have been instrumental for the development of the ITU-T Recommendation P.919. Finally, the annotated dataset from the tests is made publicly available for the research community.

In this contribution a study dedicated to understanding the influence of the presence of human faces in a 360° image on human perception is presented. Extensive research on saliency estimation in 2D images has shown that the presence of faces attracts human attention. Following these studies, recent 2D image quality assessment methods exploit face detection systems in their models. The application of these concepts to 360° image is not straightforward. Furthermore, existing literature lacks a comparative study between the performance of face detection algorithms on various types of images (2D, fisheye, and omnidirectional) and how detected faces affect the procedure of saliency estimation. In this direction, we analyze the importance of person faces in a scene, by performing a set of subjective tests. From the performed analysis, it results that, even in 360° images, human faces represent an important factor for image saliency. However, giving equal importance to all the detected faces does not lead to a better saliency estimation. Therefore, in this work, a study on the possible use of the face detector in estimating the salience of the 360° image is performed.

The evaluation of the quality of light field images is a demanding task given the peculiarities of this media. In the literature, attempts to assess quality have been done by considering specific coding approaches or visualization techniques. In this paper we intend to i) investigate whether the distortions of the light field are reflected in the distortion of the saliency map and ii) propose a metric for image quality assessment of light fields based on a convolutional neural network that exploits the measure of the distortion of the saliency map. In our tests, the annotated SMART dataset has been used. The achieved results confirm the importance of saliency for improving the performance of quality metrics.

In this paper we present a technique for saliency estimation in 360 images. Existing approaches exploit high/low-level image features, head movement, and eye-gazes of observers. However, we believe that the saliency of an image is influenced also by its content. The proposed approach consists of three modules. In the first module, a segmentation of the image is performed and its local and global features are extracted. The second module combines the extracted features for summarizing the content that the image portrays. Finally, this content knowledge is utilised for image saliency estimation. Experimental results show effectiveness of the proposed system with respect to ground truth saliency maps.

Light field imaging is increasingly spreading and its applications are reaching a larger number of users. The quality of the rendered content as perceived by the viewers is a key factor for the future development of this technology. Many factors can cause a degradation of the Light Field quality, from data acquisition/creation to the final rendering. To increase the perceived quality of the Light Field content, as usually performed for 2D and 3D content, processing techniques can be applied. In this work, an approach for enhancing Light Field videos is presented. The proposed approach is based on the characteristics of the human visual system and relies on the estimation of depth, motion and saliency. Those features guide a selective enhancement that is implemented in the Laguerre-Gauss Transform domain. The performed tests show the effectiveness of the proposed method.

In this paper, a multi-resolution method for depth estimation from dense image arrays is presented. Recent progress in consumer electronics has enabled the development of low cost hand-held plenoptic cameras. In these systems, multiple views of a scene are captured in a single shot by means of a micro-lens array placed on the focal point of the first camera lens, in front of the imaging sensor. These views can be processed jointly to obtain accurate depth maps. In this contribution, to reduce the computational complexity associated to global optimization schemes based on match cost functions, we make a local estimate based on the maximization of the total log-likelihood spatial density aggregated along the epipolar lines corresponding to each view pair. This method includes the local maximum likelihood estimation of the depth field based on epipolar plane images. To face the potential accuracy losses associated to the ambiguity problem that arises in flat surface regions while preserving bandwidth in correspondence of the edges, we adopt a multi-resolution scheme. In practice, the depth map resolution is reduced in regions where maximizing the higher resolution functional is ill-conditioned. The main benefits of the proposed system are in a reduced computational complexity and a high accuracy of the estimated depth. Experimental results show that the proposed scheme represents a good tradeoff among accuracy, robustness, and discontinuities handling.

In this contribution a system for seated posture identification is presented. The assessment tools is based on an office-chair equipped with sensors. In more details, a set of textile pressure sensors has been placed on a chair both on the chair backrest and on the seat. The position of the sensors has been selected for maximizing the possibility of sensing minimum variations of the subject's posture. To validate the system, an extensive subjective experiment has been performed in which the subject undergoes an increasing stress-level test. The collected results show that this instrument is effective in assessing the attention/fatigue of a subject in seating condition by the analysis of body posture.

The need for effective and reliable surveillance techniques is getting nowadays more and more of primary importance, especially in the actual scenario in which safety and security have become a priority. While classical techniques rely on video-based surveillance systems, such as Close-Circuit television, many studies show that also the audio signal can be effectively used for these purposes. There are many characteristics that make the audio signal particularly suited for this task and, above all, the fact that the analysis of the audio signal can greatly improve thanks to the introduction of automatic classification. Recently, a large focus has been on the use of Deep Neural Networks for classifying audio data and, in this work, we aim to test their performance in the audio surveillance field. In this contribution we propose an algorithm for audio events detection in noisy environments based on the use of deep recurrent neural network. The achieved results show satisfactory and improved performances with respect to state-of-the-art techniques.

Light Field imaging derives from the fundamentals of light field sampling, where the spatial information about a scene can be captured with angular information. That is, the Light Field imaging is based on a camera recording information about the intensity of light from the scene and about the direction of the light rays. The acquired data can be shown to the user in different ways, such as image with digitally extended depth of field, 3D, parallax, and 360 degree display. In this contribution, the impact of different rendering techniques on the Quality of Experience is addressed. The achieved results show that the visualization techniques may have different impact on the perceived quality even when the same content is considered.

In this work, the problem of dynamic video scene creation obtained by combining information extracted from multiple video sequences is considered. The main novelty of the proposed approach relies on the use of aesthetic features for automatically aggregating the inputs from different cameras in a unique video. While prior methodologies have separately addressed the issues of aesthetic feature extraction from videos and video orchestration, in this work we exploit selected features of a scene for automatically selecting the shots being characterized by the best aesthetic score. In order to evaluate the effectiveness of the proposed method, a subjective experiment has been carried out with experts from the audiovisual field. The achieved results are encouraging and show that there is space for improving the performances.

Light field technology may have a positive impact on several multimedia applications thanks to novel ways to explore the captured scenes, such as changing the parallax (horizontally and vertically) and refocusing the content. These innovative use cases require new considerations that affect the whole processing chain, from content acquisition to visualization, as well as the methodologies for quality evaluation. In particular, capturing and selecting the appropriate content is crucial for a successful development and evaluation of audiovisual technologies. Thus, this paper presents a framework addressing the reconsideration of the space of attributes for an adequate characterization of light field data. Firstly, an exhaustive characterization of light field content is described, based on various particular features, including depth and refocusing properties to traditional spatial, temporal, and color indicators. Then, based on this characterization, specific techniques are proposed for an effective selection of light field content for perceptual quality assessment.

The security of Cyber Physical Systems is a key factor since these architectures are being applied in many critical scenarios, such as power or water plants, hospitals, etc. In the state of the art, several approaches have been proposed to deal with the possible attacks that could be inferred to the Cyber Physical Systems. This contribution proposes a method for counter fighting the injection of tampered data in the communication channel. If this attack is not timely detected, it may result in severe disruption of the system or even in its complete damage. The proposed approach is based on coding the physical output of the system through permutation matrices whose scheme varies based on a randomly generated sequence. The strength of this method is in reducing the possibility of a successful injection attack while limiting the computational complexity. The experimental tests prove the effectiveness of the proposed scheme in detecting the performed attack while granting the real time constraints of the Cyber Physical Systems.

In this contribution, the design of a Light Field image dataset is presented. It can be useful for design, testing, and benchmarking Light Field image processing algorithms. As first step, image content selection criteria have been defined based on selected image quality key-attributes, i.e. spatial information, colorfulness, texture key features, depth of field, etc. Next, image scenes have been selected and captured by using the Lytro Illum Light Field camera. Performed analysis shows that the proposed set of images is sufficient for addressing a wide range of attributes relevant for assessing Light Field image quality.

In this paper, two image quality metrics based on morphological multiscale decompositions have been applied for the evaluation of free viewpoint video sequences. These sequences are synthesized using decompressed depth maps in the Depth-Image-Based Rendering synthesis process. Since the synthesis introduces edge distortion in the synthesized image/video, morphological filters are used for their ability to maintain important geometric information (i.e., edges) across different resolution levels. The edges displacement in different resolution scales is evaluated by means of the Mean Square Error. The adopted metrics show higher correlation with human judgment than state-of-the-art image quality measures used in this context.

In this paper a multi-resolution depth field estimation algorithm for plenoptic cameras is presented. To face the potential accuracy losses originated from ambiguity problems arising in flat surface regions, still preserving bandwidth in correspondence of edges, a multi-resolution scheme is proposed. The achieved results show that the proposed local optimization method outperforms state of the art more complex global optimization based methods.

This paper describes a recently created image database, TID2013, intended for evaluation of full-reference visual quality assessment metrics. With respect to TID2008, the new database contains a larger number (3000) of test images obtained from 25 reference images, 24 types of distortions for each reference image, and 5 levels for each type of distortion. Motivations for introducing 7 new types of distortions and one additional level of distortions are given; examples of distorted images are presented. Mean opinion scores (MOS) for the new database have been collected by performing 985 subjective experiments with volunteers (observers) from five countries (Finland, France, Italy, Ukraine, and USA). The availability of MOS allows the use of the designed database as a fundamental tool for assessing the effectiveness of visual quality. Furthermore, existing visual quality metrics have been tested with the proposed database and the collected results have been analyzed using rank order correlation coefficients between MOS and considered metrics. These correlation indices have been obtained both considering the full set of distorted images and specific image subsets, for highlighting advantages and drawbacks of existing, state of the art, quality metrics. Approaches to thorough performance analysis for a given metric are presented to detect practical situations or distortion types for which this metric is not adequate enough to human perception. The created image database and the collected MOS values are freely available for downloading and utilization for scientific purposes.

Scale Invariant Feature Transform (SIFT) has been widely employed in several image application domains, including Image Forensics (e.g. detection of copy-move forgery or near duplicates). Recently, a number of methods allowing to remove SIFT keypoints from an original image have been devised studying the problem of SIFT security against malicious procedures. Such techniques are quite effective in producing an attacked image with very few (or no) keypoints, but at the expense of an image distortion. Final perceptual quality has been taken in account very roughly so far. In this paper, effectiveness of the attacking methods is evaluated also from the side of perceptual image quality; a new version of a SIFT keypoint removal method, based on a perceptual metric, is presented and an extended series of perceptive experiments is reported.

In this contribution, a new image database for testing full-reference image quality assessment metrics is presented. It is based on 1700 test images (25 reference images, 17 types of distortions for each reference image, 4 levels for each type of distortion). Using this image database, 654 observers from three different countries (Finland, Italy, and Ukraine) have carried out about 400000 individual human quality judgments (more than 200 judgments for each distorted image). The obtained mean opinion scores for the considered images can be used for evaluating the performances of visual quality metrics as well as for comparison and for the design of new metrics. The database, with testing results, is freely available.