The evaluation of human responses in multimedia experiences using physiological data has a well-established presence in the academic literature. However, this field is currently undergoing transformative changes, driven by the accessibility of diverse and cost-effective devices, innovative software analysis methods, and the emergence of novel application domains such as Virtual and Augmented Reality and mulsemedia. To address the imperative of contextualizing these evolving trends in a contemporary context, this paper presents a systematic review with the objective of delineating the array of physiological data utilized in assessing Quality of Experience (QoE) and User Experience (UX) in multimedia studies. It also examines the devices employed for data collection and the analytical techniques applied to interpret the acquired data. While our review exposes both constraints and promising discoveries in these domains, it also emphasizes the escalating significance and practicality of leveraging physiological data in user assessments, especially as the boundaries between the physical and digital domains continue to blur.

Abstract In order to create immersive experiences in virtual worlds, we need to explore different human senses (sight, hearing, smell, taste, and touch). Many different devices have been developed by both industry and academia towards this aim. In this paper, we focus our attention on the researched area of thermal and wind devices to deliver the sensations of heat and cold against people’s skin and their application to human-computer interaction (HCI). First, we present a review of devices and their features that were identified as relevant. Then, we highlight the users’ experience with thermal and wind devices, highlighting limitations either found or inferred by the authors and studies selected for this survey. Accordingly, from the current literature, we can infer that, in wind and temperature-based haptic systems (i) users experience wind effects produced by fans that move air molecules at room temperature, and (ii) there is no integration of thermal components to devices intended for the production of both cold or hot airflows. Subsequently, an analysis of why thermal wind devices have not been devised yet is undertaken, highlighting the challenges of creating such devices.

Omnidirectional videos have become a leading multimedia format for Virtual Reality applications. While live 360$^\circ$videos offer a unique immersive experience, streaming of omnidirectional content at high resolutions is not always feasible in bandwidth-limited networks. While in the case of flat videos, scaling to lower resolutions works well, 360$^\circ$video quality is seriously degraded because of the viewing distances involved in head-mounted displays. Hence, in this article, we investigate first how quality degradation impacts the sense of presence in immersive Virtual Reality applications. Then, we are pushing the boundaries of 360$^\circ$technology through the enhancement with multisensory stimuli. 48 participants experimented both 360$^\circ$scenarios (with and without multisensory content), while they were divided randomly between four conditions characterised by different encoding qualities (HD, FullHD, 2.5K, 4K). The results showed that presence is not mediated by streaming at a higher bitrate. The trend we identified revealed however that presence is positively and significantly impacted by the enhancement with multisensory content. This shows that multisensory technology is crucial in creating more immersive experiences.

Situational awareness platforms support the continuous observation of interest situations. In the eHealth context, SA can improve health services by assessing and interpreting patients' current situations. For instance, it allows the healthcare providers to recognize changes and abnormalities in the patient's condition and take the necessary actions. However, SA platforms still suffer from centralized management issues, such as availability, security, and privacy. This paper presents a blockchain-based approach for SA data processing to overcome these issues. Blockchain is a decentralized and distributed technology that also brings the advantage of flexible data custody. We evaluate our system through a case study and discuss the benefits and challenges of adopting Blockchain as a solution for eHealth.

The circadian rhythm is the period of about 24 h, which is the basis of the human biological cycle. It coordinates human daytime activities and body rest, and influences cognitive learning and processing information in the brain during sleep. Light rhythmically influences these hormonal physiological procedures. External factors such as the exposure to artificial light can also interfere with the circadian rhythm, causing sleep and wake disturbances. Remote monitoring systems can be used to better understand how the artificial light influences sleep and bodily rhythms, integrating light and physiological measurements to assess their interaction points. We developed an equipment to monitor the environmental light spectrum and gather data from wrist devices, collecting data on sleep quality and daytime activities. This study shows the challenges faced during the development of this monitoring equipment and the lessons learned from its construction and installation at the Brazilian Antarctic research station.

Remote patient monitoring (RPM) is commonly employed in home environments to monitor a patient's health in compliance with their healthcare plan. Simulating real-life scenarios for testing the health applications is a complex and challenging task, particularly in the RPM scenario, for testing issues related to scalability, which can raise problems that could not be identified during development. Several tools have been proposed to support tests using clinical and biological data of patients, but there exist several gaps, mainly related to scalability and the data sources used for simulation. This work presents HSim, a simulator for the evaluation of health applications in RPM. HSim aims to simplify the simulation of biomedical patient data to provide support for the assessment of infrastructures and services for healthcare applications. System validation tests focused on scalability and the use of real databases demonstrated the potential use of HSim to validate RPM applications.

Using olfactory media to enhance traditional multimedia content opens up novel opportunities for user interactions. Whilst the influence of olfaction on user experience in mulsemedia (multiple sensorial media) environments has been previously studied, the impact of the fundamental dimensions of scent intensity and valence (odor hedonic dimension or pleasantness) have been largely unexplored. This is precisely what we target in this paper, which reports the results of an empirical investigation examining how scent intensity and valence impact mulsemedia Quality of Experience (QoE). Accordingly, 54 participants were exposed to different odor valences and scent intensity levels when viewing three short multimedia clips. In particular, we examine both subjective (self-reported) as well as objective QoE metrics, as evidenced by user heart rates and eye gaze patterns. Results show that whilst eye gaze patterns are largely unaffected by the experimental conditions, valence does have a statistically significant impact upon user heart rates, as does intensity for two of the three clips employed in our study. In terms of subjective QoE, results indicate that hedonic valence impacts on the sense of reality and enjoyment; however varying odor intensity levels do not seem to differentially impact on user experience, bringing into question the need for strong scent intensities.

A major challenge for remote patient monitoring (RPM) systems is processing data collected from a large number of healthcare devices and developing suitable algorithms and approaches to react accordingly to a wide spectrum of situations of interest, which must be properly detected. Flow-based programming, whose operation is based on state changes, is a promising approach to overcome this challenge, facilitating the personalization of data processing according to patients' health conditions. However, this approach is not suited to the detection of complex contextual situations, thereby hindering its adoption in RPM systems for data processing. In this regard, we propose an approach that combines situation awareness and flow-based programming to widen the capability of RPM systems to handle many-sided scenarios, which requires the monitoring of complex patient healthcare conditions. An evaluation of the proposed approach was conducted to demonstrate the flexibility of the solution for processing heterogeneous health information.

Previous research has shown that adding multisensory media-mulsemedia-to traditional audiovisual content has a positive effect on user Quality of Experience (QoE). However, the QoE impact of employing mulsemedia in 360° videos has remained unexplored. Accordingly, in this paper, a QoE study for watching a 360° video-with and without multisensory effects-in a full free-viewpoint VR setting is presented. The parametric space we considered to influence the QoE consists of the encoding quality and the motion level of the transmitted media. To achieve our research aim, we propose a wearable VR system that provides multisensory enhancement of 360° videos. Then, we utilise its capabilities to systematically evaluate the effects of multisensory stimulation on perceived quality degradation for videos with different motion levels and encoding qualities. Our results make a strong case for the inclusion of multisensory effects in 360° videos, as they reveal that both user-perceived quality, as well as enjoyment, are significantly higher when mulsemedia (as opposed to traditional multimedia) is employed in this context. Moreover, these observations hold true independent of the underlying 360° video encoding quality-thus QoE can be significantly enhanced with a minimal impact on networking resources.

The introduction of Next Generation Networks (NGN) has changed significantly the network service model. Vertical structures that are designed to each type of access have been replaced by horizontal structures. However, the service architecture has to be rebuilt to support the required agility within the horizontal structures. Our proposal, named SANGN, is based on a full integration of Open Services Gateway Initiative (OSGi) with Java Agent DEvelopment (JADE) framework for providing multimedia services of IP Multimedia Subsystem (IMS). We have introduced a Televote service as a use case in order to validate the SANGN architecture ensuring the massive scalability of these services to meet the stringent multimedia requirements. Our implementation is able to adapt with a high dynamic load of the system, making use of a minimal set of computational resources seamlessly for the clients, providing IMS core components in the network infrastructure.

The use of multiple senses in interactive applications has become increasingly feasible due to the upsurge of commercial, off-the-shelf devices to produce sensory effects. Creating Multiple Sensorial Media (MulSeMedia) immersive systems requires understanding their digital ecosystem. Mulsemedia systems encompass a set of applications, and devices of different types assembled to communicate or express feelings from the virtual world to the real world. Despite existing standards, tools, and recent research devoted to them, there is still a lack of formal and explicit representation of what mulse-media is. Misconceptions could eventually lead to the construction of solutions that might not take into account reuse, integration, standardization, among other design features. In this paper, we propose to establish a common conceptualization about mulsemedia systems through a reference ontology, named MulseOnto, covering their main notions. To evaluate it, we applied ontology verification and validation techniques, including assessment by humans and a data-driven approach. The results showed that MulseOnto can be used as a consensual conceptual model for exploring the knowledge about the whole chain of mulsemedia systems.

Human perception is inherently multisensory involving sight, hearing, smell, touch, and taste. Mulsemedia systems include the combination of traditional media (text, image, video, and audio) with non-traditional ones that stimulate other senses beyond sight and hearing. Whilst work has been done on some user-centred aspects that the distribution of mulsemedia data raises, such as synchronisation, and jitter, this paper tackles complementary issues that temporality constraints pose on the distribution of mulsemedia effects. It aims at improving response time interval in networked event-based mulsemedia systems based upon prior findings in this context. Thus, we reshaped the communication strategy of an open distributed mulsemedia platform called PlaySEM to work more efficiently with other event-based applications, such as games, VR/AR software, and interactive applications, wishing to stimulate other senses to increase the immersion of users. Moreover, we added lightweight communication protocols in its interface to analyse whether they reduce network overhead. To carry out the experiment, we developed mock applications for different protocols to simulate an interactive application working with the PlaySEM, measuring the delay between them. The results showed that by pre-processing sensory effects metadata before real-time communication, and selecting the appropriate protocol, response time interval in networked event-based mulsemedia systems can decrease remarkably.

Mulsemedia applications have become increasingly popular. There have been many efforts to increase the Quality of Experience (QoE) of users by using them. From the users' perspective, it is crucial that systems produce high levels of enjoyment and utility. Thus, many experimental tools have been developed and applied to different purposes such as entertainment, health, and culture. Despite that, little attention is paid to the evaluation of mulsemedia tools and platforms. In this paper, we present a time evaluation of the integration between a distributed mulsemedia platform called PlaySEM and an interactive application whereby users interact by gestures, in order to discover how long this process takes. We describe the test scenario and our approach for measuring this integration. Then, we discuss the results and point out aspects that bring implications to be taken into account for future similar solutions. The results showed values in the range of 27ms to 67ms on average spent throughout the process before the effective activation of the sensory effect devices on a wired network.

Content Centric Networking (CCN) represents an important change in the current operation of the Internet, prioritizing content over the communication between end nodes. Routers play an essential role in CCN, since they receive the requests for a given content and provide content caching for the most popular ones. They have their own forwarding strategies and caching policies for the most popular contents. Despite the number of works on this field, experimental evaluation of different forwarding algorithms and caching policies yet demands a huge effort in routers programming. In this paper we propose SDCCN, a SDN approach to CCN that provides programmable forwarding strategy and caching policies. SDCCN allows fast prototyping and experimentation in CCN. Proofs of concept were performed to demonstrate the programmability of the cache replacement algorithms and the Strategy Layer. Experimental results, obtained through implementation in the Mininet environment, are presented and evaluated.

The paper presents an event driven approach for building interactive environments with sensory effects (such as RGB color light, wind and vibration) described through MPEGV standard. The most important differences of this work from previous ones are: (1) it divides the design process into well defined stages, showing the behavior of the system under a new perspective and their interactions, (2) it follows an event driven paradigm instead of a timeline one to send commands to sensory devices and (3) it decouples rendering of MPEG-V effects from the presentation software component. The method to validate the proposed approach was a case study, which has integrated multiples sensory effects to an interactive environment based on gesture recognizing.

Over the years some models for multimedia spatio-temporal synchronization has been proposed. In most of these models, a known drawback is that they generally abstract the media content and consider only start and end events content presentation. Thus, it is not possible to specify synchronization relations involving "within" media content events. Highlights in a live TV show are examples of these events. In such cases, it may be expected that highlights "may occur", but it is not possible to determine "when". In order to solve this drawback, this article presents an approach for modeling live Interactive TV shows that support the definition of synchronization constraints related to events not known at authoring time. These events are structured in objects, using an object oriented paradigm, and broadcasted to TV receivers as way to assure the synchronization relations

The paper generalizes previous works of the authors by revisiting the concept of temporal consistency of hypermedia documents. This generalization relies on the distinction of internal and external non-deterministic events.

Hypermedia authoring tools usually suffer ikom a lack of validation capabilities that would make it possible to check a document against temporal inconsistencies.The document design methodology proposed in the paper overcomes this problem by translating a high-level model of an hypermedia document into a RT-LOTOS formal specificatio~on which standard reachability analysis may be applied for verification purposes.The propod approa~extends an early work to verify both intrinsic and extrinsic temporal consistency properties of an hypermedia documenti 1.1

The paper presents a methodology by which documents authored with the Nested Context Model (NCM) are automatically translated into a RT-LOTOS specification. The latter is then checked against intrinsic and extrinsic consistencies using standard RT-LOTOS reachability analysis.