Qianli Wang, Van Bach Nguyen, Yihong Liu, Fedor Splitt, Nils Feldhus, Christin Seifert, Hinrich Schuetze, Sebastian Möller, Vera Schmitt. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Fact-checking articles encode rich supporting evidence and reasoning, yet this evidence remains largely inaccessible to automated verification systems due to unstructured presentation. We introduce PrimeFacts, a methodology and resource for extracting fine-grained evidence from full fact-checking articles. We compile 13,106 PolitiFact articles with claims, verdicts, and all referenced sources, and we identify 49,718 in-article hyperlinks as natural anchors to pinpoint key evidence. Our framework leverages large language models (LLMs) to rewrite these anchor sentences into stand-alone, context-independent premises and investigates the extraction of additional implicit evidence. In evaluations on cross-article evidence retrieval and claim verification, the extracted premises substantially improve performance. Decontextualized evidence yields higher retrievability, achieving up to a 30 percent relative gain in Mean Reciprocal Rank over verbatim sentences, and using the evidence for verdict prediction raises Macro-F1 by 10-20 points over the baseline. These gains are consistent across different verdict granularities (2-class vs. 5-class) and model architectures. A qualitative analysis indicates that the decontextualized premises remain faithful to the original sources. Our work highlights the promise of reusing fact-checkers' evidence for automation and provides a large-scale resource of structured evidence from real-world fact-checks.

Natural language explanations (NLEs) are vital for elucidating the reasoning behind large language model (LLM) decisions. Many techniques have been developed to generate NLEs using LLMs. However, like humans, LLMs might not always produce optimal NLEs on first attempt. Inspired by human learning processes, we introduce Cross-Refine, which employs role modeling by deploying two LLMs as generator and critic, respectively. The generator outputs a first NLE and then refines this initial explanation using feedback and suggestions provided by the critic. Cross-Refine does not require any supervised training data or additional training. We validate Cross-Refine across three NLP tasks using three state-of-the-art open-source LLMs through automatic and human evaluation. We select Self-Refine (Madaan et al., 2023) as the baseline, which only utilizes self-feedback to refine the explanations. Our findings from automatic evaluation and a user study indicate that Cross-Refine outperforms Self-Refine. Meanwhile, Cross-Refine can perform effectively with less powerful LLMs, whereas Self-Refine only yields strong results with ChatGPT. Additionally, we conduct an ablation study to assess the importance of feedback and suggestions. Both of them play an important role in refining explanations. We further evaluate Cross-Refine on a bilingual dataset in English and German.

Counterfactual examples are widely employed to enhance the performance and robustness of large language models (LLMs) through counterfactual data augmentation (CDA). However, the selection of the judge model used to evaluate label flipping, the primary metric for assessing the validity of generated counterfactuals for CDA, yields inconsistent results. To decipher this, we define four types of relationships between the counterfactual generator and judge models: being the same model, belonging to the same model family, being independent models, and having an distillation relationship. Through extensive experiments involving two state-of-the-art LLM-based methods, three datasets, four generator models, and 15 judge models, complemented by a user study (n = 90), we demonstrate that judge models with an independent, non-fine-tuned relationship to the generator model provide the most reliable label flipping evaluations. Relationships between the generator and judge models, which are closely aligned with the user study for CDA, result in better model performance and robustness. Nevertheless, we find that the gap between the most effective judge models and the results obtained from the user study remains considerably large. This suggests that a fully automated pipeline for CDA may be inadequate and requires human intervention.

Rapid advancements in generative modeling have made synthetic audio generation easy, making speech-based services vulnerable to spoofing attacks. Consequently, there is a dire need for robust countermeasures more than ever. Existing solutions for deepfake detection are often criticized for lacking generalizability and fail drastically when applied to real-world data. This study proposes a novel method for generalizable spoofing detection leveraging non-semantic universal audio representations. Extensive experiments have been performed to find suitable non-semantic features using TRILL and TRILLsson models. The results indicate that the proposed method achieves comparable performance on the in-domain test set while significantly outperforming state-of-the-art approaches on out-of-domain test sets. Notably, it demonstrates superior generalization on public-domain data, surpassing methods based on hand-crafted features, semantic embeddings, and end-to-end architectures.

Virtual environments (VEs) are increasingly used for immersive experiences, training simulations, and entertainment, yet factors such as height perception and user stance can significantly influence user experience (UX). Height perception in VEs plays a crucial role in shaping UX, particularly in immersive applications such as climbing simulations. This study investigates the effects of height in various VEs and examines how user stance, sitting or standing, impacts immersion, perceived height, and motion sickness. A user study was conducted with 25 participants who played through five randomized climbing scenarios, ranging from indoor climbing gyms to outdoor cityscapes and mountainous terrains. Participants’ UX was assessed using standardized questionnaires, including the IPQ for general presence, spatial presence, involvement, and experienced realism; as well as the SSQ to evaluate motion sickness symptoms such as nausea, oculomotor strain, and disorientation. Results indicate that seated participants experienced slightly higher immersion but were also more susceptible to motion sickness compared to those standing. While standing participants maintained consistent scores across different environments, seated participants reported increased immersion and discomfort as the VEs became larger, more physically demanding, and visually complex.

This paper explores the effects of visual aspects in virtual environments on user input performance and perception, when conducting a target selection task with raycasting as the interaction technique. Four virtual environments based on combinations of two visual modalities have been created with four different types of objects to conduct a selection task based user study. Each environment was designed to vary according to two key visual elements: the presence or absence of spatial reference points and the inclusion of dynamic elements. Participants completed a questionnaire, which included items from the General Presence and Spatial Presence subscales of the iGroup Presence Questionnaire, as well as the short version of the User Experience Questionnaire, which assessed the perceived qualities of the virtual environment. The results revealed that spatial reference points significantly improved accuracy by reducing selection errors, reinforcing their role as essential visual guides in virtual environments. Dynamic elements, on the other hand, increased selection time, indicating higher cognitive demands during task execution. Additionally, immersion and user experience metrics were positively correlated with both selection time and hit offset, indicating a relationship between higher presence or enjoyment and slower, less precise interactions.

Standardized speech quality assessments should optimally occur in soundproof environments with minimal noise interference. However, poor ventilation in such environments can lead to increased CO2concentrations, which may affect cognitive function and judgement. This study investigates whether CO2levels affect speech quality ratings and response behaviour in listening tests. Across over 5,000 trials with 24 participants, we found three key effects: (1) higher CO2levels were associated with faster response times, indicating more impulsive decision-making; (2) ratings of poor-quality samples became more favorable, and high-quality samples less favorable under high CO2, suggesting reduced perceptual discrimination; (3) rating variability decreased, implying diminished sensitivity. These findings highlight a previously overlooked factor in speech quality research: ambient air quality may introduce bias into subjective ratings.

Traditional 2D video telephony systems often fail to convey critical non-verbal cues and spatial depth, limiting naturalistic remote communication and contributing to user fatigue. This study systematically investigates the experiential and interactive benefits of stereoscopic 3D video telephony compared to conventional 2D video calls. Thirty participants engaged in two communicative tasks—a free conversation and a game of Taboo—using a shutter-glasses-based stereoscopic setup and a matched 2D control. Measures included user satisfaction (UEQ), non-verbal immediacy (NIS-O), communication quality (audiovisual ratings, comfort, cognitive workload), and immersion (Film-IEQ). Results consistently demonstrated significantly higher user satisfaction, richer non-verbal communication, improved perceived audiovisual quality, and greater immersion in the 3D condition without increased cognitive workload or physical discomfort. These findings highlight the value of stereoscopic 3D video to enhance remote interpersonal interactions by providing depth cues that support more natural and engaging communication. The study contributes empirical evidence guiding the design and evaluation of emerging immersive telepresence technologies, offering a benchmark for future volumetric and point-cloud-based 3D communication systems.

The ability to establish eye contact is a central aspect of human communication and interaction. In video communication, however, it is impaired by the physical separation between camera and screen. This study investigates the effects of three video telephony configurations – standard video calls, teleprompter-assisted calls, and AI-based gaze correction – on user experience and perceived communication quality. In a collaborative task with 32 participants, we evaluated subjective dimensions such as Engagement, Naturalness of Eye Contact, and Social Presence across four conditions, including a face-to-face baseline. The results indicate that face-to-face interactions significantly outperform all video conditions in central dimensions. Notably, technical attempts to replicate natural eye contact, particularly through AI-based gaze correction, proved ineffective, leading to reduced perceived Naturalness of Eye Contact and increased Anxiety. These findings point to a heightened user sensitivity to artificial modifications in socially salient contexts. Although teleprompter-based solutions elicited less Anxiety, their technical constraints limit practical applicability. We conclude that current implementations of AI gaze correction, specifically the version in our study, are not yet suitable for enhancing conversational quality. Future efforts might benefit from focusing on design that build on the unique affordances of digital interaction, rather than aiming to replicate face-to-face dynamics.

Large Language Models (LLMs) hold promise for addressing the shortage of qualified therapists in mental health care. While chatbot-based Cognitive Behavioral Therapy (CBT) tools exist, their efficacy in sensitive contexts remains underexplored. This study examines the potential of LLMs to support therapy sessions aimed at reducing Child Sexual Abuse Material (CSAM) consumption. We propose a Retrieval-Augmented Generation (RAG) framework that leverages a fine-tuned BERT-based retriever to guide LLM-generated responses, better capturing the multi-turn, context-specific dynamics of therapy. Four LLMs—Qwen2-7B-Instruct, Mistral-7B-Instruct-v0.3, Orca-2-13B, and Zephyr-7B-Alpha—were evaluated in a small-scale study with 14 domain-expert psychotherapists. Our comparative analysis reveals that, in certain scenarios, LLMs like Mistral-7B-Instruct-v0.3 and Orca-2-13B were preferred over human therapist responses. While limited by sample size, these findings suggest that LLMs can perform at a level comparable to or even exceeding that of human therapists, especially in therapy focused on reducing CSAM consumption. Our code is available online: https://git.tu-berlin.de/neha.deshpande/therapy_responses/-/tree/main

This study explores how prior exposure to physical objects influences the quality and realism perception of Digital Twins (DT) with varying levels of fidelity in Virtual Reality (VR). In a mixed experimental design, 24 participants were divided into two equal groups: an exposure group, in which members were shown physical objects before inspecting and rating their replicas in VR, and a control group without prior knowledge. Three objects were presented, each under four fidelity conditions with varying texture resolution and geometric detail. Participants rated perceived quality and realism through in-VR self-reports. Statistical analysis revealed that texture resolution significantly affected realism and quality perception, whereas geometric detail only influenced quality ratings. Investigating the between-factor, no significant effect of exposure on quality and realism perception was found. These findings raise important questions about the cognitive relationship between physical objects and their digital counterparts and how fidelity influences the perception of DTs in VR.

User-generated data sources have gained significance in uncovering Adverse Drug Reactions (ADRs), with an increasing number of discussions occurring in the digital world. However, the existing clinical corpora predominantly revolve around scientific articles in English. This work presents a multilingual corpus of texts concerning ADRs gathered from diverse sources, including patient fora, social media, and clinical reports in German, French, and Japanese. Our corpus contains annotations covering 12 entity types, four attribute types, and 13 relation types. It contributes to the development of real-world multilingual language models for healthcare. We provide statistics to highlight certain challenges associated with the corpus and conduct preliminary experiments resulting in strong baselines for extracting entities and relations between these entities, both within and across languages.

In this study, we introduce KIALOPRIME, a novel large-scale dataset comprising 5,687 argument discussion graphs with a total of 1,088,801 of supporting, attacking, and neutral argument relations, derived from the structured debates of the online discussion platform Kialo.com. This dataset facilitates in-depth analysis of argument structures and the dynamics of discourse, serving as a substantial resource for computational argumentation research. We explore argument inference through traditional sequence classification and a modern generative reasoning based approach, employing an open-source mixture of experts LLM to interpret and enrich each argument pair with high-quality synthetic elaborations about the argumentative interaction. We achieve baseline results of F1 .899 and .840 within discussions and F1 .908 and .840 across discussions for the argument relation and elaboration classification models, respectively. While the elaboration-based model scores slightly lower on the classification task, we highlight areas of improvement to better capture the hidden complexities of argumentative text. These initial findings are promising as they not only establish robust benchmarks for future studies but also demonstrate the potential for using generative reasoning to provide a more insightful analysis of argument relations.

We investigate the impact of large language models (LLMs) on political discourse with a particular focus on the influence of generated personas on model responses.We find an echo chamber effect from LLM chatbots when provided with German-language biographical information of politicians and voters in German politics, leading to sycophantic responses and the reinforcement of existing political biases.Findings reveal that personas of certain political party, such as those of the 'Alternative für Deutschland' party, exert a stronger influence on LLMs, potentially amplifying extremist views.Unlike prior studies, we cannot corroborate a tendency for larger models to exert stronger sycophantic behaviour.We propose that further development should aim at reducing sycophantic behaviour in LLMs across all sizes and diversifying language capabilities in LLMs to enhance inclusivity. 1

This study explored a concept for using Augmented Reality (AR) glasses to customize augmented table setup and lighting in a restaurant. The aim was to provide insights into AR usage in restaurants and contribute to existing research by introducing an extendable and versatile concept for scholars and restaurateurs. A controlled laboratory study, using a within-subjects design, was conducted to investigate the effects of a customizable augmented table setup and lighting on user experience (UX), perceived waiting time, psychological ownership and social acceptability. A simulated restaurant environment was created using 360° image in Virtual Reality (VR). The study implemented default and customizable table setup and lighting. Results from a paired samples t-test showed a statistically significant effect of table setup and lighting on pragmatic quality of UX, hedonic quality of UX, overall UX, Valence, Dominance, psychological ownership and affect. Furthermore, table setup had a significant effect on Arousal and perceived waiting time. Moreover, table setup significantly affected AR Interaction-, Isolation-, and Safety acceptability, while lighting only affected AR Interaction acceptability. Findings suggest that these investigated variables are worth considering for AR applications in a restaurant, especially when offering customizable augmented table setup and lighting.

Nowadays, Mixed Reality (MR) headsets are a game-changer for knowledge work. Unlike stationary monitors, MR headsets allow users to work with large virtual displays anywhere they wear the headset, whether in a professional office, a public setting like a cafe, or a quiet space like a library. This study compares four different layouts (eye level-close, eye level-far, below eye level-close, below eye level-far) of virtual displays regarding feelings of safety, perceived productivity, and social acceptability when working with MR in public. We test which layout is most preferred by users and seek to understand which factors affect users’ layout preferences. The aim is to derive useful insights for designing better MR layouts. A field study in a public library was conducted using a within-subject design. While the participants interact with a layout, they are asked to work on a planning task. The results from a repeated measure ANOVA show a statistically significant effect on productivity but not on safety and social acceptability. Additionally, we report preferences expressed by the users regarding the layouts and using MR in public.

One of the most promising technologies inside the Extended Reality (XR) spectrum is Augmented Reality. This technology is already in people’s pockets regarding Mobile Augmented Reality with their smartphones. The scientific community still needs answers about how humans could and should interact in environments where perceived stimuli are different from fully physical or digital circumstances. Moreover, it is still being determined if people accept these new technologies in different social environments and interaction settings or if some obstacles could exist. This paper explores the impact of the Social Environment and the Focus of social interaction on users while playing a location-based augmented reality game, measuring it with user experience and social acceptance indicators. An empirical study in a within-subject fashion was performed in different social environments and under different settings of social interaction focus with N = 28 participants compiling self-reported questionnaires after playing a Scavenger Hunt in Augmented Reality. The measures from two different Social Environments (Crowded vs. Uncrowded) resulted in statistically relevant mean differences with indicators from the Social Acceptability dimension. Moreover, the analyses show statistically relevant differences between the variances from different degrees of Social Interaction Focus with Overall Social Presence, Perceived Psychological Engagement, Perceived Attentional Engagement, and Perceived Emotional Contagion. The results suggest that a location-based AR game played in different social environments and settings can influence the user experience’s social dimension. Therefore, they should be carefully considered while designing immersive technological experiences in public spaces involving social interactions between players.

Objectively measuring affective states remains a recurring challenge in psychology and user experience research. A promising proxy for perceived arousal is the momentary assessment of the pupil diameter. However, the pupillometric signal is highly susceptible to variations in stimulus luminance, which can substantially confound the results. We introduce a method to improve the accuracy of pupillometric signals as an arousal predictor by eliminating frame-by-frame luminance variations in visual stimuli (i.e. scaling the brightness value for all pixels such that each frame has the same average luminance). This process was termed "equalization". We tested our approach in a study with 31 participants between the ages of 21 and 64. Our study focused on two metrics: (1) the perceived quality of the stimuli should not be affected by the process, and (2) the predictive power of the pupillometric signal for the perceived arousal should be greater for the equalized videos than for the non-equalized ones. We used a within-subjects design, where each participant rated their affective response and five distinct quality dimensions of four videos (two in the equalized and two in the non-equalized condition). Our results indicate that the equalization process does improve the predictive power of the pupillometric signal by a substantial amount. Statistical analyses also show that the participants rated all quality dimensions equivalently for an equalized and non-equalized variant of the same stimulus, with only small differences that are limited to a subset of the perceptual dimensions. While potentially limiting the efficacy of the process in some scenarios, the strengthened explanatory power of the pupillometric signal leaves room for a wide range of possible applications.

While current Quality of Experience (QoE) formation models recognize the impact of user states on perception, they often overlook the subjective nuances and individual variations in these experiences. We discuss the integration of both situation-dependent and independent user states, show how dependent states are influenced by varying multimedia content quality, and reflect back into the QoE assessment. To address this gap, we conducted a comprehensive within-subjects lab study (N=92) in a video-telephony setting, employing variables representative of both dependent and independent states, such as affective states, social relationship, sympathy, and bodily needs. Dependent states were assessed per trial and independent states before or after the experiment. Our study investigates two key questions: How does video-telephony call quality influence user-dependent states, and how do both dependent and independent states collectively impact QoE ratings? Our findings reveal a substantial influence of call quality on user emotional states, underscoring the importance of considering these factors in QoE assessments. Moreover, a structural equation model comparison favored the dependent+independent state structure model, highlighting the significant impact of both independent and preceding dependent states on QoE ratings. This research advances QoE models by incorporating a more nuanced mental model of user states, leading to more personalized and accurate multimedia assessment tools, potentially enhancing users’ degree of delight or annoyance and engagement.

This study investigates the feasibility of remote virtual reality (VR) studies conducted at home using VR headsets and video conferencing by deploying an experiment on emotion ratings. 20 participants used head-mounted displays to immerse themselves in 360° videos selected to evoke emotional responses. The research compares continuous ratings using a graphical interface to retrospective questionnaires on a digitized Likert Scale for measuring arousal and valence, both based on the self-assessment manikin (SAM). It was hypothesized that the two different rating methods would lead to significantly different values for both valence and arousal. The goal was to investigate whether continuous ratings during the experience would better reflect users’ emotions compared to the post-questionnaire by mitigating biases such as the peak-end rule. The results show significant differences with moderate to strong effect sizes for valence and no significant differences for arousal with low to moderate effect sizes. This indicates the need for further investigation of the methods used to assess emotion ratings in VR studies. Overall, this study is an example of a remotely conducted VR experiment, offering insights into methods for emotion elicitation in VR by varying the timing and interface of the rating.

In this paper, we present QoEXplainer, a QoE dashboard for supporting humans in understanding the internals of an explainable, data-driven Quality of Experience model. This tool leverages Large Language Models and the concept of Mediators to convey relevant explanations to the user in an understandable, chatbot-like fashion. For this purpose, our tool QoEXplainer integrates a data-driven video streaming QoE model and techniques from Explainable Artificial Intelligence. The resulting data-driven model explanations are illustrated in the dashboard and users can interact with the chatbot to ask questions about the data and QoE model and control the dashboard to enhance model understanding. With this hybrid demo, we aim to conduct a live study at QoMEX 2024 to evaluate Mediators in the context of (data-driven) QoE modelling with domain experts.

The EyeSight feature, introduced with the new Apple Vision Pro XR headset, promises to revolutionize user interaction by simulating real human eye expressions on a digital display. This feature could enhance XR devices’ social acceptability and social presence when communicating with others outside the XR experience. In this pilot study, we explore the implications of the EyeSight feature by examining social acceptability, social presence, emotional responses, and technology acceptance. Eight participants engaged in conversational tasks in three conditions to contrast experiencing the Apple Vision Pro with EyeSight, the Meta Quest 3 as a reference XR headset, and a face-to-face setting. Our preliminary findings indicate that while the EyeSight feature improves perceptions of social presence and acceptability compared to the reference headsets, it does not match the social connectivity of direct human interactions.

Relation extraction (RE) is a fundamental task in information extraction, whose extension to multilingual settings has been hindered by the lack of supervised resources comparable in size to large English datasets such as TACRED (Zhang et al., 2017).To address this gap, we introduce the MultiTACRED dataset, covering 12 typologically diverse languages from 9 language families, which is created by machine-translating TACRED instances and automatically projecting their entity annotations.We analyze translation and annotation projection quality, identify error categories, and experimentally evaluate fine-tuned pretrained mono-and multilingual language models in common transfer learning scenarios.Our analyses show that machine translation is a viable strategy to transfer RE instances, with native speakers judging more than 83% of the translated instances to be linguistically and semantically acceptable.We find monolingual RE model performance to be comparable to the English original for many of the target languages, and that multilingual models trained on a combination of English and target language data can outperform their monolingual counterparts.However, we also observe a variety of translation and annotation projection errors, both due to the MT systems and linguistic features of the target languages, such as pronoun-dropping, compounding and inflection, that degrade dataset quality and RE model performance.

Estimating the quality of remote speech communication is a complex task influenced by the speaker, transmission channel, and listener. For example, the degradation of transmission quality can increase listeners’ cognitive load, which can influence the overall perceived quality of the conversation. This paper presents a framework that isolates quality-dependent changes and controls most outside influencing factors like personal preference in a simulated conversational environment. The performed statistical analysis finds significant relationships between stimulus quality and the listener’s valence and personality (agreeableness and openness) and, similarly, between the perceived task load during the listening task and the listener’s personality and frustration intolerance. The machine learning model of the task load prediction improves the correlation coefficients from 0.48 to 0.76 when listeners’ individuality is considered. The proposed evaluation framework and results pave the way for personalized audio quality assessment that includes speakers’ and listeners’ individuality beyond conventional channel modeling.

Preservation of private user data is of paramount importance for high Quality of Experience (QoE) and acceptability, particularly with services treating sensitive data, such as IT-based health services. Whereas anonymization techniques were shown to be prone to data re-identification, synthetic data generation has gradually replaced anonymization since it is relatively less time and resource-consuming and more robust to data leakage. Generative Adversarial Networks (GANs) have been used for generating synthetic datasets, especially GAN frameworks adhering to the differential privacy phenomena. This research compares state-of-the-art GAN-based models for synthetic data generation to generate time-series synthetic medical records of dementia patients which can be distributed without privacy concerns. Predictive modeling, autocorrelation, and distribution analysis are used to assess the Quality of Generating (QoG) of the generated data. The privacy preservation of the respective models is assessed by applying membership inference attacks to determine potential data leakage risks. Our experiments indicate the superiority of the privacy-preserving GAN (PPGAN) model over other models regarding privacy preservation while maintaining an acceptable level of QoG. The presented results can support better data protection for medical use cases in the future.

Although there are several aspects that might impact one's feeling of immersion or presence in a virtual environment, noises in the actual world are often overlooked. Distractions (such as noises or lights) may have an effect on a person's mental or emotional state in real life, and these interruptions can be disturbing while trying to focus on work. With a steady stream of distractions, one may eventually lose interest and make mistakes or stop the activity. The goal of this research is to better understand the impact of various ambient sounds on presence, immersion, and decision-making by using virtual reality and audio rendering technology. This study's sound sources were obtained and recorded from several real-life sources before being mixed into the virtual world. These audio recordings have been shown to influence a person's decision-making skills by generating a sudden shift in different emotional states (such as annoyance and anger). In certain circumstances, binaural recording is used to provide the listener with a 3D stereo sound experience. They were triggered at random intervals while the user was in the virtual environment. Based on the findings of the study presented in this paper, some types of noises, such as continuous and impulsive noise, have a negative impact on user experience by reducing immersion in the virtual reality environment. It was also discovered that the more interaction there was, the more mental work was required to complete the activities.

This manuscript investigates the suitability of video-based conversation simulations for studying human reactions to quality degradations, focusing on facial configurations as a proxy for QoE. We analyze data from two distinct studies: a video-simulated video-telephony scenario using the storytime dataset, where participants passively watched videos, and a second study involving real conversations between participants. In both studies, facial features were continuously recorded using Apple's iOS ARKit API. We identify a factor structure of facial features that significantly relates to participants' QoE ratings in the first study and validate its robustness by replicating it in the second, independent study. Our findings suggest statistically significant estimations of QoE ratings across both paradigms, demonstrating the suitability of passive conversation simulations for studying human reactions to quality degradation. We assess the value of the proposed approach at its present stage and conclude that it can be a valuable tool when used in conjunction with other methods, as its predictive capabilities are still not robust enough to rely solely on this analysis technique.

The subjective quality of experience (QoE) in multimedia contexts is influenced by various factors, including individual differences among raters and experimental setups. While the latter has been extensively studied, the former remains relatively unexplored. This paper investigates the impact of hangriness - a mental state of irritability and frustration caused by hunger - on QoE ratings. In our analysis, hangriness appears to be prevalent in a specific time interval, where individuals have not consumed any food between five and eleven hours. Our analysis, comprising ratings from 100 participants, reveals a significant, non-linear effect of hangriness on QoE ratings, specifically for multimedia stimuli with subpar quality. Participants in the hangry state rated such stimuli significantly worse compared to those who had eaten recently or abstained from food for more than eleven hours. Interestingly, this effect was not observed for high-quality multimedia content. Our findings highlight the importance of considering individual differences, such as hangriness, in QoE research, as they can significantly impact subjective ratings. Further research is needed to corroborate these results and explore other factors that may influence QoE ratings. This work contributes to a better understanding of individual variability in multimedia quality perception and provides insights for designing more reliable QoE assessment methods.

Nowadays, Augmented Reality (AR) is available on almost all smartphones creating some exciting interaction opportunities but also challenges. For example, already after the famous AR app Pokemon GO was released in July 2016, numerous accidents related to the use of the app were reported by users. At the same time, the spread of AR can be noticed in the tourism industry, enabling tourists to explore their surroundings in new ways but also exposing them to safety issues. This preliminary study explores users' sense of safety when manipulating the amount and UI elements visualization parameters of Point of Interest (POI) markers in a developed AR application. The results show that the amount of POI markers that are displayed is significant for participants' sense of safety. The influence of manipulating UI elements in terms of “transparency”, “color”, and “size” cannot be proven. Nevertheless, most tested people stated that manipulating transparency and size somehow influences their sense of safety, so a closer look at them should be taken in future studies.

Nowadays, many apps use location data to estimate the user's behavior for targeted advertising, predicting significant locations, personal preferences, state of health, and sports activities. Users of location-based services are often left with no other choice than to accept or reject location tracking when they want to use various applications. Especially, users with higher privacy concerns may reduce the frequency of location tracking by turning it off in the settings. However, most users are unaware that many applications installed on their phones are continuously tracking them. Therefore, this study attempts to answer how (obviously) being tracked over one-week influences a user's privacy concerns. The study was implemented using an iOS app, which participants could install on their smartphones. Moreover, over one week, the participants were requested to answer daily mini-questionnaires about how much they would be willing to pay for the protection of their location information on a monthly basis and how much money they were willing to accept in exchange for their location information. Hereby, the context was an important criterion to determine how the monetary values vary among different location types for, among others, home location, work location, and meeting family and friends. The participants (N=51) interacted with the app on a daily basis by filling out various daily mini-surveys based on their significant locations visited. The results show a significant difference between the monetary valuating of willingness to pay and to accept for all location types except work location and sharing scenarios contributing to further empirical evidence for the endowment effect. The obvious fact of continuously being tracked did not increase the privacy concern of participants.

Android applications request specific permissions from users during the installations to perform required functionalities by accessing system resources and personal information. Usually, users must approve the permissions requested by applications (apps) during the installation process and before the apps can collect privacy- or security-relevant information. However, recent studies have shown that users are overwhelmed with the information provided in privacy policies and do not understand permission requests and which functionalities are necessary for certain applications. Hereby, the collection of personal information remains mostly hidden, as the task of verifying to which information different apps have access to can be very complicated. Therefore, it is necessary to develop frameworks and apps that enable the user to perform informed decisions about apps’ run-time permission access to facilitate the control over sensitive information collected by various apps on smartphones. In this work, we conducted an online study with 70 participants who interacted with a mockup app that enables advanced control over permission requests. The selected permissions are based on the apps’ run-time permission access patterns and explanations, and commonly known visual cues are used to facilitate the user’s understanding and privacy-conscious decision making. Furthermore, the effects of perceived control over information sharing and privacy awareness are examined in combination with the permission manager mockup app to investigate if increased control over information sharing increases general privacy awareness.

With the advances in speech communication systems such as online conferencing applications, we can seamlessly work with people regardless of where they are. However, during online meetings, speech quality can be significantly affected by background noise, reverberation, packet loss, network jitter, etc. Because of its nature, speech quality is traditionally assessed in subjective tests in laboratories and lately also in crowdsourcing following the international standards from ITU-T Rec. P.800 series. However, those approaches are costly and cannot be applied to customer data. Therefore, an effective objective assessment approach is needed to evaluate or monitor the speech quality of the ongoing conversation. The ConferencingSpeech 2022 challenge targets the non-intrusive deep neural network models for the speech quality assessment task. We open-sourced a training corpus with more than 86K speech clips in different languages, with a wide range of synthesized and live degradations and their corresponding subjective quality scores through crowdsourcing. 18 teams submitted their models for evaluation in this challenge. The blind test sets included about 4300 clips from wide ranges of degradations. This paper describes the challenge, the datasets, and the evaluation methods and reports the final results.

Vocabulary learning is vital to foreign language learning. Correct and adequate feedback is essential to successful and satisfying vocabulary training. However, many vocabulary and language evaluation systems perform on simple rules and do not account for real-life user learning data. This work introduces Multi-Language Vocabulary Evaluation Data Set (MuLVE), a data set consisting of vocabulary cards and real-life user answers, labeled indicating whether the user answer is correct or incorrect. The data source is user learning data from the Phase6 vocabulary trainer. The data set contains vocabulary questions in German and English, Spanish, and French as target language and is available in four different variations regarding pre-processing and deduplication. We experiment to fine-tune pre-trained BERT language models on the downstream task of vocabulary evaluation with the proposed MuLVE data set. The results provide outstanding results of > 95.5 accuracy and F2-score. The data set is available on the European Language Grid.

This paper presents a fine-grained test suite for the language pair German–English. The test suite is based on a number of linguistically motivated categories and phenomena and the semi-automatic evaluation is carried out with regular expressions. We describe the creation and implementation of the test suite in detail, providing a full list of all categories and phenomena. Furthermore, we present various exemplary applications of our test suite that have been implemented in the past years, like contributions to the Conference of Machine Translation, the usage of the test suite and MT outputs for quality estimation, and the expansion of the test suite to the language pair Portuguese–English. We describe how we tracked the development of the performance of various systems MT systems over the years with the help of the test suite and which categories and phenomena are prone to resulting in MT errors. For the first time, we also make a large part of our test suite publicly available to the research community.

In this work, we present the first corpus for German Adverse Drug Reaction (ADR) detection in patient-generated content. The data consists of 4,169 binary annotated documents from a German patient forum, where users talk about health issues and get advice from medical doctors. As is common in social media data in this domain, the class labels of the corpus are very imbalanced. This and a high topic imbalance make it a very challenging dataset, since often, the same symptom can have several causes and is not always related to a medication intake. We aim to encourage further multi-lingual efforts in the domain of ADR detection and provide preliminary experiments for binary classification using different methods of zero- and few-shot learning based on a multi-lingual model. When fine-tuning XLM-RoBERTa first on English patient forum data and then on the new German data, we achieve an F1-score of 37.52 for the positive class. We make the dataset and models publicly available for the community.

For different reasons, text can be difficult to read and understand for many people, especially if the text’s language is too complex. In order to provide suitable text for the target audience, it is necessary to measure its complexity. In this paper we describe subjective experiments to assess the readability of German text. We compile a new corpus of sentences provided by a German IT service provider. The sentences are annotated with the subjective complexity ratings by two groups of participants, namely experts and non-experts for that text domain. We then extract an extensive set of linguistically motivated features that are supposedly interacting with complexity perception. We show that a linear regression model with a subset of these features can be a very good predictor of text complexity.

With the rapid increase of video streaming content, high-quality video quality metrics, mainly signal-based video quality metrics, are emerging, notably VMAF, SSIMPLUS, and AVQM. Besides signal-based video quality metrics, within the standardization body, ITU-T Study Group 12, two well-known bitstream-based video quality metrics are developed named P.1203 and P.1204.3. Due to the low complexity and low level of access to the bitstream data, these models gained attention from network providers and service providers. In this paper, we proposed a new bitstream-based model named Deep-BVQM, which outperforms the standard models on the tested datasets. While the model comes with slightly higher computational complexity, it offers a frame-level quality prediction which is essential diagnostic information for some video streaming services such as cloud gaming. Deep-BVQM is developed in two layers; first, the frame quality was predicted using a lightweight CNN model. Next, the latent features of the CNN were used to train an LSTM network to predict the video quality in a short-term duration.

Subjective assessments are a key component of speech quality research. Traditionally, the assessments are conducted in laboratories in controlled conditions and following international standards like ITU-T Rec.P.800. However, even before the current pandemic, more speech quality research used crowdsourcing-based approaches for collecting subjective ratings. Crowdsourcing allows researchers to collect data even without a dedicated test laboratory, to collect data from a huge and diverse group of participants, and to perform the assessment in various real-life settings. Still, this approach raises questions about the reliability and validity of the subjective ratings, especially when comparing the ratings with data collected in standardized procedures. One step to approach these challenges was the development of the ITU-T Rec.P.808 standard. This standard helps practitioners implement best practices from speech quality studies and crowdsourcing studies in their crowdsourced speech quality assessments. However, even with the ITU-T Rec.P.808 in action, it is unclear how much background knowledge is necessary to successfully “implement” this standard. Therefore, this paper aims to assess the data quality differences between two P.808 implementations. One implementation is from a co-author of the P.808 standard, and the other is a researcher with only a little background in crowdsourcing and speech quality assessments. Both implementations are used in a large-scale crowdsourcing study with about two hundred users from Amazon Mechanical Turk. The collected ratings are compared to gold-standard data from a certified laboratory. Also, the two implementations are compared to analyze whether they lead to the same conclusions. The results show that both implementations correlate strongly with the laboratory and with each other. Thus, suggesting that the ITU-T Rec.P.808 is robust enough to be implemented by non-experts in speech evaluation or crowdsourcing.

To study people's natural behavior during different conditions of audiovisual quality, we usually invite people into a lab and let them talk to each other. In such conversation settings, not only the media quality impacts the quality perception, but, e.g., social aspects of a real conversation are reflected by individual conversational and rating behavior. Hence, to study quality perception in conversational settings, we try to create an environment that isolates the media quality from such outside factors and is consistent for each participant in the lab. Therefore, we created a dataset of simulated videotelephony clips to act as stimuli in quality perception research. The dataset consists of four different stories in the German language that are told through ten consecutive parts, each about 10 seconds long. Each of these parts is available in four different quality levels, ranging from perfect to stalling. All clips (FullHD, H.264 / AAC) are actual recordings from end-user video-conference software to ensure ecological validity and realism of quality degradation. To ensure consistency among different clips of the same quality level, each video has been scored using VMAF and POLQA and selected to match predefined selection criteria. To analyze the perceived quality of the clips, we conducted a user study (N=25) and evaluated perceived quality, interest in the stories, and speaker engagement. Results validate the consistency of the quality levels of the video clips. Apart from a detailed description of the methodological approach, we contribute the entire stimuli dataset containing 160 videos and all rating scores for each file.

Over the past five years, the interest in Augmented Reality (AR) technologies has significantly increased. Different industries adopted it, such as gaming, tourism, e-commerce, and entertainment, to name a few. Due to its non-fully immersive property, easy implementation, and compatibility with the most recent smartphones, AR seems to be suitable for daily usage. This poses a significant challenge in creating a good User Experience (UX), as this kind of technology needs to be designed for private scenarios and public ones. In its context, it is essential to understand how users are influenced in their social interactions while using AR. Through the use of humanoid Virtual Characters, this paper is intended to better the understanding of how people perceive personal space in AR. By varying Virtual Character Gender (male, female) and its Placement Modality (predefined, dynamic distance) in the real environment, the effects on UX, Emotions, and Social Presence are investigated. As Personal Space is a dynamic psychological construct, which could depend on individual factors, this paper further investigates some possible co-variant effects of users' preferences towards a female or a male character based on their Sexual Attraction. Results show that Placement Modality significantly affects Personal Space boundaries in terms of position, width, and hedonic quality of the experience. Moreover, an effect of Virtual Character Gender was found on users' Personal Space inner boundary, and Social Presence in its emotional contagion dimension. Finally, an effect of Sexual Attraction on UX, Social Presence in its emotional contagion dimension, and dominance were discovered.

Chatbots are increasingly used to automate operational processes in customer service.However, most chatbots lack adaptation towards their users which may results in an unsatisfactory experience.Since knowing and meeting personal preferences is a key factor for enhancing usability in conversational agents, in this study we analyze an adaptive conversational agent that can automatically adjust according to a user's personality type carefully excerpted from the Myers-Briggs type indicators.An experiment including 300 crowd workers examined how typifications like extroversion/introversion and thinking/feeling can be assessed and designed for a conversational agent in a job recommender domain.Our results validate the proposed design choices, and experiments on a user-matched personality typification, following the so-called law of attraction rule, show a significant positive influence on a range of selected usability criteria such as overall satisfaction, naturalness, promoter score, trust and appropriateness of the conversation.

Abstract Gaming video streaming services are growing rapidly due to new services such as passive video streaming of gaming content, e.g. Twitch.tv, as well as cloud gaming, e.g. Nvidia GeForce NOW and Google Stadia. In contrast to traditional video content, gaming content has special characteristics such as extremely high and special motion patterns, synthetic content and repetitive content, which poses new opportunities for the design of machine learning-based models to outperform the state-of-the-art video and image quality approaches for this special computer generated content. In this paper, we train a Convolutional Neural Network (CNN) based on an objective quality model, VMAF, as ground truth and fine-tuned it based on subjective image quality ratings. In addition, we propose a new temporal pooling method to predict gaming video quality based on frame-level predictions. Finally, the paper also describes how an appropriate CNN architecture can be chosen and how well the model performs on different contents. Our result shows that among four popular network architectures that we investigated, DenseNet performs best for image quality assessment based on the training dataset. By training the last 57 convolutional layers of DenseNet based on VMAF values, we obtained a high performance model to predict VMAF of distorted frames of video games with a Spearman’s Rank correlation (SRCC) of 0.945 and Root Mean Score Error (RMSE) of 7.07 on the image level, while achieving a higher performance on the video level leading to a SRCC of 0.967 and RMSE of 5.47 for the KUGVD dataset. Furthermore, we fine-tuned the model based on subjective quality ratings of images from gaming content which resulted in a SRCC of 0.93 and RMSE of 0.46 using one-hold-out cross validation. Finally, on the video level, using the proposed pooling method, the model achieves a very good performance indicated by a SRCC of 0.968 and RMSE of 0.30 for the used gaming video dataset.

Software developers build complex systems using plenty of third-party libraries. Documentation is key to understand and use the functionality provided via the libraries’ APIs. Therefore, functionality is the main focus of contemporary API documentation, while cross-cutting concerns such as security are almost never considered at all, especially when the API itself does not provide security features. Documentations of JavaScript libraries for use in web applications, e.g., do not specify how to add or adapt a Content Security Policy (CSP) to mitigate content injection attacks like Cross-Site Scripting (XSS). This is unfortunate, as security-relevant API documentation might have an influence on secure coding practices and prevailing major vulnerabilities such as XSS. For the first time, we study the effects of integrating security-relevant information in non-security API documentation. For this purpose, we took CSP as an exemplary study object and extended the official Google Maps JavaScript API documentation with security-relevant CSP information in three distinct manners. Then, we evaluated the usage of these variations in a between-group eye-tracking lab study involving N=49 participants. Our observations suggest: (1) Developers are focused on elements with code examples. They mostly skim the documentation while searching for a quick solution to their programming task. This finding gives further evidence to results of related studies. (2) The location where CSP-related code examples are placed in non-security API documentation significantly impacts the time it takes to find this security-relevant information. In particular, the study results showed that the proximity to functional-related code examples in documentation is a decisive factor. (3) Examples significantly help to produce secure CSP solutions. (4) Developers have additional information needs that our approach cannot meet. Overall, our study contributes to a first understanding of the impact of security-relevant information in non-security API documentation on CSP implementation. Although further research is required, our findings emphasize that API producers should take responsibility for adequately documenting security aspects and thus supporting the sensibility and training of developers to implement secure systems. This responsibility also holds in seemingly non-security relevant contexts.

Speech communication systems based on Voice-over-IP technology are frequently used by native as well as non-native speakers of a target language, e.g. in international phone calls or telemeetings. Frequently, such calls also occur in a noisy environment, making noise suppression modules necessary to increase perceived quality of experience. Whereas standard tests for assessing perceived quality make use of native listeners, we assume that noise-reduced speech and residual noise may affect native and non-native listeners of a target language in different ways. To test this assumption, we report results of two subjective tests conducted with English and German native listeners who judge the quality of speech samples recorded by native English, German, and Mandarin speakers, which are degraded with different background noise levels and noise suppression effects. The experiments were conducted following the standardized ITU-T Rec. P.835 approach, however implemented in a crowdsourcing setting according to ITU-T Rec. P.808. Our results show a significant influence of language on speech signal ratings and, consequently, on the overall perceived quality in specific conditions.

In this paper, we present an update to the NISQA speech quality prediction model that is focused on distortions that occur in communication networks. In contrast to the previous version, the model is trained end-to-end and the time-dependency modelling and time-pooling is achieved through a Self-Attention mechanism. Besides overall speech quality, the model also predicts the four speech quality dimensions Noisiness, Coloration, Discontinuity, and Loudness, and in this way gives more insight into the cause of a quality degradation. Furthermore, new datasets with over 13,000 speech files were created for training and validation of the model. The model was finally tested on a new, live-talking test dataset that contains recordings of real telephone calls. Overall, NISQA was trained and evaluated on 81 datasets from different sources and showed to provide reliable predictions also for unknown speech samples. The code, model weights, and datasets are open-sourced.

Mobile gaming has the largest market shares of all gaming domains, accounting for an estimated $ 77.2 billion in 2020. In recent times, one can witness an increase in highly interactive mobile online games. However, the gaming Quality of Experience (QoE) can be strongly influenced by network degradations, concretely by delay and packet loss. Thus, network providers need to ensure fast and reliable connections between the gaming servers and the users' clients. To maintain a satisfying user experience, QoE prediction models are fundamental. Aiming at the development of such a model, a detailed parameter space consisting of various delay and packet loss conditions will be investigated in this paper. Here, especially the importance of jitter is of interest. Next, it will be examined whether a recently published opinion model for cloud gaming, the ITU-T Rec. G.1072, can also be used for online mobile gaming. Finally, a new proposal for a model targeting online mobile gaming services will be presented and evaluated concerning its performance.

Humans are fascinated by video games for many years, which intrinsically immerse players in their virtual environments. Apart from the challenges offered by steadily new game concepts, aesthetically pleasing environments and characters, stories, and sound effects are highly important for the player experience. Recently, the new concept of cloud gaming, which offers users to play games executed on a cloud server remotely, is becoming increasingly popular. While cloud gaming offers many advantages, additional audio and video streaming poses many technical challenges. To ensure a satisfying Quality of Experience (QoE) of their customers, all stakeholders are interested in finding out which aspects of a gaming experience are of high importance and how resources can be optimally allocated. However, gaming QoE is a multidimensional construct including hedonic and pragmatic aspects and could be strongly influenced by interaction quality, video quality, and audio quality. While the impact of network and video encoding parameters of cloud gaming services was investigated in much detail in recent years, not many studies about the effect of audio quality on gaming experiences are available. Thus, in this paper, the impact of audio quality on gaming experience under different bitrate and packet loss conditions using two popular games is investigated. Therefore, a subjective experiment adhering to the ITU-T Rec. P.809 was conducted. The results show a significant impact of packet loss on audio quality and the overall gaming QoE. However, no significant effect of the bitrate, which was reduced to a minimum of 32 kbps, was revealed. Additionally, the influence of audio quality on gaming QoE was stronger for a game, which contained mainly diegetic effect sounds compared to a game containing few diegetic sounds and affect sound as background music.

The past few years has witnessed a promising surge in immersive media technologies, however, a widespread adoption is still some years away. Recently, virtual reality (VR) head-mounted displays with inside-out tracking and hand-tracking are becoming popular in various fields. Representing users' hands and fingers in VR is crucial for many tasks. This is because hand-tracking enables a more natural, direct, interaction with a virtual environment (VE) which, compared to traditional handheld controllers, improves the VR immersive experience. This paper investigates user-perceived quality of experience for a reach-pick-place task inside VR. It presents a performance assessment methodology based on in-game analytics to evaluate user experience for different VR interaction types. We ran a controlled experiment (N=33) comparing two commodities: controller vs. hand-tracking. Results indicate a significant difference between the two data sets. The performance analytics show lower play-durations and trigger frequencies for the handheld controller compared to hand-tracking. The perceived mental workload for the tasks was also evaluated using Rated Scale Mental Effort (RSME). The reported scores were higher when using hand-tracking. The results help our understanding of the two interaction modalities in terms of their viability for naturalistic experiences akin to real-world scenarios.

Subjective speech quality scores needed to train models for the automatic evaluation of telecommunication systems have generally been collected by conducting demanding laboratory tests. Alternatively, crowdsourcing has emerged as a valid method to address user-centered studies to a large pool of users over the Internet. However, crowdsourcing users often do not follow the instructions and may execute the assigned task in noisy environments. The validity of the data collected in a disturbed environment is questionable, especially in speech quality assessment and other audio-related tasks. This work investigates the applicability of different ensemble-based and non-linear models to correct the bias found in speech quality ratings given to a German speech dataset in noisy crowdsourcing environments. Such a model would help to avoid throwing away quality scores that could be used. The model was trained with data collected in a speech quality assessment study conducted in a simulated crowdsourcing environment in the laboratory. Two groups of listeners rated the quality of speech stimuli in the presence of environmental noise at different levels. The noise under test was street traffic, and the levels ranged from 36dBA to 65. 5dBA. A fine-tuned gradient boosting regressor yielded the best results with a R2score of 0.90 and RMSE of 0.416.

The quality of the speech signal is essential as it influences the user experience of voiced interactive systems. Speech quality studies have traditionally been conducted in restricted laboratory rooms with professional audio equipment. Nowadays, crowd-sourcing represents a valid alternative for the rapid assessment of large speech databases at a fraction of the cost and time of traditional laboratory practices. However, crowd-sourcing users perform tasks in an unsupervised manner. Thus, it is challenging to control whether their skills match those of the study's intended audience. This is important in speech quality evaluations as some listeners may end up participating in a listening test of a target language other than their mother tongue. This paper investigates the influence of assessing the quality of a German speech dataset with native English and Spanish speakers. To this end, three studies were conducted in crowdsourcing where listeners evaluated the quality of speech stimuli following the ITU-T Rec. P.808. A strong Pearson correlation and low RMSE was found between the laboratory ratings and the scores collected in all crowdsourcing studies, despite the listeners' mother tongue. Still, a bias was seen between the mean opinion scores from the German crowd-workers and the native English and Spanish speakers. The non-German participants tended to overestimate the quality of the speech stimuli.

The ground truth used for training image, video, or speech quality prediction models is based on the Mean Opinion Scores (MOS) obtained from subjective experiments. Usually, it is necessary to conduct multiple experiments, mostly with different test participants, to obtain enough data to train quality models based on machine learning. Each of these experiments is subject to an experiment-specific bias, where the rating of the same file may be substantially different in two experiments (e.g. depending on the overall quality distribution). These different ratings for the same distortion levels confuse neural networks during training and lead to lower performance. To overcome this problem, we propose a bias-aware loss function that estimates each dataset's biases during training with a linear function and considers it while optimising the network weights. We prove the efficiency of the proposed method by training and validating quality prediction models on synthetic and subjective image and speech quality datasets.

Traditionally, Quality of Experience (QoE) for a communication system is evaluated through a subjective test. The most common test method for speech QoE is the Absolute Category Rating (ACR), in which participants listen to a set of stimuli, processed by the underlying test conditions, and rate their perceived quality for each stimulus on a specific scale. The Comparison Category Rating (CCR) is another standard approach in which participants listen to both reference and processed stimuli and rate their quality compared to the other one. The CCR method is particularly suitable for systems that improve the quality of input speech. This paper evaluates an adaptation of the CCR test procedure for assessing speech quality in the crowdsourcing set-up. The CCR method was introduced in the ITU-T Rec. P.800 for laboratory-based experiments. We adapted the test for the crowdsourcing approach following the guidelines from ITU-T Rec. P.800 and P.808. We show that the results of the CCR procedure via crowdsourcing are highly reproducible. We also compared the CCR test results with widely used ACR test procedures obtained in the laboratory and crowdsourcing. Our results show that the CCR procedure in crowdsourcing is a reliable and valid test method.

Nowadays, Virtual Reality (VR) technology can be potentially used everywhere through wearable head-mounted displays. Nevertheless, it is still uncommon to see VR devices used in public settings. In these contexts, unaware bystanders in the surroundings might influence the User Experience (UX) and create concerns about the social acceptability of this technology. The user acts in a Social Environment (SE), characterized by surrounding people's number, proximity, and behavior. Simultaneously, VR applications often require a different degree of interactivity concerning body movements and controllers interaction. In this paper, the influence of Social Environments, and degree of interactivity on User Experience and social acceptability is investigated. Four Social Environments were simulated employing 360° Videos, and two VR games developed with two levels of interactivity. Results showed a statistically significant influence of Social Environments on Overall UX as well as Public VR, Interaction, Isolation, Privacy and Safety acceptability, and of the degree of interactivity on Presence, Valence, Arousal, Overall UX, UX Hedonic quality, and Safety acceptability. Findings indicate that Social Environments and degree of interactivity should be taken into account while designing VR applications.

Advancing technology and higher availability of Virtual Reality (VR) devices sparked its application in various research fields. For instance, health-related research showed that simulated nature environments in VR could reduce arousal and increase valence levels. This study investigates how the amount of possible interactivity influences the presence in nature environments and consequences on arousal and valence. After inducing fear (high arousal and low valence) through a VR-horror game, it was tested how participants recovered if they played a VR-nature game with either no, limited, or extensive interaction. The horror game proved to be a valid stimulus for inducing high arousal and low valence with a successful manipulation check. Igroup presence questionnaire (IPQ) scores showed that more interaction with the virtual environment increases spatial presence. A beneficial effect of experiencing nature can also be concluded. Results from the Self-Assessment Manikin questionnaire (SAM) scores for valence indicate a significant increase in the conditions with extensive and limited interaction compared to the control group. The VR Nature experience did significantly decrease arousal and increase valence compared to the post-horror game ratings. The physiological responses support this finding. These results can increase the effectiveness of health-related VR-applications to elevate mood levels by either implementing plenty of interactions and consequently increasing spatial presence or doing the opposite and leaving out any interactions at all.

Telecommunication service providers have to guarantee acceptable speech quality during a phone call to avoid a negative impact on the users’ quality of experience. Currently, there are different speech quality assessment methods. ITU-T Recommendation G.107 describes the E-model algorithm, which is a computational model developed for network planning purposes focused on narrowband (NB) networks. Later, ITU-T Recommendations G.107.1 and G.107.2 were developed for wideband (WB) and fullband (FB) networks. These algorithms use different impairment factors, each one related to different speech communication steps. However, the NB, WB, and FB E-model algorithms do not consider wireless techniques used in these networks, such as Multiple-Input-Multiple-Output (MIMO) systems, which are used to improve the communication system robustness in the presence of different types of wireless channel degradation. In this context, the main objective of this study is to propose a general methodology to incorporate wireless network parameters into the NB and WB E-model algorithms. To accomplish this goal, MIMO and wireless channel parameters are incorporated into the E-model algorithms, specifically into the $I_{e,eff}$ and $I_{e,eff,WB}$ impairment factors. For performance validation, subjective tests were carried out, and the proposed methodology reached a Pearson correlation coefficient (PCC) and a root mean square error (RMSE) of 0.9732 and 0.2351, respectively. It is noteworthy that our proposed methodology does not affect the rest of the E-model input parameters, and it intends to be useful for wireless network planning in speech communication services.

The creation of the NPTEL platform in India has led to a vast population of engineering students getting access to quality online content for Signal Processing. These courses are globally accessible, free of cost, and also provide a means of obtaining certificates of proficiency by taking a proctored examination. Recently, a European Union funded project, MIELES, has supported the activity of creating online courses in the fields related to Signal Processing. This paper presents the details and experiences of creating course content and presents guidelines for prospective content creators.

In this paper, we present a full-reference speech quality prediction model with a deep learning approach. The model determines a feature representation of the reference and the degraded signal through a Siamese recurrent convolutional network that shares the weights for both signals as input. The resulting features are then used to align the signals with an attention mechanism and are finally combined to estimate the overall speech quality. The proposed network architecture represents a simple solution for the time-alignment problem that occurs for speech signals transmitted through Voice-Over-IP networks and shows how the clean reference signal can be incorporated into speech quality models that are based on end-to-end trained neural networks.

In this paper, we present a new objective prediction model for synthetic speech naturalness. It can be used to evaluate Text-To-Speech or Voice Conversion systems and works language independently. The model is trained end-to-end and based on a CNN-LSTM network that previously showed to give good results for speech quality estimation. We trained and tested the model on 16 different datasets, such as from the Blizzard Challenge and the Voice Conversion Challenge. Further, we show that the reliability of deep learning-based naturalness prediction can be improved by transfer learning from speech quality prediction models that are trained on objective POLQA scores. The proposed model is made publicly available and can, for example, be used to evaluate different TTS system configurations.

Question classification is an important component of Question Answering Systems responsible for identifying the type of an answer a particular question requires. For instance, “Who is the prime minister of the United Kingdom?” demands a name of a PERSON, while “When was the queen of the United Kingdom born?” entails a DATE. This work makes an extensible review of the most recent methods for Question Classification, taking into consideration their applicability in low-resourced languages. First, we propose a manual classification of the current state-of-the-art methods in four distinct categories: low, medium, high, and very high level of dependency on external resources. Second, we applied this categorization in an empirical comparison in terms of the amount of data necessary for training and performance in different languages. In addition to complementing earlier works in this field, our study shows a boost on methods relying on recent language models, overcoming methods not suitable for low-resourced languages.

The intrinsic and extrinsic quality evaluation is an essential part of the summary evaluation methodology usually conducted in a traditional controlled laboratory environment. However, processing large text corpora using these methods reveals expensive from both the organizational and the financial perspective. For the first time, and as a fast, scalable, and cost-effective alternative, we propose micro-task crowdsourcing to evaluate both the intrinsic and extrinsic quality of query-based extractive text summaries. To investigate the appropriateness of crowdsourcing for this task, we conduct intensive comparative crowdsourcing and laboratory experiments, evaluating nine extrinsic and intrinsic quality measures on 5-point MOS scales. Correlating results of crowd and laboratory ratings reveals high applicability of crowdsourcing for the factors overall quality, grammaticality, non-redundancy, referential clarity, focus, structure & coherence, summary usefulness, and summary informativeness. Further, we investigate the effect of the number of repetitions of assessments on the robustness of mean opinion score of crowd ratings, measured against the increase of correlation coefficients between crowd and laboratory. Our results suggest that the optimal number of repetitions in crowdsourcing setups, in which any additional repetitions do no longer cause an adequate increase of overall correlation coefficients, lies between seven and nine for intrinsic and extrinsic quality factors.

Many people share information in social media or forums, like food they eat, sports activities they do or events which have been visited. Information we share online unveil directly or indirectly information about our lifestyle and health situation. Particularly when text input is getting longer or multiple messages can be linked to each other. Those information can be then used to detect possible risk factors of diseases or adverse drug reactions of medications. However, as most people are not medical experts, language used might be more descriptive rather than the precise medical expression as medics do. To detect and use those relevant information, laymen language has to be translated and/or linked against the corresponding medical concept. This work presents baseline data sources in order to address this challenge for German language. We introduce a new dataset which annotates medical laymen and technical expressions in a patient forum, along with a set of medical synonyms and definitions, and present first baseline results on the data.

The streaming of gaming content, both passive and interactive, has increased manifolds in recent years. Gaming contents bring with them some peculiarities which are normally not seen in traditional 2D videos, such as the artificial and synthetic nature of contents or repetition of objects in a game. In addition, the perception of gaming content by the user is different from that of traditional 2D videos due to its pecularities and also the fact that users may not often watch such content. Hence, it becomes imperative to evaluate whether the existing video quality models usually designed for traditional 2D videos are applicable to gaming content. In this paper, we evaluate the applicability of the recently standardized bitstream-based video-quality model ITU-T P.1204.3 on gaming content. To analyze the performance of this model, we used 4 different gaming datasets (3 publicly available + 1 internal) not previously used for model training, and compared it with the existing state-of-the-art models. We found that the ITU P.1204.3 model out of the box performs well on these unseen datasets, with an RMSE ranging between 0.38 - 0.45 on the 5-point absolute category rating and Pearson Correlation between 0.85 - 0.93 across all the 4 databases. We further propose a full-HD variant of the P.1204.3 model, since the original model is trained and validated which targets a resolution of 4K/UHD-1. A 50:50 split across all databases is used to train and validate this variant so as to make sure that the proposed model is applicable to various conditions.

Existing works in the field of quality assessment focus separately on gaming and non-gaming content. Along with the traditional modeling approaches, deep learning based approaches have been used to develop quality models, due to their high prediction accuracy. In this paper, we present a deep learning based quality estimation model considering both gaming and non-gaming videos. The model is developed in three phases. First, a convolutional neural network (CNN) is trained based on an objective metric which allows the CNN to learn video artifacts such as blurriness and blockiness. Next, the model is fine-tuned based on a small image quality dataset using blockiness and blurriness ratings. Finally, a Random Forest is used to pool frame-level predictions and temporal information of videos in order to predict the overall video quality. The light-weight, low complexity nature of the model makes it suitable for real-time applications considering both gaming and non-gaming content while achieving similar performance to existing state-of-the-art model NDNetGaming. The model implementation for testing is available on GitHub1.

Cloud Gaming (CG) is an immersive multimedia service that promises many benefits. In CG, the games are rendered in a cloud server, and the resulted scenes are streamed as a video sequence to the client. Using CG users are not forced to update their gaming hardware frequently, and available games can be played on any operating system or suitable device. However, cloud gaming requires a reliable and low-latency network, which makes it a very challenging service. Transmission latency strongly affects the playability of a cloud game and consequently reduces the users' Quality of Experience (QoE). In this paper, we propose a latency compensation technique using game adaptation that mitigates the influence of delay on QoE. This technique uses five game characteristics for the adaptation. These characteristics, in addition to an Aim-assistance technique, were implemented in four games for evaluation. A subjective study using 194 participants was conducted using a crowdsourcing approach. The results showed that the majority of the proposed adaptation techniques lead to significant improvements in the cloud gaming QoE.

The gaming industry is one of the largest digital markets for decades and is steady developing as evident by new emerging gaming services such as gaming video streaming, online gaming, and cloud gaming. While the market is rapidly growing, the quality of these services depends strongly on network characteristics as well as resource management. With the advancement of encoding technologies such as hardware accelerated engines, fast encoding is possible for delay sensitive applications such as cloud gaming. Therefore, already existing video quality models do not offer a good performance for cloud gaming applications. Thus, in this paper, we provide a gaming video quality dataset that considers hardware accelerated engines for video compression using the H.264 standard. In addition, we investigate the performance of signal-based and parametric video quality models on the new gaming video dataset. Finally, we build two novel parametric-based models, a planning and a monitoring model, for gaming quality estimation. Both models are based on perceptual video quality dimensions and can be used to optimize the resource allocation of gaming video streaming services.

Traditionally, the Quality of Experience (QoE) is assessed in a controlled laboratory environment where participants give their opinion about the perceived quality of a stimulus on a standardized rating scale. Recently, the usage of crowdsourcing micro-task platforms for assessing the media quality is increasing. The crowdsourcing platforms provide access to a pool of geographically distributed, and demographically diverse group of workers who participate in the experiment in their own working environment and using their own hardware. The main challenge in crowdsourcing QoE tests is to control the effect of interfering influencing factors such as a user's environment and device on the subjective ratings. While in the past, the crowdsourcing approach was frequently used for speech and video quality assessment, research on a quality assessment for gaming services is rare. In this paper, we present a method to measure gaming QoE under typically considered system influence factors including delay, packet loss, and framerates as well as different game designs. The factors are artificially manipulated due to controlled changes in the implementation of games. The results of a total of five studies using a developed evaluation method based on a combination of the ITU-T Rec. P.809 on subjective evaluation methods for gaming quality and the ITU-T Rec. P.808 on subjective evaluation of speech quality with a crowdsourcing approach will be discussed. To evaluate the reliability and validity of results collected using this method, we finally compare subjective ratings regarding the effect of network delay on gaming QoE gathered from interactive crowdsourcing tests with those from equivalent laboratory experiments.

Recently, streaming of gameplay scenes has gained much attention, as evident with the rise of platforms such as Twitch.tv and Facebook Gaming. These streaming services have to deal with many challenges due to the low quality of source materials caused by client devices, network limitations such as bandwidth and packet loss, as well as low delay requirements. Spatial video artifact such as blockiness and blurriness as a result of as video compression or up-scaling algorithms can significantly impact the Quality of Experience of end-users of passive gaming video streaming applications. In this paper, we investigate solutions to enhance the video quality of compressed gaming content. Recently, several super-resolution enhancement techniques using Generative Adversarial Network (e.g., SRGAN) have been proposed, which are shown to work with high accuracy on non-gaming content. Towards this end, we improved the SRGAN by adding a modified loss function as well as changing the generator network such as layer levels and skip connections to improve the flow of information in the network, which is shown to improve the perceived quality significantly. In addition, we present a performance evaluation of improved SRGAN for the enhancement of frame quality caused by compression and rescaling artifacts for gaming content encoded in multiple resolution-bitrate pairs.

Combining interconnected wearables provides fascinating opportunities like augmenting exergaming with virtual coaches, feedback on the execution of sports activities, or how to improve on them. Breathing rhythm is a particularly interesting physiological dimension since it is easy and unobtrusive to measure and gained data provide valuable insights regarding the correct execution of movements, especially when analyzed together with additional movement data in real-time. In this work, we focus on indoor rowing since it is a popular sport that's often done alone without extensive instructions. We compare a visual breathing indication with haptic guidance in order for athletes to maintain a correct, efficient, and healthy breathing-movement-synchronicity (BMS) while working out. Also, user experience and acceptance of the different modalities were measured. The results show a positive and statistically significant impact of purely verbal instructions and purely tactile feedback on BMS and no significant impact of visual feedback. Interestingly, the subjective ratings indicate a strong preference for the visual modality and even an aversion for the haptic feedback, although objectively the performance benefited most from using the latter.

Quality of Experience is traditionally evaluated by using short stimuli usually representing parts or single usage episodes. This opens the question on how the overall service perception involving multiple usage episodes can be evaluated—a question of high practical relevance to service operators. Despite initial research on this challenging aspect of multi-episodic perceived quality, the question of the underlying quality formation processes and its factors are still to be discovered. We present a multi-episodic experiment of an Audio-on-Demand service over a usage period of 6 days with 93 participants. Our work directly extends prior work investigating the impact of time between usage episodes. The results show similar effects — although the recency effect is not statistically significant. In addition, we extend prediction of multi-episodic judgments by accounting for the observed saturation.

Crowdsourcing is a valid approach to collect and annotate data efficiently and cost-effectively. This approach permits us to reach a large and diverse pool of users that usually work from home employing their computers and headphones. Still, there is insufficient information about the users' surroundings. Specifically, little knowledge about the background noise to which users might be exposed to when executing crowd-work. The validity of the data gathered in a disturbed environment is questionable, especially in speech quality assessment and other audio-related tasks. This work presents the results of a simulated crowdsourcing study conducted in the laboratory. We investigate the influence of environmental background noise in speech quality assessment tests. Three groups of listeners were recruited to rate the quality of speech files under the influence of background noise at different levels. Two types of noise were tested, i.e., street noises and TV-Show. Our findings suggest that the threshold at which an environmental background noise would significantly affect the speech quality ratings in crowdsourcing is between 43dB(A) and 50dB(A). Additionally, listeners tolerated more the TV-Show noise. They provided more accurate ratings while conducting the test under the influence of higher levels of the TV-Show noise, than at lower levels of the street noise. We also found that the presence of background noise does not cause a constant bias of the quality scores; instead, its impact depends on the speech degradation condition under test.

Virtual Reality (VR) has an increasing impact on the market in many fields, from education and medicine to engineering and entertainment, by creating different applications that replicate or in the case of augmentation enhance real-life scenarios. Intending to present realistic environments, VR applications are including text that we are surrounded by every day. However, text can only add value to the virtual environment if it is designed and created in such a way that users can comfortably read it. With the aim to explore what values for text parameters users find comfortable while reading in virtual reality, a study was conducted allowing participants to manipulate text parameters such as font size, distance, and contrast. Therefore two different standalone virtual reality devices were used, Oculus Go and Quest, together with three different text samples: Short (2 words), medium (21 words), and long (51 words). Participants had the task of setting text parameters to the best and worst possible value. Additionally, participants were asked to rate their experience of reading in virtual reality. Results report mean values for angular size (the combination of distance and font size) and color contrast depending on the different device used as well as the varying text length, for both tasks. Significant differences were found for values of angular size, depending on the length of the displayed text. However, different device types had no significant influence on text parameters but on the experiences reported using the self-assessment manikin (SAM) scale.

This paper describes the development and validation of a continuous pictographic scale for self-reported assessment of affective states in virtual environments. The developed tool, called Morph A Mood (MAM), consists of a 3D character whose facial expression can be adjusted with simple controller gestures according to the perceived affective state to capture valence and arousal scores. It was tested against the questionnaires Pick-A-Mood (PAM) and Self-Assessment Manikin (SAM) in an experiment in which the participants (N = 32) watched several one-minute excerpts from music videos of the DEAP database within a virtual environment and assessed their mood after each clip. The experiment showed a high correlation with regard to valence, but only a moderate one with regard to arousal. No statistically significant differences were found between the SAM ratings of this experiment and MAM, but between the valence values of MAM and the DEAP database and between the arousal values of MAM and PAM. In terms of user experience, MAM and PAM hardly differ. Furthermore, the experiment showed that assessments inside virtual environments are significantly faster than with paper-pencil methods, where media devices such as headphones and display goggles must be put on and taken off.

The E-model is the only recommended parametric tool for planning the quality of speech communication services, and its fullband version has recently been standardized by the International Telecommunication Union, ITU-T. In this paper, we analyze and extend the model by comparing its predictions for random and bursty packet loss as well as for delay to the results of a signal-based model, POLQA, as well as to the result of a subjective conversation test. The analysis shows that, by extending the fullband model to account for the burstiness, a reasonable prediction accuracy can be reached for random as well as bursty loss. The results are discussed and limitations of the current model are pointed out.

Crowdsourcing micro-task platforms facilitate subjective media quality assessment by providing access to a highly scaleable, geographically distributed and demographically diverse pool of crowd workers. Those workers participate in the experiment remotely from their own working environment, using their own hardware. In the case of speech quality assessment, preliminary work showed that environmental noise at the listener's side and the listening device (loudspeaker or headphone) significantly affect perceived quality, and consequently the reliability and validity of subjective ratings. As a consequence, ITU-T Rec. P.808 specifies requirements for the listening environment of crowd workers when assessing speech quality. In this paper, we propose a new Just Noticeable Difference of Quality (JNDQ) test as a remote screening method for assessing the suitability of the work environment for participating in speech quality assessment tasks. In a laboratory experiment, participants performed this JNDQ test with different listening devices in different listening environments, including a silent room according to ITU-T Rec. P.800 and a simulated background noise scenario. Results show a significant impact of the environment and the listening device on the JNDQ threshold. Thus, the combination of listening device and background noise needs to be screened in a crowdsourcing speech quality test. We propose a minimum threshold of our JNDQ test as an easily applicable screening method for this purpose.

The rank correlation coefficients and the ranked-based statistical tests (as a subset of non-parametric techniques) might be misleading when they are applied to subjectively collected opinion scores. Those techniques assume that the data is measured at least at an ordinal level and define a sequence of scores to represent a tied rank when they have precisely an equal numeric value. In this paper, we show that the definition of tied rank, as mentioned above, is not suitable for Mean Opinion Scores (MOS) and might be misleading conclusions of rank-based statistical techniques. Furthermore, we introduce a method to overcome this issue by transforming the MOS values considering their 95% Confidence Intervals. The rank correlation coefficients and ranked-based statistical tests can then be safely applied to the transformed values. We also provide open-source software packages in different programming languages to utilize the application of our transformation method in the quality of experience domain.

The subjective quality of transmitted speech is traditionally assessed in a controlled laboratory environment according to ITU-T Rec. P.800. In turn, with crowdsourcing, crowdworkers participate in a subjective online experiment using their own listening device, and in their own working environment. Despite such less controllable conditions, the increased use of crowdsourcing micro-task platforms for quality assessment tasks has pushed a high demand for standardized methods, resulting in ITU-T Rec. P.808. This work investigates the impact of the number of judgments on the reliability and the validity of quality ratings collected through crowdsourcing-based speech quality assessments, as an input to ITU-T Rec. P.808 . Three crowdsourcing experiments on different platforms were conducted to evaluate the overall quality of three different speech datasets, using the Absolute Category Rating procedure. For each dataset, the Mean Opinion Scores (MOS) are calculated using differing numbers of crowdsourcing judgements. Then the results are compared to MOS values collected in a standard laboratory experiment, to assess the validity of crowdsourcing approach as a function of number of votes. In addition, the reliability of the average scores is analyzed by checking inter-rater reliability, gain in certainty, and the confidence of the MOS. The results provide a suggestion on the required number of votes per condition, and allow to model its impact on validity and reliability.

Cloud Gaming is an emerging service that is considered by many as the future of the gaming industry. This service requires a highly reliable network with low latency and high bandwidth. If these requirements are not satisfied, cloud gaming services cannot create a good Quality of Experience (QoE) for its users. However, gaming QoE can vary significantly among different game scenarios and users. For an optimal resource allocation and quality estimation, it is highly important for cloud providers, game developers, and network planners to consider the influence of the game content and gamers. This paper presents the result of a subjective study that investigated the impact of different player strategies and user inputs on their perceived delay. The results indicated that the user input characteristics vary among the games but stays the same between different users and different strategies. In addition to the users' inputs, the input quality and the overall gaming experience of the users were also investigated, and results did not show any main effect of user strategy on the delay sensitivity of the games.

This study introduces a Quality of Experience (QoE) model of loudspeaker-based spatial speech reproduction, which specifies quality elements and quality features relevant to overall listening experience and Quality of Service (QoS), respectively. Assumptions about the relations between selected quality elements and quality features were validated in a listening-only test. Participants had the task to behaviorally identify the voices of two talkers and provided subjective judgments on category rating scales. Sentences uttered by the two talkers were alternately presented through only a central loudspeaker (non-spatial mode) versus through either the central or one talker-specific lateral loudspeaker (spatial mode). The transmitted speech signals were clean, superimposed with background noise or bandpass-filtered. Results showed that transmission quality, but not reproduction mode significantly affected evaluative (speech quality, speech intelligibility), immersion (voice naturalness, spatial presence, social presence) and task-related (voice similarity, talker identification effort) aspects of overall listening experience. Contrary to previous findings, the spatial mode did not reduce the effort of talker identification. This suggested that noticeable advantages from spatial reproduction may not be evident in relatively simple listening situations, but could manifest if complexity is increased. Besides their task-related influence, noise and spectral distortion should be mitigated when targeting a higher level of immersion.

With the rising interest in Virtual Reality and the fast development and improvement of available devices, new features of interactions are becoming available. One of them that is becoming very popular is hand tracking, as the idea to replace controllers for interactions in virtual worlds. This experiment aims to compare different interaction types in VR using either controllers or hand tracking. Participants had to play two simple VR games with various types of tasks in those games — grabbing objects or typing numbers. While playing, they were using interactions with different visualizations of hands and controllers. The focus of this study was to investigate user experience of varying interactions (controller vs. hand tracking) for those two simple tasks. Results show that different interaction types statistically significantly influence reported emotions with Self-Assessment Manikin (SAM), where for hand tracking participants were feeling higher valence, but lower arousal and dominance. Additionally, task type of grabbing was reported to be more realistic, and participants experienced a higher presence. Surprisingly, participants rated the interaction type with controllers where both where hands and controllers were visualized as statistically most preferred. Finally, hand tracking for both tasks was rated with the System Usability Scale (SUS) scale, and hand tracking for the task typing was rated as statistically significantly more usable. These results can drive further research and, in the long term, contribute to help selecting the most matching interaction modality for a task.

In recent years 360° videos have been becoming more popular. For traditional media presentations, e.g., on a computer screen, a wide range of assessment methods are available. Different constructs, such as perceived quality or the induced emotional state of viewers, can be reliably assessed by subjective scales. Many of the subjective methods have only been validated using stimuli presented on a computer screen. This paper is using 360° videos to induce varying emotional states. Videos were presented 1) via a head-mounted display (HMD) and 2) via a traditional computer screen. Furthermore, participants were asked to rate their emotional state 1) in retrospect on the self-assessment manikin scale and 2) continuously on a 2-dimensional arousal-valence plane. In a repeated measures design, all participants (N = 18) used both presentation systems and both rating systems. Results indicate that there is a statistically significant difference in induced presence due to the presentation system. Furthermore, there was no statistically significant difference in ratings gathered with the two presentation systems. Finally, it was found that for arousal measures, a statistically significant difference could be found for the different rating methods, potentially indicating an underestimation of arousal ratings gathered in retrospect for screen presentation. In the future, rating methods such as a 2-dimensional arousal-valence plane could offer the advantage of enabling a reliable measurement of emotional states while being more embedded in the experience itself, enabling a more precise capturing of the emotional states.

Conversations over the telephone require timely turn-taking cues that signal the participants when to speak and when to listen.When a two-way transmission delay is introduced into such conversations, the immediate feedback is delayed, and the interactivity of the conversation is impaired.With delayed speech on each side of the transmission, different conversation realities emerge on both ends, which alters the way the participants interact with each other.Simulating conversations can give insights on turn-taking and spoken interactions between humans but can also used for analyzing and even predicting human behavior in conversations.In this paper, we simulate two types of conversations with distinct levels of interactivity.We then introduce three levels of two-way transmission delay between the agents and compare the resulting interactionpatterns with human-to-human dialog from an empirical study.We show how the turn-taking mechanisms modeled for conversations without delay perform in scenarios with delay and identify to which extend the simulation is able to model the delayed turn-taking observed in human conversation.

This article investigates the possibility to estimate the perceived Quality of Experience (QoE) automatically and unobtrusively by analyzing the face of the consumer of video streaming services, from which facial expression and gaze direction are extracted. If effective, this would be a valuable tool for the monitoring of personal QoE during video streaming services without asking the user to provide feedback, with great advantages for service management. Additionally, this would eliminate the bias of subjective tests and would avoid bothering the viewers with questions to collect opinions and feedback. The performed analysis relies on two different experiments: i) a crowdsourcing test, where the videos are subject to impairments caused by long initial delays and re-buffering events; ii) a laboratory test, where the videos are affected by blurring effects. The facial Action Units (AU) that represent the contractions of specific facial muscles together with the position of the eyes' pupils are extracted to identify the correlation between perceived quality and facial expressions. An SVM with a quadratic kernel and a k-NN classifier have been tested to predict the QoE from these features. These have also been combined with measured application-level parameters to improve the quality prediction. From the performed experiments, it results that the best performance is obtained with the k-NN classifier by combining all the described features and after training it with both the datasets, with a prediction accuracy as high as 93.9% outperforming the state of the art achievements.

The quality of speech communication networks has recently improved significantly by extending the available audio bandwidth from narrowband, firstly to wideband, and then to super-wideband. This bandwidth extension marks the end of the typically muffled sound we know from plain old telephone services. Another reason for increased speech quality is the fully digitally packet-based transmission. However, so far, no speech quality prediction model is able to estimate super-wideband quality without a clean reference signal. In this paper, we present a non-intrusive speech quality assessment model NISQA, which - in contrast to current state-of-the-art models - can predict the quality of super-wideband speech transmission. Furthermore, it is able to accurately predict the quality impact of packet loss concealment of modern codecs, such as Opus and EVS. The model uses a novel approach, where a CNN firstly estimates the per-frame quality, and subsequently, an RNN aggregates the per-frame values over time, to estimate the overall speech quality. Averaged over a comprehensive test set, the model achieves an RMSE*3rd of 0.29 with subjective MOS.

High cost and time consumption are concurrent barriers for research and application of automated summarization. In order to explore options to overcome this barrier, we analyze the feasibility and appropriateness of micro-task crowdsourcing for evaluation of different summary quality characteristics and report an ongoing work on the crowdsourced evaluation of query-based extractive text summaries. To do so, we assess and evaluate a number of linguistic quality factors such as grammaticality, non-redundancy, referential clarity, focus and structure & coherence. Our first results imply that referential clarity, focus and structure & coherence are the main factors effecting the perceived summary quality by crowdworkers. Further, we compare these results using an initial set of expert annotations that is currently being collected, as well as an initial set of automatic quality score ROUGE for summary evaluation. Preliminary results show that ROUGE does not correlate with linguistic quality factors, regardless if assessed by crowd or experts. Further, crowd and expert ratings show highest degree of correlation when assessing low quality summaries. Assessments increasingly divert when attributing high quality judgments.

Crowdsourcing has become a convenient instrument for addressing subjective user studies to a large amounts of users. Data from crowdsourcing can be corrupted due to users' neglect, and different mechanisms has been proposed to address the users' reliability and to ensure valid experiments' results. Users that are consistent in their answers or present a high intra-rater reliability score, are desired for subjective studies. This work investigates the relationship between the intra-rater reliability and the user performance in the context of a speech quality assessment task. To this end, a crowdsourcing study has been conducted in which users were requested to rate speech stimuli with respect to their overall quality. Ratings were collected on a 5-point scale in accordance with the ITU-T Rec. P.808. The speech stimuli were taken from the database ITU-T Rec. P.501 Annex D, and the results are to be contrasted with ratings collected in a laboratory experiment. Furthermore, a model as a function of intra-rater reliability, root-mean-squared-deviation between the listeners ratings and age, has been built to predict the listener performance. Such a model is intended to provide a measure of how valid the crowdsourcing results are, when there is no laboratory results to compare to.

Crowdsourcing has been used extensively for gathering and annotating data cost efficiently. Nowadays, there are multiple platforms offering crowd-sourced workforce, still most of these users are from Asia or English speaking countries, and not so many native German speakers. Thus, there is a lack of information regarding the conditions in which German users execute tasks, neither about their habits when taking part in crowdsourcing campaigns. Which is of main importance to address properly user studies to German crowd-workers. This paper reports on a survey that investigated the environments' characteristics of users from German speaking countries. To this end, a study has been conducted in which users were asked to provide details about the surroundings in which they normally execute crowdsourcing tasks. Audio and visual data was collected per user which contributed to aggregate even more information on the users' input. We provide insights aimed at easing the decision making process when designing subjective user studies.

One of the fields where Virtual Reality (VR) is finding a potentially growing market is in the combination of exercising and gaming - also called exergaming. When it comes to competition in gaming, is important to investigate how different levels of delay influence overall quality of experience (QoE) in VR multiplayer exergames. The experimental setup consisted of a VR application coupled with a rowing ergometer, allowing races between the user and an artificially created opponent that is following the player with a similar speed and keeping the race tight. To investigate the influence of the delay, on both user's and opponent's side three levels of network delay were introduced (30ms, 100ms, and 500ms) and mixed throughout different conditions. After each session, participants rated perceived flow, sense of presence, and the degree to which they have noticed the delay in their or the opponent's system. Interestingly, results show different perception of delay and QoE depending on user's own delay. Participants perceived the opponent's player as being delayed even if only the player itself had network delay along with significantly lower rating of QoE only when their delay was high.

In past years, as virtual reality (VR) technology is extensively developing, more and more people are using it in different fields. One of the fast developing fields in VR are exergames, a combination of physical exercise and game. With a goal to engage people in physical activity, VR exergames should look and feel good for users. Therefore, user interface (UI) in VR is important and has to be built in a way to enhance user experience. In this paper, ergometer is used together with VR rowing environment as VR exergame for a study aiming to explore possibilities of UI in VR. Accordingly, different metrics commonly used to quantify rowing action (e.g, speed and distance) were visualized. The visualizations that were created had different positioning (closer or further away from the player) and different level of complexity (more or less metrics shown as only numbers or in a gamified design). Participants (N = 27) during experiment for this study rowed four times in different conditions depending were metrics of the game were shown: 1) as a cockpit at the front of the rowing e.g, with gamified visualization of metrics; 2) as a coach boat that follows the player with gamified visualization of metrics on a screen; 3) as a cockpit at the front of the rowing e.g, with digital visualization of metrics; 4) as a coach boat that follows the player with digital visualization of metrics on a screen. Results show that level of UI complexity has a significant influence on readability, while positioning of UI elements significantly influences users' perception of support from system. Furthermore, participants preferred the opposite level of complexity depending on the position where metrics were shown.

There are numerous instrumental tools available to monitor the perceived quality of speech communication networks. However, these tools give no insight into the cause of a quality degradation. In this paper, we present a new method for quality diagnosis of speech communication networks that builds upon recent developments in the field of semantic image segmentation. The proposed model works non-intrusively, without the need for a clean reference signal. We use the deep convolutional network architecture SegNet and label each pixel of a speech spectrogram image as either clean or with its corresponding distortion. This way, quality degradations can directly be located in the time and frequency domain. To train the model, we created a large database with four different distortion types: packet-loss, background noise, GSM buzz, and bandwidth limitation. While processing the speech files, we also generated corresponding ground-truth labels with which we trained SegNet. Our experiments show promising results of this new diagnostic approach with a mIoU of 0.75.

Data-driven approaches towards readability assessment, using automated linguistic analysis and machine learning methods, is a viable road forward for readability rankings. This paper describes the development of an automated readability assessment estimator based on supervised learning algorithms over German text corpora. For this purpose, natural language processing tools are used to extract 73 linguistic features grouped in traditional, lexical and morphological features. Feature engineering approaches are employed to select informative features. Different supervised learning models are implemented, with the top-ranked features fed as input. The results obtained depict that Random Forest Regressor yielding best result (0.847) for RMSE measure.

As known from everyday contexts of multimedia usage, suddenly occurring quality impairments are capable of causing strong negative emotions in human users. This is particularly the case if the displayed content is highly relevant to current motives and behavioral goals. The present study investigated the effects of visual degradations on quality perception and emotional state of participants who were exposed to a series of short video clips. After each video playback, participants had to decide whether a certain event happened in the video. For data collection, subjective measures of quality and emotion were complemented by behavioral measures derived from capturing participants' spontaneous facial expressions. For data analysis, two general approaches were combined: First, a multivariate analysis of variance approach allowed to examine the effects of visual degradation factors on perceived quality and subjective emotional dimensions. It mainly revealed that perceived quality and emotional valence were both sensitive to degradation intensity, whereas the impact of degradation length was limited when task-relevant video content had already been obscured. Second, using a machine learning approach, an automatic Video Quality of Experience (VQoE) prediction system based on the recorded facial expressions was derived, demonstrating a strong correlation between facial expressions and perceived quality. Hereby, estimates of VQoE might be delivered in an objective, continuous and concealed manner, thus diminishing any further need for subjective self-reports.

In communication services, speech quality plays an important role to achieve user expectations. Nowadays, there are different objective methods to estimate speech quality. Parametric models consider different factors, such as network parameters, acoustic characteristics, communication equipment, among others. The most representative parametric models for telephone service are described in ITU-T Rec. G.107 and G.107.1, mostly known as E-model and WB E-model, respectively. However, they do not consider wireless network parameters as inputs. In this context, this research proposes a speech quality parametric model (SQPM) based on artificial neural networks that considers both wireless network degradation characteristics and the techniques used to improve the transmission quality. For this purpose, a network simulator was built, in which two forward error correction (FEC) codes and four different antenna configurations in a multiple-input-multiple-output (MIMO) system are implemented. To validate the results obtained by the simulator, the ITU-T Rec. P.863 and the WB E-model are used. Experimental results show how different wireless network configurations impact on speech quality. Performance evaluation results demonstrated a high correlation between the proposed SQPM and ITU-T Rec. P.863 results, reaching an PCC and an RMSE of 0.9901 and 0.1492, respectively. Therefore, our proposal intends to be useful for wireless network planning tasks.

Both online and cloud gaming services require a very low network delay to create a good Quality of Experience (QoE) for their users. The required network latency cannot be guaranteed due to the current best effort-nature of the network, and as a result, network latency often degrades the gamers performance and QoE. In this paper, the adaptability of gamers to different variations on delay is investigated both subjectively and objectively using three self-developed games. The results show that gamers can adapt to constant delay while they are playing and change their behavior if the actions in a game are predictable. Such adaptation leads to a significant increase in gamers performance and QoE. The paper also provides evidence that regardless of performance frequent delay switching annoys gamers. The result of this study can be used to create a network resource allocation technique which controls a congested network by giving more priority and resource to the unadaptable games than the adaptable games.

Crowdsourcing (CS) has established as a powerful tool to collect human input for data acquisition and labeling. However, it remains the question about the validity of the data collected in a CS platform. Sometimes, the users work carelessly or they try to tweak the system to maximize their profits. This paper reports on whether the number of speech stimuli presented to the listeners has an impact on the user perception of certain degradation conditions applied to the speech signal. To this end, a crowdsourcing study has been conducted with 209 listeners that were divided in three non-overlapping user groups, each of which was presented with tasks containing a different number of stimuli: 10, 20, or 40. Listeners were asked to rate speech stimuli with respect to their overall quality and the ratings were collected on a 5-point scale in accordance with ITU-T Rec. P.800. Workers assessed the speech stimuli of the database 501 from ITU-T Rec. P.863. Additionally, the influence of certain speech signal characteristics, such as interruptions and bandwidth, on the quality perception of the workers was investigated.

In this paper, we analyze whether deep convolutional neural networks can be used to detect lost packets in speech communication systems. The speech quality of modern communication networks has significantly improved recently, for example through higher available audio bandwidth. This was, among other reasons, possible through the use of packet-based networks, which allow a fully digital transmission from the sender to the receiver terminal. However, these networks often suffer from frequent interruptions caused by lost packets due to transmission errors. Consequently, the packet loss rate is one of the main indicators for the quality of speech communication services. In spite of that, the information of how many packets are lost in a network is not always available. To estimate the amount of lost packets, we calculate spectrograms of the transmitted speech signals and use them as input of a convolutional neural network. This approach has recently gained popularity in the field of detection and recognition tasks for music and speech. The interruptions caused by lost packets can often clearly be seen in the spectrogram of the degraded signal. Therefore, it seems natural to interpret the spectrograms as images and use deep learning methods that are common for image classification. The proposed model allows for estimating the packet loss rate of a communication system by simply using the recorded speech file from the receiver side, without the need of the reference speech signal that was originally sent through the channel. Our results show that the model reduces the prediction error by more than 75% when compared to a model that is based on MFCC features.

With emerging delay sensitive gaming services such as cloud gaming and online gaming, the importance of understanding and reducing the effect of delay on the gamer's Quality of Experience (QoE) becomes highly important for the success of these services. In this paper, the findings of two subjective experiments investigating the relationship between delay and QoE are reported. In the first study, it was shown that in addition to the direct effect of the delay on QoE, there is a significant indirect effect between delay and QoE through the relationship with performance. In the second part of the paper, we illustrate that adapting characteristics of a game can strongly mitigate the negative effect of delay on gaming QoE due to increased player performance. This adaptation in addition to compensation the effect of the delay, in contrast to the other difficulty adjustment systems, does not require to track the gamer's interaction, behaviors, and profile.

This paper reports an examination of the perceptual video quality space for video degradations. Therefore, we conducted an experiment, which investigates whether the perceptual video quality dimensions obtained in a head-and-shoulder videotelephony setting are also valid for other video content. The method of Direct Scaling is used for directly assessing the identified perceptual quality dimensions. This method makes it possible to assess more test conditions in the same time, due to the reduced experimental effort. Besides the ratings of the perceptual video quality dimensions, the test participants are also asked to rate the overall video quality. This allows drawing conclusions of the composition of the underlying quality dimensions to form overall quality for video. The results of the experiment are compared to the ratings obtained from subjective testing in the domain of video telephony.

Gaming as a popular system has recently expanded the associated services, by stepping into live streaming services. Live gaming video streaming is not only limited to cloud gaming services, such as Geforce Now, but also include passive streaming, where the players' gameplay is streamed both live and ondemand over services such as Twitch.tv and YouTubeGaming. So far, in terms of gaming video quality assessment, typical video quality assessment methods have been used. However, their performance remains quite unsatisfactory. In this paper, we present a new No Reference (NR) gaming video quality metric called NR-GVQM with performance comparable to state-of-the-art Full Reference (FR) metrics. NR-GVQM is designed by training a Support Vector Regression (SVR) with the Gaussian kernel using nine frame-level indexes such as naturalness and blockiness as input features and Video Multimethod Assessment Fusion (VMAF) scores as the ground truth. Our results based on a publicly available dataset of gaming videos are shown to have a correlation score of 0.98 with VMAF and 0.89 with MOS scores. We further present two approaches to reduce computational complexity.

Video games and sport are an essential part in the life of millions of people. With recent advances of immersive virtual reality devices such as the HTC Vive, Oculus Rift, or PlayStation VR, the use of virtual environments (VE) for exergames is becoming more and more popular. An exergame combines a physical activity with video game elements by tracking body movements or reactions of user, attempting to engage users in a more enjoyable system. In this paper, we present the results of a subjective experiment carried out with the aim to compare different kinds of virtual environments with each other. A rowing ergometer, connected either to a virtual reality system using a head-mounted display (HMD) or to a CAVE environment, was used as an exergame device. While for rowing experts, fitness and performance improvements are of major interest, we wanted to focus on the motivation and engagement of non-professionals. By means of a series of questionnaires and a follow-up interview, the Quality of Experience of participants using the system was assessed. Measurements include concepts such as flow, presence, video quality and well-being. Results show significant advantages of the HMD as well as of the CAVE compared to a system without a VE for the overall quality, system feedback, and flow. While the CAVE and HMD system mainly differed in their autotelic experience, the HMD was favored by the majority of participants due to a superior feeling of presence.

Subjective tests to assess the Quality of Experience (QoE) of gaming services are necessary to enable service providers to ensure the satisfaction of their customers. Since gaming is an interactive activity, interactive tests are typically conducted to measure the full spectrum of the player experience. However, carrying out such tests is expensive and time-consuming. Furthermore, the results can be influenced by the behavior and abilities of participants. For this reason, it is of high interest whether such interactive tests can be partially replaced with passive viewing-and-listening tests. In this paper, we present a comparison of an interactive gaming test with passive tests using two different durations. To investigate the differences between the test paradigms, we assessed the overall quality, the video quality and the reactiveness of the game as well as other player experience aspect for different frame rates and bit rates. Results show that once certain requirements are fulfilled, passive tests offer indeed a valuable quality assessment method. However, if the duration of the presented video material is too short, the passive test overestimated the gaming and video quality. Furthermore, we show that the player performance has no impact on the video quality ratings.

Recent years have seen a tremendous increase in video traffic with the rise of Over The Top (OTT) services. Along with traditional Video on demand (VoD) streaming services (e.g., Netflix, YouTube), live video services (e.g., Twitch. tv, YouTubeGaming, Facebook Live) have also resulted in a tremendous share of Internet traffic. Among the live streaming services, gaming video streaming has a major share, with Twitch.tv alone currently responsible for the fourth highest peak Internet traffic in the US. As a consequence of this, and due to the fact that gaming videos are artificial and synthetic, it is worth investigating the specificity of gaming videos in relation to compression and the consequent end user QoE. In this paper, we present an objective and subjective quality comparison study for regular videos and gaming videos, with 30 video sequences (15 per type), encoded using the state of the art encoder HEVC. We discuss the similarity and dissimilarity between the two video types and also discuss how these observations can be used to improve the end user QoE.

This short paper presents the recently published Dagstuhl Manifesto `QoE Vadis?`. The Manifesto is the result of a set of three Dagstuhl Seminars and one Dagstuhl Perspectives Workshop, aimed at shaping understanding, development and application of the Quality of Experience (QoE) notion and concept. Its task is to convey the current status, promising developments and future projections for different stakeholders. The latter are summarised in a set of eleven recommendations to academia, industry and funding organisations.

In telephone conversations, transmitted speech of good to excellent quality is desired for enhanced Quality of Experience and to sustain lasting customer loyalty. Subjective mean opinion scores account for perceived transmitted quality, while instrumental models, such as POLQA, are able to estimate the subjective judgments. To perform subjective or instrumental quality measurements, the International Telecommunication Union recommends to employ two sentences from both, male and female speakers as speech material. In this paper, we have examined whether subjective and instrumental MOS ratings are affected by perceptual voice likability. A listening test has been conducted over 8 degradations with 12 extremely likable and unlikable male and female speakers. Statistically significant effects of gender and of voice likability have been detected on subjective MOS, whereas instrumental MOS was only affected by gender differences. These results can contribute to further improvements needed in the POLQA perceptual modeling, as well as to the selection of speakers for speech quality assessment tests.

Nowadays, crowdsourcing provides an exceptional opportunity for conducting subjective user tests on the Internet with a demographically diverse audience. Previous work has pointed out that the offered tasks should be kept short in time, therefore, participants evaluate at once just a portion of the dataset. Aspects like users' workload and fatigue are important as they relate to a main question: how to optimize study design without compromising results quality by tiring the test participants? This work investigates the influence of the number of presented speech stimuli on the reliability of listeners' ratings in the context of subjective speech quality assessment. A crowdsourcing study have been conducted with 209 listeners that were asked to rate speech stimuli with respect to their overall quality. Participants were randomly assigned to one of three user groups, each of which was confronted with tasks consisting of a different number of stimuli: 10, 20, or 40. The results from the three groups are highly correlated to existing laboratory ratings, the group with the largest number of samples offering the highest correlation. However, participant retention decreased while the study completion time increased. Thus, it might be desirable to offer tasks with less speech stimuli sacrificing ratings' accuracy to some extent.

Virtual Reality (VR) multiplayer games are currently developed more frequently, while in parallel VR is becoming more accessible to the broader public. Also exergames, which refer to computer games combined with a physical activity, are getting widely available, driven by developments in sensor technologies. In this paper, we present results of a multiplayer exergame experiment where two players race against each other four times on two rowing ergometers. Experimental conditions were type of environment (with VR vs. without VR) and the possibility of conversation with the opponent (conversation enabled vs. conversation disabled). To measure subjective experience, participants rated perceived flow, sense of presence and social presence on standardized questionnaires. Results show that the type of environment has a significant influence on the perceived presence, as participants rated the presence higher in VR. The possibility of conversation resulted in a perception of a significantly higher social presence. This illustrates that the usage of virtual environments as well as the possibility of conversations between users are highly valuable to increase the Quality of Experience during exergames. In our future research, we plan to further investigate means to improve the user experience by applying gamification, different representations of opponents, and to assess the impact of network impairments during multiplayer scenarios.

Voice transmission networks are commonly planned with the help of computational quality models, which give an estimate of the expected quality that a user will experience. The most popular of these tools is the E-model. When certain parameters are known, such as the applied codec and its bitrate, the model is able to predict the perceived quality of a communication system. Up to now, the E-model is only available for narrowband telephony (300–3400 Hz) and limited also for wideband telephony (100–7000 Hz). With the extension of voice networks to super-wideband telephony (50–14000 Hz), and the introduction of the super-wideband codec EVS to mobile networks and state of the art smartphones, an update of the E-model has become necessary. To this end, we firstly examined the quality improvement of super-wideband over wideband with results from mixed-band listening-only tests, where we found that the quality is improved by 15%. Then, we calculated impairment factors for the EVS codec and analyzed its robustness towards packet loss, by using auditory and instrumental methods.

Measuring the quality experienced during online gaming activities is an important prerequisite for managing gaming services, but up to now no standardized methods were available for this purpose. This paper presents the results from two study items promoted by Study Group 12 of the International Telecommunication Union. Work on these items resulted in one draft Recommendation on subjective methods for evaluating gaming Quality of Experience (QoE), as well as one approved Recommendation on influence factors on gaming QoE which might be considered in service planning and management. Future work which is necessary to predict gaming QoE is addressed in the end of the paper.

Recent advancements of network architecture such as 5G networks, promise cloud services with strict network constrains a bright future. Cloud gaming as an interactive service has strict end-to-end delay constraints. Therefore, many studies investigated the impact of network parameters such as delay or packet loss on gaming QoE. However, they mostly compared games or genres with each other and neglected the fact even two levels of the same game may have different sensitivity toward delay. In order to understand the game characteristics that cause this difference in delay sensitivity, a bottom-up approach by means of modifiable open source games can be of high value. In this paper we present a game designed to tackle this issue. The game allows to artificially change characteristics of the game, such as the pace and size of objects, and also simulate influences like delay, packet loss or a reduced frame rate. This allows the usage of the game also for crowdsourcing studies, where it is not possible to control the different network conditions of the participants, and to investigate the impact of spatial and temporal accuracy in respect to the sensitivity towards impairments.

This paper reports the further investigation of the perceptual quality space for video degradations [1], [2]. First, a new method is presented for directly assessing the identified perceptual quality dimensions. This method makes it possible to assess more test conditions in the same time, due to the reduced experimental effort. We tested this method in an user experiment. The method itself as well as the results are presented. Besides the rating of the perceptual quality dimensions, the test participants are also asked to rate the overall quality of the video. This allows drawing conclusions of the composition of the underlying quality dimensions to form overall quality for video. Second, an overall quality prediction model on the basis of the before investigated perceptual dimensions is presented. The results of the model are compared to the ratings obtained from subjective testing.

Obtaining reliable quality ratings for applications is a time-consuming and expensive process. Touch interaction data, however, can be recorded without additional effort when using mobile applications. It could be used to assess usability without having to conduct dedicated tests. This work investigates the possibility to estimate quality ratings for two mobile games solely from touch interactions without relying on performance or game logic measures. To capture pragmatic as well as hedonic quality aspects of usability as ground truth, the AttrakDiff Mini questionnaire was chosen. Two different simple mobile games were used to capture a variety of touch interactions (taps and swipes). To influence the user ratings, three versions of the games were presented: one in original quality, the two others with manipulated interaction quality. Touch interaction features were extracted from the recorded interaction data and used for training models using linear regression and multivariate adaptive regression splines (MARS). The models were built with three times 10-fold cross-validation on each game data set separately. The results indicate that the ratings for pragmatic quality are estimated significantly better than for hedonic quality and attractiveness. Furthermore, the transferability of the models from one game to the other was examined and the touch features that were most important for estimating quality ratings in this study were identified. Touch interactions seem to carry some information on usability, especially pragmatic quality, and it seems to be possible to estimate a broad direction of the user's usability perception. Just a few touch features carry the most information and can be used for simple but fast and powerful models. This could be applied as an automatic analytic tool for mobile applications as well as in adaptive applications that optimize themselves in terms of usability perceived by individual users.

End-to-end delay is an important factor when studying the quality of modern packet-based telephone conversations. Other than most degradations, delay cannot be assessed by listen-only tests and mainly impacts the structure and interactivity of the conversation. Because of this, the modeling of the perceived quality of conversation under delay is dependent on various instrumental as well as human-related influencing factors. Physiological methods in particular might provide additional insights into how human participants are affected by such quality degradations during natural conversations. In the present experiment, a novel dual-electroencephalo-graphy (dual-EEG) method was employed to investigate the effects of delay on 10 pairs of participants engaging in dyadic conversations. In each test session, neurophysiological activity was registered simultaneously in both interlocutors, who interacted in accordance with short conversation test scenarios (SCIs) through an audio network. Meanwhile, delays of different magnitudes (0, 800, 1600 ms) were inserted into the network, which was expected to interfere with the conversation and cause changes in the participants' internal processing and state. Analysis of subjective and behavioral measures shows a decrease in perceived quality and of conversational interactivity with higher levels of delay. Moreover, an initial intra-brain analysis of the recorded dual-EEG data (N = 18) revealed significant modulations in beta and gamma frequency bands by varying delay level. This suggests that attentional load increased with high delay, probably due to less amounts of single talk and focusing more on the adaptation of the own turn-taking behavior.

The present study utilized electroencephalography (EEG) to explore the neuro-electrical correlates of perceptual dimensions underlying speech quality. Specific focus lay on the P300 event-related brain potential (ERP) component to provide indication for internal processes related to attention and stimulus categorization. A high-quality (HQ) recording of a spoken word was impaired on each of three perceptual dimensions at a time, “discontinuity” (F), “noisiness” (N) and “coloration” (C), with F being realized through random erasure of distinct frames in the speech signal parts of the audio file. In an active three-stimulus oddball task, repeated presentations of the HQ stimulus led to the formation of a sensory/perceptual HQ reference, which was interrupted by infrequent occurrences of degraded “oddball” stimuli (F, N, C). Initial analysis of the obtained subjective and electrophysiological data suggested the following conclusions: 1) Participants perceived the three degraded stimuli as clearly impaired, but equal in terms of degradation intensity. Thus, variations in neural responses were assumed to reflect changes in the perceptual dimension along which the speech degradation had been induced. 2) Timing of the evoked P300 corresponded with temporal differences in the impairments, implying a later onset for “discretely” (F) compared to “continuously” (N, C) degraded stimuli after being categorized as task-irrelevant. Hence, P300 peak latency might prove useful to dissociate both classes of speech quality impairments on a neural level of analysis.

The experience of web services depends on multiple factors such as the usability of the service itself as well as influences on Quality of Experience (QoE) like waiting times. Existing studies so far have investigated both experimental factors (usability and QoE) in isolation, thus it is not possible to investigate dependencies between the two. In this study, we manipulated the usability and QoE of a news web page in the same experiment. Participants had to solve 9 tasks (three usability level and three QoE level, all combinations). Data (N=44) from two different sessions with different QoE manipulations were recorded: Simulation of initial page display delay via 1) a complete overlay and 2) a semi-transparent overlay. Our preliminary results show that the usability manipulation resulted in significant effects along multiple dimensions (QoE, aesthetic, and usability). Furthermore, also the implementation of the QoE manipulation had an effect on, e.g., performance parameters. Surprisingly, our results reveal that participants seem to be unable to rate the constructs usability and QoE independently from each other. Even after adding pronounced explanations of the constructs to the introductory phase of the second session of the experiment, the results still show a strong correlation between the two constructs. We thus recommend for similar future experiments to check beforehand whether Quality of Experience and Usability can be reliably judged and distinguished by naive participants.

Recent advances of streaming services and the upcoming new generation of mobile networks, 5G, offering low end-to-end delay as well as high bandwidths, promise a bright future for cloud gaming applications. Cloud gaming, in the contrary to traditional gaming services, suffers not only from system factors on the client, but is also affected significantly by the network, server specification and encoding parameters. In this paper, we present the results of a subjective experiment aiming to investigate the impact of two influencing factors, frame rate and bit rate, on the gaming Quality of Experience. The results reveal that a trade-off between an acceptable video quality and interaction quality exists. In case of very low bit rates, lowering the frame rate can improve the video quality while at some point, jerkiness becomes visible which affects the video quality negatively and the control over the game will be strongly reduced. Furthermore, even though in the gaming community a frame rate of 60 fps is desired, no significant difference for quality ratings, as well as performance ratings, was found between 60 fps and 25 fps. Therefore, it would be highly valuable for service providers to find an ideal strategy for this issue. In addition, we investigate the impact of video encoding on gaming experience dimensions. Finally, a first attempt to model the impact of two influencing factors on overall quality will be presented.

The performance of automatic speech recognition based on coded-decoded speech heavily depends on the quality of the transmitted signals, determined by channel impairments. This paper examines relationships between speech recognition performance and measurements of speech quality and intelligibility over transmission channels. Different to previous studies, the effects of super-wideband transmissions are analyzed and compared to those of wideband and narrowband channels. Furthermore, intelligibility scores, gathered by conducting a listening test based on logatomes, are also considered for the prediction of automatic speech recognition results. The modern instrumental measurement techniques POLQA and POLQA-based intelligibility have been respectively applied to estimate the quality and the intelligibility of transmitted speech. Based on our results, polynomial models are proposed that permit the prediction of speech recognition accuracy from the subjective and instrumental measures, involving a number of channel distortions in the three bandwidths. This approach can save the costs of performing automatic speech recognition experiments and can be seen as a first step towards a useful tool for communication channel designers. Copyright © 2017 ISCA. Amazon Alexa; Apple; DiDi; et al.; Furhat Robotics; Microsoft

This paper takes a view on the Quality-of-Experience (QoE) perspective of Cyber-Physical System (CpS) applications involving users. We provide an overview of characteristics of CPS applications, their relation to QoE and implications for assessing QoE of such systems. Furthermore, we incorporate eudaimonic aspects such as meaningfulness and personal growth that need to be considered in order to grasp the strongly multi-dimensional aspects of QoE in this context.

Crowdsourcing has established itself as a powerful tool to collect human input for data acquisition and labeling. Conventional laboratory experiments can now be addressed to a wider and diverse audience. This paper presents a study performed both in a laboratory and on a mobile-crowdsourcing platform, adopting a paired-comparison setup to obtain ratings of voice likability. We show considerations taken to adequately adapt the laboratory-based test to the remote-labour approach. Once all pair-comparison answers were collected, preference choice matrices were built and the Bradley-Terry-Luce probabilistic choice model was applied to estimate a ratio scale of preferences, reflecting the voice likability scores. Our results show a strong correlation between the scores obtained by the two approaches considered, which indicates the validity of crowdsourcing for the acquisition of voice likability ratings. This is of great benefit when datasets need to be quickly and reliably labeled for speech applications relying on detection or on synthesis of speaker and voice characteristics.

In several Quality of Experience (QoE) assessment disciplines it has become common practice to not only assess the overall QoE but also underlying quality dimensions. However, in most cases the relation between the underlying quality dimensions and the overall QoE is not clear. In addition, it is not known which method provides the best results when trying to model the overall QoE on the basis of its underlying quality dimensions. To provide new ideas and applicable approaches for the QoE community, four different approaches to model the overall QoE are presented in this paper. To this end, the use case QoE of transmitted speech with its underlying perceptual quality dimensions is used. Based on three available databases, linear regression, multivariate adaptive regression splines, peak rule, and a combination of linear regression and peak rule are presented and compared. The results and the discussion reveal new insights into the overall QoE modeling process of transmitted speech and allows for drawing conclusions regarding other QoE disciplines.

A diagnostic approach is presented to analyze interaction phases for describing the quality of speech communication services. The quality of individual phases is assessed in a first stage, and integrated to form a quality estimation of the entire interactive usage episode, and also across usage episodes. Constituents of such an approach for either subjective evaluation methods or instrumental prediction models are outlined. They would enable a focused and efficient diagnosis and monitoring of interactive speech service quality.

Recent advances in Virtual Reality (VR) technologies have resulted in a wider availability of Head Mounted Displays (HMDs). However, it is still unclear if VR gaming offers a substantial added value to players. For this reason a comparison of gaming experiences on VR HMD to those on mobile and PC, two other popular gaming platforms, is performed by conducting a user study via two games available on all three platforms. We explore the QoE of gaming by investigating momentous dimensions using the Player Experience of Need Satisfaction (PENS) questionnaire. The results show higher Presence and Autonomy obtained by using HMD when compared to the two other platforms. However, these factors alone did not improve the Overall Quality. To take advantage of the new technology, satisfaction of all psychological needs, especially Competency, must be assured.

Decision support systems increasingly support experts' work, but errors caused by them may have severe consequences, especially in the medical domain. To better understand how these inaccuracies affect experts' behavior and decisions, a quality of context model was constructed. Based on the model a controlled online experiment was conducted in which physicians had to treat hypothetical cases while seeing inaccurate medication suggestions. Initial results indicate that raw information quality and cover rate significantly improve patient outcomes and reduce required time, whereas high misleadingness leads to poorer patient outcomes and more time being required.

This paper reports the further investigation of the perceptual quality space for video degradations. A study was conducted to explore the space using a Paired Comparison (PC) test paradigm with a subsequent Multidimensional Scaling (MDS). This paper presents the test set-up as well as the analysis of the results. Furthermore, this paper reports the conclusions to further understand, diagnose, and analyse the perceptual quality space of video degradations mainly in the domain of video-telephony.

Cloud gaming promises many advantages from the user and game developer perspective. But besides its benefits, cloud gaming suffers from two bottlenecks: bandwidth and latency. Although, many researchers have identified delay as an important factor on the QoE in gaming, the cause of varying tolerances towards delay in different games is not yet well understood. In this paper, we want to show that delay sensitivity should not be generalized for an entire game or genre. Instead, a specific scenario with its underlying characteristics should be considered, since differences within the same game can be higher than differences to another game. To investigate the delay sensitivity of games, we assessed quality features such as control, difficulty, delay perception, annoyance, and fairness for highly similar scenarios only differing in their pace or perspective. Results show that changing the pace within the same game can lead to stronger differences in respect to the delay sensitivity than using another game type. Furthermore, we explain how quality features might be used in the judgement process of a player and define a set of input action metrics for a future delay sensitivity based classification of games.

Much work has been conducted to investigate the obstacles that keep users from using mitigations against security and privacy threats on smartphones. By contrast, we conducted in-depth interviews ( N = 19) to explore users’ motivations for voluntarily applying security and privacy actions on smartphones. Our work focuses on analyzing intrinsic motivation in terms of psychological need fulfillment. The findings from the interview study provide first insights on the salience of basic psychological needs in the context of smartphone security and privacy. They illustrate how security and privacy actions on smartphones are motivated by a variety of psychological needs, only one of them being the need for Security . We further conducted an online survey ( N = 70) in which we used questionnaires on psychological need fulfillment from the literature. The online survey is a first attempt to quantify psychological need fulfillment for security and privacy actions on smartphones. Whereas the results of the interview study indicate that Security and other needs play a role as motivators for employing security and privacy actions on smartphones, the online study does not support the need for Security as an outstanding motivator. Instead, in the online study, other needs such as Keeping the meaningful, Stimulation, Autonomy , and Competence show to be rather salient as motivators for security and privacy actions. Furthermore, the mean need fulfillment for security and privacy actions is in general rather low in the online survey. We conclude that there is scope for improvement to maximize psychological need fulfillment with security and privacy actions. In order to achieve a positive user experience with security and privacy technologies on smartphones, we suggest addressing additional psychological needs, beyond the need for Security , in the design of such technologies.

With the move to ubiquitous computing, user interfaces (UI) are no longer bound to specific devices. While this problem can be tackled using the model-based UI development (MBUID) process, the usability of the device-specific interfaces is still an open question. We are presenting a combined system that integrates MBUID with a cognitive modeling framework in order to provide usability predictions at development time. Because of their potential impact, our focus within usability problems lies on user errors. These are captured in a cognitive model that capitalizes on meta-information provided by the MBUID system such as the abstract role of a UI element within a task sequence (e.g., input, output, command). The free parameters of the cognitive model were constrained using data from two previous studies. A validation experiment featuring a new application and UI yielded an unexpected error pattern that was nonetheless consistent with the model predictions.

Technical Causes Analysis (P.TCA) is a method for identifying technical causes of sub-optimum speech transmission quality. Originally created as an expert procedure for the annotation of speech samples, its applicability to naïve listener was also studied. Due to the low agreement of naïve listener annotations, it was suggested that detailed training methods are necessary to lift naïve annotations to an agreement level of experts. The aim of this work was to develop training methods for naïve annotators. For this, two different training procedures were developed and tested in two separate annotation experiments. The results are analyzed and discussed regarding the effects of the trainings and their implications for the P.TCA annotation scheme. The outcome shows that these training methods did not meet the expectations for improving the inter-rater agreement of naïve annotators. It is concluded that trainings of 15 to 20 minutes rather confuse naïve annotators by conveying too much information in too little time, and that they are not sufficient to prepare naïve annotators. It is argued that much more extensive training is needed to raise naïve annotators to expert level, and that such a training must include both, in-depth introduction to the annotation process as well as detailed presentation and exercise regarding the P.TCA degradations.

This paper reports on ongoing work in ITU-T Study Group 12 on developing a universal scale quantifying quality across different types of speech communication services. In contrast to a quality rating scale, this scale should be deliberated from the judgment context as far as possible. As a consequence, it should be possible to compare different types of services on such a scale, in order to justify investments made in comparison to QoE gains to be expected. The paper explains the underlying rationale of such a scale, defines the requirements it has to fulfill, and outlines the way to its derivation. It invites further discussion to this work item of ITU-T Study Group 12, and could serve as an example for other types of services.

State-of-the-art approaches on text-to-speech (TTS) synthesis like unit selection and HMM synthesis are data-driven. Therefore, they use a prerecorded speech corpus of natural speech to build a voice. This paper investigates the influence of the size of the speech corpus on five different perceptual quality dimensions. Six German unit selection voices were created based on subsets of different sizes of the same speech corpus using the MaryTTS synthesis platform. Statistical analysis showed a significant influence of the size of the speech corpus on all of the five dimensions. Surprisingly the voice created from the second largest speech corpus reached the best ratings in almost all dimensions, with the rating in the dimension fluency and intelligibility being significantly higher than the ratings of any other voice. Moreover, we could also verify a significant effect of the synthesized utterance on four of the five perceptual quality dimensions.

In an interdisciplinary team, we developed personas and use cases addressing higher-level needs of residents in a smart home, such as safety, self reflection or evocation of memories. For one of the use cases, we describe a concrete usage scenario in which photos are taken automatically in predefined situations. We show how we implemented the required technology, a multi-camera person localization framework, and present results from a lab study to evaluate the performance and user experience of the technology.

It is known that nasal consonants and vowels are more effective than other phonemes for human speaker recognition. However, the influence of channel transmissions on the speakerdiscriminative capabilities of phonemes has not yet been examined. Specifically, the speech bandwidth has a strong effect on the human speaker recognition performance and also on the speech intelligibility. The phonemes that permit more accurate human speaker recognition are determined in this study by means of a speaker verification auditory test, focusing on the differences in performance when the stimuli are presented to listeners in narrowband and in wideband. The speech intelligibility is also investigated via an intelligibility test employing the same speech stimuli. Finally, the possible relationship between phonemes offering better human speaker recognition and more intelligible phonemes in the transition to an enhanced bandwidth is discussed.

Location-based mobile tools provide effective means for mobile participation, allowing better targeting of users and faster opinion gathering for municipalities and organizations. For participation purposes it is essential to reach the users as widely as possible, and for this, smartphones with ever-increasing adoption rates offer an optimal tool. However, privacy concerns might prevent the use of location-based applications, which results in a bias in opinion-gathering. In this paper, we present results of a field study tackling the issue of how privacy concern affects the adoption and use of a location-based mobile participation application, and what motivates the use of such applications. We find that users have varying motivations for participating in location-based polls, and that these motivations depend on the type of the poll in question. We show that privacy concerned users are less likely to be motivated by a monetary benefit, but rather by the importance of the topic, and that the intrinsic motivators that drive the privacy concerned users differ from those that drive the unconcerned. Finally, we show that, based on our study, it seems that privacy concern might be an inhibiting factor in adoption of a location-based mobile participation application.

Past studies have shown evidence of important speakerspecific content in the higher frequencies of the spectrum, which are filtered out by narrowband channels. Besides, wideband transmissions, which are gaining ground over narrowband communications, offer an extended range of frequencies which account not only for better speech quality and intelligibility, but also for an improved speaker recognition performance. In this work, different phoneme classes (fricatives, nasals, and vowels) were removed from speech of different bandwidths, and a series of i-vector based speaker verification experiments were conducted. Our results show that the performance enhancement with clean wideband speech with respect to clean narrowband speech is principally due to the presence of unvoiced fricative consonants. The effects of codec schemes of different bandwidths on the aforementioned speech are discussed.

Wideband communications permit the transmission of an extended frequency range compared to the traditional narrowband. While benefits for automatic speaker recognition can be expected, the extent of the contribution of the additional bandwidth in wideband is still unclear. This work compares the i-vector speaker verification performances employing speech signals of 0-4 kHz, 4-8 kHz, and 0-8 kHz and different sets of cepstral features extracted using linearlyand a mel-spaced filterbanks. Analyses of clean speech and of speech transmitted through commonly employed codecs are conducted separately for male and for female speech. Our evaluation on two different datasets shows the improved speaker verification performance with the extended bandwidth, and also that the linear scale can lead to better results for narrowband signals. The advantages of linear- over mel-scaled features for wideband depend on the speakers' gender and on the channel distortion.

Most telecommunication systems are used for communication between two people which interact during a conversation. In general, the quality of conversational speech is the major indicator for telecommunication-service providers to evaluate their systems. In this context, not only the assessment of the overall quality but also the analysis of the conversational speech quality is essential. We present an initial approach towards analyzing the conversational quality by separating a conversation in phases, and extracting individual corresponding perceptual dimensions of quality, as they are subjectively perceived by the system users. These dimensions can be combined for overall quality estimation and may separately be used to diagnose the technical reasons of quality degradation. For this reason, we review known and identify new dimensions of quality perception on the basis of subjective experiments. This enables to deeply analyze conversational speech quality for diagnosis and optimization of telecommunication systems. Index Terms: conversation, speech quality, perceptual quality dimensions, multidimensional analysis

This position paper claims that a major obstacle of offering video lectures for public universities appears to be the fact that they intend to compete with prestigious private universities regarding quality of the videos and complexity of the installed platform without being able to provide the additional resources required to do so. We argue that in other areas of teaching this issue has been acknowledged for a long time, and lacking resources are usually compensated for by primarily two means: individually offering provisory course material (manuscripts), and active participation of the student body in administering those. Based on this, a simple system is proposed that mostly draws on existing platforms and tools, and refrains from extensive video editing prior to publishing. We discuss technical and non-technical requirements and possible research directions that result from establishing such low-fidelity video lectures.

Subjective speech quality assessment depends on listener “quality” opinions after hearing a particular test speech stimulus. Subjective scores are given based on a perception and quality judgment process that is unique to a particular listener. These processes are postulated to be dependent on the listener's internal reference of what good and bad quality sounds like, as well as their mental and emotional states. To overcome this variability, subjective listening tests often average scores over several listeners. In this paper, we use electroencephalography (EEG) and self-assessment tools to investigate the neural and affective correlates of speech quality perception of reverberant speech, with the goal of obtaining new insights into human speech quality perception in complex listening environments. We show that EEG event related potentials (ERP) are a useful tool to monitor the conscious stages of neural-processing during a speech quality assessment task. Significant correlations were obtained between the so-called P300 ERP component and the reverberation time of the room, as well as between the P300 peak amplitude and emotional self-assessment ratings. These insights could lead to more effective ways of characterizing room acoustics for improved speech quality and intelligibility.

Together with the variety of networks, diverse terminals and devices, such as telephones with handset or hands-free mode, mobile phones and headsets, are commonly available for everyday calls. We conducted an auditory test to examine the combined influence of these user interfaces, audio bandwidths, coding schemes and packet loss on human speaker identification of previously known voices. The effects of the user interfaces on transmission and reception were tested separately with the different channel impairments. Our study confirms that the identification task is facilitated if the voices are transmitted through wideband instead of narrowband channels, and that headsets and hands-free phones take greater advantage of the improved bandwidth that is gaining ground rapidly.

We extract 1495 speech features from 2 subjectively evaluated text-to-speech (TTS) databases. These features are extracted from pitch, loudness, MFCCs, spectrals, formants, and intensity. The speech material is synthesized using up to 15 different TTS systems, some of them with up to 8 different voices. We develop quality predictors for TTS signals following two different approaches to handle the huge set of speech features: a three-step feature selection followed by a stepwise multiple linear regression and an approach based on support vector machines. The predictors are cross-validated via 3-fold cross validation (CV) and leave-one-test-out (LOTO) CV. Due to the high number of features we apply a strict CV method where the partitioning is realized prior to the feature scaling and feature selection steps. In comparison we also follow a semi-strict approach where the partitioning effectively takes place after these steps. In the 3-fold CV case we achieve correlations as high as .75 for strict CV and .89 for semi-strict CV. The more ambitious LOTO CV yields correlations around .80 for the male speakers whereas the results for the female voices show the need for improvement.

During the transition period from narrowband to wideband speech transmission services, Artificial Bandwidth Extension (ABE) algorithms are able to reduce the perceptual degradation of narrowband-transmitted speech signals by extending the audio bandwidth. In this paper, we analyze whether the resulting speech quality can be predicted reliably with instrumental models. Estimations from the new ITU standard POLQA, its predecessor WB-PESQ and the diagnostic DIAL model are compared to subjective listener judgments. This comparison reveals that the instrumental measures are not fully able to cope with ABE-processed speech, particularly in predicting ABE rank orders reliably. Reasons for this finding and corresponding diagnoses are discussed. Index Terms: speech quality, artificial bandwidth extension, instrumental quality prediction, speech transmission, diagnosis

This paper provides an overview of instrumental models for predicting the quality of speech signals. On the basis of perceptual and cognitive characteristics of the human auditory system which are relevant for quality judgment, approaches are presented which aim at predicting overall quality, intelligibility, or other quality dimensions from measurable parameters or signal characteristics. The approaches are discussed with respect to their underlying principles, showing that perception modeling can significantly improve prediction accuracy. Application examples are presented which make use of these algorithms for offline or online prediction, adaptation, or intelligibility improvement.

In this paper, we present ongoing experiments and insights regarding automatic assessment of perceived personality. While within the INTERSPEECH Speaker Trait Challenge participants will train systems in order to recognize binary targets along the Big 5 personality trait, we will analyze and discuss properties of the data, the labeling scheme and the predictive quality. Conducting factor analyses, estimating reliability, and building regression models capturing dimensions of personality we compare all results to our former and current work and introduce a new extension of our personality database. Eventually, this paper contributes in methodology and understanding on how to asses the perceived personality from an unknown speaker by humans and machines.

To give feedback on mobile devices, sound is commonly used in different ways. Much research has focused on the learnability and user performance with systems that have audio feedback. But so far, there is no standardized method to evaluate the subjective quality of auditory feedback messages. We describe a study to investigate the affective impression of short audio feedback on mobile devices and their functional connotation in three different contexts. Results show that context influences the affective impression of sounds and that there is a relation between ratings according affective quality and functional applicability. We conclude that sounds can be unobtrusive, but still convey their intended meaning in a working context as well as in a leisure time situation without being perceived as disturbing.

This paper presents research on perceptual quality dimensions of synthetic speech. We generated 57 stimuli from 16/19 female/male German text-to-speech systems (TTS) and asked listeners to judge the perceptual distances between them in a sorting task. Through a subsequent multidimensional scaling algorithm, we extracted three dimensions. Via expert listening and a comparison to ratings gathered on 16 attribute scales, the three dimensions can be assigned to naturalness of voice, temporal distortions and calmness. These dimensions are discussed in detail and compared to the perceptual quality dimensions from previous multidimensional analyses. Moreover, the results are analyzed depending on the type of TTS system. The identified dimensions will be used in the future to build a dimension-based quality predictor for synthetic speech.

Formal parameters of speech prosody are investigated concerning their ability to estimate the perceptual quality of text-to-speech (TTS) signals. The study is carried out for the German language using a broad databasis comprising a wide range of TTS systems and text materials. 18 purely acoustic markers, derived from Foand vocalic/consonantal durations, are analysed individually and in conjunction via cross-validated regression models. The Foslope within voiced segments proves particularly useful when integrated in a nonlinear fashion, whereas measures of durational variation perform comparably weak. The results highlight a strong potential for instrumental estimation techniques of TTS quality.

This paper presents work in progress which aims at analyzing the origins of interaction problems which certain users have when interacting with new technology. Our analysis is based on device models which categorize certain classes of devices via a pre-defined set of features. We provide examples which show that usability problems are partially caused by an erroneous transfer of device features to new/unknown devices.

The aim of this paper is to analyze the perceptual quality dimensions of state-of-the-art text-to-speech systems (TTS). Therefore, several pretests were conducted to determine a suitable set of attribute scales. The resulting 16 scales were used in a semantic differential on a diverse database containing 16 different TTS systems. A subsequent multidimensional analysis (Principal Axis Factor analysis with Promax rotation) resulted in three underlying quality dimensions. They were labeled naturalness, disturbances, and temporal distortions. A mapping of these factors onto the perceived overall quality revealed that naturalness contributes the most to the quality of TTS signals. Index Terms: speech synthesis, quality dimensions, multidimensional analysis

This contribution presents a new approach towards nonintrusive quality assessment of text-to-speech (TTS) signals. Perturbation measures which capture the degree of excitationspecific aperiodicity in voiced speech are investigated concerning their quality implications in synthesized speech. Based on two independent TTS databases for which formal attributebased listening tests have been conducted, we show that perturbation measures are sensitive to quality aspects of prosody and voice characteristic. Furthermore a dominant dependency on TTS type, namely non-uniform unit-selection and diphone synthesis, is identified. Yet, considerable differences between male and female TTS samples are recognized, emphasizing the need for gender-specific quality assessment.

In this paper, we validate the application of an established personality assessment and modeling paradigm to speech input, and extend earlier work towards text independent speech input. We show that human labelers can consistently label acted speech data generated across multiple recording sessions, and investigate further which of the 5 scales in the NEO-FFI scheme can be assessed from speech, and how a manipulation of one scale influences the perception of another. Finally, we present a clustering of human labels of perceived personality traits, which will be useful in future experiments on automatic classification and generation of personality traits from speech.

Video quality has turned out to be a crucial aspect of multimodal transmission services. Most common video quality tests rely on a conscious judgment of test participants reflecting their internal quality perception. But it is not completely clear how this conscious rating is formed, neither in the auditory nor in the visual domain. Initial audio tests with Electroencephalography (EEG) have shown that EEG recordings can be used as a sensitive and non-intrusive method for quality assessment. In this paper we conducted first experiments of pure video quality tests with EEG to complement this approach in the visual domain. One of the goals for this experiment was to show that there is a different pattern in the EEG data for cases with no distortion compared to cases when there was a distortion and the subject recognized this in the subjective test.

Video-telephony and mobile TV are typical multimedia services which are becoming a part of the everyday life due to the increase in bandwidth availability and also viewing devices with larger screen sizes (smart phone, PDA, etc). To ensure high quality, packet layer parametric quality prediction models for audio-visual services like video-telephony and IPTV video streaming have emerged and are still under development. Those parametric models depend on a set of parameters which have to be tuned for every specific application. In this work, we carry out an experiment to analyze the impact of video resolution and up scaling operation on perceived quality. We could show that the current parametric models can be modified to explicitly integrate the joined effect of resolution and display video size.

Mobile phones are rapidly becoming small-size general purpose computers, so-called smartphones. However, applications and data stored on mobile phones are less protected from unauthorized access than on most desktop and mobile computers. This paper presents a survey on users' security needs, awareness and concerns in the context of mobile phones. It also evaluates acceptance and perceived protection of existing and novel authentication methods. The responses from 465 participants reveal that users are interested in increased security and data protection. The current protection by using PIN (Personal Identification Number) is perceived as neither adequate nor convenient in all cases. The sensitivity of data stored on the devices varies depending on the data type and the context of use, asking for the need for another level of protection. According to these findings, a two-level security model for mobile phones is proposed. The model provides differential data and service protection by utilizing existing capabilities of a mobile phone for authenticating users.

In this paper, the quality gain of super-wideband (SWB) speech, transmitted in the much wider frequency range of 50-14000 Hz compared to the standard 300-3400 Hz narrowband, is quantified employing the E-model framework, a parametric tool for speech quality prediction. Based on two listening experiments, a linear extrapolation of the E-model transmission rating scale was found that leads to a maximum quality advantage of 39% relative to wideband (50-7000 Hz) transmission, and 79% relative to narrowband. Furthermore, narrowband, wideband, and super-wideband conditions can be quantified on this universal quality scale. Equipment Impairment Factors were derived and discussed for several SWB codecs. It will further be shown that a model quantifying the quality impact of linear distortions, reflected by the Bandwidth Impairment Factor, can successfully applied to SWB conditions. The correlation between the overall impairment and the model predictions amounts to r = 0.977 for linearly distorted speech samples.

Speech quality in Next Generation Mobile Networks (NGMN) is critical. Accordingly, we analyze in this paper, how the quality estimation provided by wideband PESQ, a signal-based speech quality prediction model, differs from the user perception. We find that the model underestimates the auditory quality in certain NGMN conditions: 1) wideband-narrowband speech codec switching, 2) speech signal fading during codec switching, and 3) talk-spurt internal time-shifting due to jitter buffer instability. By pointing out the impact of these degradations on the speech signal, this paper contributes to potential improvements and adaptation of the wideband PESQ model for NGMNs.

International audience

Models for the prediction of user judgments from interaction data can be used in different contexts such as system quality assessment, monitoring of deployed systems, or as a reward function in learned dialog managers. Such models still show a considerable lack with respect to their generalizability [6]. This paper specifically addresses this issue. We propose to use a Mixture of Experts approach for cross-database predictions. In Mixture of Experts, several classifiers are trained on subsets of the data showing specific characteristics. Predictions of each expert model are combined for the overall prediction result. We show that such an approach can improve the cross-database prediction accuracy. Index Terms: user judgments, quality prediction, evaluation, spoken dialog systems

In this paper, we compare and combine different approaches for instrumentally predicting the perceived quality of Text-to-Speech systems. First, a log-likelihood is determined by comparing features extracted from the synthesized speech signal with features trained on natural speech. Second, parameters are extracted which capture quality-relevant degradations of the synthesized speech signal. Both approaches are combined and evaluated on three auditory test databases. The results show that auditory quality judgments can in many cases be predicted with a sufficiently high accuracy and reliability, but that there are considerable differences, mainly between male and female speech samples.

In this paper, we present first results on applying a personality assessment paradigm to speech input, and comparing human and automatic performance on this task. We cue a professional speaker to produce speech using different personality profiles and encode the resulting vocal personality impressions in terms of the Big Five NEO-FFI personality traits. We then have human raters, who do not know the speaker, estimate the five factors. We analyze the recordings using signal-based acoustic and prosodic methods and observe high consistency between the acted personalities, the raters' assessments, and initial automatic classification results. This presents a first step towards being able to handle personality traits in speech, which we envision will be used in future voice-based communication between humans and machines.

This paper describes a user study about the influence of efficiency on modality selection (speech vs. virtual keyboard/ speech vs. physical keyboard) and perceived mental effort. Efficiency was varied in terms of interaction steps. Based on previous research it was hypothesized that the number of necessary interaction steps determines the preference for a specific modality. Moreover the relationship between perceived mental effort, modality selection and efficiency was investigated. Results showed that modality selection is strongly dependent on the number of necessary interaction steps. Task duration and modality selection showed no correlation. Also a relationship between mental effort and modality selection was not observed.

An ideal system should be usable and secure. However, increasing the security of a system often makes its use more cumbersome and less efficient. This tradeoff between usability and security poses major challenges for system designers. System security may be impaired when users override or ignore security features to facilitate the use of the system. Little empirical data are available on user behavior regarding the tradeoff between security and usability. To obtain such data we developed a controlled research environment (i.e., a microworld) for studying users’ tendency to take precautionary actions as a function of the tradeoff between a system’s usability and the level of security the system provides. It is a modified version of a “Tetris” game and includes an alert system that warns about possible virus attacks, which, if not prevented, can cause losses of monetary earnings. Users could alter the threshold settings of the security system. The system allows us to manipulate the usability cost of using a security feature, the severity of the consequences of an attack, the likelihood that a threat will occur, and the statistical properties of the security system. In a preliminary experiment two groups of 10 participants each used the system for three 20-minutes sessions. The likelihood for an attack was 4 times higher for one group than for the other group. The likelihood of an attack clearly affected participants' behavior. When attacks were more likely, participants altered thresholds more frequently, selected more cautious thresholds, and tended to respond more to security system alerts. This microworld is a step towards the development of quantitative predictive models of user interactions with security features while using a system.

In NGNs, handovers between different wireless access technologies provide seamless roaming during voice calls. The resulting speech quality depends on the audio bandwidth of the speech codecs used in the respective networks, as well as on degradations resulting from the handover, coding, and packet loss. We present the results of four listening experiments where speech quality is quantified as a function of network and codec characteristics, and compare them to estimations obtained from instrumental models. The results show when and under which circumstances a network handover and/or codec changeover should be scheduled in order to obtain better speech quality. This is important for the development of high-quality roaming strategies.

The increase of online services, such as eBanks, WebMails, in which users are verified by a username and password, is increasingly exploited by identity theft procedures. Identity Theft is a fraud, in which someone pretends to be someone else is order to steal money or get other benefits. To overcome the problem of identity Theft an additional security layer is required. Within the last decades the option of verifying users based on their keystroke dynamics was proposed during login verification. Thus, the imposter has to be able to type in a similar way to the real user in addition to having the username and password. However, verifying users upon login is not enough, since a logged station/mobile is vulnerable for imposters when the user leaves her machine. Thus, verifying users continuously based on their activities is required. Within the last decade there is a growing interest and use of biometrics tools, however, these are often costly and require additional hardware. Behavioral biometrics, in which users are verified, based on their keyboard and mouse activities, present potentially a good solution. In this paper we discuss the problem of identity theft and propose behavioral biometrics as a solution. We survey existing studies and list the challenges and propose solutions.

Technological developments and the addition of new features to existing applications or services require the inclusion of security mechanisms to protect the user. When using these mechanisms the user faces a tradeoff between more risky and more efficient or safer and less efficient use of the system. We discuss this tradeoff and present a novel complementary experimental system which provides researchers and corporations the ability to explore and model the usability and security tradeoff in the context of user interaction with security systems and psychological acceptability, even before the actual development and implementation processes have ended.

With the advent of audio-visual IP clients, video telephony becomes a realistic option in many application scenarios. In order to guarantee an adequate quality to its users, providers of audio-visual telephony services need to know the impact of the audio and video transmission channel characteristics on perceived Quality of Experience (QoE) in a realistic interactive setting. For this aim, a conversational video telephony experiment was conducted where the audio and video channel settings were adjusted in a controlled way, and participants were asked about the perceived audio, video and overall quality after carrying out a conversation over the audio-visual channel. We analyze the results with respect to the impact the two modalities have, as well as with respect to the impact of the conversation scenario.

We present a method to estimate the perceived listening quality by a subscriber at the end of a common voice telephony conversation. This method was recently introduced in ETSI STQ mobile and was approved as TR 102 506 'Speech Quality per Call'. The idea is to calculate this "speech-quality-per-call" value based on short-term listening quality scores (so-called Mean Opinion Scores, MOS), as they are usually derived by subjective listening-only tests, or based on predictions of short-term scores by means of objective measures. It is shown that a pure linear averaging of short-term scores will not predict the perceived quality of the entire call sufficiently well in case of a non-stationary quality over the call. Mainly the "recency effect" and the out-weighting of very bad parts in a call have to be considered in an adequate way. An algorithm was developed that allows the obtained "speech-quality-per-call" score to be predicted on the basis of the MOS of the individual utterances. The algorithm can be applied for various lengths of call and numbers of individual utterances. Since speech quality is usually objectively predicted in real networks the approach was also proven and confirmed for objectively obtained quality scores. This paper follows widely the work and the decisions taken within ETSI STQ mobile.

The paper presents a parameter-based model for predicting the perceived quality of transmitted video for IPTV applications. The core model we derived can be applied both to service monitoring and network or service planning. In its current form, the model covers H.264 and MPEG-2 coded video (standard and high definition) transmitted over IP-links. The model includes factors like the coding bit-rate, the packet loss percentage and the type of packet loss handling used by the codec. The paper provides an overview of the model, of its integration into a multimedia model predicting audio-visual quality, and of its application to service monitoring. A performance analysis is presented showing a high correlation with the results of different subjective video quality perception tests. An outlook highlights future model extensions.

Vertical handovers in Next Generation Networks enable a new experience of mobility, since application layer sessions are maintained while roaming across different access networks. For real-time media services like Voice-over-IP, a change in the underlying network technology is particulary challenging. Ongoing calls are not suspended during the handover, however, the handover may go along with an audible gap during the transition time due to lost or delayed packets and an adaptation in call parameters such as those of the employed speech codec. This, in turn, may translate in an unfamiliar speech quality perception. This contribution introduces an ergonomic technique that aims at seamlessly switching the speech codec in Voice-over-IP calls during vertical handovers, based on SIP/SDP session renegotiation, the establishment of a parallel media stream and RTP packet filtering. Evaluation results are presented, showing that the proposed approach does not cause any interruption of the audio stream in about 90% of the test cases, clearly outperforming simple re-negotiation of session parameters that does not take a seamless transition into account (interruptions in all test cases). PESQ speech quality estimates reveal a quality advantage of 5.4% on average for the considered scenario.

In this paper we report the results of a user study evaluating talking heads in the smart home domain. Three noncommercial talking head components are linked to two freely available speech synthesis systems, resulting in six different combinations. The influence of head and voice components on overall quality is analyzed as well as the correlation between them. Three different ways to assess overall quality are presented. It is shown that these three are consistent in their results. Another important result is that in this design speech and visual quality are independent of each other. Furthermore, a linear combination of both quality aspects models overall quality of talking heads to a good degree.

In this paper we present a new approach to the automation of usability evaluation for interactive systems. Design ideas or complete systems are modeled as a conditional state machine. Then, user interactions with the system are simulated on the basis of tasks, by first searching for possible solution paths and then generating deviations from these paths under consideration of user groups and system attributes. The approach has been implemented into a workbench which supports the modeling of the system and the evaluation of the simulations. We present first results for the reliability of the approach in modeling interactions with a spoken dialog system. Index Terms: usability evaluation, user simulation, spoken dialog system

In this letter, the first steps toward the development of a signal-based instrumental quality measure for text-to-speech (TTS) systems are described. Hidden Markov models (HMM), trained on naturally-produced speech, serve as artificial text- and speaker-independent reference models against which synthesized speech signals are assessed. A normalized log-likelihood measure, computed between perceptual features extracted from synthesized speech and a gender-dependent HMM reference model, is proposed and shown to be a reliable parameter for multidimensional TTS quality diagnosis. Experiments with subjectively scored synthesized speech data show that the proposed measure attains promising estimation performance for quality dimensions labeled overall impression, listening effort, naturalness, continuity/fluency, and acceptance.

Proper usability evaluations of spoken dialogue systems are costly and cumbersome to carry out. In this paper, we present a new approach for facilitating usability evaluations which is based on user error simulations. The idea is to replace real users with simulations derived from empirical observations of users ’ erroneous behavior. The simulated errors must cover both system-driven errors (e.g., due to poor speech recognition) as well as conceptual errors and slips of the user, because neither alone is predictive of perceived usability. The simulation is integrated into a workbench which produces reports of typical and rare errors, and which allows usability ratings to be predicted. If successful, this workbench will help designers in making choices between system versions and lower testing costs at early phases of development. Challenges to the approach are discussed and solutions proposed. Index Terms: spoken-dialogue system, evaluation, usability 1.

It is the aim of the present paper to analyze the perceptual quality dimensions of modern telephone connections. Such connections differ from standard connections in their timevariant characteristics (e.g., due to Voice-over-IP transmission or due to noise reduction algorithms) and their user interfaces (e.g., hands-free terminals). With the help of two independent auditory experiments with subsequent multidimensional analyses, three perceptual dimensions were identified for a diverse set of stimuli. These dimensions were labeled “directness/frequency content”, “continuity”, and “noisiness”. Overall listening quality scores were collected in a separate experiment. A mapping of the obtained dimensions onto the overall listening quality scores by means of a linear model revealed that “continuity” appears to be the most important dimension in terms of overall listening quality. Index Terms: assessment and modeling of speech quality, quality dimensions, multidimensional analyses

A new method is described for quantifying the quality degradation introduced by wide-band speech codecs via a one-dimensional impairment factor. The method is based on auditory listening-only tests, but the resulting impairment factors may be used for predicting speech quality in an instrumental way, e.g., for network planning purposes. Following the method, auditory test results are first transformed to an overall quality rating scale, and then adjusted to rule out test-specific effects. The derived impairment factors fit into the common framework which is defined by the E-model for narrow-band telephone networks, and which is hereby extended towards wide-band speech transmission. This paper presents the necessary auditory test data, describes the derivation and adjustment methodology, and provides numerical values for a range of wide-band speech codecs. The values are tested for their robustness in case of codec tandems and adjusted to represent the effects of packet loss

This paper presents an evaluation of the speech output component of a smart home system developed under the European INSPIRE project. In particular, it is investigated how the 'personality' of the system which is conveyedby the output speech is accepted by the user. Three different metaphors are compared: An assistant visualized by a talking head, an invisible assistant, and multiple intelligent devices. The results show that a personal assistant metaphor leads to more homogenous judgments for the individual voices. Potential reasons for this finding are discussed.

This paper reports on the evaluation of the speech output component of a dialogue system that enables the control of different home appliances via speech. It was simulated that participants called the system from the car and listened to feedback messages from the system. Their task was to judgethe quality of the system’s output while performing a driving task in the TNO driving simulator. The following effects on the quality judgements were studied: type of system prompt (natural, synthesized), transmission degradations (circuit noise, speech codec, packet loss), and the driving task. The results are presented in terms of these aspects.

The impairment factor methodology has been adopted by telecommunication experts (ITU-T, ETSI) for describing the relative impact of telephone transmission degradations on the overall quality of transmitted speech. I nput parameters to this methodology are mainly instrumentally measurable characteristics of the transmission path, with the exception of low bit-rate codecs, whose perceptual characteristics still have to be determined in auditory tests. In this paper, we describe a new approach for deriving impairment factors for low bit-rate codecs in a purely instrumental way. Using instrumental quality prediction models like PESQ or TOSQA, quality estimations are obtained for the codec under test, as well as for several reference codecs. The reference conditions are subsequently used for anchoring the speech codec of interest, so that the impairments it introduces can be combined with other degradations in order to obtain an overall quality estimation for the whole transmission channel. A comparison with defined values for well-known codecs shows a high correlation of instrumentally derived impairment factors with the corresponding defined values, as well as with auditory test data.

Abstract The basic hypothesis is that cry vocalizations of hearing-impaired infants differ from those of their counterparts with normal hearing abilities due to the lack of auditory feedback. This assumption, based on observations made by clinical experts, is investigated by means of auditory experiments with naive and expert listeners, and by signal analysis of the cries. The listening experiment shows that it is possible for experts to auditorily classify cries for both infant groups, based on the voice related and melodic cry features. The cries of profoundly hearing-impaired infants are different regarding their perceived sound, rhythm and melody. The sound may well be correlated to spectral characteristics, and melodic and rhythmic parameters are extracted which differ significantly for the two infant groups. The findings are discussed in the context of a cry production model. The extracted signal parameters enable an automatic classification of the cries by means of topological feature maps, which may later be used as the basis for an early supplementary diagnostic tool.