While robotics and game development have independently achieved significant progress in creating interactive and intelligent systems, a deeper collaboration between these fields could be mutually beneficial. This paper argues for more collaboration, highlighting current limited interactions and proposing directions for future research. We discuss shared foundations such as Artificial Intelligence, Extended Reality, and the increasing use of common tools and standards. We then propose opportunities where game development methodologies can advance robotics (e.g., gamified data collection and richer simulation environments) and where robotics research can contribute to games (e.g., improved NPC autonomy and embodied intelligence). This cross-disciplinary interaction can accelerate innovation and lead to more intelligent and usercentered technologies in both domains.

Imitation learning is an effective approach for training game-playing agents and, consequently, for efficient game production. However, generalization-the ability to perform well in related but unseen scenarios-is an essential requirement that remains an unsolved challenge for game AI. Generalization is difficult for imitation learning agents because it requires the algorithm to take meaningful actions outside of the training distribution. In this paper we propose a solution to this challenge. Inspired by the success of data augmentation in supervised learning, we augment the training data so the distribution of states and actions in the dataset better represents the real state-action distribution. This study evaluates methods for combining and applying data augmentations to observations, to improve generalization of imitation learning agents. It also provides a performance benchmark of these augmentations across several 3D environments. These results demonstrate that data augmentation is a promising framework for improving generalization in imitation learning agents.

This article proposes a novel deep reinforcement learning algorithm to perform automated analysis and detection of gameplay issues in complex 3-D navigation environments. The curiosity-conditioned proximal trajectories (CCPT) method combines curiosity and imitation learning to train agents that methodically explore in the proximity of known trajectories derived from expert demonstrations. We show how our new algorithm can explore complex environments, discovering gameplay issues, and design oversights in the process, and recognize and highlight them directly to game designers. We also propose a visual analytics interface to aid interpretation of results from the method. This interface transforms information from complex models into interpretable and interactive visual forms. We further demonstrate the effectiveness of the algorithm in a novel 3-D navigation environment, which reflects the complexity of modern video games. Our results show a higher level of coverage and bug discovery than baseline methods, demonstrating that our method can be a useful tool for game designers to automatically identify design issues. Moreover, our experiments show that the visual explanations provided by the analytics interface result in a significant increase in user trust and acceptance of automated playtesting and increased confidence in the use of machine learning techniques for video game development.

Reinforcement learning has been widely successful in producing agents capable of playing games at a human level. However, this requires complex reward engineering, and the agent’s resulting policy is often unpredictable. Going beyond reinforcement learning is necessary to model a wide range of human playstyles, which can be difficult to represent with a reward function. This paper presents a novel imitation learning approach to generate multiple persona policies for playtesting. Multimodal Generative Adversarial Imitation Learning (Multi-GAIL) uses an auxiliary input parameter to learn distinct personas using a single-agent model. MultiGAIL is based on generative adversarial imitation learning and uses multiple dis-criminators as reward models, inferring the environment reward by comparing the agent and distinct expert policies. The reward from each discriminator is weighted according to the auxiliary input. Our experimental analysis demonstrates the effectiveness of our technique in two environments with continuous and discrete action spaces.

In games, as in many other domains, design validation and testing is a significant challenge as systems are growing in size and manual testing is becoming infeasible. In this position paper we outline an approach to automated game validation based on an imitation learning technique, and provide an analysis of the potential benefits to automated game testing. The method leverages a data-driven technique, which requires little effort and time and no knowledge of machine learning or programming, that designers can use to efficiently train game testing agents. We evaluate the validity of our claim by conducting a user study with industry experts. The survey results presented in this paper demonstrate the potential of a data-driven approach to reduce effort and enhance the quality of game testing. Moreover, the survey reveals several open challenges. To this end, we analyze the identified challenges and provide a basis for further research and discussion, as well as to help guide the development of imitation learning for game testing.

Modern video games are rapidly growing in size and scale, and to create rich and interesting environments, a large amount of content is needed. As a consequence, often several thousands of detailed 3D assets are used to create a single scene. As each asset’s polygon mesh can contain millions of polygons, the number of polygons that need to be drawn every frame may exceed several billions. Therefore, the computational resources often limit how many detailed objects that can be displayed in a scene. To push this limit and to optimize performance one can reduce the polygon count of the assets when possible. Basically, the idea is that an object at farther distance from the capturing camera, consequently with relatively smaller screen size, its polygon count may be reduced without affecting the perceived quality. Level of Detail (LOD) refers to the complexity level of a 3D model representation. The process of removing complexity is often called LOD reduction and can be done automatically with an algorithm or by hand by artists. However, this process may lead to deterioration of the visual quality if the different LODs differ significantly, or if LOD reduction transition is not seamless. Today the validation of these results is mainly done manually requiring an expert to visually inspect the results. However, this process is slow, mundane, and therefore prone to error. Herein we propose a method to automate this process based on the use of deep convolutional networks. We report promising results and envision that this method can be used to automate the process of LOD reduction testing and validation.

Abstract We present Voice2Face: a Deep Learning model that generates face and tongue animations directly from recorded speech. Our approach consists of two steps: a conditional Variational Autoencoder generates mesh animations from speech, while a separate module maps the animations to rig controller space. Our contributions include an automated method for speech style control, a method to train a model with data from multiple quality levels, and a method for animating the tongue. Unlike previous works, our model generates animations without speaker‐dependent characteristics while allowing speech style control. We demonstrate through a user study that Voice2Face significantly outperforms a comparative state‐of‐the‐art model in terms of perceived animation quality, and our quantitative evaluation suggests that Voice2Face yields more accurate lip closure in speech with bilabials through our speech style optimization. Both evaluations also show that our data quality conditioning scheme outperforms both an unconditioned model and a model trained with a smaller high‐quality dataset. Finally, the user study shows a preference for animations including tongue. Results from our model can be seen at https://go.ea.com/voice2face .

Training RL agents to solve novel environments is a notoriously difficult task. Here we present a new approach ARLPCG: Adversarial Reinforcement Learning for Procedural Content Generation, which procedurally generates and tests previously unseen environments with an auxiliary input as a control variable. The procedurally generated environments induces state diversity which increases the generalizability of the trained agents. ARLPCG deploys an adversarial model with one PCG RL agent (called Generator) and one solving RL agent (called Solver). The Generator receives a reward signal based on the Solver's performance, which encourages the environment design to be challenging but not impossible. To further drive diversity and control of the environment generation, we propose using auxiliary inputs for the Generator. The benefit is two-fold: Firstly, the Solver achieves better generalization through the Generator's generated challenges. Secondly, the trained Generator can be used as a creator of novel environments that, together with the Solver, can be shown to be solvable. We create two types of 3D environments to validate our model, representing two popular game genres: a third-person platformer and a racing game. In these cases, we show that ARLPCG has a significantly better solve ratio, and that the auxiliary inputs renders the levels creation controllable to a certain degree. For a video compilation of the results please visit https://youtu.be/z7q2PtVsT0I.

As modern games continue growing both in size and complexity, it has become more challenging to ensure that all the relevant content is tested and that any potential issues are properly identified and fixed. Attempting to maximize testing coverage using only human participants, however, results in a tedious and hard to orchestrate process which normally slows down the development cycle. Complementing playtesting via autonomous agents has shown great promise accelerating and simplifying this process. This paper addresses the problem of automatically exploring and testing a given scenario using reinforcement learning agents trained to maximize game state coverage. Each of these agents is rewarded based on the novelty of its actions, thus encouraging a curious and exploratory behaviour on a complex 3D scenario where previously proposed exploration techniques perform poorly. The curious agents are able to learn the complex navigation mechanics required to reach the different areas around the map, thus providing the necessary data to identify potential issues. Moreover, the paper also investigates different visualization strategies and evaluates how to make better use of the collected data to drive design decisions and to recognize possible problems and oversights.

General game testing relies on the use of human play testers, play test scripting, and prior knowledge of areas of interest to produce relevant test data. Using deep reinforcement learning (DRL), we introduce a self-learning mechanism to the game testing framework. With DRL, the framework is capable of exploring and/or exploiting the game mechanics based on a user-defined, reinforcing reward signal. As a result, test coverage is increased and unintended game play mechanics, exploits and bugs are discovered in a multitude of game types. In this paper, we show that DRL can be used to increase test coverage, find exploits, test map difficulty, and to detect common problems that arise in the testing of first-person shooter (FPS) games.

This demo features an embodiment of Smart Eye-tracking Enabled Networking (SEEN), a novel content delivery method for optimizing the provision of 360° video streaming. SEEN relies on eye-gaze information from connected eye trackers to provide high quality, in real time, in the proximity of users' fixations points, while lowering the quality at the periphery of the users' fields of view. The goal is to exploit the characteristics of the human vision to reduce the bandwidth required for the mobile provision of future data intensive services in Virtual Reality (VR). This demo provides a tangible experience of the tradeoffs among bandwidth consumption, network performances (RTT) and Quality of Experience (QoE) associated with SEEN's novel content provision mechanisms.

This paper presents a novel approach to content delivery for video streaming services. It exploits information from connected eye-trackers embedded in the next generation of VR Head Mounted Displays (HMDs). The proposed solution aims to deliver high visual quality, in real time, around the users' fixations points while lowering the quality everywhere else. The goal of the proposed approach is to substantially reduce the overall bandwidth requirements for supporting VR video experiences while delivering high levels of user perceived quality. The prerequisites to achieve these results are: (1) mechanisms that can cope with different degrees of latency in the system and (2) solutions that support fast adaptation of video quality in different parts of a frame, without requiring a large increase in bitrate. A novel codec configuration, capable of supporting near-instantaneous video quality adaptation in specific portions of a video frame, is presented. The proposed method exploits in-built properties of HEVC encoders and while it introduces a moderate amount of error, these errors are indetectable by users. Fast adaptation is the key to enable gaze-aware streaming and its reduction in bandwidth. A testbed implementing gaze-aware streaming, together with a prototype HMD with in-built eye tracker, is presented and was used for testing with real users. The studies quantified the bandwidth savings achievable by the proposed approach and characterize the relationships between Quality of Experience (QoE) and network latency. The results showed that up to 83% less bandwidth is required to deliver high QoE levels to the users, as compared to conventional solutions.

This paper explores the key tradeoffs for the design and optimization of eye-gaze based content provision for video streaming services. The proposed end-to-end solution, called “foveated content provision”, uses real-time information from connected eye-trackers to dynamically deliver optimized video frames, with higher resolution in areas corresponding to the users' fovea while lowering the quality at the periphery. In this novel approach, the main system constraint is the achievable latency (RTT) in the communication link between content servers and user clients. To cope with various latency levels, several design choices are presented, including varying the size of the high quality region or the resolution for the areas in the user's peripheral field of view. The paper presents a set of experimental results, obtained with real users via a novel event-driven experience sampling method, which is specifically developed to address Quality of Experience (QoE) in foveated content delivery. The results show that several operating points within the system parameter space allows to deliver high levels of QoE, even at latency levels comparable to current 4G networks.

In this paper, we study the influence of video stalling on QoE. We provide QoE models that are obtained in realistic scenarios on the smartphone, and provide energy-saving approaches for smartphone by leveraging the proposed QoE models in relation to energy. Results show that approximately 5J is saved in a 3 minutes video clip with an acceptable Mean Opinion Score (MOS) level when the video frames are skipped. If the video frames are not skipped, then it is suggested to avoid freezes during a video stream as the freezes highly increase the energy waste on the smartphones.

This paper is an introduction to the "Future IKEA Catalogue", enclosed here as an example of a design fiction produced from a long standing industrial-academic collaboration. We introduce the catalogue here by discussing some of our experiences using design fiction` with companies and public sector bodies, giving some background to the catalogue and the collaboration which produced it. We have found design fiction to be a useful tool to support collaboration with industrial partners in research projects - it provides a way of thinking and talking about present day concepts, and present day constraints, without being overly concerned with contemporary challenges, or the requirements of academic validation. In particular, there are two main aspects of this we will discuss here, aspects that are visible in the enclosed catalogue itself. The first is the potential of design fiction as a sort of 'boundary object' in industry and academic collaboration, and second the role of critique. After this introduction to the paper we enclose the output of our collaboration in the form of the catalogue itself.

Abstract—This paper describes a novel QoE-aware adaptive video streaming method that enhances the viewing experience on mobile devices and reduces cellular network bandwidth consumed by Dynamic Adaptive Streaming over HTTP (DASH) by consid-ering perceptual video quality and data rate channel conditions in the bitrate adaptation process. By streaming an optimized video for the particular video quality and channel conditions to a mobile device, we can improve the worst video qualities caused by DASH streaming and reduce quality variations using fewer number of bits. Keywords—QoE, Adaptive video streaming, DASH I.

This paper describes a study of how social media could be integrated and used in Mobile Experience Sampling. As addressed in previous studies, Experience Sampling Method relies on high response frequency. However participants may experience it as a burden which may cause delay or even suspension of data collection. We have developed a system that allows participants in Mobile Experience Sampling studies to share their questions and answers on social media. We tested our system in a group of 40 participants. The study shows that enabled sharing of ESM questions significantly increased response and participation rates, in our test by +43%, which also indicates its influence on participants' compliance and motivation levels. This paper presents the study and discusses some further use and influence of social media in Experience Sampling.

The explosive growth of Over-the-top (OTT) online video strains capacity of operators' networks, which severely threatens video quality perceived by end users. Since video is very bandwidth consuming, its distribution costs are becoming too high to scale with network investments that are required to support the increasing bandwidth demand. Content providers and operators are searching for solutions to reduce this video traffic load, without degrading their customers' perceived Quality of Experience (QoE). This paper proposes a method that can programmatically optimize video content for desired QoE according to perceptual video quality and device display properties, while achieving bandwidth and storage savings for content providers, operators, and end users. The preliminary results obtained with Samsung Galaxy S3 phone show that up to 60% savings can be achieved by optimizing movies without compromising the perceptible video quality, and up to 70% for perceptible, but not annoying video quality difference. Tailoring video optimization to individual user perception can provide seamless QoE delivery across all users, with a low overhead (i.e., 10%) required to achieve this goal. Finally, two applications of video optimization: QoE-aware delivery and storage, are proposed and examined.

We present Communiplay, a public display media space. People passing by see their own contour mirrored on a public display and can start to play with virtual objects. At the same time, they see others playing at remote displays within the same virtual space. We are interested whether people would use such a public display media space, and if so, how and why. We evaluate Communiplay in a field study in six connected locations and find a remote honey-pot effect, i.e. people interacting at one location attract people at other locations. The conversion rate (percentage of passers-by starting to interact) rose by +136% when people saw others playing at remote locations. We also provide the first quantification of the (local) honey-pot effect (in our case it raised the conversion rate by +604% when people saw others playing at the same location). We conclude that the integration of multiple public displays into a media space is a promising direction for public displays and can make them more attractive and valuable.

Self-logging is a critical component to many wellbeing systems. However, self-logging often is difficult to sustain at regular intervals over many weeks. We demonstrate the power of passive mobile notifications to increase logging of wellbeing data, particularly food intake, in a mobile health service. Adding notifications increased the frequency of logging from 12% in a one-month, ten-user pilot study without reminders to 63% in the full 60-user study with reminders included. We will discuss the benefits of passive notifications over existing interruptive methods.

People now have access to many sources of data about their health and wellbeing. Yet, most people cannot wade through all of this data to answer basic questions about their long-term wellbeing: Do I gain weight when I have busy days? Do I walk more when I work in the city? Do I sleep better on nights after I work out? We built the Health Mashups system to identify connections that are significant over time between weight, sleep, step count, calendar data, location, weather, pain, food intake, and mood. These significant observations are displayed in a mobile application using natural language, for example, “You are happier on days when you sleep more.” We performed a pilot study, made improvements to the system, and then conducted a 90-day trial with 60 diverse participants, learning that interactions between wellbeing and context are highly individual and that our system supported an increased self-understanding that lead to focused behavior changes.

The arrival of smart phones and tablets, along with a flat rate mobile Internet pricing model have caused increasing adoption of mobile data services. According to recent studies, video has been the main driver of mobile data consumption, having a higher growth rate than any other mobile application. However, streaming a medium/high quality video files can be an issue in a mobile environment where available capacity needs to be shared among a large number of users. Additionally, the energy consumption in mobile devices increases proportionally with the duration of data transfers, which depend on the download data rates achievable by the device. In this respect, adoption of opportunistic content pre-fetching schemes that exploit times and locations with high data rates to deliver content before a user requests it, has the potential to reduce the energy consumption associated with content delivery and improve the user's quality of experience, by allowing playback of pre-stored content with virtually no perceived interruptions or delays. This paper presents a family of opportunistic content pre-fetching schemes and compares their performance to standard on-demand access to content. By adopting a simulation approach on experimental data, collected with monitoring software installed in mobile terminals, we show that content pre-fetching can reduce energy consumption of the mobile devices by up to 30% when compared to the on demand download of the same file, with a time window of 1 hour needed to complete the content prepositioning.

Mobile devices are now able to connect to a variety of sensors and provide personalized information to help people reflect on and improve their health. For example, pedometers, heart-rate sensors, glucometers, and other sensors can all provide real-time data to a variety of devices. Collecting and interacting with personal health or well-being data is a growing research area. This workshop will focus on the ways in which our mobile devices can aggregate and visualize these types of data and how these data streams can be presented to encourage interaction, increased awareness and positive behavior change.

This paper describes a novel tool for eliciting user requirements early in the design process of mobile applications. The "Mobile Oracle", as we have called it, is intended to help developers and designers obtain a better understanding of what the user wants at different points in space and time. It is an extension of a lo-fi version of the well-established Wizard of Oz technique, but it adds an "on demand" component to force users to explicitly request the information they need. The technique has been tested in an investigation involving 15 users (sighted, visually impaired, and elderly). Our preliminary results show it to generate valuable information concerning the ways people ask about directions and distances, as well as the services they would like to have in future mobile applications.

Images of objects as queries is a new approach to search for information on the web. Image-based information retrieval goes beyond only matching images, as information in other modalities also can be extracted from data collections using image search. We have developed a new system that uses images to search for web-based information. This paper has a particular focus towards exploring user's experience of general mobile image-based Web searches to find what issues and phenomena it contains. This was achieved in a multi-part study by creating and letting respondents test prototypes of mobile image-based search systems and collecting data using interviews, observations, video-observations, and questionnaires. We observed that searching for information only based on visual similarity and without any assistance is sometimes difficult, especially on mobile devices with limited interaction bandwidth. Most of our subjects preferred a search tool that guides the users through the search result based on contextual information, compared to presenting the search result as a plain ranked list.

A user study of video-mediated communication (VMC) involving six persons with mild intellectual disability is presented. It took place at comHOME, a full-scale model of an apartment of the future, showing innovative architectural and technical designs with regard to the integration of VMC into the domestic environment. Two different zones for VMC, comZONES, in the apartment were tested, the videoTORSO (a large-screen set-up for informal everyday communication) and the workPLACE(a place for professional work tasks). The purpose of the study was to get a deeper understanding of how people use these comZONES. The final discussion points out that the comZONES seem to be interpreted correctly and to function aptly in relation to the participants in the study. An assumed explanation is that spatial recognition is a very fundamental human function and thus less significant with regard to the mental capacity of the individual.

Location-based context is important for many applications. Previous systems offered only coarse room-level features or used manually specified room regions to determine fine-scale features. We propose a location context mechanism based on activity maps, which define regions of similar context based on observations of 3-D patterns of location and motion in an environment. We describe an algorithm for obtaining activity maps using the spatio-temporal clustering of visual tracking data. We show how the recovered maps correspond to regions for common tasks in the environment and describe their use in some applications.

This paper describes our studies of mediaspaces which are embedded within public places in organizations so as to examine the hypothesis that individuals might benefiteven when working apart- from opportunities for light informal interaction. A set of full-scale prototypes were used and assessed over extensive periods of time. The informal observations and reflection in design of these places have been supplemented by formal studies. We found that great care needs to be taken when designing these places from an architectural point of view. For some of the places, we would like to suggest using architectural features when altering the room rather than technology. In other settings, the artful deployment of communication media might be more effective.

Article Free Access Share on Supporting social awareness @ work design and experience Authors: Konrad Tollmar Interaction and Presentation Laboratory, Royal Institute of Technology, S-100 44 Stockholm, Sweden Interaction and Presentation Laboratory, Royal Institute of Technology, S-100 44 Stockholm, SwedenView Profile , Ovidiu Sandor Interaction and Presentation Laboratory, Royal Institute of Technology, S-100 44 Stockholm, Sweden Interaction and Presentation Laboratory, Royal Institute of Technology, S-100 44 Stockholm, SwedenView Profile , Anna Schömer Interaction and Presentation Laboratory, Royal Institute of Technology, S-100 44 Stockholm, Sweden Interaction and Presentation Laboratory, Royal Institute of Technology, S-100 44 Stockholm, SwedenView Profile Authors Info & Claims CSCW '96: Proceedings of the 1996 ACM conference on Computer supported cooperative workNovember 1996 Pages 298–307https://doi.org/10.1145/240080.240309Published:16 November 1996Publication History 64citation1,719DownloadsMetricsTotal Citations64Total Downloads1,719Last 12 Months237Last 6 weeks52 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF