Recent RL research has utilized reward shaping--particularly complex shaping rewards such as intrinsic motivation (IM)--to encourage agent exploration in sparse-reward environments. While often effective, ``reward hacking'' can lead to the shaping reward being optimized at the expense of the extrinsic reward, resulting in a suboptimal policy. Potential-Based Reward Shaping (PBRS) techniques such as Generalized Reward Matching (GRM) and Policy-Invariant Explicit Shaping (PIES) have mitigated this. These methods allow for implementing IM without altering optimal policies. In this work we show that they are effectively unsuitable for complex, exploration-heavy environments with long-duration episodes. To remedy this, we introduce Action-Dependent Optimality Preserving Shaping (ADOPS), a method of converting intrinsic rewards to an optimality-preserving form that allows agents to utilize IM more effectively in the extremely sparse environment of Montezuma's Revenge. We also prove ADOPS accommodates reward shaping functions that cannot be written in a potential-based form: while PBRS-based methods require the cumulative discounted intrinsic return be independent of actions, ADOPS allows for intrinsic cumulative returns to be dependent on agents' actions while still preserving the optimal policy set. We show how action-dependence enables ADOPS's to preserve optimality while learning in complex, sparse-reward environments where other methods struggle.

Recently there has been a proliferation of intrinsic motivation (IM) reward shaping methods to learn in complex and sparse-reward environments. These methods can often inadvertently change the set of optimal policies in an environment, leading to suboptimal behavior. Previous work on mitigating the risks of reward shaping, particularly through potential-based reward shaping (PBRS), has not been applicable to many IM methods, as they are often complex, trainable functions themselves, and therefore dependent on a wider set of variables than the traditional reward functions that PBRS was developed for. We present an extension to PBRS that we show preserves the set of optimal policies under a more general set of functions than has been previously demonstrated. We also present Potential-Based Intrinsic Motivation (PBIM), a method for converting IM rewards into a potential-based form that are useable without altering the set of optimal policies. Testing in the MiniGrid DoorKey environment, we demonstrate that PBIM successfully prevents the agent from converging to a suboptimal policy and can speed up training.

Developments in Reinforcement Learning and the availability of healthcare data sources such as Electronic Health Records (EHR) provide an opportunity to derive data-driven treatment dose recommendations for patients and improve clinical outcomes. Recent studies have focused on deriving discretized dosages using offline historical data extracted from EHR. In this paper, we propose an Actor-Critic framework to infer continuous dosage for treatment recommendation and demonstrate its advantage in numerical stability as well as interpretability. In addition, we incorporate a Bayesian Neural Network as a simulation model and probability-based regularization techniques to alleviate the distribution shift in off-line learning environments to increase practical safety. Experiments on a real-world EHR data set, MIMIC-III, show that our approach can achieve improved performance while maintaining similarity to expert clinician treatments in comparison to other baseline methods.

In Multiplayer Online Battle Arena (MOBA) games, teams of players compete in combat to complete an objective and defeat the opposing team. To stay alive, players must closely monitor their character's status, especially remaining health. Understanding how health may change in the near future can be vital in determining what tactics a player may use. We analyzed replay logs of the game Defense of the Ancients 2 (DotA 2) to discover methods to predict how players' health evolves over time. For DotA 2, our results suggest that forecasting changes in a player's health can be done by viewing gameplay as two separate processes: normal gameplay flow in which changes in health are smaller and more regular, and less frequent but higher-impact events in which players experience larger changes in their health, such as team battles. We accomplished this by considering health data as two separate, but interleaved, time series in which separate processes govern low magnitude changes in health from high magnitude changes. In this paper, we present a value-split approach to predicting changes in health and describe the results of our approach using autoregressive moving-average models for low magnitude health changes and a combination of statistical models for the larger changes.

This paper investigates the problem of interactively learning behaviors communicated by a human teacher using positive and negative feedback. Much previous work on this problem has made the assumption that people provide feedback for decisions that is dependent on the behavior they are teaching and is independent from the learner’s current policy. We present empirical results that show this assumption to be false—whether human trainers give a positive or negative feedback for a decision is influenced by the learner’s current policy. Based on this insight, we introduce Convergent Actor-Critic by Humans (COACH), an algorithm for learning from policy-dependent feedback that converges to a local optimum. Finally, we demonstrate that COACH can successfully learn multiple behaviors on a physical robot.

This article describes an emerging direction in the intersection between human–computer interaction and cognitive science: the use of cognitive models to give insight into the challenges of cybersecurity (cyber-SA). The article gives a brief overview of work in different areas of cyber-SA where cognitive modeling research plays a role, with regard to direct interaction between end users and computer systems and with regard to the needs of security analysts working behind the scenes. The problem of distinguishing between human users and automated agents (bots) interacting with computer systems is introduced, as well as ongoing efforts toward building Human Subtlety Proofs (HSPs), persistent and unobtrusive windows into human cognition with direct application to cyber-SA. Two computer games are described, proxies to illustrate different ways in which cognitive modeling can potentially contribute to the development of HSPs and similar cyber-SA applications.

Although numerous advances have been made in instrumental odor detection systems, these still cannot match the efficient sampling, odor discrimination, agile mobility and the olfactory acuity of odor detection dogs. A limiting step in using dogs to detect odors is the subjectivity of the translation of odor information processed by the dog to its handler. We present our preliminary efforts towards a wireless wearable system for continuous auscultation of respiratory behavior by recording internal sounds at the neck and chest by means of a commercially available electronic stethoscope to provide objective decision support for handlers. We have identified discrete features of sniffing and panting in the time domain and utilize event duration, event rate, event mean energy, and the number of consecutive events in a row to build a decision tree classifier. Since feature extraction requires segmentation of individual sniffing and panting events, we developed an adaptive method using short-time energy contour and an adaptive threshold. The performance of the system was evaluated on recordings from a Greyhound and a Labrador Retriever and achieved high classification accuracies.

We present a study that investigates the heretofore unexplored relationship between a player's sense of her narrative role in an interactive narrative role-playing game and the options she selects when faced with choice structures during gameplay. By manipulating a player's knowledge over her role, and examining in-game options she preferred in choice structures, we discovered what we term the Mimesis Effect: when players were explicitly given a role, we found a significant relationship between their role and their in-game actions; participants role-play even if not instructed to, exhibiting a preference for actions consistent with their role. Further, when players were not explicitly given a role, participants still role-played -- they were consistent with an implicit role -- but did not agree on which role to implicitly be consistent with. We discuss our findings and broader implications of our work to both game development and games research.

One of the challenges to working with canines is that whereas humans are primarily vocal communicators, canines are primarily postural and behavioral communicators. It can take years to gain some level of proficiency at reading canine body language, even under the best of circumstances. In the case of guide dogs and visually-impaired handlers, this task is even more difficult. Luckily, new technology designed to help monitor canines may prove useful in helping handlers, especially those with visual impairments, to better understand and interpret what their working partners are feeling or saying.

Training animals is a process that requires a significant investment of time and energy on the part of the trainer. One of the most basic training tasks is to train dogs to perform postures on cue. While it might be easy for a human trainer to see when an animal has performed the desired posture, it is much more difficult for a computer to determine this. Most work in this area uses accelerometer and/or gyroscopic data to produce data from an animal's current state, but this has limitations. Take for example a normal standing posture. From an accelerometer's perspective, it closely resembles the "laying down" posture, but the posture can look very different if the animal is standing still, versus walking, versus running, and might look completely different from a "standing on incline" posture. A human trainer can instantly tell the difference between these postures and behaviors, but the process is much more difficult for a computer.

This paper shows how game analytics can be used to dynamically adapt casual game environments in order to increase session-level retention. Our technique involves using game analytics to create an abstracted game analytic space to make the problem tractable. We then model player retention in this space and use these models to make guided changes to game analytics in order to bring about a targeted distribution of game states that will, in turn, influence player behavior. Experiments performed showed that the adaptive versions of two different casual games, Scrabblesque and Sidequest: The Game, were able to better fit a target distribution of game states while also significantly reducing the quitting rate compared to the nonadaptive version of the games. We showed that these gains were not coming at the cost of player experience by performing a psychometric evaluation in which we measured player intrinsic motivation and engagement with the game environments. In both cases, we showed that players playing the adaptive version of the games reported higher intrinsic motivation and engagement scores than players playing the nonadaptive version of the games.

This paper introduces two novel algorithms for learning behaviors from human-provided rewards. The primary novelty of these algorithms is that instead of treating the feedback as a numeric reward signal, they interpret feedback as a form of discrete communication that depends on both the behavior the trainer is trying to teach and the teaching strategy used by the trainer. For example, some human trainers use a lack of feedback to indicate whether actions are correct or incorrect, and interpreting this lack of feedback accurately can significantly improve learning speed. Results from user studies show that humans use a variety of training strategies in practice and both algorithms can learn a contextual bandit task faster than algorithms that treat the feedback as numeric. Simulated trainers are also employed to evaluate the algorithms in both contextual bandit and sequential decision-making tasks with similar results.

In order to be useful in real-world situations, it is critical to allow non-technical users to train robots. Existing work has considered the problem of a robot or virtual agent learning behaviors from evaluative feedback provided by a human trainer. That work, however, has treated feedback as a numeric reward that the agent seeks to maximize, and has assumed that all trainers will provide feedback in the same way when teaching the same behavior. We report the results of a series of user studies that indicate human trainers use a variety of approaches to providing feedback in practice, which we describe as different “training strategies.” For example, users may not always give explicit feedback in response to an action, and may be more likely to provide explicit reward than explicit punishment, or vice versa. If the trainer is consistent in their strategy, then it may be possible to infer knowledge about the desired behavior from cases where no explicit feedback is provided. We discuss a probabilistic model of human-provided feedback that can be used to classify these different training strategies based on when the trainer chooses to provide explicit reward and/or explicit punishment, and when they choose to provide no feedback. Additionally, we investigate how training strategies may change in response to the appearance of the learning agent. Ultimately, based on this work, we argue that learning agents designed to understand and adapt to different users' training strategies will allow more efficient and intuitive learning experiences.

In this paper, we present computational models for generating influence that allow story managers to shape players' decisions in interactive narrative experiences. Our approach uses concepts from social psychology, discourse analysis, and natural language generation. We describe an abstract formalism to operationalize tools of social psychological influence described by Cialdini (Influence: The Psychology of Persuasion, New York, NY, USA: Harper-Collins, 1998) and evaluate two example implementations that enable a storytelling system to generate influence on the fly (with varying degrees of success), thereby adapting stories to realize goals specified by authors. These implementations are used in an interactive story where influence is generated dynamically as players' experiences unfold. We present the results of a user study to characterize the effectiveness of these models. Results did not indicate the presence of any significant differences in players' sense of control over the story with, or without, the use of influence. Further, the use of influence resulted in a set of stories experienced by players that more closely matched the author's goals.

Conflict is an essential element of interesting stories. In this paper, we operationalize a narratological definition of conflict and extend established narrative planning techniques to incorporate this definition. The conflict partial order causal link planning algorithm (CPOCL) allows narrative conflict to arise in a plan while maintaining causal soundness and character believability. We also define seven dimensions of conflict in terms of this algorithm's knowledge representation. The first three-participants, reason, and duration-are discrete values which answer the “who?” “why?” and “when?” questions, respectively. The last four-balance, directness, stakes, and resolution-are continuous values which describe important narrative properties that can be used to select conflicts based on the author's purpose. We also present the results of two empirical studies which validate our operationalizations of these narrative phenomena. Finally, we demonstrate the different kinds of stories which CPOCL can produce based on constraints on the seven dimensions.

Training and handling working dogs is a costly process and requires specialized skills and techniques. Less subjective and lower-cost training techniques would not only improve our partnership with these dogs but also enable us to benefit from their skills more efficiently. To facilitate this, we are developing a canine body-area-network (cBAN) to combine sensing technologies and computational modeling to provide handlers with a more accurate interpretation for dog training. As the first step of this, we used inertial measurement units (IMU) to remotely detect the behavioral activity of canines. Decision tree classifiers and Hidden Markov Models were used to detect static postures (sitting, standing, lying down, standing on two legs and eating off the ground) and dynamic activities (walking, climbing stairs and walking down a ramp) based on the heuristic features of the accelerometer and gyroscope data provided by the wireless sensing system deployed on a canine vest. Data was collected from 6 Labrador Retrievers and a Kai Ken. The analysis of IMU location and orientation helped to achieve high classification accuracies for static and dynamic activity recognition.

We report on the experience of teaching an industry-validated course on Artificial Intelligence in Computer Games within the Simulation and Game Design department at a two-year community college during a 16-week semester. The course format used a blended learning just-in-time teaching approach, which included active learning programming exercises and one-on-one student interactions. Moskal's Attitudes Toward Computer Science survey showed a positive and significant increase in students in both interest (W(10) = 25, p = 0.011) and professional (W(10) = 49.5, p = 0.037) constructs. The Felder-Soloman Index of Learning Styles (n = 14) failed to identify any statistically significant differences in learning styles when compared to a four-year CS1 class. In the final class evaluation, 8 out of 13 students (62%) strongly or very strongly preferred the blended learning approach. We validated this course through four semi-structured interviews with game companies. The interview results suggest that companies are strongly favorable to the course content and structure. The results of this work serve as a template that community colleges can adopt for their curriculum.

In this paper, we present a data-driven technique for designing models of user behavior. Previously, player models were designed using user surveys, small-scale observation experiments, or knowledge engineering. These methods generally produced semantically meaningful models that were limited in their applicability. To address this, we have developed a purely data-driven methodology for generating player models based on past observations of other players. Our underlying assumption is that we can accurately predict what a player will do in a given situation if we examine enough data from former players that were in similar situations. We have chosen to test our method on achievement data from the MMORPG World of Warcraft. Experiments show that our method greatly outperforms a baseline algorithm in both precision and recall, proving that this method can create accurate player models based solely on observation data.