Humans expect their collaborators to look beyond the explicit interpretation of their words. Implicature is a common form of implicit communication that arises in natural language discourse when an utterance leverages context to imply information beyond what the words literally convey. Whereas computational methods have been proposed for interpreting and using different forms of implicature, its role in human and artificial agent collaboration has not yet been explored in a concrete domain. The results of this paper provide insights to how artificial agents should be structured to facilitate natural and efficient communication of actionable information with humans. We investigated implicature by implementing two strategies for playing Hanabi, a cooperative card game that relies heavily on communication of actionable implicit information to achieve a shared goal. In a user study with 904 completed games and 246 completed surveys, human players randomly paired with an implicature AI are 71% more likely to think their partner is human than players paired with a non-implicature AI. These teams demonstrated game performance similar to other state of the art approaches.

Phishing attacks are a major problem, as evidenced by the DNC hackings during the 2016 US presidential election, in which staff were tricked into sharing passwords by fake Google security emails, granting access to confidential information. Vulnerabilities such as these are due in part to insufficient and tiresome user training in cybersecurity. Ideally, we would have more engaging training methods that teach cybersecurity in an active and entertaining way. To address this need, we introduce the game What.Hack, which not only teaches phishing concepts but also simulates actual phishing attacks in a role-playing game to encourage the player to practice defending themselves. Our user study shows that our game design is more engaging and effective in improving performance than a standard form of training and a competing training game design (which does not simulate phishing attempts through role-playing).

Designing educational games is difficult, and ideally designers should be able to rely on tools that take some of the burden off them by generating content automatically. Previous work in automatic level generation and sequencing for educational games has primarily focused on achieving a gradual increase of difficulty. However, engagement often comes from a sense of accomplishment after completing hard tasks [27]. For this reason, many games feature "boss levels" that are more intense. In this paper, we propose that a good progression should be goal-based, meaning that it should build up towards some hard tasks (goals) as soon as possible to create a sense of satisfaction. To achieve this, we propose a graph-based algorithm for automatically synthesizing goal-based learning progressions of user-specific length. We also introduce Katchi, a Korean language learning puzzle game that is designed to be highly parameterizable, to evaluate our synthesized progression. In an evaluation of a Korean learning game with 248 participants, our synthesized progression performed similarly to an expert-designed progression in terms of both our engagement and learning metrics, demonstrating that our algorithm is capable of automatically synthesizing goal-based progressions that are comparable to the manually created progressions.

K-8 mathematics students must learn many procedures, such as addition and subtraction. Students frequently learn "buggy' variations of these procedures, which we ideally could identify automatically. This is challenging because there are many possible variations that reflect deep compositions of procedural thought. Existing approaches for K-8 math use manually specified variations which do not scale to new math algorithms or previously unseen misconceptions. Our system examines students' answers and infers how they incorrectly combine basic skills into complex procedures. We evaluate this approach on data from approximately 300 students. Our system replicates 86% of the answers that contain clear systematic mistakes (13%). Investigating further, we found 77% at least partially replicate a known misconception, with 53% matching exactly. We also present data from 29 participants showing that our system can demonstrate inferred incorrect procedures to an educator as successfully as a human expert.

Learning at scale (LAS) systems like Massive Open Online Classes (MOOCs) have hugely expanded access to high quality educational materials however, such material are frequently time and resource expensive to create. In this work we propose a new approach for automatically and adaptively sequencing practice activities for a particular learner and explore its application for foreign language learning. We evaluate our system through simulation and are in the process of running an experiment. Our simulation results suggest that such an approach may be significantly better than an expert system when there is high variability in the rate of learning among the students and if mastering prerequisites before advancing is important, and is likely to be no worse than an expert system if our generated curriculum approximately describes the necessary structure of learning in students.

Dominant approaches to programming education emphasize program construction over language comprehension. We present Reduct, an educational game embodying a new, comprehension-first approach to teaching novices core programming concepts which include functions, Booleans, equality, conditionals, and mapping functions over sets. In this novel teaching strategy, the player executes code using reduction-based operational semantics. During gameplay, code representations fade from concrete, block-based graphics to the actual syntax of JavaScript ES2015. We describe our design rationale and report on the results of a study evaluating the efficacy of our approach on young adults (18+) without prior coding experience. In a short timeframe, novices demonstrated promising learning of core concepts expressed in actual JavaScript. We also present results from an online deployment. Finally, we discuss ramifications for the design of future computational thinking games.

Many people want to learn a language but find it difficult to stay engaged. Ideally, we would have language learning tools that can make language learning more enjoyable by simulating immersion in a foreign language environment. Therefore, we adapted Crystallize, a 3D video game for learning Japanese, so that it can be played in virtual reality with the Oculus Rift. Specifically, we explored whether we could leverage virtual reality technology to teach embodied cultural interaction, such as bowing in Japanese greetings. To evaluate the impact of our virtual reality game designs, we conducted a formative user study with 68 participants. We present results showing that the virtual reality design trained players how and when to bow, and that it increased participants' sense of involvement in Japanese culture. Our results suggest that virtual reality technology provides an opportunity to leverage culturally-relevant physical interaction, which can enhance the design of language learning technology and virtual reality games.

Authentic foreign language videos are effective for developing pragmatic competence, or sensitivity to meanings expressed by tone and word choice, and the ability to effectively express these meanings. However, established methods for learning from foreign language videos are primarily text-based (e.g.captioning). Using text, learners do not practice aspects of oral performance (e.g. intonation, pausing, and pitch) that are important to pragmatic competence. In this paper we present a voice-driven system where learners practice and learn a foreign language by repeating phrases out loud from any video. Utterances are transcribed and translated and, if captions are available, the system indicates the correctness of the utterance. In an evaluation with 27 participants, we show that participants more frequently used the voice-driven system than a comparison text-based system. Furthermore, ina field study of 130 independent learners, we show potential for community-driven resource collection.

Designing engaging learning content is important but difficult, and typically involves a lot of manual specification. We present a unified framework that utilizes automatic problem decomposition and partial ordering graph construction to facilitate multiple workflows: knowledge assessment and progression analysis and design. We present results from a study with 847 participants in an online Japanese-language assessment tool demonstrating that our framework can efficiently measure student ability and predict student performance on specific problems. We also present results from analysis of curricula showing that the progressions of two different textbooks are surprisingly similar, and that our framework can lead to the discovery of general principles of expert progression design. Finally, we demonstrate automatic progression generation with desired sequencing and pacing, allowing for tailoring of progressions and mapping of parameters extracted from one curriculum onto another.

Analysis of student data is critical for improving education. In particular, educators need to understand what approaches their students are taking to solve a problem. However, identifying student strategies and discovering areas of confusion is difficult because an educator may not know what queries to ask or what patterns to look for in the data. In this paper, we present a visualization tool, PathViewer, to model the paths that students follow when solving a problem. PathViewer leverages ideas from flow diagrams and natural language processing to visualize the sequences of intermediate steps that students take. Using PathViewer, we analyzed how several students solved a Python assignment, discovering interesting and unexpected patterns. Our results suggest that PathViewer can allow educators to quickly identify areas of interest, drill down into specific areas, and identify student approaches to the problem as well as misconceptions they may have.

Learning from captioned foreign language videos is highly effective, but the availability of such videos is limited. By using speech-to-text technology to generate partially correct transcripts as a starting point, we see an opportunity for learners to build accurate foreign language captions while learning at the same time. We present a system where learners correct captions using automatic transcription and machine-generated suggested alternative words for scaffolding. In a lab study of 49 participants, we found that compared to watching the video with accurate caption, learning and quality of experience were not significantly impaired by the secondary caption correction task using interface designs either with or without scaffolding from speech-to-text generated alternative words. Nevertheless, aggregating corrections reduced word error rate from 19% to 5.5% without scaffolding from suggested-alternatives, and 1.8% with scaffolding. Feedback from participants suggest that emphasizing the learning community contribution aspect is important for motivating learners and reducing frustration.

Learning a second language is challenging. Becoming fluent requires learning contextual information about how language should be used as well as word meanings and grammar. The majority of existing language learning applications provide only thin context around content. In this paper, we present work in Crystallize, a language learning game that combines traditional learning approaches with a situated learning paradigm by integrating a spaced-repetition system within a language learning roleplaying game. To facilitate long-term engagement with the game, we added a new quest paradigm, "jobs," that allow a small amount of design effort to generate a large set of highly-scaffolded tasks that grow iteratively. A large-scale evaluation of the language learning game "in the wild" with a diverse set of 186 people revealed that the game was not only engaging players for extended amounts of time but that players learned an average of 8.7 words in an average of 40.5 minutes.

Language students are most engaged while reading texts at an appropriate difficulty level. However, existing methods of evaluating text difficulty focus mainly on vocabulary and do not prioritize grammatical features, hence they do not work well for language learners with limited knowledge of grammar. In this paper, we introduce grammatical templates, the expert-identified units of grammar that students learn from class, as an important feature of text difficulty evaluation. Experimental classification results show that grammatical template features significantly improve text difficulty prediction accuracy over baseline readability features by 7.4%. Moreover,we build a simple and human-understandable text difficulty evaluation approach with 87.7% accuracy, using only 5 grammatical template features.

Learning a second language is challenging. Becoming fluent requires learning contextual information about how language should be used as well as word meanings and grammar. The majority of existing language learning applications provide only thin context around content. In this paper, we present Crystallize, a collaborative 3D game that provides rich context along with scaffolded learning and engaging gameplay mechanics. Players collaborate through joint tasks, or quests. We present a user study with 42 participants that examined the impact of low and high levels of task interdependence on language learning experience and outcomes. We found that requiring players to help each other led to improved collaborative partner interactions, learning outcomes, and gameplay. A detailed analysis of the chat-logs further revealed that changes in task interdependence affected learning behaviors.

A long-term goal of game design research is to achieve end-to-end automation of much of the design process, one aspect of which is creating effective level progressions. A key difficulty is getting the player to practice with interesting combinations of learned skills while maintaining their engagement. Although recent work in task generation and sequencing has reduced this effort, we still lack end-to-end automation of the entire content design process. We approach this goal by incorporating ideas from intelligent tutoring systems and proposing progression strategies that seek to achieve mastery of not only base concepts but arbitrary combinations of these concepts. The input to our system is a model of what the player needs to do to complete each level, expressed as either an imperative procedure for producing solutions or a representation of features common to all solutions. The output is a progression of levels that can be adjusted by changing high-level parameters. We apply our framework to a popular math puzzle game and present results from 2,377 players showing that our automatic level progression is comparable to expert-crafted progression after a few design iterations based on a key engagement metric.

Interactive learning environments such as intelligent tutoring systems and software tutorials often teach procedures with step-by-step demonstrations. This instructional scaffolding is typically authored by hand, and little can be reused across problem domains. In this work, we present a framework for generating interactive tutorials from an algorithmic representation of the problem-solving thought process. Given a set of mappings between programming language constructs and user interface elements, we step through this algorithm line-by-line to trigger visual explanations of each step. This approach allows us to automatically generate tutorials for any example problem that can be solved with this algorithm. We describe two prototype implementations in the domains of K-12 mathematics and educational games, and present results from two user studies showing that educational technologists can author thought-process procedures and that generated tutorials can effectively teach a new procedure to students.

A key challenge in teaching a procedural skill is finding an effective progression of example problems that the learner can solve in order to internalize the procedure. In many learning domains, generation of such problems is typically done by hand and there are few tools to help automate this process. We reduce this effort by borrowing ideas from test input generation in software engineering. We show how we can use execution traces as a framework for abstracting the characteristics of a given procedure and defining a partial ordering that reflects the relative difficulty of two traces. We also show how we can use this framework to analyze the completeness of expert-designed progressions and fill in holes. Furthermore, we demonstrate how our framework can automatically synthesize new problems by generating large sets of problems for elementary and middle school mathematics and synthesizing hundreds of levels for a popular algebra-learning game. We present the results of a user study with this game confirming that our partial ordering can predict user evaluation of procedural difficulty better than baseline methods.

One of the key challenges of video game design is teaching new players how to play. Although game developers frequently use tutorials to teach game mechanics, little is known about how tutorials affect game learnability and player engagement. Seeking to estimate this value, we implemented eight tutorial designs in three video games of varying complexity and evaluated their effects on player engagement and retention. The results of our multivariate study of over 45,000 players show that the usefulness of tutorials depends greatly on game complexity. Although tutorials increased play time by as much as 29% in the most complex game, they did not significantly improve player engagement in the two simpler games. Our results suggest that investment in tutorials may not be justified for games with mechanics that can be discovered through experimentation.

One of the most promising ways that games for learning can improve education is by adapting to each child individually. However, it is often difficult to instrument game mechanics so that they can be controlled to promote learning. Furthermore, even if this parameterization is possible, there is little knowledge of how to generate adaptive level progressions that optimize engagement and learning. We have taken the first step towards enabling adaptivity in an educational game for teaching fractions through the automatic generation of levels in a way that allows for multiple axes of mathematical and spatial difficulty to be controlled independently. We propose to expand on this work by developing a framework for representing conceptual knowledge. This framework will keep track of each player's knowledge, generate game levels that are tailored to the player's knowledge and skill level, and create progressions of these levels that allow players to learn new concepts through experimentation. We will compare multiple adaptive concept sequencing algorithms by evaluating their effects on player learning and engagement through multivariate tests with tens of thousands of players.

Some problems in procedural content generation for games involve hard constraints (e.g. that a generated puzzle is necessarily solvable). Common techniques for generator design lack a way to specify crisp (yes/no) constraints on what counts as a valid content artifact and guarantee these constraints are satisfied in the generator's output. In this paper we present two independent implementations of three diverse level design automation tools for the popular online educational game Refraction. All of the systems guarantee key properties of their output. Applying a constraint-focused generator design perspective in depth, we found that even emergent aesthetic style properties were straightforward to directly control. Our results with Refraction provide further concrete evidence for the claim that the expressive power of constraints and the ease with which they can be incorporated into suitably designed generative processes makes them a powerful tool for producing reliably-controllable generators for game content.

Many subjects such as languages, history, biology, geography, law, and medicine require the memorization of large numbers of unstructured facts and vocabulary. While there are many techniques and tools for memorizing such information, sustaining motivation remains a major barrier towards progress in these subjects. In order to help fill this gap, we brainstormed, experimented, and paper-prototyped to create Picard, a Facebook game that intertwines creative and social elements to turn rote learning into a creative, engaging experience. Picard, inspired by Pictionary, encourages players to come up with creative stories and images to help remember a particular card. Players compete and cooperate with their friends on Facebook to create the most memorable and funny mnemonics, called Picards, gaining points when they help themselves or others remember flashcards. Players use these Picards to learn and prove knowledge of the material at spaced intervals, which research has shown to be maximally effective for learning. Our game is the novel combination of an established learning methodology with player creativity and social support for learning. Picard increases player investment in the process of learning by allowing players to create and share useful visual mnemonics, while at the same time motivating and structuring their study habits through spaced-repetition quizzes.

Game designers frequently invest in aesthetic improvements such as music, sound effects, and animations. However, their exact value for attracting and retaining players remains unclear. Seeking to estimate this value in two popular Flash games, we conducted a series of large-scale A/B tests in which we selectively removed aesthetic improvements and examined the effect of each component on play time, progress, and return rate. We found that music and sound effects had little or no effect on player retention in either game, while animations caused users to play more. We also found, counterintuitively, that optional rewards caused players to play less in both games. In one game, this gameplay modification affected play time three times as much as the largest aesthetic variation. Our methodology provides a way to determine where resources may be best spent during the game design and development process.

Secondary game objectives, optional challenges that players can choose to pursue or ignore, are a fundamental element of game design. Still, little is known about how secondary objectives affect player behavior. It is commonly believed that secondary objectives such as coins or collectible items can increase a game's flexibility, replayability, and depth. In contrast, we present results from analysis of two popular online Flash games showing that secondary objectives can easily harm the retention of many players. We support our findings with data collected from over 27,000 players through large-scale A/B tests in which we measured play time, progress, and return rate. We show that while secondary objectives can encourage long-term players to extend their playtime, they can also cause many players to play for less time. By modifying secondary objectives so that they reinforce the primary goal of the game instead of distracting from it, we are able to avoid negative consequences and still maintain the retention of long-term players. Our results suggest that secondary objectives that support the primary goal of the game are consistently useful, while secondary objectives that do not support the main goal require extensive testing to avoid negative consequences.

Visual data mining is a powerful technique allowing game designers to analyze player behavior. Playtracer, a new method for visually analyzing play traces, is a generalized heatmap that applies to any game with discrete state spaces. Unfortunately, due to its low discriminative power, Playtracer's usefulness is significantly decreased for games of even medium complexity, and is unusable on games with continuous state spaces. Here we show how the use of state features can remove both of these weaknesses. These state features collapse larger state spaces without losing salient information, resulting in visualizations that are significantly easier to interpret. We evaluate our work by analyzing player data gathered from three complex games in order to understand player behavior in the presence of optional rewards, identify key moments when players figure out the solution to the puzzle, and analyze why players give up and quit. Based on our experiences with these games, we suggest general principles for designers to identify useful features of game states that lead to effective play analyses.

Analysis of gameplay data is crucial for evaluating design decisions and refining a game experience. However, identifying player strategies and finding areas of confusion is difficult because a designer may not know what queries to ask or what patterns to look for in the data. To make this task easier, we present Playtracer, a method for visually analyzing play traces that is independent of a specific game's structure. Playtracer applies multidimensional scaling to cluster players and game states, providing a detailed visual representation of the paths the players take through a game. We evaluate our method by analyzing an educational puzzle game and highlighting common hypotheses, pitfalls, confusing elements, and anomalies. Our results suggest that Playtracer can be an effective tool for game analysis and improvement.

The incorporation of randomness is critical for the believability and effectiveness of controllers for characters in competitive games. We present a fully automatic method for generating intelligent real-time controllers for characters in such a game. Our approach uses game theory to deal with the ramifications of the characters acting simultaneously, and generates controllers which employ both long-term planning and an intelligent use of randomness. Our results exhibit nuanced strategies based on unpredictability, such as feints and misdirection moves, which take into account and exploit the possible strategies of an adversary. The controllers are generated by examining the interaction between the rules of the game and the motions generated from a parametric motion graph. This involves solving a large-scale planning problem, so we also describe a new technique for scaling this process to higher dimensions.