Inability to regulate affective states can impact one's capacity to engage in higher-order thinking like scientific reasoning with game-based learning environments. Many efforts have been made to build affect-aware systems to mitigate the potentially detrimental effects of negative affect. Yet, gaps in research exist since accurately capturing and modeling affect as a state that changes dynamically over time is methodologically and analytically challenging. In this paper, we calculated multilevel mixed effects growth models to assess whether seventy-eight participants’ (n= 78) time engaging in scientific reasoning (via logfiles and eye gaze) were related to time facially expressing confused, frustrated, and neutral states (via facial recognition software) during game-based learning with Crystal Island. The fitted model estimated significant positive relations between the time learners facially expressed confusion, frustration, and neutral states and time engaging in scientific-reasoning actions. The time individual learners facially expressed frustrated, confused, and neutral states explained a significant amount of variation in time engaging in scientific reasoning. Our finding emphasize that individual differences and agency may play a important role on relations between affective states, their dynamics, and higher-order cognition during game-based learning. Designing affect-aware game-based learning environments that track the dynamics within individual learners’ affective states may best support cognition.

Significant efforts are currently being made to design affect-aware systems to classify, monitor, and scaffold emotional experiences across a range of settings. However, most investigations are limited in their view of emotions due to less sophisticated methodologies and analytical techniques. To address these issues, we captured 174 undergraduates’ emotions over time and defined them by multiple dimensions: (1) temporality, (2) valence, and (3) activation during learning with an intelligent tutoring system called MetaTutor. Latent growth models revealed the stability of negative activating emotions over time was related to performance, and changes in negative deactivating emotions were related to time engaging in cognitive strategies. Finally, a random forest classifier revealed high accuracy in predicting high (top 30%) and low performance groups (bottom 30%) using pre-test scores, changes in negative deactivating emotions, and time engaging in cognitive strategies. These findings have important implications for designing affect-aware systems that can potentially leverage emotion interventions based on if, when, and how an emotion changed (or remained stable) to optimize cognition and performance with emerging technologies.