Automated grading systems, or auto-graders, have become ubiquitous in programming education, and the way they generate feedback has become increasingly automated as well. However, there is insufficient evidence regarding auto-grader feedback's effectiveness in improving student learning outcomes, in a way that differentiates students who utilized the feedback and students who did not. In this study, we fill this critical gap. Specifically, we analyze students' interactions with auto-graders in an introductory Python programming course, offered at five community colleges in the United States. Our results show that students checking the feedback more frequently tend to get higher scores from their programming assignments overall. Our results also show that a submission that follows a student checking the feedback tends to receive a higher score than a submission that follows a student ignoring the feedback. Our results provide evidence on auto-grader feedback's effectiveness, encourage their increased utilization, and call for future work to continue their evaluation in this age of automation

The accelerating pace of developments in Artificial Intelligence (AI) and the increasing role that technology plays in society necessitates substantial changes in the structure of the workforce. Besides scientists and engineers, there is a need for a very large workforce of competent AI technicians (i.e., maintainers, integrators) and users (i.e., operators). As traditional 4-year and 2-year degree-based education cannot fill this quickly opening gap, alternative training methods have to be developed. We present the results of the first four years of the AI Technicians program which is a unique collaboration between the U.S. Army's Artificial Intelligence Integration Center (AI2C) and Carnegie Mellon University to design, implement and evaluate novel rapid occupational training methods to create a competitive AI workforce at the technicians level. Through this multi-year effort we have already trained 59 AI Technicians. A key observation is that ongoing frequent updates to the training are necessary as the adoption of AI in the U.S. Army and within the society at large is evolving rapidly. A tight collaboration among the stakeholders from the army and the university is essential for successful development and maintenance of the training for the evolving role. Our findings can be leveraged by large organizations that face the challenge of developing a competent AI workforce as well as educators and researchers engaged in solving the challenge.

This research full paper describes the augmented reality (AR) application AR-Classroom that combines a physical and virtual environment to teach 3D geometric rotations in an engaging and simplified manner. The AR-Classroom contains a virtual workshop where users can perform rotations by manipulating the application's X, Y, and Z-axis sliders to rotate a virtual LEGO model and a physical workshop where users perform rotations using a physical LEGO model. Guided by previous findings and an iterative approach to usability, the present usability study focused on assessing the usability of the AR-Classroom in its most recent version using a new physical LEGO model (i.e., airplane) and reflecting on how discoverability and usability can be assessed using different types of user experience measures and qualitative analysis. Participants were 22 undergraduate students who completed a pre-test with demographic information, watched a video on geometric transformations, and were randomly assigned to interact with either workshop. While interacting with the AR app, participants were instructed to provide feedback and a single ease-of-use question score. Participants then completed a post-test with two measures of usability. Descriptive statistics of the UX measures were explored, and a thematic analysis was conducted to identify and code themes in human-computer interaction. Findings suggest that the AR-Classroom has reached satisfactory usability, and users can navigate the app's features effectively. Discussion includes insight into which aspects of the app need improvement, how to promote self-directed support in the app, the development of future app efficacy experiments, and how to evaluate the usability of AR technology for learning.

This research full paper describes the AR-Classroom application that utilizes augmented reality (AR) and physical and virtual manipulatives to enable undergraduate students to build intuition about the relation between spatial transformations and their mathematical representations. To further build on the app's usability and functionality, additional features are being prototyped to continue improving the user-app interaction with the AR-Classroom. Some of the challenges the students faced when using AR-Classroom were recalling basic matrix operations without geometric context, basic trigonometric functions and their applications in the two-dimensional space, loss of AR registration for not understanding the AR environment, and User Interface (UI) issues. To address these issues, a conversational Artificial Intelligence (AI)-based multi-sensory and interactive assistance has been added to the AR-Classroom. Integrating sophisticated language processing and response generation of AI with immersive three-dimensional capabilities of AR can create a more engaging learning experience than the previous versions of the app. This integration focuses on creating a symbiosis between AR and AI. It creates an elevated user experience by offering real-time, personalized assistance to students dealing with issues related to understanding mathematical concepts and functionalities of the app. A qualitative exploratory usability study was done to assess the user's interaction with the AI implemented in the AR-Classroom, aiming to explore the AI's ability to guide students in using AR technology and aid in introductory matrix algebra learning, to effectively serve the students' learning. Based on the thematic analysis of the user experiment we found four main themes related to users' perceptions of AR-Classroom AI features usability: (1) AI chatbot ease-of-use, (2) Need for answer elaboration from AI, (3) Desire for visual information, and (4) Increased understanding of the content area. The scores of ease of use indicate AI's ability to guide complex tasks in an AR environment using AI features with less concern for the cognitive load. The overall result suggests the need for further investigation on incorporating AI-guided visual cues in an AR environment.

The AR-Classroom application utilizes augmented reality technology (AR) to make the three-dimensional (3D) rotations underlying matrix algebra visible and interactive. The AR-Classroom has physical and virtual versions, where users can perform rotations using a physical LEGO model or by manipulating the application's x, y, and z axes sliders to rotate a virtual model. Both versions provide 3D matrices, color-coded axes lines, and a green wireframe superimposed onto a LEGO model to represent transformations. To ensure that the AR-Classroom makes learning 3D matrix algebra more engaging and accessible, two usability tests were used to evaluate the discoverability and usability of the app. The benchmark test assessed usability in the AR-Classroom's original format, and recommendations were made to improve the app., such as adding additional instructions on model set-up, restructuring, and updating the instructions, and turning the 'visualization type 'function into a button to make it easier to find. After the improvements, the updated usability test assessed usability again so that the impact of the modifications could be evaluated. Participants followed similar procedures in both the benchmark$(\mathrm{N}=12)$and updated usability$(\mathrm{N}=12)$tests. Participants completed a pre-test assessing their math abilities and confidence, watched a video on geometric transformations, and then were randomly assigned to interact with either the physical or virtual version of the app. While interacting with the app, participants were given tasks to complete while thinking out loud and provided an ease-of-use rating from$1=\text{very}$easy to$7=\text{very}$difficult (i.e., SEQ score). Once done interacting with the app, participants completed a post-test assessing their math abilities and confidence and provided feedback on their overall experience with the app (i.e., SUS). A thematic analysis was conducted after each test to identify and code themes in interaction and compare findings from the benchmark and updated tests. Results indicated that after changes were made to the app, the usability of both versions significantly improved: users were better able to set up the space shuttle model, effectively utilize the in-app instructions, and quickly access all of the app's features. Findings from the updated usability test contribute to enhancing the AR-Classroom app and further its use in higher education classrooms for learning matrix algebra.

Project AR-Classroom aims to enhance undergraduate students learning spatial transformations and their mathematical representations. Understanding closely allied spatial and mathematical concepts significantly contributes to STEM learning in fields of computer graphics, computer-aided design, computer vision, robotics, and many more. The technology and learning innovations of this research include novel AR features and their implications for learning. In AR-Classroom, a student can hold and manipulate a 3D physical model (a LEGO space shuttle as an example) while simultaneously interacting with AR visualization of 3D rotations. Two usability tests with 24 participants total have been conducted for AR-Classroom leading to promising results and recommendations for improvements. The project contributes to advancing our knowledge in (1) the role of interplay between physical and virtual manipulatives to engage students in embodied learning and (2) the features of AR to make difficult, invisible concepts visible for supporting an intuitive and formal understanding of spatial reasoning and mathematical formulation.