This full-length research study investigated first-year engineering students “trust in generative artificial intelligence (GenAl) before and after course instruction. Pre-and post-surveys were conducted with questions on students” experience with GenAl tools as well as trust in GenAl. The trust questions had students evaluate the likelihood of GenAl generating a correct response to various prompts such as “explain the unit circle” (correct response likely) to “solving this system of equations …” (correct response unlikely for tools available in Fall 2023). Within-subjects analyses indicated that lessons in the course significantly increased trust in ChatGPT for correct-response-likely items. In addition, the lessons significantly decreased students “trust in ChatGPT for correct-response-unlikely items. There were no significant interactions between prior experience level and change in trust. These results show that guided exposure to GenAl helped first-year engineering students begin to understand capabilities and limitations of GenAl. These results are promising because student trust in GenAl output will directly impact decisions to engage with it for different tasks. More work is needed to optimize instruction and understand students” use of the tool beyond the classroom integration, including their full ethical decision-making process, but the malleability of trust at this level is an indication that engineering educators can impact student perspectives of GenAl.

This research work in progress research paper examines student perceptions after completing an exploratory learning lesson before instruction on an introductory programming concept. During exploratory learning activities, students explore a novel concept prior to instruction-the reverse of typical instruct-then-practice methods. Exploratory learning before instruction can help students activate prior knowledge, become aware of their knowledge gaps, and discern important problem features to improve conceptual understanding. Students in a first-year engineering course (N=402) learned about Python error messages in one of two conditions. In the explore-first condition, students completed a collaborative activity prior to instruction. In the instruct-first condition, students received instruction prior to the activity. Following the activity and instruction, students completed a survey to assess their perceptions of the activities. Survey items (e.g. cognitive load, self-efficacy, belonging, knowledge gaps) were chosen as potential factors that could explain learning outcomes between the two conditions. In prior work, we found higher posttest scores in the instruct-first compared to explore-first condition, contrary to the majority of previous studies. Cognitive load and knowledge gaps were higher in the explore-first condition than the instruct-first condition. Self-efficacy and competence were lower in the explore-first condition. No other significant differences were found. Exploring before instruction might disrupt learning and perceived efficacy and competence if the activity is too challenging, or if the instruction does not fully resolve gaps in students' knowledge.

This full-length research paper describes a retrospective analysis of engineering students' sense of belonging following a summer bridge program (SBP) and as related to their first-semester math performance. The Brown Forman Engineering Academy (BFEA) is a two-week, no-cost SBP designated to enhance college readiness for first-year students at the University of Louisville who are underrepresented and underprepared (e.g., low-income, race, gender) for engineering school. The program aims to support students in calculus, foster community growth, and develop a sense of belonging on campus. This study assessed the success of the BFEA program by evaluating participants' sense of belonging at the beginning of the first semester. In addition, this study examined how students' sense of belonging was influenced by their first-semester math grades. Results indicated that at the beginning of the fall semester, there was no significant difference in the sense of belonging between BFEA participants and the broader student body. Additionally, math performance affected students' sense of belonging, even following the SBP, similarly to the full incoming student cohort. It is possible that students entered this program with very low belonging, and that it raised them to the average cohort level. It is also possible, however, that an improvement in belonging did not occur during the program. It is not possible to fully determine the impact of BFEAfrom this data, because there was no pre-program assessment of belonging. More research is needed to investigate the SBP as a causal force on students' sense of belonging.

This work-in-progress research paper describes a study of different categorical data coding procedures for machine learning (ML) in engineering education. Often left out of methodology sections, preprocessing steps in data analysis can have important ramifications on project outcomes. In this study, we applied three different coding schemes (i.e., scalar conversion, one-hot encoding, and binary) for the categorical variable of Race across three different ML models (i.e., Neural Network, Random Forest, and Naïve Bayes classifiers) looking at the four standard measures of ML classification models (i.e., accuracy, precision, recall, and F1-score). Results showed that in general, the coding scheme did not affect predictive outcomes as much as ML model type did. However, one-hot encoding - the strategy of transforming a categorical variable with$k$possible values to$k$binary nodes, a common practice in educational research - does not work well with a Naïve Bayes classifier model. Our results indicate that such sensitivity studies at the beginning of ML modeling projects are necessary. Future work includes performing a full range of sensitivity studies on our complete, grant-funded project dataset that has been collected, and publishing our findings.

This WIP paper disseminates the results of an anonymous survey given to first-year engineering students in February of 2023 about ChatGPT, the recently-developed artificial intelligence chatbot. Survey results showed that some engineering students had used ChatGPT to complete their homework assignments. Furthermore, only a few of those who used it for homework felt like they were “cheating,” or acting unethically, by doing so. These results indicate that the higher education community should carefully consider the potential learning benefits and threats of this new tool, as it may be utilized by at least some portion of students on their assignments. Much more work is needed to understand the potential uses, threats, and limitations of large language model chatbots in engineering education.

This WIP paper presents new research on exploratory learning, an educational technique that reverses the order of standard lecture-based instruction techniques. In exploratory learning, students are presented with a novel activity first, followed by instruction. Exploratory learning has been observed to benefit student learning in foundational math and science courses such as calculus, physics, and statistics; however, it has yet to be applied to engineering topics such as programming. In two studies, we tested the effectiveness of exploratory learning in the programming unit of a first-year undergraduate engineering course. We designed a new activity to help students learn about different python error types, ensuring that it would be suitable for exploration. Then we implemented two different orders (the traditional instruct-first versus exploratory learning's explore-first) across the six sections of the course. In Study 1 ($N$=406), we did not detect a difference between the instruct-first and explore-first conditions. In Study 2 (N=411), we added more scaffolding to the activity. Students who received the traditional order of instruction followed by the activity scored significantly higher on the assessment. These findings contradict the exploratory learning benefits typically shown, shedding light on potential boundary conditions to this effect.

Spaced retrieval practice is an evidence-based learning technique that has the potential to improve student learning and memory in undergraduate courses. However, studies in STEM courses are limited, and results from available studies are inconsistent [1]. This paper presents such a study in an introductory physics course for engineering students, answering the following research question: Does spaced retrieval practice improve engineering student learning in physics? Students answered quiz questions that were administered in massed and spaced conditions throughout the semester, and then took a criterial test at the end of the semester. Results showed that spacing significantly improved student performance, with a mean gain of 3.4% in the spaced condition over the massed condition. This result is particularly interesting because a null effect of spacing was previously obtained in a similar physics course taken by non-engineering students [2]. Potential implications are discussed.

This Work-In-Progress research paper presents the investigation of a decision tree model that was trained to predict engineering students' first-semester math performance using demographic and survey data. This is a small step in a larger project that will develop a predictive AI model that can identify students at risk of leaving engi-neering. Ultimately, we will pair a predictive model with an explanation method to identify targeted interventions that can be implemented within the first year of engineering school. Our findings from this project indicate that we may be able to successfully identify students at risk of low performance in first-semester math courses, and design effective individualized interventions.

This full-length, research-to-practice paper discusses an ongoing NSF project aimed at implementing spaced retrieval practice in Science, Technology, Engineering, and Mathematics (STEM) classes. By implementing evidence-based learning practices in the classroom, educational performance, and thereby retention and graduation rates, are thought to be improved. Retrieval practice refers to the repeated recall of information from memory to increase the information’s future accessibility. Retrieval practice can be massed or spaced. In massed practice, all acts of recall occur during a single short temporal window. In spaced practice, acts of recall are temporally distributed. This project implements spaced retrieval practice in STEM barrier courses in the fields of biology, chemistry, engineering, mathematics, physics, and psychology. The current dissemination focuses on spaced retrieval practice in one section of Calculus for Engineers and two sections of Fundamentals of Physics I, with similarly delivered implementations. Spaced retrieval practice significantly improved student performance in calculus but not in the two sections of physics. Potential explanations for these results are discussed, as well as implications.

This Research to Practice Full Paper presents an investigation of engineering retention using machine learning models. We use random forests and artificial neural networks in the form of multilayer perceptrons to analyze the interaction between different factors, such as demographic information, standardized test scores, first semester grades, and surveys to predict student retention in engineering. We find that obtained models can predict with good accuracy if students will remain in engineering, with F1 scores of at least 75 percent. We find that each model places different levels of importance on distinct factors.

This Full-Length Research Paper investigates the difficulty imposed by spaced retrieval practice in nine introductory Science, Technology, Engineering, and Mathematics (STEM) courses. By improving student performance in these courses, evidence-based pedagogical practices have the potential to increase graduation and success in STEM fields. Spaced retrieval practice is a technique in which questions on the same topic are asked repeatedly over time with intermittent delays. Spacing may initially make retrieval more difficult because it requires learners to recall information from long-term as opposed to short-term memory. However, this difficulty may ultimately be “desirable” because spacing often produces memory benefits in the long-term. The current paper examines the difficulty imposed by spaced retrieval in the nine STEM courses, using data collected from a 3-year project funded by the National Science Foundation. Results indicated that the magnitude of the difficulty imposed by spacing varied widely across the diverse STEM barrier courses. We anticipate that we will find similarly wide variability in the effectiveness of spaced retrieval practice in students' final learning outcomes, which will be investigated in future work.

This Full-Length Research Paper presents findings from a scoping review of diversity research in engineering and computing education in the 2019–2020 FIE conference proceedings. The purpose of this review was to determine present-day themes and contributions of diversity research and identify opportunities for further research. Through the scoping review process, we identified 21 out of 776 papers that focused on a diverse population in engineering or computing, utilized an established methodology, and presented results. From these selected papers, three themes emerged. The first was about minority student experiences of discrimination and biases, and their resulting feelings of inclusivity and belonging. Results showed that these experiences and feelings negatively impacted success factors such as GPA and graduation in engineering or computing. A second theme was that financial need is a substantial deterrent to degrees in engineering and computing. Lastly, there were several investigations of the experiences of diverse students during active learning interventions in the classroom. The mixed results do not present a clear picture of how active learning might impact minority student success. Based on the existing work, we make recommendations for future diversity research. In general, much more quantitative, intervention-based research is needed. Several difficulties have been established, but almost no solutions have been identified.

This WIP Research Paper investigates the temporal nature of the difficulty imposed by spacing. Spaced retrieval practice is an evidence-based strategy for improving memory and consists of asking multiple questions on a topic with intermittent delays. Spacing is often thought to impose difficulty by making questions harder to answer. However, this difficulty may be desirable, since spacing ultimately improves memory. In this paper, we (a) outline an implementation of spaced retrieval practice in an engineering mathematics classroom, (b) describe the development of an SQL database to organize and manage the large and complex dataset, and (c) discuss a brief but interesting dive into the rich data we have collected. Statistical analyses revealed that, when three questions targeting the same topic are spaced over multiple quizzes, versus being massed on a single quiz, students are less likely to answer the first and second questions correctly. Spacing does not affect students' ability to answer the third questions. This suggests that spacing may impose difficulty when students are first learning to perform mathematical operations, rather than when they are trying to retrieve memories of how to perform those operations.

This Full-Length, Research-to-Practice paper discusses intrinsic requirements that may challenge instructors if they attempt to implement spaced retrieval practice in their courses. During the first year of National Science Foundation (NSF) grant #1912253, project leaders led nine Science, Technology, Engineering, and Mathematics (STEM) instructors through a series of five interactive workshops to develop learning objectives and quiz questions. Most of the STEM instructors had to redefine their existing, multifaceted learning objectives into more specific objectives with an appropriately fine grain-size for the practice. They also worked to develop multiple questions that test the same objective with (1) comparable difficulty and (2) similar cognitive processes. Project leaders noticed that a within-subjects, counterbalanced study design presents additional challenges to implementation. Instructors were able to work around difficulties during break-out sessions in workshops and in one-on-one sessions, especially when given examples from their own discipline in one-on-one feedback.In this paper, we first describe the current state of spaced retrieval practice research and the purpose and plan of our active NSF grant. We then detail the implementation requirements we have discovered. Lastly, we summarize our findings with bullet-point, STEM-practitioner-centered statements about implementing spaced retrieval practice in the classroom. Identifying potential challenges of implementation and solutions to these challenges is an important step in getting the powerful memory tool of spaced retrieval practice into the STEM classroom.

This Full-Paper in the Research Category analyzed longitudinal data from the University of Louisville's J. B. Speed School of Engineering to better understand the relationship between 1) progression and performance in the required mathematics courses for all engineering majors, 2) retention in engineering (year 1 and year 2) and 3) persistence to graduation. Analysis revealed that engineering students who complete the required sequence of four mathematics courses have a 93% graduation rate. What we call math completion is a compelling statistic when compared to the more common first year retention metric, which in our case, leads to a graduation rate of 68%. While few student's complete math by the second fall semester, many students do complete math during the second fall semester, allowing math completion to be a comparable and possibly more relevant metric than first- and second-year retention rates. This paper further investigated paths to mathematics sequence completion, finding that the first mathematics course (Calculus I or Precalculus) and first-semester mathematics grade are significant predictors of math completion, and hence graduation.

Physics problem solving requires the use of verbal, pictorial, diagrammatic, and mathematical representations in order to translate a problem into its underlying mathematical components. This study investigated the effect of explicit deliberate practice on representation skills in introductory physics at the University of Louisville. In a controlled, randomized design, physics students received either deliberate practice or traditional solve-though practice weekly throughout the semester. The deliberate practice instruction focused on translating from verbal to diagrammatic and diagrammatic to mathematical representations. The deliberate practice problem solving intervention had no impact on exam performance or an encoding task developed to measure facility with diagrammatic representations of physics problems. However, performance on the encoding task positively predicted exam performance across conditions. These results suggest that deliberate practice on multiple representations did not demonstrably improve exam scores. However, the ability to encode problem features predicts exam performance more generally.

Mathematics learning is critical in STEM degrees. In engineering specifically, advanced courses depend on the derivation and application of higher-order equations, and long-term retention of early mathematical concepts is mandatory. Cognitive science research has shown that learning and memory can be improved with simple manipulations of material retrieval. One major finding has been that increasing retrieval practice improves material retention more than restudying the material (the retrieval-practice effect). Independently, spacing retrieval over time is known to improve material retention (the spacing effect). To date, no research has studied these two effects in conjunction. This NSF-funded study investigated retrieval practice and spacing in an engineering mathematics classroom. Separately, increasing retrieval practice and increasing spacing both improved final exam performance. In combination, they produced the greatest learning gains.