This Work-in-Progress research paper evaluates the validity of Large Language Models (LLMs) as conversational AI tutors for computer science learning. While current engineering education literature has predominantly emphasized the rapid evolution of LLMs as conversational AI tutors for programming languages, the exploration into their effectiveness within general STEM topics remains comparatively scarce. This WIP study thus centers on evaluating the potential of LLMs to facilitate understanding of core hardware design concepts critical to computer science and engineering (CSE) education. By cross-checking the responses from generative AI chatbots to an openended CSE-based question, we aimed to uncover how LLMs, such as ChatGPT-3.5, Claude, Gemini, and Meta AI, can contribute to teaching and learning of general CSE courses instead of a specifically coding-based one. Our method involved simulating a student query on the popular debate between CISC vs. RISC related to computer architecture and analyzing the chatbots' responses. This initial collection of data served as the foundation for a continual comparative analysis aimed at determining the inherent instructional value of each LLM and its validity and reliability. To systematically assess the responses, we introduced an evaluation framework focusing on metrics, such as response accuracy, persuasiveness, and depth of explanation. The current work anticipates not only enriching our understanding of how these advanced LLMs can support general CSE education but also identifying areas where further development is needed for a more holistic integration of LLM-based chatbots in assisting student comprehension in the overarching engineering education.

This research full paper describes engineering doctoral students' nonresponse patterns on a departmental climate survey. As the U.S. engineering workforce does not reflect the diversity of the U.S. population, the departmental climate can be one lever that higher education leaders can use to identify specific policies, practices, and procedures in doctoral programs to increase the retention and success of students from historically excluded groups. During the summer and fall of 2023, 355 engineering doctoral students from 28 institutions in the U.S. responded to a climate scale that we developed to assess multiple climate factors associated with organizational commitment or member retention. Items included a six-point Likert-type response option and “not sure.” While most students responded adequately to the climate scale items with the Likert scale, a significant number of students also responded to the “not sure” option. Based on the climate and survey research in the literature, we hypothesized that these item nonresponses may stem from (a) the contextual characteristics of climate constructs or items and/or (b) student characteristics. Descriptive and inferential statistical data analyses showed that “not sure” item nonresponse differed by climate constructs and items as well as student characteristics. Among the six climate factors, on average, authenticity climate had the highest item nonresponse rates, followed by performance climate and diversity climate. While the item nonresponse rates increased by student age, at the item level analysis, item nonresponse rates varied by residency, gender, and first-generation status. There were no significant differences in the item nonresponse rates on underrepresented minority (URM) status, student disability, and LGBTQIA+ status.

The global nature of the engineering marketplace requires students to have the ability to work across teams of different cultures. Study abroad programs associated with experiential learning have traditionally been used to offer students opportunities that enhance their abilities to contribute to the global workforce. In response to the COVID-19 pandemic, there were several changes in course delivery creating an opportunity for virtual experiences as an alternative to study abroad programs. Such opportunities are being considered as formal courses involving international projects, or as informal opportunities for students to interact with students at other foreign universities. This research paper considers a Virtual International Collaborative Experiential Program (VICEP) involving engineering students from a US Midwestern research R1 institution and engineering students from a similar institution in Ethiopia and Ghana. The paper aims to explore the transformative potential of VICEP using Mezirow's transformative learning framework. The Learning Activities Survey data showed that students underwent various stages of transformative learning and reported changes in their habits of mind.

Spatial ability plays a crucial role in several fields, such as science, technology, engineering, and mathematics (STEM) for student performance, and its assessment is critical to identify the strengths and weaknesses of students in STEM performance. This study systematically examined a sample of spatial tests collected from the Northwestern Spatial Intelligence and Learning Center (SILC) website, a place for resources for research. Out of 37 tests on the website, 18 spatial tests were located and reviewed from two different perspectives of test-takers and a psychometrician and evaluated based on usability and psychometric characteristics. The explored usability characteristics of the spatial tests were the types of spatial ability to be assessed, target population, test format, availability of the directions, and usage in STEM education research. The reviewed psychometric characteristics were validity and reliability evidence and item-level characteristics of the spatial tests. Finally, this study appraised the usage of 18 spatial tests in STEM education research. The findings of this study are expected to provide researchers with the information needed to make informed decisions when selecting spatial tests that best match their specific research goals in STEM education.