This Innovative Practice Full Paper presents findings from the implementation of a virtual reality-based learning module. In the Fall of 2020, a prototype for a novel intervention namely, Virtual/Augmented-Reality-Based Discipline Exploration Rotations (VADERs), was offered as part of the first-year Introduction to Architectural Engineering (AE) classes at two universities. VADERs will ultimately be a collection of modules that are designed to improve student engagement and diversity-awareness by providing interactive virtual explorations of an engineering discipline and its sub-disciplines. VADERs utilize an open source, device-agnostic, and browser-based three-dimensional Virtual Reality (VR) platform, creating unique accessibility, synchronous social affordances, and media asset flexibility. The conjecture explored in this paper is: Having first-year engineering students experience Architectural Engineering and its sub-disciplines through an interactive VADER module, will improve their self-efficacy in regards to their commitment to studying the discipline. A total of 89 students participated in the VADER pilot in Fall 2020. Complete data was collected from 67 of these participants in the form of pre- and post-surveys, and final project deliverables. Results tied to the hypothesis and recommendations for future related work are discussed.

This research paper examines how engineering instructors initially experienced a forced change from in-person teaching to emergency remote teaching by collecting instructors' self-reported successes and challenges. Participants included engineering instructors from an R1 institution who completed weekly online surveys during the last seven weeks of the Spring 2020 semester. An inductive thematic analysis of the instructors' responses to two open-ended items was completed from which the three most frequently cited themes from each of the successes and the challenges are presented and discussed. Understanding the most common successes and challenges experienced by instructors during emergency remote instruction lays the foundation for designing supportive strategies for future instructional change initiatives in engineering education.

In this Research Category Full Paper, weekly structured reflections were given to students as a means to engage them in planning and acting to improve their learning. First-year engineering students struggle to self-regulate their study habits. The intention was to investigate students' planning and follow-through of different study strategies to gain a better understanding of how students formulate their study plans. This study was conducted in one section (n=114) of a first-year computer tools and problem-solving course. Students were given weekly structured reflections about the learning strategies they used to complete problem sets. The results showed that students overestimated the number of study strategies they planned to use. Frequently completed strategies were less time-intensive, while less frequently completed strategies were more time intensive and involved more deep-thinking. These findings can help inform instructors on how to support student learning and self-regulation.

This Research paper explores engineering instructors' ability to emotionally adapt to remote teaching and whether they felt their emotions were typical compared to a non-COVID semester. Participants, including engineering professors of practice, tenure-track and tenured faculty from five engineering disciplines completed weekly online surveys during the last seven weeks of the Spring 2020 semester. Adaptability, as a framework, was used to examine instructors’ emotions over the initial weeks of remote instruction. Descriptive statistics were used to explore engineering instructors’ emotions across the seven weeks and typicality of these emotions as compared to a non-COVID semester. The findings revealed that over the seven-week period, instructors reported their emotions as being more positive than negative. More than half of the instructors felt their emotions were atypical during the first four weeks and typical during the last three weeks. In addition, many instructors (73%) felt accomplished by the end of the semester. Understanding instructors’ feelings during this forced change to emergency remote teaching can serve to identify emotional support needed for faculty to be able to change.

This Research-To-Practice Category Full Paper is a qualitative study of four students' reflections on their performance and feedback on assignments in a required first-year engineering course. The intention was to gain a deep understanding of how students engage with a course's standards-based grading (SBG) system to plan and act via an analysis of their structured reflections. The individual student cases are presented. A cross-case analysis focused on self-ratings compared to performance, actions and connections made to learning objectives, plans and connections made to learning objectives, and other insights is also presented. Implications for the design of SBG systems and accompanying reflections are discussed.

In this Research-to-Practice Full Paper, two different reflection prompts in the form of unstructured and structured reflections were implemented in an engineering course to guide students towards improved self-regulated learning behaviors, specifically accessing feedback on their assignments. The course setting for this study employed standards-based grading (SBG) which provides rich feedback on students' proficiency with the course learning objectives. Low student access to feedback, delivered through a learning management system (LMS), was seen as a considerable learning opportunity loss. Negative binomial regression models were used to investigate whether semester (2016 or 2017) and refection (presence or absence) had an impact on the (1) number of days students accessed the LMS gradebook or rubrics, (2) the number of times students accessed the gradebook to view grades, or (3) the number of times students accessed rubrics to view feedback. Semester, which relates to implementation changes in the SBG system, significantly increased students' number of days of access and number of gradebook and rubric accesses. Reflection, particularly structured reflection, significantly improved students' access to the rubrics. Pairing reflection with a well-developed SBG system has the potential to improve students' access to feedback.

Massive Open Online Courses (MOOCs) attract learners with different learner intentions, background knowledge, and skills as compared to traditional, closed enrollment settings. This level of diversity in learners introduces new factors that impact student persistence and engagement. Previous research has analyzed MOOC learners who are either fully engaged in a course in its entirety or at least consistently accessing course materials. Sporadic users, those who access course content randomly, have not been studied as in depth, even though they comprise the largest number of learners in a highly advanced engineering MOOC. In this study, we use clickstream data and pre-survey data to understand sporadic learners in a highly advanced nanoelectronics MOOC. We used a MOOC on nanotechnology with a total enrolment of close of 10,000 users offered for a duration of 8-weeks. We identified that academic preparedness, learner intentions, and expected time commitment could be used to predict sporadic users. Finally, an effect size analysis was performed.

Despite the remarkable expansion of massive open online courses (MOOCs) over the last decade, the extent to which they achieve their potential as a “disruptive force” in education is questionable, with limited evidence of true impact. There is a wide range of desired outcomes from both learners themselves and those that invest in MOOCs. Therefore, to truly evaluate the effectiveness of a MOOC, it is necessary to understand the needs and goals of all stakeholder groups involved. While some studies have explored these factors for students who take MOOCs and the institutions offering MOOCs, little research has investigated what the instructors who choose to teach advanced science, technology, engineering, and mathematics (STEM) MOOCs hope to gain from the experience. In order to inform this component of a contextual evaluation framework of MOOCs, this study seeks to answer the question, “What factors drive teaching a MOOC?” in the context of advanced STEM courses. This study explores instructor perspectives through semi-structured interviews with 14 instructors of advanced STEM MOOCs who were identified through purposive sampling. Following transcription, we performed thematic analysis to identify underlying themes. A set of nine themes fell into two main categories: personal factors and situational factors. While the expression of these themes varied from one instructor to the next, the consistent presence of each theme suggests the value of incorporating the themes into a framework for evaluating the effectiveness of a MOOC.

There have been numerous efforts to increase students' academic success. One data-driven approach is to highlight the important learning objectives in a course. In this paper, we used visualization and three feature selection methods to highlight the important learning objectives in a course. Identifying important learning objectives as well as the relation among the learning objectives have multiple educational advantages. First, it informs the instructors and students of the important topics in the course; without learning them properly students will not be successful. Second, it highlights any inconsistencies in defining the learning objective, how they are being assessed, and design of the course. Thus, this approach can be used as a course design diagnostic tool.

Standards-based grading (SBG) provides students with feedback on their achievement of specific course learning objectives. What training students need to make meaning of and plan to act on this feedback is not known. This work in progress took place in a required first-year engineering course in which a SBG system is used. Following two problem, students were asked to respond to reflection prompts concerning their beliefs about their learning, how representative their learning objective based feedback was of their learning, and the actions they planned to take as a result of their feedback. Initial findings suggest that students do not engage in deep thinking on the feedback, though students' ability to reflect or the nature of the reflective prompts and their timing may also have had an impact on the depth of students' reflections. Pairing instruction on reflection and interpretation of SBG feedback is presented as a strategy to overcome the lack of deep thinking on SBG feedback and improve the quality of the reflection data.

Massive Open Online Courses (MOOCs) offer the ability to educate large numbers of diverse learners who might not have access, time, or the financial resources necessary for more formal coursework. While some studies have focused primarily on understanding MOOC learners purely through their access rates to course materials, others have sought to understand learners through surveys. We combined these two sources of data to address two research questions: (1) What are the patterns of user behavior in an advanced, technical MOOC? and (2) What are the characteristics of fully engaged learners? By analyzing clickstream and pre-survey data for a nanotechnology-related MOOC, we identified differences and similarities between fully engaged learners and other groups. The lack of strong indicators to predict fully engaged learners suggests a need for improved data from pre-course surveys.

In our continued efforts to characterize feedback on design, we analyzed experienced engineers' feedback on design. This analysis enabled us to characterize and compare experienced engineers' feedback on design to students' and educators' feedback. Participates were 10 experienced systems engineers, with an average of 28 years of industry experience, who attended the ninth International Council on Systems Engineering (INCOSE) Great Lakes Regional Conference in Cleveland, Ohio. A coding scheme consisting of three domains (User-centered design, Substance, and Focus of feedback) was used to analyze the feedback. The experienced engineers' feedback profile was different than both students' and educators' profiles. However, there were some similarities between engineers' and educators' feedback.

Highly technical STEM MOOCs have recently become widely available, but little is known about the motivations of the various groups of learners participating. In this work, we perform a detailed survey of 1,624 learners to examine their motivations in detail. These learners exhibited overall high levels of intrinsic motivation, but varied in their extrinsic motivation, according to their current position as students, workers, or unemployed individuals. Students generally reported the highest levels of extrinsic motivation compared to other groups (p<;0.001). The results from this analysis indicate that additional factors about learners in each group, such as their course participation and performance, should be examined in future work to help better understand the various needs of those enrolling in highly technical STEM MOOCs.

Most studies on feedback and design use thematic analysis. In this paper, we apply three text analysis methods that are not commonly used in feedback and design studies to encourage researchers to think outside of the box and try different methods. We show how word cloud representation, word frequency, and reading ease and grade level tests can be applied to a feedback on design dataset in order to highlight the strengths and limitations of each approach. In general, findings of these three methods are aligned with those from our previous thematic analysis and provide new insights about the feedback. We recommend that these methods be used in addition to thematic analysis in analyzing feedback on design research.

Educational interventions in first-year engineering programs can positively affect students' awareness of nanotechnology by introducing students to basic nanotechnology concepts and motivating them to follow nanotechnology-related career paths. The research question examined in this study is: What are the differences in exposure, awareness, and motivation between students in classrooms where a mathematical modeling activity is implemented and where a mathematical modeling activity and a simulation design project are implemented? In a pre/posttest quasi-experimental design, first-year engineering course sections were split into experimental and control groups. The experimental course sections (6 sections n=496) implemented a mathematical modeling activity and a simulation design project related to nanotechnology. The control course sections (8 sections n=703) implemented only the mathematical modeling activity. Results show that when implemented together, the simulation design project and the mathematical modeling activity are more effective than only the mathematical modeling activity in terms of increasing student awareness, exposure and motivation related to nanotechnology. Although the change in motivation is statistically significant, the effect size is low. Therefore, further research for uncovering factors linked to motivation is necessary.

Different types of instruction are used at the college-level for increasing students' knowledge of nanotechnology. Purdue University introduced nanotechnology to first-year engineering students in an introductory engineering course through two nanotechnology-related projects. The students were split into two groups. Group 1 students (8 sections, n=703) implemented only a mathematical modeling project related to nanotechnology. Group 2 students (6 sections, n=496) implemented both the mathematical modeling project and a simulation design project related to nanotechnology. Students were asked to complete a nanotechnology assessment at the start and end of the semester. In this assessment, one prompt asked students to describe five things they know about nanotechnology. This work-in-progress describes the development of the coding scheme for categorizing the student responses. Researchers had ten meetings to revise the coding scheme, the final meeting resulting in the inter-rater reliability (TRR) estimate of 82% One of the researchers applied the coding scheme to 2000 responses and initial results are reported here. The coding scheme will be used to inform educators about the impact of their nanotechnology-related projects and attain their desired learning goals.

The concepts of size and scale are fundamental to nanotechnology education but can be difficult for beginning undergraduate students to grasp. It is important to develop curricular interventions to increase students' conceptual understanding of size and scale, before moving on to more advanced subject matter. The purpose of this paper is to present the initial steps in the development of the Nano Size and Scale Instrument (NSSI), an assessment of students' understanding of size and scale. Ten items were developed for the NSSI. The NSSI was administered to 118 first year engineering students and studied using Classical Test Theory. Results suggest that six of the ten items show appropriate difficulty and discrimination. Five of the items were of moderate difficulty and students' mean scores overall were low (4.97 out of 10), indicating that students in this study found size and scale concepts difficult. Potential modifications to the NSSI are discussed.

Providing feedback on design projects to other students (peer feedback) helps students develop a better understanding of design and helps students practice providing feedback, which is an engineering professional skill. In this study, we analyzed first-year engineering students' feedback on sample student design work in order to understand the characteristics of the feedback students provide and highlight the differences, if any, between students with and without prior design experience. A coding scheme for feedback on design was developed and used in this study. Feedback on Milestone 1, which focused on problem scoping, was analyzed for this work-in-progress paper. Overall, students' feedback focused on the weaknesses of the design work and provided specific directions for change. Students with previous design experience provided more comments than students with no experience. These comments were mostly focused on weaknesses and were related to communication aspects of design.

Developing engineers for a changing world presents a greater need for students to understand simulations. Understanding mathematical models and simulations are skills that all engineers need for industry and research, but many engineering students lack a conceptual understanding of how to develop them. This workshop facilitates conversations about the future of engineering education with an emphasis on the importance of mathematical modeling and simulations. The team will introduce pedagogy used in Purdue's First-Year Engineering Program that enables students to learn about mathematical modeling and simulation. The team will introduce model-eliciting activities (MEAs), which enable students to learn how to build mathematical models in an open-ended learning environment. The team will also introduce a well-developed design project that focuses on using mathematical models to build simulations. The team will also introduce nanoHUB.org, an online community that enables students to interact with expert-developed simulations and presents a platform for students to disseminate their own simulations in the future. We will facilitate conversations about how nanoHUB.org can be used in informal and formal learning environments. We will also discuss how nanoHUB.org presents a place for collaborative engineering education research. All of the materials from this workshop will also be incorporated on the Nano Education Research page on nanoHUB.org to further facilitate conversation after the conference. The context for all of the discussed content is connected to nanotechnology, which enables students from various STEM backgrounds to work together in a cross-disciplinary environment. Overall, our mini-workshop will compliment FIE objectives by looking forward to the future of STEM education through nanotechnology education and innovative, online learning environments.

Teaching assistants (TAs) are crucial for student learning, especially in large science, technology, engineering, and mathematics (STEM) programs with student-centered learning environments. To understand TAs' perspectives of their ability to facilitate learning in these environments, this research team conducted a mixed-methods study. This study was set in the First-Year Engineering Program at Purdue University. The TAs for this program are responsible for scaffolding student teams' learning through feedback on open-ended problems, along with some other traditional TA duties (e.g., grading student work, answering student emails). The study began with a qualitative component in which eight TAs were interviewed about their abilities to execute their various responsibilities. Based on these interview findings and literature, a survey was developed to further investigate key elements that the TAs discussed (e.g., motivation and feedback). Forty-three TAs completed the survey. Based on the analysis of TAs' responses, two regression models were made that describe the factors that effect TAs' self-reported likeliness to give effective feedback and ineffective feedback. TAs with intrinsic motivation were more likely to give more effective feedback, while TAs with extrinsic motivation were more likely to give more ineffective feedback. Also female TAs reported giving less feedback in general than male TAs.

During 2008, the Institute for P-12 Engineering Research and Learning (INSPIRE) began a research project that started with 19 teachers and more than 200 students who were educated, interviewed, and assessed for STEM learning. This longitudinal study grew fast, gathering numerous pieces of data from 168 teachers and more than 3,000 students during the five-year treatment. This information growth led to a need for data management improvement. However, there are few database structures available for gathering school-related research. This paper presents lessons learned during the process of database construction and improvement for supporting project data consolidation. The process of construction and development included structural changes to provide an organization commensurate with schools' clusters, domains, and strata, all of which occur organically in the school setting. The database construction also included program development for uploading and cleaning large amounts of information; application analysis for defining an appropriate tool; and improvements for user information visualization. The analysis of lessons learned includes a discussion of major changes within the database and benefits of creating program codes within this tool. The discussion ends addressing the implications of having a structured and automatized yet flexible database tool designed for school settings.

This paper presents an investigation into First-Year Engineering (FYE) students' ability to identify the direct user in open-ended client-driven problem solving activities. To guide FYE students in formulating an open-ended problem, students are asked a series of questions about the stakeholders, the direct user of the solution, and its needs. The purpose of these questions is to help students think about the problem individually by listing stakeholders including the direct user and the stakeholders' relationships to the problem and its solution prior starting to solve the problem in teams. One hundred (out of ~1600) students' responses to three open-ended problem solving activities during two successive semesters were randomly selected and analyzed. Results showed improvement in students' ability to identify the direct user over time. The majority of students' responses shifted from identifying non-users instead of direct user in the first MEA, to identifying indirect-users in the second MEA, and to identifying direct user correctly in the third MEA. This shows a clear improvement in students' ability to understand these problems over time. However, even for the third MEA, only 60% of students identified the direct user correctly, raising concerns about comprehension of the problem by about 40% of the students.

Effective teaching assistants (TAs) are crucial for effective student learning. This is especially true in science, technology, engineering, and mathematics (STEM) programs, where TAs are enabling large programs to transition to more student-centered learning environments. To ensure that TAs are able to support these types of learning environments, their perspectives of training, their abilities, and other work related aspects must be understood. In this paper a survey that was created based on interviews conducted with eight TAs is discussed. The survey has four primary categories of content that are critical for understanding TAs' perspectives: (1) background, (2) motivation, (3) training, and (4) grading and feedback. This research team is first utilizing this survey at Purdue University to test for validity and reliability of the instrument, as well as identifying ways to improve the experiences and effectiveness of the First-Year Engineering Program's TAs' support system, training, hiring process, and any other relevant components of the infrastructure. The more generalizable goal of this research is to further develop this survey to be used by any STEM program as a diagnostic tool for identifying opportunities to enhance the TA support systems and therefore improve student learning.

Nanotechnology as a research field presents many new opportunities and challenges for educating the next generation of engineers. In this paper, we attempt to understand the impact of this research team's initiatives to increase engineering students' awareness and understanding of nanotechnology. This is a case study of how four student teams changed their iteratively-developed design solutions for a nanotechnology-based design project. The goal is to investigate how first-year engineering (FYE) students developed their demonstrated knowledge of nanotechnology. We found that all four teams discussed nanotechnology concepts from the beginning of their solutions. Although they had difficulty relating their nanotechnology applications to science and mathematics concepts, they had consistent ideas for the nanotechnology-specific content they wanted to focus on throughout their project. The results show that all four project cases have a clear potential connection to a field of engineering. Connecting nanotechnology to potential engineering fields may present a better content focus than science and mathematics concepts.

Interview transcripts from students (grades 2-4) were examined to measure change in understanding of engineering constructs using a coding scheme that is in development. The participants were students of teachers from a large, urban school-district that participated in two summer teacher professional development (TPD) engineering education academies and received continued support from a university center devoted to P-12 engineering education. To analyze the interviews, the overarching themes were broken into eight constructs that were points of focus during the TPD. Sample results show change in all mean scores. The inter-coder reliability for this iteration of scheme development was 91.3%. Scored level of understanding of the constructs (on a scale of 0-3) produced 75% agreement. The coding scheme continues to be refined in an effort to provide a form of evaluation of TPD and teacher effectiveness that identifies weaknesses in curriculum design and implementation.

When developing mathematical models for authentic problems that draw on basic descriptive statistics, many first-year engineering students do not address data distribution when it is appropriate to do so. An assignment was created in association with such a problem that asks students to develop their own test cases as a means of getting students to think more deeply about the data provided and the models they are developing. In this work in progress, these test cases are analyzed. The means and standard deviations of the test cases are quite different than those provided with the problem. However, there is evidence that students may not be attending to the original problem context. Next steps are briefly indicated.

The addition of engineering to many state standards has created a need for effective teacher professional development. The Institute for P-12 Engineering Research and Learning (INSPIRE) facilitated summer engineering workshops for teachers in grades two to four. Participants completed the Noticing Activity, an assessment meant to gauge teachers' ability to notice and discuss engineering and its role in society. This work in progress describes the coding system being developed to analyze teacher responses. Analysis of teacher responses to the Noticing Activity will be used to determine what teachers notice about engineering in society and, later, help study how teachers discuss engineering with their students. Results from the coding will be used for programmatic evaluation and revision.

Peer feedback is a key component of STEM education, but there is not an accepted pedagogy for teaching students effective critical feedback skills. The quality and nature of current peer reviews must be understood to establish what students need to learn to give effective feedback. This research begins to identify the nature and quality of peer feedback on mathematical models through a mixed-methods approach. The quantitative analysis shows that students typically score models higher than an expert, and the qualitative analysis shows that most students do not address the mathematics used in solutions but rather focus on the presentation of solutions. This research indicates that students need better guidance on how to engage in a more critical feedback process on solutions to open-ended problems. This research also raises some new research questions.

While providing opportunities for students to engage in open-ended, ambiguous problems allows students to develop skills for responding to ambiguity and applying knowledge in practical ways, it can be difficult for instructors to assess student work when students' solutions are very different from each other. However, to promote student learning, it is essential that instructors provide feedback - particularly in the form of formative feedback that allows students to revise their work before submitting their final solution. In this study, we investigate the challenges associated with providing feedback, the challenges that students face in responding to feedback, and the impact of the feedback on student work. We do this by investigating the role of feedback through multiple data collection methods: collection of artifacts - copies of student work (and copies of instructor and peer feedback), interviews conducted with students and with teaching assistants, and video recordings of the students as they work to address the feedback and revise their solution. Through these multiple lenses, we are beginning to uncover challenges and strategies for success in both providing and responding to feedback. These findings provide opportunities to design and re-design instructional materials that equip teaching assistants and students with skills to provide feedback, and to equip student with skills to respond to feedback.

The Draw-an-Engineer Test (DAET) is an age-appropriate instrument for collecting elementary students' conceptions of engineers and engineering. However, analysis of drawings can be challenging. To make using the instrument more manageable for measuring the impact of engineering integration into classrooms, it is desirable to code each drawing with one of a limited set of defined conceptions. Four conceptions were identified in prior work with students with little to no engineering instruction: laborer, mechanic, technician, and designer. In this prior work, few students demonstrated the conception of designer. As engineering has been integrated into classrooms, more drawings could be coded designer but the drawings also represent a wider variety of ideas about engineering. This Work in Progress (WIP) will look at representative samples of student drawings coded with the designer conception to highlight the emergent engineering ideas.

Assessing student work on realistic, open-ended problems is challenging when multiple solutions of varying quality can be expected. A balance needs to be sought between school-like and engineering-practice-like feedback and assessment of student work if an instructor wishes to capitalize on the authenticity of a problem. This workshop will work with the ideas of authentic assessment for open-ended engineering problems — the idea that not only the problem but also the feedback and assessment strategy can reflect engineering practice. Participants will learn about a four dimensional feedback and assessment model being used with Model-Eliciting Activities, a special form of open-ended problem. They will learn to apply the feedback and assessment model to sample student work and how the model can be more generally applied to open-ended problems.