With the rise of generative Artificial Intelligence (AI), Machine Learning (ML) developers are becoming less reliant on real data to train their models. Data insufficiency can be resolved by using synthetic data generated by a diffusion model. However, beyond ad hoc interpretation of a generative model's outputs, there is little assurance of the synthetic data's adherence to the data requirement specifications. Adherence of synthetic data to these specifications is critical given that they describe desired downstream model behavior. Therefore, without proper verification methods for this synthetic data, ML developers cannot be confident in the behavior of the downstream model. This paper presents a verification method for generating synthetic data to train downstream ML models by prompting the generative model using requirement specifications and tracing elements of the output back to the prompt. The purpose of this research is to embed requirements engineering into the data augmentation process to increase the rigor and acceptance of these generative AI models to train downstream ML models. This improves the transparency of the data augmentation process, potentially increasing the trust of stakeholders in the generated data, and the use of generative models for data augmentation in a wider range of applications. This also provides a more traditional approach to synthetic data generation to guide ML developers in augmenting their datasets, thus incorporating a more rigorous engineering process into the ML development, i.e., ML Engineering.

During the Software Development Lifecycle (SDLC), the first stage entails the Requirement Engineering phase. In this phase, engineers gather, analyze, and specify the requirements for a software system. Requirements playa crucial role in the SDLC as they establish the foundation for the entire system by defining the expected behaviors of the software system to be built. The resulting specifications are captured in a Software Requirement Specification (SRS) document. As part of the validation process, requirement specifications are traced. Requirement tracing involves linking the requirement to the artifacts where the customer requested the high-level requirement. Teaching proper requirements tracing can be challenging in a traditional classroom setting. It is essential to educate future software engineers on the proper process of developing an SRS document and of tracing requirements back to the originating artifact, which is also challenging due to the complexity and large scope of applying the complete requirements engineering process. Understanding how changes in customer needs can impact requirements is an imperative learning opportunity. In this work, we aim to incorporate the use of AI in the teaching of requirements tracing using Large Language Models. In this experiment, both GPT -3.5 and GPT -4 are provided the transcript of an interview between the customer and the engineering team, as well as the subsequent requirements elicited from that meeting and other customer provided artifacts. The GPTs are then instructed to determine which requirements can be traced back to the interview transcript. At the same time, the students (the requirements engineering team) conduct their own effort to trace requirements back to the original interview. The experiment was taken one step further to assess students' and the GPTs abilities to address requirements modifications. After another interview with the customer, where some needs were changed, some requirements were modified, and students, and GPTs were asked to trace the modified requirements to the new interview. The results proved that students are better than both GPT versions at tracing modified requirements, yet GPTs again identified requirements that students didn't trace back. The findings, illustrate that AI can help in the teaching of requirement tracing; these results suggest that while no AI model is currently capable of replacing real requirement engineers as they don't outperform students, it can be used as a tool to test the completeness of the requirement tracing process. We posit that GPT can be a tool for students to self-assess the degree to which their own requirements tracing is exhaustive.

This innovative practice full paper describes the integration of Ph.D. students into departmental operations using Agile approaches, i.e., Serum, Serum is an agile framework originally developed for managing software development projects, but in recent years has been adapted into a framework used in classroom experiences and a management technique for non-engineering project-based work. In this work, Serum was adapted at the department operations level to enhance the efficacy of department projects and processes. A Ph.D. student acted as the Scrum Master for one of the teams. This provided the Ph.D. student with a unique experiential learning opportunity to apply theoretical knowledge in a practical setting, significantly enhancing their learning experience. The benefits for the Ph.D. student included the development of soft skills that made them a more well-rounded Ph.D. student and the creation of meaningful relationships with the faculty members on their team which increased their sense of belonging within the department. Additionally, the Ph.D. student gained insight into the intricacies of university operations, which is valuable training for a student who seeks a career in academia. This paper aligns with a desire to improve the experience of engineering and computing education of graduate students outside of the typical classroom and research domains.

Large Language Models (LLMs) are extensive aggregations of human language, designed to understand and generate sophisticated text. LLMs are becoming ubiquitous in a range of applications, from social media to code generation. With their immense size, LLMs face scalability challenges, making testing methods particularly difficult to implement effectively. Traditional machine learning and software testing methods, derived and adapted for LLMs, test these models to a point; however, they still struggle to accurately capture the full complexity of model behavior. This paper aims to capture the current efforts and techniques in testing LLMs, specifically focusing on stress testing, mutation testing, regression testing, metamorphic testing, and adversarial testing. This survey focuses on how traditional testing methods must be adapted to fit the needs of LLMs. Furthermore, while this area is fairly novel, there are still gaps in the literature that have been identified for future research.

This innovate-practice, work in progress paper presents a blueprint on the adoption of agile process in an academic engineering department. The blueprint offered in this paper can be used by engineering departments to introduce an agile approach in their teaching, research, and service activities. This paper provides a framework for using Scrum as it has been implemented in the EECS department at Embry-Riddle Aeronautical University. For each of the common facets of academia, i.e., teaching, research, and service, we report on our experiences, lessons learned, and proposed approaches in adopting Scrum. Furthermore, we highlight the value added by integrating this Scrum approach, such as the increased marketability of students with agile experience, a greater accountability from students and faculty, and/or building a much stronger communities between faculty, and between faculty and students. We also discuss possible obstacles and required mitigations to facilitate adoption. Results of adopting agility in all three facets of the academic department will be reported, including how it benefits computing and engineering education and the academic department.

Over the years, Agile approaches have been proven successful in industry settings, as documented in the literature. In response to this success, education professionals have developed ways to introduce Agile practices into engineering classrooms with similar success. These practices have been most popular in project-based courses because they enhance student learning and prepare students for using Agile practices in industry after graduation. The Scrum approach is one of the most popular Agile methods in industry and classroom adoption. Modified versions of Scrum are utilized within the classroom to align with student needs, familiarity with Scrum, and the materials presented within the class. As a result of this adaptation, many different Scrum-based implementations are found in classrooms. The popularity of this approach has led to numerous publications detailing individual experiments using Scrum in the classroom, with most of these adoptions occurring in engineering classrooms. This paper presents a literature review of Scrum applied in the classroom. This work explores the advantages of using Scrum in the classroom, providing details on the type and level of university classroom used for implementation. Information on methods of implementation, appropriate class subjects, and student educational levels are provided within this paper. This guide can be used by those looking to utilize Scrum within their classroom as a stand-alone practice. The findings of this paper demonstrate that Scrum can be used in correlation with a wide variety of classroom structures and topics. This paper is intended to guide future educators who wish to implement Scrum into classes and educational programs.

Universities, faculty, and students use Learning Outcomes (LO) to create a shared understanding of the content provided in an individual course, known as Outcome-Based Education (OBE). One area of interest in OBE is evaluating whether the instructor and individual student performance have met the LO, which is integral to ensuring all invested parties are on the same page about class content and student performance. This work proposes a system for the management and evaluation of LO. Primarily, this work defines an ontology to support the management and evaluation of LO via Knowledge Graphs (KG). The KG links individual LO with individual assessment items. Two state-of-the-art Natural Language Processing models, BERT and ChatGPT, are evaluated in respect to their effectiveness in automating this linking. This data allows the educational professional to reflect on how well their assessments match the course's LO. The second part of this system harnesses student data to measure performance in relation to LO. In this Work-in-Progress paper, the system is prototyped and tested on the midterm results of a course in the Software Engineering curriculum. Student performance is documented in relation to each assessment question on the exams to measure student mastery of course material. Through this approach, courses can be evaluated and improved to deliver better quality education to all students. This includes improvements at the course level and possibilities for early intervention to ensure student success. This paper details the development of this system and through its implementation shows how it benefits engineering educators and their students.

In this paper, we propose an innovative practice based on agile software development methods. This research approach introduces agility into learning of research in an academic environment, resulting in an Agile Research Team. Such a research team follows an agile approach, based on modifications to the Scrum approach, to collaboratively learn about research, and to manage research projects and the researchers involved. Success in research requires self-motivation, collaboration, and knowledge exchange. Traditional research occurs in top-down research groups that are led by a leading researcher, who oversees postdoctoral researchers and Ph.D. students, who in turn manage graduate and undergraduate level students. It is up to individual researchers to stay motivated, to acquire the necessary skills to conduct research, and, oftentimes, to decide what the following steps are. Much like effective research groups, agile software development approaches rely on individuals to form self-organizing and motivated teams to deliver technical excellence. Agile software development teams also require an environment of sharing knowledge between senior and junior developers. Agile approaches can facilitate the efficient exchange of knowledge due to a strong dependency on face-to-face communication and teamwork. With the emerging adoption of agile methods for software development in industry and its ability to expedite projects’ delivery, we argue that such approaches can potentially provide similar benefits for researchers and students in academia. The advantages that agile methods provide are twofold: the ability to respond faster to change, and a shorter feedback loop, which facilitates the learning of how to conduct research. This paper explores the impactful benefits of using an agile approach to manage research team projects to keep researchers motivated, enhance the learning of knowledge and research skills, increase scalability, and foster inclusivity. This paper will also present the roles, responsibilities, and processes defined for managing an Agile Research Team to support adoption of the approach with other research teams. In addition, results and lessons learned are presented following our experience with using the approach as described in this work.

This innovate practice work in progress paper presents our initial findings about the application of agile process in the day to day operation of an academic department. The benefits of agile methods, which are processes that are lightweight and people-oriented, are well-known. Agile methods provide the ability to respond faster to the changing needs of stakeholders, improved product quality, team over individual goals, and better transparency. Therefore, one would expect adaption of the agile process within academic environment would be welcomed, however, unique issues exist within faculty that impact adoption; faculty have different goals that change through their tenure, and faculty have their own priorities that drive their time investment. For the last 18 months, the Electrical Engineering and Computer Science department at the Embry-Riddle Aeronautical University has been involved in adopting the use of agile process in its day-to-day operations, namely Scrum, which is one type of Agile method. To assess the feasibility of departmental adaptation of agile process, we established two pilot projects to identify advantages and disadvantages such adaptation. In this paper we report on the outcome of the projects in addition to the observations, challenges, and opportunities regarding the adaptation of the agile processes in the day-to-day operation of an academic department.

This Work in Progress paper in the Innovative Practice Category presents an approach to incorporate the use of a virtual reality environment in the teaching of analysis of software requirements through modeling. Common models include use case diagrams, state charts, data flow diagrams, etc., all serving their own role in defining different perspectives of a system. These different perspectives of the same system can be a powerful technique to facilitate the analysis of requirements about the system. Using a virtual reality environment allows for the linking of multiple perspectives from the separate analysis models to be viewed in three dimensions. This approach can be particularly beneficial to students in learning how to analyze models by providing a mechanism to examine the visual relationship between the perspectives. Using this approach allows the student to view, create, manipulate, and alter models within the native three-dimensional space, which can provide an indepth experience better suited for learning. The virtual reality environment also allows models to be nested to provide better abstractions and layered to provide a level of complexity impossible to replicate in the traditional two-dimensional space. This paper explores a prototype virtual reality environment to facilitate the use of software requirement models for undergraduate software engineering students. As part of this work, students will be exposed to the tool and tasked with using the virtual reality environment to analyze the models of systems. Assessment of the use of the approach will be conducted and results will be presented including feedback from the students, benefits found, insights and potential ways to evolve the approach for other areas such as design or testing.

In this paper, we explore the benefits of teaching heavyweight process life cycle models to young software engineering students to better prepare them in the use of agile methods. The benefits of agile methods compared to process-heavy models include the ability to respond faster to the changing needs of customers and the short feedback loop between customers and developers. Since agile methods depend significantly on the competence of the individual, teaching traditional approaches (e.g. waterfall) can be advantageous to the understanding of students about what the major activities associated with software development are, by providing a well-defined structure that naturally aligns with the novice student. Towards this goal, we conducted a survey with different types of software engineers, ranging from novice undergraduates to software engineering professionals. The survey provides data on the exposure of traditional and agile process, the level of experience in using traditional and agile process models, and the perceived most optimal learning sequence. Based on the analysis of the collected data, we argue that teaching traditional software engineering process-oriented approaches prior to introducing agile methods, is highly beneficial to students' understanding and optimal use of agile techniques.

In this paper we present an approach used to enhance students' competency in software verification. Students were asked to apply software verification techniques to a complex formal specification system. The complexity of the system stems from its sophisticated requirements. Selecting such system for this study was intentional for the following two reasons 1) the system is difficult to understand and analyze because of the domain knowledge required to generate formal specifications in temporal logic and 2) the system is large and complex which lends itself to a wide range of applicable verification techniques, and thus highlights the differences in the capabilities of each of the software verification approaches. Students were assessed using multiple criteria including; examination in applying learned techniques, students' attitude toward the technique, perceived efficiency of the techniques in discovering software defects, and the ability of the technique to locate errors in the code beyond simply indicating their presence. The results of this work show that the students applied the learned techniques successfully and their attitudes towards software verification improved.