This research investigates the effectiveness of randomized computer-based exams used in Carnegie Mellon's Introduction to Computer Systems course. Students received exams with 7-8 multiple choice and short answer questions that were automatically graded with regular expressions. Each question is randomly drawn from a pool of questions within its respective category. The exam system collects data about how each student progresses throughout their exam. This data [1] includes the order in which students view and answer the exam questions, the time spent viewing each question, and the score they have achieved at any given point during their exam. Our analysis of student scores quantifies and validates the fairness of these exams by reproducing prior methodology [2] that uses the MIRT multi-dimensional Generalized Partial Credit Model. The model allows us to estimate question difficulty, student ability, and question discrimination for any given student and exam question. Further analysis examines the correlation of higher-level student behaviors (e.g. order of solving questions) with ability and score. Key features include which question categories students spent the longest on, which ones students scored the highest on, and which ones the majority of students started the exam with. These insights were derived with statistical methods as well as supervised and unsupervised machine learning methods. This research supports the value of administering computer-based exams and informs future exam design. This research also motivates future interventions to test for identifiable behavioral features and students' test-taking strategies.

Liquid types can express richer verification properties than simple type systems. However, despite their advantages, liquid types have yet to achieve widespread adoption. To understand why, we conducted a study analyzing developers’ challenges with liquid types, focusing on LiquidHaskell. Our findings reveal nine key barriers that span three categories, including developer experience, scalability challenges with complex and large codebases, and understanding the verification process. Together, these obstacles provide a comprehensive view of the usability challenges to the broader adoption of liquid types and offer insights that can inform the current and future design and implementation of liquid type systems.