Reproducibility is crucial to the advancement of science; it strengthens confidence in seemingly contradictory results and expands the boundaries of known discoveries. Computer Security has the natural benefit of creating artifacts that should facilitate computational reproducibility, the ability for others to use someone else's code and data to independently recreate results, in a relatively straightforward fashion. While the Security community has recently increased its attention on reproducibility, whether the current approach to increasing reproducible research is effective remains an open question. In this dissertation, we measure the impact of current approaches to reproducible research, construct frameworks and tools to increase reproducible outcomes, and analyze the approach we have taken. The goal of this dissertation is to provide tools for security researchers that simplify and increase the reproducibility of research.

In response to the growing sophistication of censorship methods deployed by governments worldwide, the existence of open-source censorship measurement platforms has increased. Analyzing censorship data is challenging due to the data’s large size, diversity, and variability, requiring a comprehensive understanding of the data collection process and applying established data analysis techniques for thorough information extraction. In this work, we develop a framework that is applicable across all major censorship datasets to continually identify changes in censorship data trends and reveal potentially unreported censorship. Our framework consists of control charts and the Mann-Kendall trend detection test, originating from statistical process control theory, and we implement it on Censored Planet, GFWatch, the Open Observatory of Network Interference (OONI), and Tor data from Russia, Myanmar, China, Iran, Türkiye, and Pakistan from January 2021 through March 2023. Our study confirms results from prior studies and also identifies new events that we validate through media reports. Our correlation analysis reveals minimal similarities between censorship datasets. However, because our framework is applicable across all major censorship datasets, it significantly reduces the manual effort required to employ multiple datasets, which we further demonstrate by applying it to four additional Internet outage-related datasets. Our work thus provides a tool for continuously monitoring censorship activity and acts as a basis for developing more systematic, holistic, and in-depth analysis techniques for censorship data.

Audio deepfakes represent a rising threat to trust in our daily communications. In response to this, the research community has developed a wide array of detection techniques aimed at preventing such attacks from deceiving users. Unfortunately, the creation of these defenses has generally overlooked the most important element of the system - the user themselves. As such, it is not clear whether current mechanisms augment, hinder, or simply contradict human classification of deepfakes. In this paper, we perform the first large-scale user study on deepfake detection. We recruit over 1,200 users and present them with samples from the three most widely-cited deepfake datasets. We then quantitatively compare performance and qualitatively conduct thematic analysis to motivate and understand the reasoning behind user decisions and differences from machine classifications. Our results show that users correctly classify human audio at significantly higher rates than machine learning models, and rely on linguistic features and intuition when performing classification. However, users are also regularly misled by pre-conceptions about the capabilities of generated audio (e.g., that accents and background sounds are indicative of humans). Finally, machine learning models suffer from significantly higher false positive rates, and experience false negatives that humans correctly classify when issues of quality or robotic characteristics are reported. By analyzing user behavior across multiple deepfake datasets, our study demonstrates the need to more tightly compare user and machine learning performance, and to target the latter towards areas where humans are less likely to successfully identify threats.

Reproducibility is crucial to the advancement of science; it strengthens confidence in seemingly contradictory results and expands the boundaries of known discoveries. Computer Security has the natural benefit of creating artifacts that should facilitate computational reproducibility, the ability for others to use someone else's code and data to independently recreate results, in a relatively straightforward fashion. While the Security community has recently increased its attention on reproducibility, an independent and comprehensive measurement of the current state of reproducibility has not been conducted. In this paper, we perform the first such study, targeting reproducible artifacts generated specifically by papers on machine learning security (one of the most popular areas in academic research) published in Tier 1 security conferences over the past ten years (2013-2022). We perform our measurement study of indirect and direct reproducibility over nearly 750 papers, their codebases, and datasets. Our analysis shows that there is no statistically significant difference between the availability of artifacts before and after the introduction of Artifact Evaluation Committees in Tier 1 conferences. However, based on three years of results, artifacts that pass through this process work at a higher rate than those that do not. From our collected findings, we offer data-driven suggestions for improving reproducibility in our community, including five common problems observed in our study. In so doing, we demonstrate that significant progress still needs to be made in computational reproducibility in Computer Security research.

SMS-based two-factor authentication (2FA) is the most widely deployed 2FA mechanism, despite the fact that SMS messages are known to be vulnerable to rerouting attacks, and despite the availability of alternatives that may be more secure. This is for two reasons. First, it is very effective in practice, as evidenced by reports from Google and Microsoft. Second, users prefer SMS over alternatives, because text messaging is already part of their daily communication. Accepting this practical reality, we developed a new SMS-based protocol that makes rerouting attacks useless to adversaries who aim to take over user accounts. Our protocol delivers one-time passwords (OTP) via text message in a manner that adds minimal overhead (to both the user and the server) over existing SMS-based methods, and is implemented with only small changes to the stock text-message applications that already ship on mobile phones. The security of our protocol rests upon a provably secure authenticated key exchange protocol that, crucially, does not place significant new burdens upon the user. Indeed, we carry out a user study that demonstrates no statistically significant difference between traditional SMS and our protocol, in terms of usability.