This work presents an evaluation of six prominent commercial endpoint malware detectors, a network malware detector, and a file-conviction algorithm from a cyber technology vendor. The evaluation was administered as the first of the Artificial I ntelligence Applications t o Autonomous Cybersecurity (AI ATAC) prize challenges, funded by / completed in service of the US Navy. The experiment employed 100K files (50/50% benign/malicious) with a stratified distribution of file types, including ~1K zero-day program executables (increasing experiment size two orders of magnitude over previous work). We present an evaluation process of delivering a file to a fresh virtual machine donning the detection technology, waiting 90s to allow static detection, then executing the file and waiting another period for dynamic detection; this allows greater fidelity in the observational data than previous experiments, in particular, resource and time-to-detection statistics. To execute all 800K trials (100K files × 8 tools), a software framework is designed to choreograph the experiment into an automated, time-synced, and reproducible workflow with substantial parallelization. Software with base classes for this framework are provided. A cost-benefit model was configured to integrate the tools’ detection statistics into a comparable quantity by simulating costs of use. This provides a ranking methodology for cyber competitions and a lens for reasoning about the varied statistical results. The results provide insights on state of commercial malware detection.

There is limited information regarding how users employ password managers in the wild and why they use them in that manner. To address this knowledge gap, we conduct observational interviews with 32 password manager users. Using grounded theory, we identify four theories describing the processes and rationale behind participants’ usage of password managers. We find that many users simultaneously use both a browser-based and a third-party manager, using each as a backup for the other, with this new paradigm having intriguing usability and security implications. Users also eschew generated passwords because these passwords are challenging to enter and remember when the manager is unavailable, necessitating new generators that create easy-to-enter and remember passwords. Additionally, the credential audits provided by most managers overwhelm users, limiting their utility and indicating a need for more proactive and streamlined notification systems. We also discuss mobile usage, adoption and promotion, and other related topics.

Password managers help users more effectively manage their passwords, encouraging them to adopt stronger passwords across their many accounts. In contrast to desktop systems where password managers receive no system-level support, mobile operating systems provide autofill frameworks designed to integrate with password managers to provide secure and usable autofill for browsers and other apps installed on mobile devices. In this paper, we evaluate mobile autofill frameworks on iOS and Android, examining whether they achieve substantive benefits over the ad-hoc desktop environment or become a problematic single point of failure. Our results find that while the frameworks address several common issues, they also enforce insecure behavior and fail to provide password managers sufficient information to override the frameworks’ insecure behavior, resulting in mobile managers being less secure than their desktop counterparts overall. We also demonstrate how these frameworks act as a confused deputy in manager-assisted credential phishing attacks. Our results demonstrate the need for significant improvements to mobile autofill frameworks. We conclude the paper with recommendations for the design and implementation of secure autofill frameworks.

Despite efforts to replace them, passwords remain the primary form of authentication on the web. Password managers seek to address many of the problems with passwords by helping users generate, store, and fill strong and unique passwords. Even though experts frequently recommend password managers, there is limited information regarding their usability. To aid in designing such usability studies, we systematize password manager use cases, identifying ten essential use cases, three recommended use cases, and four extended use cases. We also systematize the system designs employed to satisfy these use cases, designs that should be examined in usability studies to understand their relative strengths and weaknesses. Finally, we describe observations from 136 cognitive walkthroughs exploring the identified essential use cases in eight popular managers. Ultimately, we expect that this work will serve as the foundation for an explosion of new research into the usability of password managers.

Secure messaging tools are an integral part of modern society. While there is a significant body of secure messaging research generally, there is a lack of information regarding users’ security and privacy perceptions and requirements for secure group chat. To address this gap, we conducted a survey of 996 respondents in the US and UK. The results of our study show that group chat presents important security and privacy challenges, some of which are not present in one-to-one chat. For example, users need to be able to manage and monitor group membership, establish trust for new group members, and filter content that they share in different chat contexts. Similarly, we find that the sheer volume of notifications that occur in group chat makes it extremely likely that users ignore important security or privacy notifications. We also find that respondents lack mechanisms for determining which tools are secure and instead rely on non-technical strategies for protecting their privacy—for example, self-filtering what they post and carefully tracking group membership. Based on these findings, we provide recommendations on how to improve the security and usability of secure group chat.