Large language models (LLMs) are prone to hallucinations and sensitive to prompt perturbations, often resulting in inconsistent or unreliable generated text.Different methods have been proposed to mitigate such hallucinations and fragility, one of which is to measure the consistency of LLM responses-the model's confidence in the response or likelihood of generating a similar response when resampled.In previous work, measuring LLM response consistency often relied on calculating the probability of a response appearing within a pool of resampled responses, analyzing internal states, or evaluating logits of resopnses.However, it was not clear how well these approaches approximated users' perceptions of consistency of LLM responses.To find out, we performed a user study (n = 2, 976) demonstrating that current methods for measuring LLM response consistency typically do not align well with humans' perceptions of LLM consistency.We propose a logit-based ensemble method for estimating LLM consistency and show that our method matches the performance of the bestperforming existing metric in estimating human ratings of LLM consistency.Our results suggest that methods for estimating LLM consistency without human evaluation are sufficiently imperfect to warrant broader use of evaluation with human input; this would avoid misjudging the adequacy of models because of the imperfections of automated consistency metrics.

Online targeted ads are those shown only to certain users based on interests, demographics, or behaviors. Because targeted ads raise many privacy concerns, many platforms provide ad transparency systems (ATSs) to inform users about this practice. To better understand what current ATSs are communicating to users—and how—we first taxonomized the design and content of 22 of the most popular English-language websites' ATSs as presented to users in the United States. We found substantial differences across ATSs in both the prevalence of transparency-enhancing features (e.g., whether they show users what has been inferred about them) and the presentation of information (e.g., the terminology used, where settings are located). Across all platforms, however, we observed consistent ambiguity about what data is used to target ads and the actual impact of altering settings. To gauge how these different design choices impact users, we conducted an online user study in which 198 participants used their own account to explore the ATS of one of eight representative platforms. We found that many of the questions participants hoped the ATS would answer remained unanswered after exploring the ATS. More broadly, participants found current ATSs simultaneously complex and lacking key details. We pinpoint ATS design decisions that best support users.

Cryptographers have long been concerned with secure messaging protocols threatening deniability. Many messaging protocols—including, surprisingly, modern email— contain digital signatures which definitively tie the author to their message. If stolen or leaked, these signatures make it impossible to deny authorship. As illustrated by events surrounding leaks from Hilary Clinton’s 2016 U.S. presidential campaign, this concern has proven well founded. Deniable protocols are meant to avoid this very outcome, letting politicians and dissidents alike safely disavow authorship. Despite being deployed on billions of devices in Signal and WhatsApp, the effectiveness of such protocols in convincing people remains unstudied. While the absence of cryptographic evidence is clearly necessary for an effective denial, is it sufficientƒWe conduct a survey study (n = 1, 200) to understand how people perceive evidence of deniability related to encrypted messaging protocols. Surprisingly, in a world of "fake news" and Photoshop, we find that simple denials of message authorship, when presented in a courtroom setting without supporting evidence, are not effective. In contrast, participants who were given access to a screenshot forgery tool or even told one exists were much more likely to believe a denial. Similarly, but to a lesser degree, we find an expert cryptographer’s assertion that there is no evidence is also effective.

One widespread, but frequently overlooked, source of security information is influencer marketing ads on YouTube for security and privacy products such as VPNs. This paper examines how widespread these ads are, where on YouTube they are found, and what kind of information they convey. Starting from a random sample of 1.4% of YouTube, we identify 243 videos containing VPN ads with a total of 63 million views. Using qualitative analysis, we find that these ads commonly discuss broad security guarantees as well as specific technical features, frequently focus on internet threats, and sometimes emphasize accessing otherwise unavailable content. Different VPN companies tend to advertise in different categories of channels and emphasize different messages. We find a number of potentially misleading claims, including overpromises and exaggerations that could negatively influence viewers’ mental models of internet safety.

At-risk users are people who experience risk factors that augment or amplify their chances of being digitally attacked and/or suffering disproportionate harms. In this systematization work, we present a framework for reasoning about at-risk users based on a wide-ranging meta-analysis of 95 papers. Across the varied populations that we examined (e.g., children, activists, people with disabilities), we identified 10 unifying contextual risk factors —such as marginalization and access to a sensitive resource —that augment or amplify digital-safety risks and their resulting harms. We also identified technical and non-technical practices that at-risk users adopt to attempt to protect themselves from digital-safety risks. We use this framework to discuss barriers that limit at-risk users’ ability or willingness to take protective actions. We believe that researchers and technology creators can use our framework to identify and shape research investments to benefit at-risk users, and to guide technology design to better support at-risk users.