The rise of AI-powered tools like ChatGPT enables non-programmers to bypass programming screening questions, undermining internal validity in usable security and privacy, and software engineering studies. Past ChatGPT-resistant tasks proposed static visual questions, which ChatGPT can now circumvent. Therefore, we tested alternative approaches such as video- and audio-based screeners that reveal key information step by step under strict time constraints to distinguish programmers from non-programmers. To this end, we conducted a study with 74 participants across three groups: programmers, non-programmers without AI assistance, and non-programmers using ChatGPT. Our results showed that audio-based screeners were robust against ChatGPT-based cheating, as non-programmers struggled to find correct answers within time limits, whereas programmers demonstrated high accuracy with minimal time pressure. Based on our findings, we recommend six audio-based ChatGPT-resistant screening questions that maximize screening effectiveness and efficiency and suggest a 215-second instrument that includes 95.87% of programmers while excluding 99.69% of non-programmers.

Past research showed that software developers often require explicit instructions to implement security measures. With the rapid rise of AI assistant tools such as ChatGPT, it remains unclear whether AI assistance supports or undermines secure practices, whether explicit security instructions are still essential, and how developers behave without guidance. To investigate these research questions, we conducted a qualitative lab study with 21 computer science students and a quantitative online study with 80 freelance developers. We focused on secure password storage and asked participants to implement registration logic under four conditions: without instructions, with AI assistance, with security instructions, or with both AI assistance and security instructions. Our study reveals a clear behavioral shift: In our task, many participants relied on AI-assisted code generation for security-related tasks, often prioritizing convenience over security. However, explicit security-focused instructions can redirect this behavior toward secure outcomes, demonstrating that AI tools alone are insufficient without targeted guidance.

Effective communication is crucial for meeting security needs, yet gender-related communication challenges faced by women security experts within software development remain underexplored.In an interview study with 25 women security experts, we investigated gender-related communication challenges hindering the adoption of security requirements, and strategies to overcome these.Key challenges included the undervaluation of women's security expertise, communication barriers, resistance to women's security-related suggestions, and instances of hostility.Communication challenges with stakeholders who were men disrupted team collaboration, resulting in delays, weakened security measures, and increased organizational risk.Consequently, women security experts often had to adopt strategies, such as leveraging allied men and overpreparing, to assert their security competence.We further offer insights into women's participation in security studies.Based on our findings, we provide recommendations on how to address gender-related challenges.

The high number of cyber threats poses significant challenges, with impactful software exploits ranging from data theft to ransomware deployment. Unfortunately, past research highlighted limited security expertise within development teams. Collaboration between developers and security experts, therefore, emerges as one of the few workable means to address this gap. In this paper, we explore the complex interplay between developers and security experts within Scrum, one of the most widely adopted frameworks which actively promotes collaboration, to shed light on their working relationship, challenges, and potential avenues for improvement. To this end, we conducted a qualitative interview study with 14 developers and 13 security experts. Our qualitative results reveal three communication patterns and five shared challenges between the groups affecting the develop-security expert collaboration. Top challenges include consistent interaction difficulties and the lack of workable means to balance business and security needs. As a result, we found that three core Scrum values (openness, respect, courage) are missing from this relationship. Based on our results, we propose recommendations for fostering a healthy collaboration between developers and security experts, both within and beyond Scrum.

While protecting user data is essential, software developers often fail to fulfill privacy requirements. However, the reasons why they struggle with privacy-compliant implementation remain unclear. Is it due to a lack of knowledge, or is it because of insufficient support? To provide foundational insights in this field, we conducted a qualitative 5-hour programming study with 30 professional software developers implementing 3 privacy-sensitive programming tasks that were designed with GDPR compliance in mind. To explore if and how developers implement privacy requirements, participants were divided into 3 groups: control, privacy prompted, and privacy expert-supported. After task completion, we conducted follow-up interviews. Alarmingly, almost all participants submitted non-GDPR-compliant solutions (79/90). In particular, none of the 3 tasks were solved privacy-compliant by all 30 participants, with the non-prompted group having the lowest number of 3 out of 30 privacy-compliant solution attempts. Privacy prompting and expert support only slightly improved participants' submissions, with 6/30 and 8/30 privacy-compliant attempts, respectively. In fact, all participants reported severe issues addressing common privacy requirements such as purpose limitation, user consent, or data minimization. Counterintuitively, although most developers exhibited minimal confidence in their solutions, they rarely sought online assistance or contacted the privacy expert, with only 4 out of 10 expert-supported participants explicitly asking for compliance confirmation. Instead, participants often relied on existing implementations and focused on implementing functionality and security first.

Abstract Security must be considered in almost every software system. Unfortunately, selecting and implementing security features remains a challenge due to the wide variety of security threats and possible countermeasures. While security standards are intended to help developers, they are usually too abstract and vague to help implementing security features, or they merely help configuring such. A resource that describes security features at an abstraction level that lies between high-level (i.e., rather too general) and low-level (i.e., rather too specific) security standards could facilitate secure systems development. This resource should support the selection of appropriate security features to achieve high-level security goals, allow easy retrieval of relevant low-level details, and provide pointers to suitable ways to realize the security features. To realize security features, developers typically use external security libraries or frameworks, to minimize implementation mistakes. Even when using libraries, developers still make mistakes when writing code to integrate them, often resulting in security vulnerabilities. When security incidents occur or the system needs to be audited or maintained, it is essential to know what security features have been implemented and, more importantly, where they are located. This task, commonly referred to as feature location, is often tedious and error-prone. While dedicated feature location techniques exist, they require significant manual effort or adherence to strict development processes, preventing their use. Therefore, we have to support long-term tracking of implemented security features. We present a study of security features presented in the literature and their coverage in popular security frameworks. We contribute (1) a taxonomy of 68 functional implementation-level security features including a mapping to widely used security standards, (2) an examination of 21 popular security frameworks concerning which of these security features they provide, and (3) a discussion on the representation of security features in source code. Our taxonomy aims to aid developers in selecting appropriate security features and security frameworks, as well as relating them to security standards when they need to choose and implement security features for a software system.

Solana gained considerable attention as one of the most popular blockchain platforms for deploying decentralized applications. Compared to Ethereum, however, we observe a lack of research on how Solana smart contract developers handle security, what challenges they encounter, and how this affects the overall security of the ecosystem. To address this, we conducted the first comprehensive study on the Solana platform consisting of a 90-minute Solana smart contract code review task with 35 participants followed by interviews with a subset of seven participants. Our study shows, quite alarmingly, that none of the participants could detect all important security vulnerabilities in a code review task and that 83% of the participants are likely to release vulnerable smart contracts. Our study also sheds light on the root causes of developers' challenges with Solana smart contract development, suggesting the need for better security guidance and resources. In spite of these challenges, our automated analysis on currently deployed Solana smart contracts surprisingly suggests that the prevalence of vulnerabilities - especially those pointed out as the most challenging in our developer study - is below 0.3%. We explore the causes of this counter-intuitive resilience and show that frameworks, such as Anchor, are aiding Solana developers in deploying secure contracts.

Following the recent release of AI assistants, such as OpenAI's ChatGPT and GitHub Copilot, the software industry quickly utilized these tools for software development tasks, e.g., generating code or consulting AI for advice. While recent research has demonstrated that AI-generated code can contain security issues, how software professionals balance AI assistant usage and security remains unclear. This paper investigates how software professionals use AI assistants in secure software development, what security implications and considerations arise, and what impact they foresee on secure software development. We conducted 27 semi-structured interviews with software professionals, including software engineers, team leads, and security testers. We also reviewed 190 relevant Reddit posts and comments to gain insights into the current discourse surrounding AI assistants for software development. Our analysis of the interviews and Reddit posts finds that despite many security and quality concerns, participants widely use AI assistants for security-critical tasks, e.g., code generation, threat modeling, and vulnerability detection. Their overall mistrust leads to checking AI suggestions in similar ways to human code, although they expect improvements and, therefore, a heavier use for security tasks in the future. We conclude with recommendations for software professionals to critically check AI suggestions, AI creators to improve suggestion security and capabilities for ethical security tasks, and academic researchers to consider general-purpose AI in software development.

To ensure the validity of software engineering and IT security studies with professional programmers, it is essential to identify participants without programming skills. Existing screening questions are efficient, cheating robust, and effectively differentiate programmers from non-programmers. However, the release of ChatGPT raises concerns about their continued effectiveness in identifying non-programmers. In a simulated attack, we showed that Chat-GPT can easily solve existing screening questions. Therefore, we designed new ChatGPT-resistant screening questions using visual concepts and code comprehension tasks. We evaluated 28 screening questions in an online study with 121 participants involving programmers and non-programmers. Our results showed that questions using visualizations of well-known programming concepts performed best in differentiating between programmers and non-programmers. Participants prompted to use ChatGPT struggled to solve the tasks. They considered ChatGPT ineffective and changed their strategy after a few screening questions. In total, we present six ChatGPT-resistant screening questions that effectively identify non-programmers. We provide recommendations on setting up a ChatGPT-resistant screening instrument that takes less than three minutes to complete by excluding 99.47% of non-programmers while including 94.83% of programmers.

To ensure data-privacy compliance, it is common for companies to consult privacy experts for the identification and communication of privacy requirements to software developers. However, developers often fail to fulfill those requirements resulting in companies regularly being fined for violations due to non-compliance with privacy data regulations. To investigate why software developers struggle with the implementation of privacy requirements and explore their communication modality, we conducted a qualitative semi-structured interview study with 30 participants involving 10 software developers, 10 privacy experts, and 10 team coordinators with an average experience of nine years in the privacy communication and implementation process within a company context. We found a communication gap between software developers and privacy experts, suggesting a lack of proper procedural steps during the software development process to guarantee that the privacy requirements have been adequately addressed. We also uncovered that since privacy requirements were mostly communicated in a uni-directional manner, they were often perceived as a hindrance during software development, thus fostering an adversarial relationship between privacy experts and developers. Therefore, in order to fulfill the experts' requirements, software developers requested concrete steps to take during the software development process, as observed in the security field. However, privacy experts often lacked the technical knowledge to provide such instructions. This work contributes an explanatory theory on the communication gap between software developers and privacy experts. We discuss common obstacles in the communication of privacy experts and software developers and provide guidance on how to address them.

Recruiting study participants with programming skill is essential for researchers. As programming is not a common skill, recruiting programmers as participants in large numbers is challenging. Platforms like Amazon MTurk or Qualtrics offer to recruit participants with programming knowledge. As this is self-reported, participants without programming experience could still take part, either due to a misunderstanding or to obtain the study compensation. If these participants are not detected, the data quality will suffer. To tackle this, Danilova et al. [11] developed and tested screening tasks to detect non-programmers. Unfortunately, the most reliable screeners were also those that took the most time. Since screeners should take as little time as possible, we examine whether the introduction of time limits allows us to create more efficient (i.e., quicker but still reliable) screeners. Our results show that this is possible and we extend the pool of screeners and make recommendations on how to improve the process.

Recruiting professional programmers in sufficient numbers for research studies can be challenging because they often cannot spare the time, or due to their geographical distribution and potentially the cost involved. Online platforms such as Clickworker or Qualtrics do provide options to recruit participants with programming skill; however, misunderstandings and fraud can be an issue. This can result in participants without programming skill taking part in studies and surveys. If these participants are not detected, they can cause detrimental noise in the survey data. In this paper, we develop screener questions that are easy and quick to answer for people with programming skill but difficult to answer correctly for those without. In order to evaluate our questionnaire for efficacy and efficiency, we recruited several batches of participants with and without programming skill and tested the questions. In our batch 42% of Clickworkers stating that they have programming skill did not meet our criteria and we would recommend filtering these from studies. We also evaluated the questions in an adversarial setting. We conclude with a set of recommended questions which researchers can use to recruit participants with programming skill from online platforms.

Ecological validity is a major concern in usable security studies with developers. Many studies are conducted with computer science (CS) students out of convenience, since recruiting professional software developers in sufficient numbers is very challenging. In a password-storage study, Naiakshina et al. (CHI'19) showed that CS students behave similarly to freelance developers recruited online. While this is a promising result for conducting developer studies with students, an open question remains: Do professional developers employed in companies behave similarly as well? To provide more insight into the ecological validity of recruiting students for security developer studies, we replicated the study of Naiakshina et al. with developers from diverse companies in Germany. We found that developers employed in companies performed better than students and freelancers in a direct comparison. However, treatment effects were found to be significant in all groups; the treatment effects on CS students also held for company developers.

A wide range of tools exist to assist developers in creating secure software. Many of these tools, such as static analysis engines or security checkers included in compilers, use warnings to communicate security issues to developers. The effectiveness of these tools relies on developers heeding these warnings, and there are many ways in which these warnings could be displayed. Johnson et al. [46] conducted qualitative research and found that warning presentation and integration are main issues. We built on Johnson et al.'s work and examined what developers want from security warnings, including what form they should take and how they should integrate into their workflow and work context. To this end, we conducted a Grounded Theory study with 14 professional software developers and 12 computer science students as well as a focus group with 7 academic researchers to gather qualitative insights. To back up the theory developed from the qualitative research, we ran a quantitative survey with 50 professional software developers. Our results show that there is significant heterogeneity amongst developers and that no one warning type is preferred over all others. The context in which the warnings are shown is also highly relevant, indicating that it is likely to be beneficial if IDEs and other development tools become more flexible in their warning interactions with developers. Based on our findings, we provide concrete recommendations for both future research as well as how IDEs and other security tools can improve their interaction with developers.

In 2017 and 2018, Naiakshina et al. (CCS'17, SOUPS'18) studied in a lab setting whether computer science students need to be told to write code that stores passwords securely. The authors' results showed that, without explicit prompting, none of the students implemented secure password storage. When asked about this oversight, a common answer was that they would have implemented secure storage - if they were creating code for a company. To shed light on this possible confusion, we conducted a mixed-methods field study with developers. We hired freelance developers online and gave them a similar password storage task followed by a questionnaire to gain additional insights into their work. From our research, we offer two contributions. First of all, we reveal that, similar to the students, freelancers do not store passwords securely unless prompted, they have misconceptions about secure password storage, and they use outdated methods. Secondly, we discuss the methodological implications of using freelancers and students in developer studies.

With WhatsApp's adoption of the Signal Protocol as its default, end-to-end encryption by the masses happened almost overnight. Unlike iMessage, WhatsApp notifies users that encryption is enabled, explicitly informing users about improved privacy. This rare feature gives us an opportunity to study people's understandings and perceptions of secure messaging pre-and post-mass messenger encryption (pre/post-MME). To study changes in perceptions, we compared the results of two mental models studies: one conducted in 2015 pre-MME and one in 2017 post-MME. Our primary finding is that users do not trust encryption as currently offered. When asked about encryption in the study, most stated that they had heard of encryption, but only a few understood the implications, even on a high level. Their consensus view was that no technical solution to stop skilled attackers from getting their data exists. Even with a major development, such as WhatsApp rolling out end-to-end encryption, people still do not feel well protected by their technology. Surprisingly, despite WhatsApp's end-to-end security info messages and the high media attention, the majority of the participants were not even aware of encryption. Most participants had an almost correct threat model, but don't believe that there is a technical solution to stop knowledgeable attackers to read their messages. Using technology made them feel vulnerable.

Passwords are still a mainstay of various security systems, as well as the cause of many usability issues. For end-users, many of these issues have been studied extensively, highlighting problems and informing design decisions for better policies and motivating research into alternatives. However, end-users are not the only ones who have usability problems with passwords! Developers who are tasked with writing the code by which passwords are stored must do so securely. Yet history has shown that this complex task often fails due to human error with catastrophic results. While an end-user who selects a bad password can have dire consequences, the consequences of a developer who forgets to hash and salt a password database can lead to far larger problems. In this paper we present a first qualitative usability study with 20 computer science students to discover how developers deal with password storage and to inform research into aiding developers in the creation of secure password systems.

The computer security community has advocated widespread adoption of secure communication tools to counter mass surveillance. Several popular personal communication tools (e.g., WhatsApp, iMessage) have adopted end-to-end encryption, and many new tools (e.g., Signal, Telegram) have been launched with security as a key selling point. However it remains unclear if users understand what protection these tools offer, and if they value that protection. In this study, we interviewed 60 participants about their experience with different communication tools and their perceptions of the tools' security properties. We found that the adoption of secure communication tools is hindered by fragmented user bases and incompatible tools. Furthermore, the vast majority of participants did not understand the essential concept of end-to-end encryption, limiting their motivation to adopt secure tools. We identified a number of incorrect mental models that underpinned participants' beliefs.