The research literature on employee non-/compliance with information security policies (ISPs) suffers from inconsistent findings. The aim of this paper is to investigate a promising, yet largely unexplored cause of these inconsistencies, namely, differences in how key variables have been conceptualized and operationalized into questionnaire measurement items across this literature. Specifically, by means of a survey experiment with 215 participants from a Swedish university, we formally tested whether alternative operationalizations of two key variables from Protection Motivation Theory—Perceived vulnerability and Perceived severity of a threat—lead to statistically significant differences in mean values across responder groups. Regarding the former variable, we found significant differences between operationalizations focusing on the probability and vulnerability of the threat. Regarding the latter, we found significant differences between whether the target of the consequences of the threat was unclear or clear to respondents, but not between clearly stated targets in terms of individuals and organizations . Overall, therefore, this study contributes to the field by highlighting the importance of conceptual clarity and precision in measuring key variables. It also highlights the potential of survey experiments—an underutilized method in ISP compliance research—for exploring the empirical impact of the different variable operationalizations which currently characterize much of the extant literature.

The rich stream of research focusing on employee non-/compliance with information security policies (ISPs) suffers from inconsistent results. Attempts to explain such inconsistencies have included investigation of possible contextual moderating factors. Another promising, yet not systematically investigated, explanation concerns conceptual inconsistencies in variable definitions and in questionnaire measurement items. Based on a systematic literature review covering 36 ISP non-/compliance articles using Protection Motivation Theory (PMT) and/or Theory of Planned Behavior (TPB), we found four major types of conceptual inconsistencies and unclarities within and across studies; (i) inconsistencies in variable definitions; (ii) inconsistencies between variable measurement items; (iii) inconsistencies between variable definitions and measurement items; and (iv) unclearly/vaguely worded measurement items. The review contributes to the field by demonstrating that the inconsistent results in the field may not only be due to unknown contextual moderators, but also to conceptual incongruences within and across studies.

Purpose Research on employee non-/compliance to information security policies suffers from inconsistent results and there is an ongoing discussion about the dominating survey research methodology and its potential effect on these results. This study aims to add to this discussion by investigating discrepancies between what the authors claim to measure (theoretical properties of variables) and what they actually measure (respondents’ interpretations of the operationalized variables). This study asks: How well do respondents’ interpretations of variables correspond to their theoretical definitions? What are the characteristics of any discrepancies between variable definitions and respondent interpretations? Design/methodology/approach This study is based on in-depth interviews with 17 respondents from the Swedish public sector to understand how they interpret questionnaire measurement items operationalizing the variables Perceived Severity from Protection Motivation Theory and Attitude from Theory of Planned Behavior. Findings The authors found that respondents’ interpretations in many cases differ substantially from the theoretical definitions. Overall, the authors found four principal ways in which respondents interpreted measurement items – referred to as property contextualization, extension, alteration and oscillation – each implying more or less (dis)alignment with the intended theoretical properties of the two variables examined. Originality/value The qualitative method used proved vital to better understand respondents’ interpretations which, in turn, is key for improving self-reporting measurement instruments. To the best of the authors’ knowledge, this study is a first step toward understanding how precise and uniform definitions of variables’ theoretical properties can be operationalized into effective measurement items.