This paper presents an empirical analysis of antimalware evasion techniques. A total of 29504 tests generated by 16 antimalware evasion tools were run against 100 machines protected by either Microsoft Windows Defender or Symantec Endpoint Protection. The configuration of each test was mapped to a categorization framework with 11 evasion techniques. Out of the executed tests, 54% evaded the antimalware scans and 5% provided interactive shells (i.e., remote control of victims). Out of the studied evasion techniques, using a packer, encryption (AES or RC4), or storing the payload loader in a common data format (as compared to a script or executable) had the greatest impact on evading antimalware scans, while application of custom encryption and code injection had the greatest impact on gaining shells. The results also showed that several evasion techniques curiously increased the likelihood of detection.

This article presents the red team emulation tool Lore, which uses boolean logic and trained models to automatically select and execute red team actions. Lore improves the current state of red team automation, and is the first such tool shown to provide a more fun and educational experience than a manual red team during a cyber defence exercise. In addition to the cyber defence exercise, empirical tests are performed to examine the accuracy of Lore's trained models. The results show that application of these models lead to two times more compromised machines than when applying expert-defined models, and five times more compromised machines than when randomly selecting actions.

Purpose It is often argued that the increased automation and availability of offensive cyber tools has decreased the skill and knowledge required by attackers. Some say that all it takes to succeed with an attack is to follow some instructions and push some buttons. This paper aims to tests this idea empirically through live exploits and vulnerable machines in a cyber range. Design/methodology/approach The experiment involved 204 vulnerable machines in a cyber range. Exploits were chosen based on the results of automated vulnerability scanning. Each exploit was executed following a set of carefully planned actions that enabled reliable tests. A total of 1,223 exploitation attempts were performed. Findings A mere eight exploitation attempts succeeded. All these involved the same exploit module (ms08_067_netapi). It is concluded that server-side attacks still are too complicated for novices who lack the skill or knowledge to tune their attacks. Originality/value This paper presents the largest conducted test of exploit effectiveness to date. It also presents a sound method for reliable tests of exploit effectiveness (or system vulnerability).

Requirements engineering is a vibrant and broad research area. It covers a range of activities with different objectives. By reviewing experiments previously included in systematic literature reviews, this paper provides an overview of the dependent variables used in experimental requirements engineering research. This paper also identifies the theoretical motivation for the use of these variables in the experiments. The results show that a wide range of different variables has been applied in experiments and operationalized through both subjective assessments (e.g., subjects' perceived utility of a technique) and objective measurements (e.g., the number of defects found in a requirements specification). The theoretical basis for these variables and operationalizations are unclear in most cases. Directions for theoretical work to identify suitable dependent variables are provided.

Purpose – The purpose of the study was twofold: to investigate the correlation between a sample of personal psychological and demographic factors and resistance to phishing; and to investigate if national culture moderates the strength of these correlations. Design/methodology/approach – To measure potential determinants, a survey was distributed to 2,099 employees of nine organizations in Sweden, USA and India. Then, the authors conducted unannounced phishing exercises, in which a phishing attack targeted the same sample. Findings – Intention to resist social engineering, general information security awareness, formal IS training and computer experience were identified to have a positive significant correlation to phishing resilience. Furthermore, the results showed that the correlation between phishing determinants and employees’ observed that phishing behavior differs between Swedish, US and Indian employees in 6 out of 15 cases. Research limitations/implications – The identified determinants had, even though not strong, a significant positive correlation. This suggests that more work needs to be done to more fully understand determinants of phishing. The study assumes that culture effects apply to all individuals in a nation. However, differences based on cultures might exist based on firm characteristics within a country. The Swedish sample is dominating, while only 40 responses from Indian employees were collected. This unequal size of samples suggests that conclusions based on the results from the cultural analysis should be drawn cautiously. A natural continuation of the research is therefore to further explore the generalizability of the findings by collecting data from other nations with similar cultures as Sweden, USA and India. Originality/value – Using direct observations of employees’ security behaviors has rarely been used in previous research. Furthermore, analyzing potential differences in theoretical models based on national culture is an understudied topic in the behavioral information security field. This paper addresses both these issues.

This paper presents the Predictive, Probabilistic Cyber Security Modeling Language (P2CySeMoL), an attack graph tool that can be used to estimate the cyber security of enterprise architectures. P2CySeMoL includes theory on how attacks and defenses relate quantitatively; thus, users must only model their assets and how these are connected in order to enable calculations. The performance of P2CySeMoL enables quick calculations of large object models. It has been validated on both a component level and a system level using literature, domain experts, surveys, observations, experiments and case studies.

Abstract In situations when data collection through observations is difficult to perform, the use of expert judgement can be justified. A challenge with this approach is, however, to value the credibility of different experts. A natural and state‐of‐the art approach is to weight the experts' judgements according to their calibration, that is, on the basis of how well their estimates of a studied event agree with actual observations of that event. However, when data collection through observations is difficult to perform, it is often also difficult to estimate the calibration of experts. As a consequence, variables thought to indicate calibration are generally used as a substitute of it in practice. This study evaluates the value of three such indicative variables: consensus, experience and self‐proclamation. The significances of these variables are analysed in four surveys covering different domains in cyber security, involving a total of 271 subjects. Results show that consensus is a reasonable indicator of calibration. The mean Pearson correlation between these two variables across the four studies was 0.407. No significant correlations were found between calibration and experience or calibration and self‐proclamation. However, as a side result, it was discovered that a subject that perceives itself as more knowledgeable than others likely also is more experienced.

Purpose – The purpose of the study was threefold: to understand security behaviours in practice by investigating factors that may cause an individual to comply with a request posed by a perpetrator; to investigate if adding information about the victim to an attack increases the probability of the attack being successful; and, finally, to investigate if there is a correlation between self-reported and observed behaviour. Design/methodology/approach – Factors for investigation were identified based on a review of existing literature. Data were collected through a scenario-based survey, phishing experiments, journals and follow-up interviews in three organisations. Findings – The results from the experiment revealed that the degree of target information in an attack increased the likelihood that an organisational employee falls victim to an actual attack. Further, an individual’s trust and risk behaviour significantly affected the actual behaviour during the phishing experiment. Computer experience at work, helpfulness and gender (females tend to be less susceptible to a generic attack than men), had a significant correlation with behaviour reported by respondents in the scenario-based survey. No correlation between the results from the scenario-based survey and the experiments was found. Research limitations/implications – One limitation is that the scenario-based survey may have been interpreted differently by the participants. Another is that controlling how the participants reacted when receiving the phishing mail, and what actually triggered each and every participant to click on the attached link, was not possible. Data were however collected to capture these aspects during and after the experiments. In conclusion, the results do not imply that one or the other method should be ruled out, as they have both advantages and disadvantages which should be considered in the context of collecting data in the critical domain of information security. Originality/value – Two different methods to collect data to understand security behaviours have rarely been used in previous research. Studies that add target information to understand if such information could increase the probability of attack success is sparse. This paper includes both approaches.

A frequent assumption in the domain of cybersecurity is that cyberintrusions follow the properties of a Poisson process, i.e., that the number of intrusions is well modeled by a Poisson distribution and that the time between intrusions is exponentially distributed. This paper studies this property by analyzing all cyberintrusions that have been detected across more than 260,000 computer systems over a period of almost three years. The results show that the assumption of a Poisson process model might be unoptimal - the log-normal distribution is a significantly better fit in terms of modeling both the number of detected intrusions and the time between intrusions, and the Pareto distribution is a significantly better fit in terms of modeling the time to first intrusion. The paper also analyzes whether time to compromise (TTC) increase for each successful intrusion of a computer system. The results regarding this property suggest that time to compromise decrease along the number of intrusions of a system.

The context of this article is the availability of enterprise IT services, a key concern for many enterprises. While there is a plethora of literature concerned with service availability, there is no previous systematic empirical study on IT service time to recovery following outages. The existing literature typically assumes a distribution, or builds on analogies to related areas such as software engineering. Therefore, our objective is to find the statistical distribution of IT service time to recovery. Method-wise, this investigation is based on logs of more than 1800 incidents in a large Nordic bank, corresponding to more than 11000 hours of recorded downtime. Five possible distributions of time to recovery from the literature were investigated using the Akaike Information Criterion to find the distribution offering the best fit. The results show that the log-normal distribution outperformed the others for all tested service channels (collections of IT services). It is concluded that the log-normal distribution offers the best fit of IT service time to recovery. Using this distribution in simulation and decision-support tools offers the prospect of better predictions of downtime and downtime costs to the practitioner community.

Purpose – The purpose of this paper is to estimate the effectiveness of web application firewalls (WAFs) at preventing injection attacks by professional penetration testers given presence or absence of four conditions: whether there is an experienced operator monitoring the WAF; whether an automated black box tool has been used when tuning the WAF; whether the individual tuning the WAF is an experienced professional; and whether significant effort has been spent tuning the WAF. Design/methodology/approach – Estimates on the effectiveness of WAFs are made for 16 operational scenarios utilizing judgments by 49 domain experts participating in a web survey. The judgments of these experts are pooled using Cooke's classical method. Findings – The results show that the median prevention rate of a WAF is 80 percent if all measures have been employed. If no measure is employed then its median prevention rate is 25 percent. Also, there are no strong dependencies between any of the studied measures. Research limitations/implications – The results are only valid for the attacker profile of a professional penetration tester who prepares one week for attacking a WA protected by a WAF. Practical implications – The competence of the individual(s) tuning a WAF, employment of an automated black box tool for tuning and the manual effort spent on tuning are of great importance for the effectiveness of a WAF. The presence of an operator monitoring it has minor positive influence on its effectiveness. Originality/value – WA vulnerabilities are widely considered a serious concern. To manage them in deployed software, many enterprises employ WAFs. However, the effectiveness of this type of countermeasure under different operational scenarios is largely unknown.

Purpose The purpose of this paper is to identify the importance of the factors that influence the success rate of remote arbitrary code execution attacks. In other words, attacks which use software vulnerabilities to execute the attacker's own code on targeted machines. Both attacks against servers and attacks against clients are studied. Design/methodology/approach The success rates of attacks are assessed for 24 scenarios: 16 scenarios for server‐side attacks and eight for client‐side attacks. The assessment is made through domain experts and is synthesized using Cooke's classical method, an established method for weighting experts' judgments. The variables included in the study were selected based on the literature, a pilot study, and interviews with domain experts. Findings Depending on the scenario in question, the expected success rate varies between 15 and 67 percent for server‐side attacks and between 43 and 67 percent for client‐side attacks. Based on these scenarios, the influence of different protective measures is identified. Practical implications The results of this study offer guidance to decision makers on how to best secure their assets against remote code execution attacks. These results also indicate the overall risk posed by this type of attack. Originality/value Attacks that use software vulnerabilities to execute code on targeted machines are common and pose a serious risk to most enterprises. However, there are no quantitative data on how difficult such attacks are to execute or on how effective security measures are against them. The paper provides such data using a structured technique to combine expert judgments.

The Common Vulnerability Scoring System (CVSS) is a widely used and well-established standard for classifying the severity of security vulnerabilities. For instance, all vulnerabilities in the US National Vulnerability Database (NVD) are scored according to this method. As computer systems typically have multiple vulnerabilities, it is often desirable to aggregate the score of individual vulnerabilities to a system level. Several such metrics have been proposed, but their quality has not been studied. This paper presents a statistical analysis of how 18 security estimation metrics based on CVSS data correlate with the time-to-compromise of 34 successful attacks. The empirical data originates from an international cyber defense exercise involving over 100 participants and were collected by studying network traffic logs, attacker logs, observer logs, and network vulnerabilities. The results suggest that security modeling with CVSS data alone does not accurately portray the time-to-compromise of a system. However, results also show that metrics employing more CVSS data are more correlated with time-to-compromise. As a consequence, models that only use the weakest link (most severe vulnerability) to compose a metric are less promising than those that consider all vulnerabilities.

Denial-of-service (DoS) attacks are an imminent and real threat to many enterprises. Decision makers in these enterprises need be able to assess the risk associated with such attacks and to make decisions regarding measures to put in place to increase the security posture of their systems. Experiments, simulations and analytical research have produced data related to DoS attacks. However, these results have been produced for different environments and are difficult to interpret, compare, and aggregate for the purpose of decision making. This paper aims to summarize knowledge available in the field by synthesizing the judgment of 23 domain experts using an establishing method for expert judgment analysis. Different system architecture's vulnerability to DoS attacks are assessed together with the impact of a number of countermeasures against DoS attacks.

Purpose The purpose of this paper is to evaluate if automated vulnerability scanning accurately identifies vulnerabilities in computer networks and if this accuracy is contingent on the platforms used. Design/methodology/approach Both qualitative comparisons of functionality and quantitative comparisons of false positives and false negatives are made for seven different scanners. The quantitative assessment includes data from both authenticated and unauthenticated scans. Experiments were conducted on a computer network of 28 hosts with various operating systems, services and vulnerabilities. This network was set up by a team of security researchers and professionals. Findings The data collected in this study show that authenticated vulnerability scanning is usable. However, automated scanning is not able to accurately identify all vulnerabilities present in computer networks. Also, scans of hosts running Windows are more accurate than scans of hosts running Linux. Research limitations/implications This paper focuses on the direct output of automated scans with respect to the vulnerabilities they identify. Areas such as how to interpret the results assessed by each scanner (e.g. regarding remediation guidelines) or aggregating information about individual vulnerabilities into risk measures are out of scope. Practical implications This paper describes how well automated vulnerability scanners perform when it comes to identifying security issues in a network. The findings suggest that a vulnerability scanner is a useable tool to have in your security toolbox given that user credentials are available for the hosts in your network. Manual effort is however needed to complement automated scanning in order to get satisfactory accuracy regarding network security problems. Originality/value Previous studies have focused on the qualitative aspects on vulnerability assessment. This study presents a quantitative evaluation of seven of the most popular vulnerability scanners available on the market.

Purpose This paper aims to assess the influence of a set of human and organizational factors in information system deployments on the probability that a number of security‐related mistakes are in the deployment. Design/methodology/approach A Bayesian network (BN) is created and analyzed over the relationship between mistakes and causes. The BN is created by eliciting qualitative and quantitative data from experts of industrial control system deployments in the critical infrastructure domain. Findings The data collected in this study show that domain experts have a shared perception of how strong the influence of human and organizational factors are. According to domain experts, this influence is strong. This study also finds that security flaws are common in industrial control systems operating critical infrastructure. Research limitations/implications The model presented in this study is created with the help of a number of domain experts. While they agree on qualitative structure and quantitative parameters, future work should assure that their opinion is generally accurate. Practical implications The influence of a set of important variables related to organizational/human aspects on information security flaws is presented. Social implications The context of this study is deployments of systems that operate nations' critical infrastructure. The findings suggest that initiatives to secure such infrastructures should not be purely technical. Originality/value Previous studies have focused on either the causes of security flaws or the actual flaws that can exist in installed information systems. However, little research has been spent on the relationship between them. The model presented in this paper quantifies such relationships.