Many Security Operations Centers (SOCs) today still heavily rely on signature-based Network Intrusion Detection Systems (NIDS) such as Suricata. The specificity of intrusion detection rules and the coverage provided by rulesets are common concerns within the professional community surrounding SOCs, which impact the effectiveness of automated alert post-processing approaches. We postulate a better understanding of factors influencing the quality of rules can help address current SOC issues. In this paper, we characterize the rules in use at a collaborating commercial (managed) SOC serving customers in sectors including education and IT management. During this process, we discover six relevant design principles, which we consolidate through interviews with experienced rule designers at the SOC.We then validate our design principles by quantitatively assessing their effect on rule specificity. We find that several of these design considerations significantly impact unnecessary workload caused by rules. For instance, rules that leverage proxies for detection, and rules that do not employ alert throttling or do not distinguish (un)successful malicious actions, cause significantly more workload for SOC analysts. Moreover, rules that match a generalized characteristic to detect malicious behavior, which is believed to increase coverage, also significantly increase workload, suggesting a tradeoff must be struck between rule specificity and coverage. We show that these design principles can be applied successfully at a SOC to reduce workload whilst maintaining coverage despite the prevalence of violations of the principles.

Organizations deploy Intrusion Detection Systems (IDSs) like Suricata to defend against threats. Although rulesets, rules, and the resulting alerts have been studied previously, little is known about the process by which rules are engineered thus far. We aim to address the previously mentioned gaps by studying how network intrusion detection rules are derived from incidents. To this end, we propose the SuriCap measurement platform and organize Jeopardy-style workshops in which participants compete to engineer Suricata rules. We collect a rich dataset consisting of over 364 rules from 28 participants. Preliminary results suggest our experimental design is viable and, together with the SuriCap measurement platform, can enable us to answer several research questions surrounding the engineering process of network intrusion detection rules.

Modern organizations are persistently targeted by phishing emails. Despite advances in detection systems and widespread employee training, attackers continue to innovate, posing ongoing threats. Two emerging vectors stand out in the current landscape: QR-code baits and LLM-enabled pretexting. Yet, little is known about the effectiveness of current defenses against these attacks, particularly when it comes to real-world impact on employees. This gap leaves uncertainty around to what extent related countermeasures are justified or needed. Our work addresses this issue. We conduct three phishing simulations across organizations of varying sizes - from small-medium businesses to a multinational enterprise. In total, we send over 71k emails targeting employees, including: a "traditional"phishing email with a click-through button; a nearly-identical "quishing"email with a QR code instead; and a phishing email written with the assistance of an LLM and open-source intelligence. Our results show that quishing emails have the same effectiveness as traditional phishing emails at luring users to the landing webpage - which is worrying, given that quishing emails are much harder to identify even by operational detectors. We also find that LLMs can be very good "social engineers": in one company, over 30% of the emails opened led to visiting the landing webpage - a rate exceeding some prior benchmarks. Finally, we complement our study by conducting a survey across the organizations' employees, measuring their "perceived"phishing awareness. Our findings suggest a correlation between higher self-reported awareness and organizational resilience to phishing attempts.

The investigations run by tier 1 (T1) analysts in a Security Operation Center are critical to the SOC operations as they represent the first gateway to alert escalation and incident response. Critically, they demand an accurate and as-complete-as-possible understanding of the events surrounding the investigated alert. This is a complex task inexperienced T1 analysts can easily lose track of. In this work, we collaborate with a commercial SOC to develop an alert investigation support tool to help inexperienced analysts identify and collect all the information relevant to the investigation of an alert. We evaluate the prototype tool with two qualitative studies. The first study employs T1 analysts from the SOC to evaluate the conformity of the tool to the underpinning analysis process. The second study employs 57 students, recruited from the same pool where the SOC acquires its junior analysts from, to evaluate whether it helps inexperienced analysts develop a complete understanding of events surrounding security alert data. Our findings suggest that employing the tool helps inexperienced analysts form a more accurate understanding of attacks, at no time cost. We discuss the wider implications for research and practice.

The interdisciplinarity of the Social Engineering (SE) domain creates crucial challenges for the development and advancement of empirical SE research, making it particularly difficult to identify the space of open research questions that can be addressed empirically. This space encompasses questions on attack conditions, employed experimental methods, and interactions with underlying cognitive aspects. As a consequence, much potential in the breadth of existing empirical SE research and in its mapping to the actual cognitive processes it aims to measure is left untapped. In this work, we carry out a systematic review of 169 articles investigating overall 735 hypotheses in the field of empirical SE research, focusing on experimental characteristics and core cognitive features from both attacker and target perspectives. Our study reveals that experiments only partially reproduce real attacks and that the exploitable SE attack surface appears much larger than the coverage provided by the current body of research. Factors such as targets’ context and cognitive processes are often ignored or not explicitly considered in experimental designs. Similarly, the effects of different pretexts and varied targetization levels are overall marginally investigated. Our findings on current SE research dynamics provide insights into methodological shortcomings and help identify supplementary techniques that can open promising future research directions.

Phishing attacks are a main threat to organizations and individuals. Current widespread defenses based on spam filters and domain blacklisting are unfortunately insufficient. Prior work identifies phishing reporting as a key, largely untapped resource to mitigate phishing threats. Yet, its practice suffers from very low reporting rates and generally too low an uptake from users. Whereas it is known that phishing reporting behavior is affected by a number of ‘human factors’, a comprehensive view of the different theories and their effects on (intent to) report is not yet developed. To address this gap, we evaluate theories and factors analyzed in the extant literature, build a cohesive theoretical view of their effects and constructs, and develop, model, and empirically evaluate (by means of an online questionnaire, n=284) the resulting hypothesis structure. We discuss both theoretical implications of our findings and research directions for practice at a research and organizational level.

Besides coming with unprecedented benefits, the Internet of Things (IoT) suffers deficits in security measures, leading to attacks increasing every year. In particular, network environments such as smart homes lack managed security capabilities to detect IoT-related attacks; IoT devices hosted therein are thus more easily targeted by threats. As such, context awareness of IoT infections is hard to achieve, preventing prompt response. In this work, we propose MUDscope, an approach to monitor malicious network activities affecting IoT systems in real-world consumer environments. We leverage the recent Manufacturer Usage Description (MUD) specification, which defines networking allow-lists for IoT devices in MUD profiles, to reflect consistent and necessarily-anomalous activities from smart things. Our approach characterizes this traffic and extracts signatures for given attacks. By analyzing attack signatures for multiple devices, we gather insights into emerging attack patterns. We evaluate our approach on both an existing dataset and a new, openly available dataset created for this research. We show that MUDscope detects several attacks targeting IoT devices with an F1-score of 95.77% and correctly identifies signatures for specific attacks with an F1-score of 87.72%.

Phishing attacks arrive in high numbers and often spread quickly, meaning that after-the-fact countermeasures such as domain blacklisting are limited in efficacy. Visual similarity-based approaches have the potential of detecting previously unseen phishing webpages. These approaches, however, require identifying the legitimate webpage(s) they reproduce. Existing approaches rely on textual feature analysis for target identification, with misclassification rates of approximately 1%; however, as most websites a user might visit are legitimate, additional research is needed to further reduce classification errors. In this work, we propose a novel method for target identification that relies on both visual features (extracted from a screenshot of the web page) and textual features (extracted from the DOM of the web page) to identify which website a phishing web page is replicating, and assess its effectiveness in detecting phishing websites using data from phishing aggregators such as OpenPhish, PhishTank and PhishStats. Compared to state-of-the-art text-based classifiers, our method reduces the phishing misclassification rate by 67% (from 1.02% to 0.34%), for an accuracy of 99.66%. This work provides a further step forwards toward semi-automated decision support systems for phishing detection.

Organizations are experiencing more and more sophisticated attacks specifically targeting their employees and customers. These attacks exploit tailored information on the victim or organization to increase their credibility. To date, no study has evaluated the role of 'traditional' phishing cognitive effects in these advanced settings. In this paper, we run a field experiment targeting 747 subjects employed in two organizations (a university and a large international consultancy company) to evaluate the interaction between phishing persuasion techniques and the success rate in a highly-tailored setting. For this purpose, we exploit well-established user notification methods to devise enhanced attack delivery techniques, and evaluate how such techniques affect success rate of our phishing campaigns. We find that the effect of 'traditional' attack techniques is widely mitigated in highly-tailored phishing settings, suggesting that current user training and detection techniques may be off-target for more sophisticated attacks. However, we find that the means by which the attack is delivered to the victim matter, and can greatly (up to three times) boost the effect of the base attack.

In this paper we introduce SAIBERSOC, a tool and methodology enabling security researchers and operators to evaluate the performance of deployed and operational Security Operation Centers (SOCs) (or any other security monitoring infrastructure). The methodology relies on the MITRE ATT&CK Framework to define a procedure to generate and automatically inject synthetic attacks in an operational SOC to evaluate any output metric of interest (e.g., detection accuracy, time-to-investigation, etc.). To evaluate the effectiveness of the proposed methodology, we devise an experiment with n = 124 students playing the role of SOC analysts. The experiment relies on a real SOC infrastructure and assigns students to either a BADSOC or a GOODSOC experimental condition. Our results show that the proposed methodology is effective in identifying variations in SOC performance caused by (minimal) changes in SOC configuration. We release the SAIBERSOC tool implementation as free and open source software.

In this paper we provide evidence of an emerging criminal infrastructure enabling impersonation attacks at scale. Impersonation-as-a-Service (IMPaaS) allows attackers to systematically collect and enforce user profiles (consisting of user credentials, cookies, device and behavioural fingerprints, and other metadata) to circumvent risk-based authentication system and effectively bypass multi-factor authentication mechanisms. We present the IMPaaS model and evaluate its implementation by analysing the operation of a large, invite-only, Russian IMPaaS platform providing user profiles for more than 260,000 Internet users worldwide. Our findings suggest that the IMPaaS model is growing, and provides the mechanisms needed to systematically evade authentication controls across multiple platforms, while providing attackers with a reliable, up-to-date, and semi-automated environment enabling target selection and user impersonation against Internet users as scale.

Abstract Assessing the risks of software vulnerabilities is a key process of software development and security management. This assessment requires to consider multiple factors (technical features, operational environment, involved assets, status of the vulnerability lifecycle, etc.) and may depend from the assessor’s knowledge and skills. In this work, we tackle with an important part of this problem by measuring the accuracy oftechnicalvulnerability assessments by assessors with different level and type of knowledge. We report an experiment to compare how accurately students with different technical education and security professionals are able to assess the severity of software vulnerabilities with the Common Vulnerability Scoring System (v3) industry methodology. Our results could be useful for increasing awareness about the intrinsic subtleties of vulnerability risk assessment and possibly better compliance with regulations. With respect to academic education, professional training and human resources selections our work suggests that measuring the effects of knowledge and expertise on the accuracy of software security assessments is feasible albeit not easy.

The Darkweb hosts a number of legal and illegal services that are hard to reach due to the lack, by design, of a regular 'name service' as those operated by DNSs on the clearnet. This difficulty, together with the continuous appearance of decoy mirror services and replicated domains, severely limits the investigation capabilities of researchers and law enforcement agencies interested in measuring the Darkweb phenomenon as a whole. To address this issue we developed MASSDEAL, a tool for the automated exploration of the Darkweb that automatically learns about new or previous unseen services and measures repeatedly across time and space (i.e. across multiple deployments of the same service). Relying on data collected over more than 20 thousand darkweb services sampled from September 2018 to January 2019, we perform and report an extensive analysis of service redundancy and measure the appearance of .onion mirrors as well as providing an estimation of the infrastructural redundancy behind those systems.

In this work, we considered two well-known bug bounty programs - HackerOne and Google Vulnerability Research - with the goal of investigating patterns of activity and comparing productivity of security researchers. HackerOne and Google's programs differ in many ways. HackerOne is one of the largest and most successful bug bounty programs, with heterogeneous membership of security researchers and software producers. Google Vulnerability Research, instead, is a closed program for selected Google employees working on a more homogeneous range of software. For the analysis, we introduced three productivity metrics, which let us study the performance of researchers under different perspectives and possible patterns of activity. A contribution of this work is to shed new light on the yet not well understood environment represented by bug bounties and software vulnerability discovery initiatives. The low-hanging fruits approach adopted by unexperienced researchers in open bug bounties has been often discussed, but less is known about the approach adopted by more experienced participants. Another result is to have shown that a generic comparison between different bug bounty programs may lead to wrong conclusions. Bug bounty programs could exhibits large variations in researcher profiles and software characteristics, which make them not comparable without a careful examination of homogeneous subsets of participants and incentive mechanisms.

Decisional processes are at the basis of several security and privacy applications. However, they are often not transparent and can be affected by human or algorithmic biases that may lead to systematically misleading or unfair outcomes. To unveil these biases, one has to identify which information was used to make the decision and to quantify to what extent such information has influenced the process outcome. Two classes of techniques are widely used to determine possible correlation between variables within decisional processes from observational data: (i) econometric techniques, in particular regression analysis, and (ii) knowledge discovery techniques, in particular association rules mining. However, these techniques, taken individually, have intrinsic drawbacks that limit their applicability. In this work, we propose an approach for unveiling biases in decisional processes, which leverages association rule mining for systematic hypothesis generation and regression analysis for model selection and recommendation extraction. We demonstrate the proposed approach in the context of discrimination detection, showing that not only it provides 'statistically significant' evidence of discrimination but it also allows for a more efficient operationalization of the recommendations extracted, upon which the decision maker can operate.

The assessment of new vulnerabilities is an activity that accounts for information from several data sources and produces a 'severity' score for the vulnerability. The Common Vulnerability Scoring System (CVSS) is the reference standard for this assessment. Yet, no guidance currently exists on which information aids a correct assessment and should therefore be considered. In this paper we address this problem by evaluating which information cues increase (or decrease) assessment accuracy. We devise a block design experiment with 67 software engineering students with varying vulnerability information and measure scoring accuracy under different information sets. We find that baseline vulnerability descriptions provided by standard vulnerability sources provide only part of the information needed to achieve an accurate vulnerability assessment. Further, we find that additional information on assets, attacks, and vulnerability type contributes in increasing the accuracy of the assessment; conversely, information on known threats misleads the assessor and decreases assessment accuracy and should be avoided when assessing vulnerabilities. These results go in the direction of formalizing the vulnerability communication to, for example, fully automate security assessments.

The security extensions to the DNS (DNSSEC) currently cover approximately 3% of all domains worldwide. In response to the low deployment of DNSSEC, a few top-level domains started offering 'per-domain' economic incentives to encourage adoption of the protocol by offering a yearly discount on each signed domain. However, it remains unclear whether these incentives are well-balanced and foster the overall security of the infrastructure as well as its deployment at scale. In this paper we argue that, in the presence of fixed costs of deployment, misaligned 'per-domain' incentives may have the collateral effect of encouraging large operators to massively deploy unsecure implementations of DNSSEC, whereas smaller operators, for which the effect of the economic incentive is negligible, may not significantly benefit from it. To investigate this, we study the security of DNSSEC deployment at scale, particularly in TLDs that offer economic incentives. We find that the security of DNSSEC implementations in the wild poorly reflects standard recommendations, particularly for tasks that cannot be solved by triggering a flag in the DNS software service (e.g. key rollover). Further, we find that, on average, large operators deploy weak DNSSEC security more frequently than small DNSSEC operators, suggesting that current incentives are ineffective in promoting a secure adoption and in deterring insecure implementations. We conclude the paper with actionable recommendations for TLD registry operators to improve the alignment of economic incentives with secure DNSSEC requirements.

Vulnerability exploitation is reportedly one of the main attack vectors against computer systems. Yet, most vulnerabilities remain unexploited by attackers. It is therefore of central importance to identify vulnerabilities that carry a high 'potential for attack'. In this paper we rely on Symantec data on real attacks detected in the wild to identify a trade-off in the Impact and Complexity of a vulnerability in terms of attacks that it generates; exploiting this effect, we devise a readily computable estimator of the vulnerability's Attack Potential that reliably estimates the expected volume of attacks against the vulnerability. We evaluate our estimator performance against standard patching policies by measuring foiled attacks and demanded workload expressed as the number of vulnerabilities entailed to patch. We show that our estimator significantly improves over standard patching policies by ruling out low-risk vulnerabilities, while maintaining invariant levels of coverage against attacks in the wild. Our estimator can be used as a first aid for vulnerability prioritisation to focus assessment efforts on high-potential vulnerabilities.

Cybercrime markets support the development and diffusion of new attack technologies, vulnerability exploits, and malware. Whereas the revenue streams of cyber attackers have been studied multiple times in the literature, no quantitative account currently exists on the economics of attack acquisition and deployment. Yet, this understanding is critical to characterize the production of (traded) exploits, the economy that drives it, and its effects on the overall attack scenario. In this paper we provide an empirical investigation of the economics of vulnerability exploitation, and the effects of market factors on likelihood of exploit. Our data is collected first-handedly from a prominent Russian cybercrime market where the trading of the most active attack tools reported by the security industry happens. Our findings reveal that exploits in the underground are priced similarly or above vulnerabilities in legitimate bug-hunting programs, and that the refresh cycle of exploits is slower than currently often assumed. On the other hand, cybercriminals are becoming faster at introducing selected vulnerabilities, and the market is in clear expansion both in terms of players, traded exploits, and exploit pricing. We then evaluate the effects of these market variables on likelihood of attack realization, and find strong evidence of the correlation between market activity and exploit deployment. We discuss implications on vulnerability metrics, economics, and exploit measurement.

(U.S.) Rule-based policies for mitigating software risk suggest using the CVSS score to measure the risk of an individual vulnerability and act accordingly. A key issue is whether the ‘danger’ score does actually match the risk of exploitation in the wild, and if and how such a score could be improved. To address this question, we propose using a case-control study methodology similar to the procedure used to link lung cancer and smoking in the 1950s. A case-control study allows the researcher to draw conclusions on the relation between some risk factor (e.g., smoking) and an effect (e.g., cancer) by looking backward at the cases (e.g., patients) and comparing them with controls (e.g., randomly selected patients with similar characteristics). The methodology allows us to quantify the risk reduction achievable by acting on the risk factor. We illustrate the methodology by using publicly available data on vulnerabilities, exploits, and exploits in the wild to (1) evaluate the performances of the current risk factor in the industry, the CVSS base score; (2) determine whether it can be improved by considering additional factors such the existence of a proof-of-concept exploit, or of an exploit in the black markets. Our analysis reveals that (a) fixing a vulnerability just because it was assigned a high CVSS score is equivalent to randomly picking vulnerabilities to fix; (b) the existence of proof-of-concept exploits is a significantly better risk factor; (c) fixing in response to exploit presence in black markets yields the largest risk reduction.