In this paper, we study the correlation between passwords across different datasets which quantitatively explains the success of existing training-based password cracking techniques. We also study the correlation between a user's password and his/her social profile. This enabled us to develop the first social profile-aware password strength meter, namely SociaLShield. Our quantification techniques and SocialShield have meaningful implications to system administrators, users, and researchers, e.g., helping them quantitatively understand the threats posed by a password leakage incident, defending against emerging profile-based password attacks, and facilitating the research of countermeasures against existing and newly developed training-based password attacks. We validate our proposed quantification techniques and SocialShield through extensive experiments by leveraging real-world leaked passwords. Experimental results demonstrate that our quantification techniques are accurate in measuring correlation among different leaked datasets and that although SocialShield is light-weight, it is effective in defending against profile-based password attacks.

In this paper, we conduct a large-scale study on the crackability, correlation, and security of 145 million real world passwords, which were leaked from several popular Internet services and applications. To the best of our knowledge, this is the largest empirical study that has been conducted. Specifically, we first evaluate the crackability of 145 million real world passwords against 6+ state-of-the-art password cracking algorithms in multiple scenarios. Second, we examine the effectiveness and soundness of popular commercial password strength meters (e.g., Google, QQ) and the security impacts of username/email leakage on passwords. Finally, we discuss the implications of our results, analysis, and findings, which are expected to help both password users and system administrators to gain a deeper understanding of the vulnerability of real passwords against state-of-the-art password cracking algorithms, as well as to shed light on future password security research topics.

Sophisticated cyber security threats, such as advanced persistent threats, rely on infecting end points within a targeted security domain and embedding malware. Typically, such malware periodically reaches out to the command and control infrastructures controlled by adversaries. Such callback behavior, called beaconing, is challenging to detect as (a) detection requires long-term temporal analysis of communication patterns at several levels of granularity, (b) malware authors employ various strategies to hide beaconing behavior, and (c) it is also employed by legitimate applications (such as updates checks). In this paper, we develop a comprehensive methodology to identify stealthy beaconing behavior from network traffic observations. We use an 8-step filtering approach to iteratively refine and eliminate legitimate beaconing traffic and pinpoint malicious beaconing cases for in-depth investigation and takedown. We provide a systematic evaluation of our core beaconing detection algorithm and conduct a large-scale evaluation of web proxy data (more than 30 billion events) collected over a 5-month period at a corporate network comprising over 130,000 end-user devices. Our findings indicate that our approach reliably exposes malicious beaconing behavior, which may be overlooked by traditional security mechanisms.

Recent years have witnessed a rampant use of domain generation algorithms (DGAs) in major botnet crimewares, which tremendously strengthens a botnet's capability to evade detection or takedown. Despite a plethora of existing studies on detecting DGA-generated domains in DNS traffic, remediating such threats still relies on vetting the DNS behavior of each individual device. Yet, in large networks featuring complicated DNS infrastructures, we often lack the capability or the resource to exhaustively investigate every part of the networks to identify infected devices in a timely manner. It is therefore of great interest to first assess the population distribution of DGA-bots inside the networks and to prioritize the remediation efforts. In this paper, we present BotMeter, a novel tool that accurately charts the DGA-bot population landscapes in large networks. Specifically, we embrace the prevalent yet challenging setting of hierarchical DNS infrastructures with caching and forwarding mechanisms enabled, whereas DNS traffic is observable only at certain upper-level vantage points. We establish a new taxonomy of DGAs that captures their characteristic DNS dynamics. This allows us to develop a rich library of rigorous analytical models to describe the complex relationships between bot populations and DNS lookups observed at vantage points. We provide results from extensive empirical studies using both synthetic data and real DNS traces to validate the efficacy of BotMeter.

Nowadays, cyber criminals often build web infrastructures rather than a single server to conduct their malicious activities. In order to continue their malevolent activities without being detected, cyber criminals make efforts to conceal the core servers (e.g., C&C servers, exploit servers, and drop-zone servers) in the malicious web infrastructure. Such deliberate invisibility of those concealed malicious servers, however, makes them particularly distinguishable from benign web servers that are usually promoted to be public. In this paper, we conduct the first large-scale measurement study to investigate the visibility of both malicious and benign servers. From our intensive analysis of over 100,000 benign servers, 45,000 malicious servers and 40,000 redirections, we identify a set of distinct features of malicious web infrastructures from their locations, structures, roles, and relationships perspectives, and propose a lightweight yet effective detection system called VisHunter. VisHunter identifies malicious redirections from visible servers to invisible servers at the entryway of malicious web infrastructures. We evaluate VisHunter on both online public data and large-scale enterprise network traffic, and demonstrate that VisHunter can achieve an average 96.2% detection rate with only 0.9% false positive rate on the real enterprise network traffic.

In this paper, we present the design, architecture, and implementation of a novel analysis engine, called Feature Collection and Correlation Engine (FCCE), that finds correlations across a diverse set of data types spanning over large time windows with very small latency and with minimal access to raw data. FCCE scales well to collecting, extracting, and querying features from geographically distributed large data sets. FCCE has been deployed in a large production network with over 450,000 workstations for 3 years, ingesting more than 2 billion events per day and providing low latency query responses for various analytics. We explore two security analytics use cases to demonstrate how we utilize the deployment of FCCE on large diverse data sets in the cyber security domain: 1) detecting fluxing domain names of potential botnet activity and identifying all the devices in the production network querying these names, and 2) detecting advanced persistent threat infection. Both evaluation results and our experience with real-world applications show that FCCE yields superior performance over existing approaches, and excels in the challenging cyber security domain by correlating multiple features and deriving security intelligence.

Consider the problem of reliable multicast over a network in the presence of adversarial errors. In contrast to traditional network error correction codes designed for a given network capacity and a given number of errors, we study an arguably more realistic setting that prior knowledge on the network and adversary parameters is not available. For this setting we propose efficient and throughput-optimal error correction schemes, provided that the source and terminals share randomness that is secret form the adversary. We discuss an application of cryptographic pseudorandom generators to efficiently produce the secret randomness, provided that a short key is shared between the source and terminals. Finally we present a secure key distribution scheme for our network setting.

Data outsourcing offers cost-effective computing power to manage massive data streams and reliable access to data. Data owners can forward their data to clouds, and the clouds provide data mirroring, backup, and online access services to end users. However, outsourcing data to untrusted clouds requires data authenticity and query integrity to remain in the control of the data owners and users.

Anti-virus systems developed by different vendors often demonstrate strong discrepancies in how they name malware, which signficantly hinders malware information sharing. While existing work has proposed a plethora of malware naming standards, most anti-virus vendors were reluctant to change their own naming conventions. In this paper we explore a new, more pragmatic alternative. We propose to exploit the correlation between malware naming of different anti-virus systems to create their consensus classification, through which these systems can share malware information without modifying their naming conventions. Specifically we present Latin, a novel classification integration framework leveraging the correspondence between participating anti-virus systems as reflected in heterogeneous information sources at instance-instance, instance-name, and name-name levels. We provide results from extensive experimental studies using real malware datasets and concrete use cases to verify the efficacy of Latin in supporting cross-system malware information sharing.

In the age of big data, key-value data updated by intensive write streams is increasingly common, e.g., in social event streams. To serve such data in a cost-effective manner, a popular new paradigm is to outsource it to the cloud and store it in a scalable key-value store while serving a large user base. Due to the limited trust in third-party cloud infrastructures, data owners have to sign the data stream so that the data users can verify the authenticity of query results from the cloud. In this paper, we address the problem of verifiable freshness for multi-version key-value data. We propose a memory-resident digest structure that utilizes limited memory effectively and can have efficient verification performance. The proposed structure is named IncBM-Tree because it can INCrementally build a Bloom filter-embedded Merkle Tree. We have demonstrated the superior performance of verification under small memory footprints for signing, which is typical in an outsourcing scenario where data owners and users have limited resources.

Cyber security attacks are becoming ever more frequent and sophisticated. Enterprises often deploy several security protection mechanisms, such as anti-virus software, intrusion detection/prevention systems, and firewalls, to protect their critical assets against emerging threats. Unfortunately, these protection systems are typically ‘noisy’, e.g., regularly generating thousands of alerts every day. Plagued by false positives and irrelevant events, it is often neither practical nor cost-effective to analyze and respond to every single alert. The main challenges faced by enterprises are to extract important information from the plethora of alerts and to infer potential risks to their critical assets. A better understanding of risks will facilitate effective resource allocation and prioritization of further investigation. In this paper, we present MUSE, a system that analyzes a large number of alerts and derives risk scores by correlating diverse entities in an enterprise network. Instead of considering a risk as an isolated and static property pertaining only to individual users or devices, MUSE exploits a novel mutual reinforcement principle and models the dynamics of risk based on the interdependent relationship among multiple entities. We apply MUSE on real-world network traces and alerts from a large enterprise network consisting of more than 10,000 nodes and 100,000 edges. To scale up to such large graphical models, we formulate the algorithm using a distributed memory abstraction model that allows efficient in-memory parallel computations on large clusters. We implement MUSE on Apache Spark and demonstrate its efficacy in risk assessment and flexibility in incorporating a wide variety of datasets.

Automatic malware clustering plays a vital role in combating the rapidly growing number of malware variants. Most existing malware clustering algorithms operate on either static instruction features or dynamic behavior features to partition malware into families. However, these two distinct approaches have their own strengths and weaknesses in handling different types of malware. Moreover, different clustering algorithms and even multiple runs of the same algorithms may produce inconsistent or even contradictory results. To remedy this heterogeneity and lack of robustness of a single clustering algorithm, we propose a novel system called DUET by exploiting the complementary nature of static and dynamic clustering algorithms and optimally integrating their results. By using the concept of clustering ensemble, DUET combines partitions from individual clustering algorithms into a single consensus partition with better quality and robustness. DUET improves existing ensemble algorithms by incorporating cluster-quality measures to effectively reconcile differences and/or contradictions between base malware clusterings. Using real-world malware samples, we compare the performance of DUET (in terms of clustering precision, recall and coverage) with individual state-of-the-art static and dynamic clustering component. The comprehensive experiments demonstrate DUET's capability of improving the coverage of malware samples by 20--40% while keeping the precision near the optimum achievable by any individual clustering algorithm.

This paper assesses the potential for highly mobile botnets to communicate and perform nefarious actions using only open WiFi networks, which we term mobile WiFi botnets. We design and evaluate a proof-of-concept mobile WiFi botnet using real-world mobility traces and actual open WiFi network locations for the urban environment of San Francisco. Our extensive simulation results demonstrate that mobile WiFi botnets can support rapid command propagation, with commands typically reaching over 75% of the botnet only 2 hours after injection-sometimes, within as little as 30 minutes. Moreover, those bots able to receive commands usually have ≈40-50% probability of being able to do so within a minute of the command being issued. Our evaluation results also indicate that even a small mobile WiFi botnet of only 536 bots can launch an effective DDoS attack against poorly protected systems. Furthermore, mobile WiFi botnet traffic is sufficiently distributed across multiple open WiFi networks-with no single network being over-utilized at any given moment-to make detection difficult.

Botnets are one of the most serious security threats to the Internet and personal computer (PC) users. Although botnets have not yet caused major outbreaks in the mobile world, with the rapidly-growing popularity of smartphones such as Apple's iPhone and Android-based phones that store more personal data and gain more capabilities than earlier generation handsets, botnets are expected to become a severe threat to smartphones soon. In this paper, we propose the design of a mobile botnet that makes the most of mobile services and is resilient to disruption. The mobile botnet utilizes SMS messages for C&C and a P2P structure as its topology. Our simulation results demonstrate that a modified Kademlia---a structured architecture---is a better choice for the mobile botnet's topology. In addition, we discuss potential countermeasures to defend against this mobile botnet threat.

This paper considers the global IP-usage patterns exhibited by different types of malicious and benign domains, with a focus on single and double fast-flux domains. We have developed and deployed a lightweight DNS probing engine, called DIGGER, on 240 PlanetLab nodes spanning 4 continents. Collecting DNS data for over 3.5 months on a plethora of domains, our global vantage points enabled us to identify distinguishing behavioral features between them based on their DNS-query results. To help us analyze the enormous amount of data, we have quantified these features and designed an effective classifier capable of accurately discriminating between different types of domains. Applying the classifier on the 3.5-month DNS data allows us to reveal the relative prevalence of different fast-flux domains and conduct detailed studies on them separately. These results provide insight into the current global state of fast-flux botnets and their range in implementation, revealing potential trends for botnet-based services. We also uncover previously-unseen domains whose name servers alone demonstrate fast-flux behavior and a new, cautious IP management strategy currently employed by criminals to evade detection.

In this paper, we explore the escalating “arms race” between fast-flux (FF) botnet detectors and the botmasters' effort to subvert them, and investigate several novel mimicry-attack techniques that allow botmasters to avoid detection. We first analyze the state-of-art FF detectors and their effectiveness against the current botnet threat, demonstrating how botmasters can - with their current resources - thwart detection strategies. Based on the realistic assumptions inferred from empirically observed trends, we create formal models for bot decay, online availability, DNS-advertisement strategies and performance, allowing us to demonstrate the effectiveness of different mimicry attacks and evaluate their effects on the overall online availability and capacity of botnets.

Cooperative (or distributed) sensing has been recognized as a viable means to enhance the incumbent signal detection by exploiting the diversity of sensors. However, it is challenging to secure such distributed sensing due mainly to the unique features of dynamic spectrum access networks-openness of low-layer protocol stacks in software-defined radio devices and the absence of interactions/coordination between primary and secondary devices. To meet this challenge, we propose an attack-tolerant distributed sensing protocol (ADSP) for DTV signal detection in IEEE 802.22 WRANs, under which sensors in close proximity are grouped as a cluster, and sensors within a cluster cooperate to safeguard the integrity of sensing. The heart of ADSP is a novel filter based on shadow-fading correlation, by which the fusion center cross-validates reports from the sensors to identify and penalize abnormal sensing reports. By realizing this correlation filter, ADSP significantly reduces the impact of an attack on the performance of distributed sensing, while incurring minimal processing and communication overheads. ADSP also guarantees the detectability requirements of 802.22 to be met even with the presence of sensing report manipulation attacks by scheduling sensing within the framework of sequential hypothesis testing. The efficacy of ADSP is validated on a realistic 2D shadow-fading field. Our extensive simulation-based study shows that ADSP reduces the false-alarm rate by 99.2 percent while achieving 97.4 percent of maximum achievable detection rate, and meets the detection requirements of IEEE 802.22 in various attack scenarios.

Bots are coordinated by a command and control (C&C) infrastructure to launch attacks that seriously threaten the Internet services and users. Most botnet-detection approaches function at the network level and require the analysis of packets' payloads, raising privacy concerns and incurring large computational overheads. Moreover, network traffic analysis alone can seldom provide a complete picture of botnets' behavior. By contrast, in-host detection approaches are useful to identify each bot's host-wide behavior, but are susceptible to the host-resident malware if used alone. To address these limitations, we consider both the coordination within a botnet and the malicious behavior each bot exhibits at the host level, and propose a C&C protocol-independent detection framework that combines host- and network-level information for making detection decisions. The framework is shown to be effective in detecting various types of botnets with low false-alarm rates.

A major challenge of the anti-virus (AV) industry is how to effectively process the huge influx of malware samples they receive every day. One possible solution to this problem is to quickly determine if a new malware sample is similar to any previously-seen malware program. In this paper, we design, implement and evaluate a malware database management system called SMIT (Symantec Malware Indexing Tree) that can efficiently make such determination based on malware's function-call graphs, which is a structural representation known to be less susceptible to instruction-level obfuscations commonly employed by malware writers to evade detection of AV software. Because each malware program is represented as a graph, the problem of searching for the most similar malware program in a database to a given malware sample is cast into a nearest-neighbor search problem in a graph database. To speed up this search, we have developed an efficient method to compute graph similarity that exploits structural and instruction-level information in the underlying malware programs, and a multi-resolution indexing scheme that uses a computationally economical feature vector for early pruning and resorts to a more accurate but computationally more expensive graph similarity function only when it needs to pinpoint the most similar neighbors. Results of a comprehensive performance study of the SMIT prototype using a database of more than 100,000 malware demonstrate the effective pruning power and scalability of its nearest neighbor search mechanisms.

Accurate sensing of the spectrum condition is of crucial importance to the mitigation of the spectrum scarcity problem in dynamic spectrum access (DSA) networks. Specifically, distributed sensing has been recognized as a viable means to enhance the incumbent signal detection by exploiting the diversity of sensors. However, it is challenging to make such distributed sensing secure due mainly to the unique features of DSA networks - openness of a low-layer protocol stack in SDR devices and non-existence of communications between primary and secondary devices. To address this challenge, we propose attack-tolerant distributed sensing protocol (ADSP), under which sensors in close proximity are grouped into a cluster, and sensors in a cluster cooperatively safeguard distributed sensing. The heart of ADSP is a novel shadow fading correlation-based filter tailored to anomaly detection, by which the fusion center prefilters abnormal sensor reports via cross-validation. By realizing this correlation filter, ADSP minimizes the impact of an attack on the performance of distributed sensing, while incurring minimal processing and communications overheads. The efficacy of our scheme is validated on a realistic two-dimensional shadow-fading field, which accurately approximates real-world shadowing environments. Our extensive simulation-based evaluation shows that ADSP significantly reduces the impact of attacks on incumbent detection performance.

Achieving secure time-synchronization in wireless sensor networks (WSNs) is a challenging, but very important problem that has not yet been addressed effectively. This paper proposes an attack-tolerant time-synchronization protocol (ATSP) in which sensor nodes cooperate to safeguard the time- synchronization service against malicious attacks. ATSP exploits the high temporal correlation existing among adjacent nodes in a WSN to achieve (1) adaptive management of the profile of each sensor's normal behavior, (2) distributed, cooperative detection of falsified clock values advertised by attackers or compromised nodes, and (3) significant improvement of synchronization accuracy and stability by effectively compensating the clock drifts with the calibrated clock. To reduce the risk of losing time-synchronization due to attacks on the reference node, ATSP utilizes distributed, mutual synchronization and confines the impact of attacks to a local area (where attacks took place). Furthermore, by maintaining an accurate profile of sensors' normal synchronization behaviors, ATSP detects various critical attacks while incurring only reasonable communication and computation overheads, making ATSP attack-tolerant and ideal for resource-constrained WSNs.

A novel behavioral detection framework is proposed to detect mobile worms, viruses and Trojans, instead of the signature-based solutions currently available for use in mobile devices. First, we propose an efficient representation of malware behaviors based on a key observation that the logical ordering of an application's actions over time often reveals the malicious intent even when each action alone may appear harmless. Then, we generate a database of malicious behavior signatures by studying more than 25 distinct families of mobile viruses and worms targeting the Symbian OS - the most widely-deployed handset OS - and their variants. Next, we propose a two-stage mapping technique that constructs these signatures at run-time from the monitored system events and API calls in Symbian OS. We discriminate the malicious behavior of malware from the normal behavior of applications by training a classifier based on Support Vector Machines (SVMs). Our evaluation on both simulated and real-world malware samples indicates that behavioral detection can identify current mobile viruses and worms with more than 96% accuracy. We also find that the time and resource overheads of constructing the behavior signatures from low-level API calls are acceptably low for their deployment in mobile devices.

Network worms pose a serious threat to the Internet infrastructure as well as end-users. Various techniques have been proposed for detection of, and response against worms. A frequently-used and automated response mechanism is to rate-limit outbound worm traffic while maintaining the operation of legitimate applications, offering a gentler alternative to the usual detect-and-block approach. However, most rate-limiting schemes to date only focus on host-level network activities and impose a single threshold on the entire host, failing to (i) accommodate network-intensive applications and (ii) effectively contain network worms at the same time. To alleviate these limitations, we propose a per-process-based containment framework in each host that monitors the fine-grained runtime behavior of each process and accordingly assigns the process a suspicion level generated by a machine-learning algorithm. We have also developed a heuristic to optimally map each suspicion level to the rate-limiting threshold. The framework is shown to be effective in containing network worms and allowing the traffic of legitimate programs, achieving lower false-alarm rates.