Automated vulnerability detection is a critical issue in software security. The advent of Deep Learning (DL) has led to numerous studies employing DL to detect vulnerabilities in software source code. However, existing approaches still perform poorly, particularly with real-world vulnerabilities, due to the difficulty in accurately capturing their properties. To this end, we introduce PVDetector, a DL-based approach that utilizes rich code semantics, incorporates vulnerability knowledge, and leverages pretrained code representations for precise vulnerability detection. At its core, PVDetector employs a new model called Vulnerability-enriched Code Semantic Graph (VCSG), which accurately characterizes functions by distinguishing the semantics of identical variables and more finely capturing control dependencies, data dependencies, and vulnerability relationships. Additionally, we introduce four pretraining tasks specifically designed to learn the semantics of control, data, vulnerability, and variables from the VCSG model. These pretraining tasks significantly enhance PVDetector’s capability to detect vulnerabilities in downstream tasks. Experimental results indicate that PVDetector outperforms SOTAs by 5.0–12.5% in precision, 0.2–9.7% in recall, and 3.0–15.1% in F1-score. Additionally, it supports six programming languages and demonstrates high efficiency (e.g., 10.6 \(\times\) faster than DeepDFA). When applied to seven software products, PVDetector discovered 55 vulnerabilities, including 10 silently patched flaws that had not been previously reported.

Firmware emulation is a critical method for re-searching embedded systems. However, current approaches to Non- Volatile Random Access Memory (NVRAM) emulation often face challenges such as strong hardware dependency, complex parameter configuration, and the need for extensive manual intervention, which result in low emulation success rates and poor network reachability. Additionally, the lack of transparency during the firmware execution process makes it difficult to track and analyze the causes of emulation failures. To address these challenges, this paper introduces FirmEE, a firmware emulation enhancement system that leverages NVRAM-Sim, which automates the modeling of NVRAM peripherals and simulates the interaction between firmware and NVRAM hardware during parameter requests and assignments. FirmEE dynamically optimizes parameter configurations by constructing an NVRAM value exploration space, utilizing the number of basic blocks executed during firmware startup as reward. This approach facilitates large-scale automated firmware emulation, significantly improving both emulation success rates and network reachability. Moreover, FirmEE provides fine-grained monitoring of the firmware execution process, offering enhanced transparency and deeper insights into system behavior. Experimental results show that FirmEE increases the emulation success rate to 79.41 % and the network reachability rate to 73.09% on a custom dataset comprising 301 firmware images from four mainstream router vendors, significantly outperforming existing methods.

Vulnerability detection remains a critical challenge in the field of security. Many existing approaches extract code representations for vulnerability detection. However, these methods often focus on the overall semantics of the code, neglecting to specifically target vulnerability-related semantics. To address this limitation, we propose a novel LLM steering method designed to steer LLMs to focus on vulnerability concepts, thereby enhancing their performance in vulnerability detection. Specifically, we introduce a vulnerability steering vector that represents the concept of vulnerability in the representation space. This vector is generated using a paired vulnerability-patch function dataset, effectively capturing the essence of vulnerabilities. Experimental results demonstrate that the proposed method significantly improves LLMs' performance and notably outperforms existing SOTA methods in vulnerability detection tasks. Furthermore, we validate the cross-language transferability of the steering vector and explore the explainability of vulnerability detection.

Encrypted traffic classification (ETC) is essential for ensuring network security and efficient management. Despite advances in deep learning, ETC remains challenging as existing models struggle to learn robust, discriminative representations from content-encrypted, highly imbalanced traffic.To address these challenges, we propose BTRFormer, a novel ETC approach that capitalizes on the inherent properties of encryption algorithms to enhance classification accuracy. At the core of BTRFormer lies a block-based, multi-layer traffic representation that adopts a 4×4 block as the fundamental unit, inspired by the encryption algorithm’s use of 16-byte blocks for encryption operations. This representation preserves the intrinsic structure of encrypted payloads, facilitating the model’s ability to learn deep semantic features. Subsequently, a transformer-based model is employed to learn from the multi-layer representation, capturing intra-block, inter-block, and inter-packet dependencies through block-wise attention mechanisms. Finally, BTRFormer leverages a pre-training phase on large-scale unlabeled data, followed by fine-tuning with a minimal amount of labeled samples to improve generalization and adaptability. Experimental results show that BTRFormer significantly outperforms SOTA methods on six real-world datasets, highlighting its effectiveness in encrypted traffic classification and secure network management.

Virtual machine snapshots are critical to service reliability and operational agility in cloud and edge infrastructures. However, under continuous snapshotting, frequent checkpoints impose severe runtime and storage costs, especially for workloads with frequent memory changes. We observe that snapshots frequently store pages never used during online rollback: empirical analysis shows approximately 45 % of pages need not be saved before the next checkpoint. In this paper, we propose Lazy-ConSnap, a VM snapshot system that combines lazy persistence with prediction-based optimization to achieve both storage efficiency and runtime performance. Our approach integrates: (1) a lazy-persistence mechanism using cross-snapshot dirty-page bitmaps to defer saves until pages are re-modified; (2) a history-set prediction algorithm that proactively persists hot pages to reduce costly VM exits; (3) an optimized rollback that reuses memory and loads only modified pages during restoration. Our evaluation with typical workloads over$\mathbf{3 0}$-minute periods shows Lazy-ConSnap achieves up to 6.8 % storage savings (up to$\mathbf{1. 5 G B}$saved), up to$\mathbf{1 4. 3 \%}$rollback speedup, and up to$\mathbf{5. 6 \%}$runtime performance improvement (up to 105s saved) compared to lazy-persistence alone, while maintaining prediction precision above$\mathbf{7 3 \%}$. These gains enable efficient continuous VM snapshots with low-latency recovery for modern cloud environments.

Software vulnerability detection is a critical challenge in cyber security. With the rise of deep learning and large language models (LLMs), numerous studies have applied these technologies to vulnerability detection. Existing approaches directly employ prompt engineering, chain-of-thought reasoning, and fine-tuning methods on LLMs, but achieve suboptimal results. To effectively leverage LLMs’ powerful reasoning capabilities for vulnerability detection, we propose VulnTeam, a novel team collaboration framework for LLM vulnerability detection inspired by human expert team collaboration. Specifically, we introduce a dual-stage fine-tuning approach where expert models are first fine-tuned using low-rank adaptation to detect vulnerabilities related to different vulnerability syntactic features, followed by instruction fine-tuning of a leader model responsible for the final decision-making. Ultimately, team members (expert models) and the team leader (leader model) collaborate to detect vulnerabilities. Our experimental evaluation across three LLMs and two datasets demonstrates that VulnTeam significantly enhances LLMs’ vulnerability detection performance (average F1-score improvement of 12.51%). Moreover, VulnTeam-enhanced LLMs substantially outperform previous state-of-the-art (SOTA) vulnerability detection methods (average F1-score improvement of 7.78%). Additionally, we analyze computational costs to validate VulnTeam’s practical applicability.

Encrypted traffic classification refers to the task of identifying the application, service or malware associated with network traffic that is encrypted. Previous methods mainly have two weaknesses. Firstly, from the perspective of word-level (namely, byte-level) semantics, current methods use pre-training language models like BERT, learned general natural language knowledge, to directly process byte-based traffic data. However, understanding traffic data is different from understanding words in natural language, using BERT directly on traffic data could disrupt internal word sense information so as to affect the performance of classification. Secondly, from the perspective of packet-level semantics, current methods mostly implicitly classify traffic using abstractive semantic features learned at the top layer, without further explicitly separating the features into different space of categories, leading to poor feature discriminability. In this paper, we propose a simple but effective Aggregator and Separator Network (ASNet) for encrypted traffic understanding, which consists of two core modules. Specifically, a parameter-free word sense aggregator enables BERT to rapidly adapt to understanding traffic data and keeping the complete word sense without introducing additional model parameters. And a category-constrained semantics separator with task-aware prompts (as the stimulus) is introduced to explicitly conduct feature learning independently in semantic spaces of different categories. Experiments on five datasets across seven tasks demonstrate that our proposed model achieves the current state-of-the-art results without pre-training in both the public benchmark and real-world collected traffic dataset. Statistical analyses and visualization experiments also validate the interpretability of the core modules. Furthermore, what is important is that ASNet does not need pre-training, which dramatically reduces the cost of computing power and time. The model code and dataset will be released inhttps://github.com/pengwei-iie/ASNET.

Malware classification involves categorizing mal-ware samples based on their characteristics. While deep learning techniques applied to malware execution traces, mainly API calls, have shown potential in this field, they still perform poorly. This is primarily because they treat all APIs equally and train classifiers directly on native APIs, which inadequately capture the under-lying family-related semantics. In this paper, we first investigate the behaviors of multiple malware families and observe that different families exhibit divergent behaviors, with each family consistently favoring certain behaviors over time. Motivated by this, we propose APIBeh, a new embedding method designed to enhance malware classification. APIBeh first utilizes Benignity Degree Algorithm to identify and exclude insignificant, likely benign APIs from sequences. Then, it introduces the concept of Behavior Inclination, which quantifies the association between an API and malicious behaviors, facilitating high-level behavior encoding for each API. This Behavior Inclination embedding is then concatenated with raw embedding to represent an API, and fed into a DL model for classifier training. Experimental results show that APIBeh outperforms existing embedding methods in classification performance, e.g., 3.18% boost in weighted f1-score over a recent study using word2vec. In addition, it offers robustness to concept drift and adversarial attacks.

Analyzing malware based on API call sequences is an effective approach, as these sequences reflect the dynamic execution behavior of malware. Recent advancements in deep learning have facilitated the application of these techniques to mine valuable information from API call sequences. However, these methods typically operate on raw sequences and may not effectively capture crucial information, especially in the case of multi-process malware, due to theAPI call interleaving problem. Furthermore, they often fail to capture contextual behaviors within or across processes, which is particularly important for identifying and classifying malicious activities. Motivated by this, we present API2Vec++, a graph-based API embedding method for malware detection and classification. First, we construct a graph model to represent the raw sequence. Specifically, we design the Temporal Process Graph (TPG) to model inter-process behaviors and the Temporal API Property Graph (TAPG) to model intra-process behaviors. Compared to our previous graph model, the TAPG model exposes operations with associated behaviors within the process through node properties and thus enhances detection and classification abilities. Using these graphs, we develop a heuristic random walk algorithm to generate numerous paths that can capture fine-grained malicious familial behavior. By pre-training these paths using the BERT model, we generate embeddings of paths and APIs, which can then be used for malware detection and classification. Experiments on a real-world malware dataset demonstrate that API2Vec++ outperforms state-of-the-art embedding methods and detection/classification methods in both accuracy and robustness, particularly for multi-process malware.

Federated Learning (FL) is a promising machine learning paradigm for collaborative training on cross-soils in a privacy-protected manner. However, the existence of non-IID data causes problems such as performance degradation and thus becomes one of the key challenges in FL recently. To address this problem, we propose a clustered personalized federated learning method named as SynCPFL. SynCPFL groups clients sharing with the similar data distribution together, thereby facilitating collaboration and producing a better-personalized model for each client. In contrast to existing clustered federated learning methods, SynCPFL does not require multiple rounds of interaction between clients and server, so that the communication overhead is reduced a lot, thereby saving resources of clients. We evaluate SynCPFL on benchmark datasets, the experimental results demonstrate that SynCPFL outperforms existing methods.

The API call sequence based malware detectors have proven to be promising, especially when incorporated with deep neural networks (DNNs). Several adversarial attack methods are proposed to fool these detectors by introducing undetectable perturbations into normal samples. However, in real-world scenarios, the malware detector provides only the predicted label for a given sample, without exposing its network architecture or output probability, making it challenging for adversarial attacks under the decision-based black-box. Existing work in this area typically relies on random-based methods that suffer high costs and low attack success rates. To address these limitations, we propose a novel decision-based black-box attack against API sequence based malware detectors, called MalAder. Our approach aims to improve the attack success rate as well as query efficiency through a directional perturbation algorithm. First, it utilizes attention-based API ranking to assess the importance of API calls in the context of different API sequences. This assessment guides the insertion position for perturbation. Then, the perturbation is carried out using benign distance perturbing, which gradually shortens the semantic distance from adversarial API sequences to a set of benign samples. Finally, our algorithm iteratively generates adversarial malware samples by performing perturbations. In addition, we have implemented MalAder and evaluated its performance against two classic malware detectors. The results show that MalAder outperforms state-of-the-art decision-based black-box adversarial attacks, proving its effectiveness.

Dynamic malware detection refers to detecting mal-ware by inferring the run-time trace of malware, i.e., a sequence of API calls. In this paper, we proposed HeteroNet, a novel dynamic malware detection model. The main idea of HeteroNet is that it integrates multiple deep learning models which use heterogeneous dynamic features of malware samples.Specifically, we implement three heterogeneous deep learning based models to learn various features from three representations, namely API name sequence, API resource graph and API call graph, respectively, each of the representation is built from the run-time trace of malware. Meanwhile, several methods such as attention mechanism and graph neural networks are applied in base models, according to the characteristics of API calls. Finally, an ensemble algorithm is used to integrate the outputs of three base models. We trained and evaluated HeteroNet on a dataset of 28,770 samples. The precision of the model on the testing set reached 98.40%, which is 1.20% higher than the best result of baselines. Moreover, HeteroNet is more robust against concept drift than other baselines.

Analyzing malware based on API call sequence is an effective approach as the sequence reflects the dynamic execution behavior of malware.Recent advancements in deep learning have led to the application of these techniques for mining useful information from API call sequences. However, these methods mainly operate on raw sequences and may not effectively capture important information especially for multi-process malware, mainly due to the API call interleaving problem.

Since many malwares disguise themselves by encrypting, obfuscating and recompiling, it is not easy for static analysis methods to recognise new or unknown malwares. This paper proposes a novel dynamic analysis technology based on hardware virtualisation to analyse more malwares with lower computational resources. Firstly, it intercepts the system-call functions to achieve on-demand behaviour analysis by setting special permissions in their physical addresses, which can be dynamically acquired when system-call functions are loaded into memory, as well as only monitoring high-risk functions, which take a small part of the whole functions. Then, this paper utilises copy-on-write technique and incremental image capability to reduce hard drive consumption and hard disk replication time. Finally, this paper proposes a novel approach to capture the return value of system-call functions to deeply analyse the poisoned results of malware samples. Meanwhile, a prototype system, called HEMC, is implemented based on QEMU/KVM . The experiments demonstrate that proposed methods outperform existing methods in efficiency and performance on malware dynamic analysis.

With the popularity of IaaS clouds, many distributed and networked applications are running in virtual machine cluster (VMC). The distributed snapshots of VMC are a practical approach to guarantee system reliability. It rewinds the system to an intermediate state from failures so that the applications can continue execution from a point near the failure. However, the applications running in the VMC suffer from long disruption and significant performance degradation due to the heavy cost distributed snapshots, especially when designed to guarantee global consistency of VMC snapshots. This article presents eHotSnap, which takes distributed snapshots of a VMC efficiently. eHotSnap divides the native snapshot into light cost transient snapshot and heavy cost memory snapshot and then coordinates the VM snapshots immediately after transient snapshots. In this way, it decouples coordination from heavy cost snapshots so that the distributed snapshots are taken (completed in logic) within a second. Then, it performs memory snapshot and optimizes it with a two-layer optimization, which first employs de-duplication to reduce the amount of snapshot data and then leverages priority queue to serve guest write operations preferentially. In addition to presenting eHotSnap, we have implemented a prototype on QEMU/KVM. The experimental results demonstrate the effectiveness and efficiency of the proposed approach.

Malware continuously evolve and become more and more sophisticated. Learning on execution behavior is proven to be effective for malware detection. In this paper, we present MalPro, a DNN based malware detection approach that performs learning on process-aware behaviors for Windows programs. It first employs logistic regression-based weighting method to assess the sensitivity of an API to malicious behavior, and weights the API following run-time arguments with varying degrees of sensitivities. Then, it constructs the process graph of inter-process interactions from which a set of attributes are extracted, for characterizing the relationship of various processes in term of invoke actions. Finally, it feeds the weighted API sequences and the process graph attributes into the DNN for training a binary classifier to detect malware. Moreover, we have implemented and evaluated MalPro on two datasets. The results demonstrate that our method outperforms naive models, verifying the effectiveness of MalPro.

With the dramatic increase in malicious software, the sophistication and innovation of malware have increased over the years. In particular, the dynamic analysis based on the deep neural network has shown high accuracy in malware detection. However, most of the existing methods only employ the raw API sequence feature, which cannot accurately reflect the actual behavior of malicious programs in detail. The relationship between API calls is critical for detecting suspicious behavior. Therefore, this paper proposes a malware detection method based on the graph neural network. We first connect the API sequences executed by different processes to build a directed process graph. Then, we apply Bert to encode the API sequences of each process into node embedding, which facilitates the semantic execution information inside the processes. Finally, we employ GCN to mine the deep semantic information based on the directed process graph and node embedding. In addition to presenting the design, we have implemented and evaluated our method on 10,000 malware and 10,000 benign software datasets. The results show that the precision and recall of our detection model reach 97.84% and 97.83%, verifying the effectiveness of our proposed method.

The Record and Replay (RnR) technology provides the ability to reproduce past execution of systems deterministically. It has many prominent applications, including fault tolerance, security analysis, and failure diagnosis. In system virtualization, previous RnR researches mainly focus on individual VM, including coherent replaying of multi-core systems, reducing performance penalty and storage overhead. However, with the emerging of distributed systems deployed in virtual machine clusters (VMC), the existing RnR technology of individual VM can not meet the requirements of analyzers and developers. The critical challenge for VMC RnR is to maintain the consistency of global state. In this paper, we propose ClusterRR, a RnR framework for VMC. To solve the inconsistency problem, we propose coordination protocols to schedule the record and replay process of VMs. Meanwhile, we employ a Hybrid RnR approach to reduce the performance penalty and storage costs caused by recording network events. Moreover, we implement ClusterRR on QEMU/KVM platform and utilize a network packets retransmission framework to guarantee the reproducibility of VMC replay. Last, we conduct a series of experiments to measure its efficiency and overhead. The results show that ClusterRR would efficiently replay the execution of the whole VMC at instruction-level granularity.

Learning on execution behaviour, i.e., sequences of API calls, is proven to be effective in malware detection. In this paper, we present CruParamer, a deep neural network based malware detection approach for Windows platform that performs learning on sequences of parameter-augmented APIs. It first employs rule-based and clustering-based classification to assess the sensitivity of a parameter to malicious behaviour, and further labels the API following the run-time parameters with varying degrees of sensitivities. Then, it encodes the APIs by concatenating the native embedding and the sensitive embedding of labelled APIs, for characterizing the relationship between successive labelled APIs and their correspondence in terms of security semantics. Finally, it feeds the sequences of API embedding into the deep neural network for training a binary classifier to detect malware. In addition to presenting the design, we have implemented CruParamer and evaluated it on two datasets. The results demonstrate that CruParamer outperforms naïve models when taking raw APIs as input, proving the effectiveness of CruParamer. Moreover, we have evaluated the impact ofmimicryand adversarial attacks on our model, and the results verify the robustness of CruParamer.

The reliability of data and services hosted on a virtual machine (VM) is a top concern in cloud environments. The Continuous Snapshots can reduce the data loss in case of failures and thus is prevailing for protecting long-running systems. However, existing methods suffer from long VM downtime, long snapshot interval and significant performance loss. In this article, we present iConSnap, a system designed to take fine-grained continuous snapshots of virtual machines without compromising VM performance. First, iConSnap adopts the copy-on-write (COW) mechanism to save the memory pages on-demand, and thus decreases the VM downtime to about 200 milliseconds. Second, we extend the idea of COW and propose a lazily incremental approach to save the delta data between two successive snapshots only once, thereby reducing the snapshot duration and snapshot data a lot. Third, we propose a scheduling mechanism to mitigate the VM performance penalty issue. Last, we introduce a method combined of compression and time-aware multi-granularity reclamation strategy to reduce the storage costs without losing performance and availability. We implement iConSnap on QEMU/KVM and evaluate it through a set of experiments. The experimental results show that iConSnap outperforms existing approaches in terms of VM downtime, snapshot duration, storage costs and VM performance.

Binary code similarity detection, which answers whether two pieces of binary code are similar, has been used in a number of applications,such as vulnerability detection and automatic patching. Existing approaches face two hurdles in their efforts to achieve high accuracy and coverage: (1) the problem of source-binary code similarity detection, where the target code to be analyzed is in the binary format while the comparing code (with ground truth) is in source code format. Meanwhile, the source code is compiled to the comparing binary code with either a random or fixed configuration (e.g.,architecture, compiler family, compiler version, and optimization level), which significantly increases the difficulty of code similarity detection; and (2) the existence of different degrees of code similarity. Less similar code is known to be more, if not equally, important in various applications such as binary vulnerability study. To address these challenges, we design BugGraph, which performs source-binary code similarity detection in two steps. First, BugGraph identifies the compilation provenance of the target binary and compiles the comparing source code to a binary with the same provenance.Second, BugGraph utilizes a new graph triplet-loss network on the attributed control flow graph to produce a similarity ranking. The experiments on four real-world datasets show that BugGraph achieves 90% and 75% true positive rate for syntax equivalent and similar code, respectively, an improvement of 16% and 24% overstate-of-the-art methods. Moreover, BugGraph is able to identify 140 vulnerabilities in six commercial firmware.

Code similarity is one promising approach to detect vulnerabilities hidden in software programs. However, due to the complexity and diversity of source code, current methods suffer low accuracy, high false negative and poor performance, especially in analyzing a large program. In this paper, we propose to tackle these problems by presenting VulDetector, a static-analysis tool to detect C/C++ vulnerabilities based on graph comparison at the granularity of function. At the key of VulDetector is a weighted feature graph (WFG) model which characterizes function with a small yet semantically rich graph. It first pinpoints vulnerability-sensitive keywords to slice the control flow graph of a function, thereby reducing the graph size without compromising security-related semantics. Then, each sliced subgraph is characterized using WFG, which provides both syntactic and semantic features in varying degrees of security. As for graph comparison, we take full usage of vulnerability graph and patch graph to improve accuracy. In addition, we propose two optimization methods based on analysis of vulnerabilities. We have implemented VulDetector to automatically detect vulnerabilities in software programs with known vulnerabilities. The experimental results prove the effectiveness and efficiency of VulDetector.

The record and replay(RnR) technology of virtual machine(VM) provides the ability to reproduce the past execution of a VM deterministically. It has many promising applications in the cloud environment, including fault tolerance, security analysis, and failure diagnosis. Existing studies in this area pay more effort in optimizing the record method, such as reducing performance penalty and storage costs. However, considering that many practical applications follow the record once, replay many mode, the optimization for the replay is more critical, especially for efficiency. In this paper, we propose pRnR, a novel parallel RnR framework, to support efficient replay. By combining the native RnR framework with an improved continuous snapshots mechanism, pRnR divides the full execution into many independent and complete slices, each of which supports arbitrary replay. In addition, it supports two replay modes to improve replay efficiency, i.e., multi-slice parallel replay and multi-dimension parallel replay. Moreover, we apply our pRnR framework to syscall-based diagnosis to demonstrate its usability. The experimental results show that pRnR is more efficient than existing RnR frameworks.

External keywords are crucial for response generation models to address the generic response problems in open-domain conversational systems. The occurrence of keywords in a response depends heavily on the order of the keywords as they are generated sequentially. Meanwhile, the order of keywords also affects the semantics of a response. Previous keywords based methods mainly focus on the composite of keywords, while the order of keywords has not been sufficiently discussed. In this work, we propose an order-sensitive keywords based model to explore the influence of the order of keywords in open-domain response generation. It automatically inferences the most suitable order that is optimized to generate a natural and relevant response, and subsequently generates the response using the ordered keywords as building blocks. We conducted experiments on a public Twitter dataset and the results show that our approach outperforms the state-of-the-art baselines in both automatic and human evaluations.

Virtual machine (VM) snapshot, enabling a VM to be resumed from a previously recorded state, is an essential part of cloud infrastructures. Unfortunately, the snapshot data are likely to be lost due to the high rate of disk failures, so that the associated VM fails to recover properly. To enhance data availability without compromising application performance upon rollback recovery, it is desired to place multiple replicas of snapshot across disperse disks. However, due to the large size of replica, it induces non-trivial storage cost when managing massive snapshots in clouds. In this paper, we investigate this problem and find out that the semantic gap existed between snapshot creation and snapshot storing is one key factor inducing high storage cost. To this end, we propose SnapFiner, a page-aware snapshot system for creating and storing massive snapshot files efficiently. First, SnapFiner acquires a fine-grained page categorization with an in-depth page exploration from three orthogonal views, thereby discovering more pages that can be excluded from the snapshot. Second, SnapFiner varies the number of replicas for different page categories based on a page-aware replication policy, achieving low storage cost without compromising availability and performance. Third, SnapFiner handles the loss of pages either intentionally dropped upon snapshot creation or unexpectedly damaged due to disk failures, enabling proper system execution after rollback recovery. We have implemented SnapFiner on QEMU/KVM to justify its practicality for Linux guests. The experimental results demonstrate that SnapFiner reduces the storage cost by 33 and 69.5 percent respectively compared to our previous work PARS and the naive approach on QEMU/KVM and HDFS.

Page reclamation is one compelling way to overcommit memory in modern operating systems. To achieve wise reclamation, the Linux kernels employ page frame reclamation algorithm (PFRA) to reclaim pages based on the page usage view. However, due to the semantic gap problem in virtualized environments, the native PFRA suffers three types of unwise evictions including false eviction, superficial eviction and omitted eviction. This will lead to high swapping I/O activity and consequently limits the ability to overcommit memory. We present SA-PFRS, a Semantics-Aware Page Frame Reclamation System, to address this problem. SA-PFRS separates the memory pages allocated for guests from reclaimable candidates in host, explores how the pages are being used by guest OS, and adjusts the reclamation order in the view of guest. Then, SA-PFRS re-arranges the reclamation sequence of guest pages and host pages, so as to reclaim the memory pages in a global semantics-aware manner. This enables SA-PFRS to eliminate a large number of swapping I/O operations when memory is overcommitted. We implement a prototype of SA-PFRS in Linux kernel, and show its effectiveness through a set of experiments.

Rollback is an effective technique to resume the system execution from a recorded intermediate state upon failures, without having to restart the entire system. However, in virtualized environments, rollback of a virtual machine cluster (VMC) produces high network traffic and long service disruption, particularly for a large cluster used for scientific computing, thereby imposing significant overhead both on network and applications. This paper proposes Piccolo, a fast and efficient rollback system, to restore a VMC from snapshot files over data center network. First, we exploit the similarity among VMC snapshots and leverage multicast to deliver the identical pages across VMs placed on disperse hosts, thereby bypassing unnecessary transmission of a large number of pages. Second, we analyze the impact on network traffic of varying VM placements in data center network, formulate the traffic aware placement as an optimization problem, and design a two-tier approximation algorithm that efficiently solves the problem. In addition to presenting Piccolo, we detail its implementation, and evaluate it by a set of experiments. The results show that Piccolo could achieve a significant reduction in terms of total sent data, network traffic and rollback latency compared to the existing generic techniques.

Rollback is an effective technique to resume the system execution from a recorded intermediate state upon failures. However, in virtualized environments, rollback of a virtual machine cluster (VMC) produces high network traffic and long service disruption, consequentially imposing significant overhead both on network and applications. In this paper, we propose Piccolo, a fast and efficient rollback system, to restore a VMC from snapshot files over datacenter network. We exploit the similarity among VMC snapshots and leverage multicast to deliver the identical pages across VMs placed on disperse hosts, thereby bypassing transmission of a large number of unnecessary pages. In addition to presenting Piccolo, we detail its implementation, and evaluate it by a set of experiments. The results show that Piccolo could achieve a significant reduction in terms of total sent data, network traffic and rollback latency compared to the existing generic rollback techniques.

Virtual machine (VM) snapshot enhances the system availability by saving the running state into stable storage during failure-free execution and rolling back to the snapshot point upon failures. Unfortunately, the snapshot state may be lost due to disk failures, so that the VM fails to be recovered. The popular distributed file systems employ replication technique to tolerate disk failures by placing redundant copies across disperse disks. However, unless user-specific personalization is provided, these systems consider the data in the file as of same importance and create identical copies of the entire file, leading to non-trivial additional storage overhead.

With the energy and power costs increasing alongside the growth of the IT infrastructures, achieving workload concentration and high availability in cloud computing environments is becoming more and more complex. Virtual machine (VM) migration has become an important approach to address this issue, particularly; live migration of the VMs across the physical servers facilitates dynamic workload scheduling of the cloud services as per the energy management requirements, and also reduces the downtime by allowing the migration of the running instances. However, migration is a complex process affected by several factors such as bandwidth availability, application workload and operating system configurations, which in turn increases the complications in predicting the migration time in order to negotiate the service-level agreements in a real datacenter. In this paper, we propose an adaptive approach named improved MIGration (iMIG), in which we characterize some of the key metrics of the live migration performance, and conduct several experiments to study the impacts of the investigated metrics on the Kernel-based VM (KVM) functionalities, as well as the energy consumed by both the destination and the source hosts. Our results reveal the importance of the configured parameters: speed limit, TCP buffer size and max downtime, along with the VM properties and also their corresponding impacts on the migration process. Improper setting of these parameters may either incur migration failures or causes excess energy consumption. We witness a few bugs in the existing Quick EMUlator (QEMU)/KVM parameter computation framework, which is one of most widely used KVM frameworks based on QEMU. Based on our observations, we develop an analytical model aimed at better predictions of both the migration time and the downtime, during the process of VM deployment. Finally, we implement a suite of profiling tools in the adaptive mechanism based on the qemu-kvm-0.12.5 version, and our experiment results prove the efficiency of our approach in improving the live migration performance. In comparison with the default migration approach, our approach achieves a 40% reduction in the migration latency and a 45% reduction in the energy consumption.

The reliability of data and services hosted in a virtual machine (VM) is a top concern in cloud computing environment. Continuous snapshots reduces the data loss in case of failures, and thus is prevailing for providing protection for long-running systems. However, existing methods suffer from long VM downtime, long snapshot interval and significant performance overhead. In this paper, we present ConSnap, a system designed to enable taking fine-grained continuous snapshots of virtual machines without compromising VM performance. First, ConSnap adopts the COW (copy-on-write) manner to save the memory pages in a lazy way, and thus decrease the snapshot interval to dozens of milliseconds. Second, we only save the incremental memory pages on the basis of the last snapshot in each epoch to reduce the snapshot duration, and thus mitigate VM performance loss. Third, we propose a multi-granularity space reclamation strategy, which merges the unused snapshot files to achieve storage space saving, as well as fast recovery. We have implemented ConSnap on QEMU/KVM and conducted several experiments to verify its effectiveness. Compared with the stop-and-copy based incremental snapshots, ConSnap reduces the performance loss by 71.1% ~ 10.2% under Compilation workload, and 14.5% ~ 4.7% for the Ftp workload, when the interval varies from 1s to 60s.

Virtual networking is vital to efficient resource management in Clouds, and it is in fact one of the main services provided by many Cloud Computing platforms. Virtual network management needs to meet specific requirements, including tenant isolation and adaption to virtual machines' lifecycle. Most of the existing schemes for virtual network management are based on the use of overlay networks in order to achieve a desirable degree of flexibility. However, these schemes suffer from a common limit, i.e. relatively high performance penalty due to a complicated forwarding process. We address this performance concern by developing a new management scheme, FENet, which makes use of Software-Defined Networks (SDN) to create virtual networks and manage them via the SDN controller programs. We present the design of an SDN controller, with the definition of flow entry rules based on the OpenFlow protocol and the specification of a routing algorithm. The results from our experimental evaluation show that our SDN-based prototype can control virtual network interconnections and tenant isolation appropriately. FENet achieves about 30% better network performance than the management scheme based on OpenVPN and lower latency in comparison with the traditional bridging scheme.

Live virtual machine migration is a technique often used to migrate an entire OS with running applications in a non-disruptive fashion. Prior works concerned with one-to-one live migration with many techniques have been proposed such as pre-copy, post-copy and log/replay. In contrast, we propose VMScatter, a one-to-many migration method to migrate virtual machines from one to many other hosts simultaneously. First, by merging the identical pages within or across virtual machines, VMScatter multicasts only a single copy of these pages to associated target hosts for avoiding redundant transmission. This is impactful practically when the same OS and similar applications running in the virtual machines where there are plenty of identical pages. Second, we introduce a novel grouping algorithm to decide the placement of virtual machines, distinguished from the previous schedule algorithms which focus on the workload for load balance or power saving, we also focus on network traffic, which is a critical metric in data-intensive data centers. Third, we schedule the multicast sequence of packets to reduce the network overhead introduced by joining or quitting the multicast groups of target hosts. Compared to traditional live migration technique in QEMU/KVM, VMScatter reduces 74.2% of the total transferred data, 69.1% of the total migration time and achieves the network traffic reduction from 50.1% to 70.3%.

Software aging has been cited in many scenarios including Operating System, Web Servers, Real-time Systems. However, few studies have been conducted in long running virtualized environments where more and more software is being delivered as a service. Furthermore, state-of-the-art methods lack the ability to deal with miscellaneous upper applications and underlying systems transparently in virtualized scenarios. In this paper, we detect aging phenomenon by conducting experiments in physical and virtual machines and identify the differences between the two, and propose a feature code-based methodology for failure prediction through system call, then implement a prototype in virtual machine manager layer to predict failure time and rejuvenate transparently, which is suitable in virtualized scenarios. The evaluation shows the prediction deviation against reality is less than 10%.

The high-availiablity of mission-critical data and services hosted in a virtual machine (VM) is one of the top concerns in a cloud computing environment. The live disk snapshot is an emerging technology to save the whole state and the data of a VM at a specific point of time, and be used for quick disaster recovery. However, the existing VM disk snapshot systems suffer from long operation time and I/O performance degradation problems during snapshots creating and managing, and thereby affecting the performance of the VM and its services. To address such issues, we designed an efficient VM disk snapshot system, named iROW (improved Redirect-on-Write). In iROW, a bitmap based light-weight index scheme is adopted to replace the existing multi-level index tree structure to reduce query cost. Additionally, through a combination of Redirect-on-Write (ROW) and Copy-on-Demand (COD) schema to avoid extra copy operation on the first write after snapshot with Copy-on-Write (COW) schema, and the file fragmentation problem caused by ROW snapshot after long-term using. Finally, iROW gives a unified disk space allocation function by the host machine's file system. We have implemented iROW in qemu-kvm 0.12.5 and conducted some experiments. The implementation of iROW completely obey the interfaces of the block device driver in QEMU, so it is transparent to the upper system or applications and original disk image formats can be also supported. The experimental results show that iROW has obvious performance advantages in snapshot creating and management operations. Compared with the existing qcow2 disk image in KVM, when the VM disk size is 50GB, and the cluster size is 64KB (the default cluster size of qcow2), the snapshot creation and rollback time is only about 6% and 3% of original qcow2's. With the increasing of the VM disk size, iROW has more performance advantages on snapshot creation and rollback operations. In addition, the I/O performance of iROW is better than qcow2. When the cluster size is 64 KB, typically the iROW's performance loss is 10% less than qcow2's, and its first write performance after snapshot creation is about 250% of qcow2's.

Virtualisation is a prevalent technology in current computing. Among the many aspects of virtualisation, it can be employed to reduce hardware costs by server consolidation, implement "green computing" by reducing power consumption and as an underpinning process for cloud computing enabling the creation of a range of virtual networks and virtual supercomputers. This paper presents performance measurements for a cloning system known as iVIC that has been developed in Beihang University, China. In an extension to earlier work, it focuses on the factors the limit the number of clones that can be successfully started. IVIC creates clusters of virtual computers that can communicate with each other through virtual switch mechanisms. The virtual switches can also allow communication between the clone environment and the physical world. Testing has been undertaken to identify the limiting factors for creating and starting numbers of clone machines, measure the power consumption of the physical system and the computational performance capability of the clones.