Short-answer based questions have been used widely due to their effectiveness in assessing whether the desired learning outcomes have been attained by students. However, due to their open-ended nature, many different answers could be considered entirely or partially correct for the same question. In the context of computer science and software engineering courses where the enrolment has been increasing recently, manual grading of short-answer questions is a time-consuming and tedious process for instructors. In software engineering courses, assessments concern not just coding but many other aspects of software development such as system analysis, architecture design, software processes and operation methodologies such as Agile and DevOps. However, existing work in automatic grading/scoring of text-based answers in computing courses have been focusing more on coding-oriented questions. In this work, we consider the problem of autograding a broader range of short answers in software engineering courses. We propose an automated grading system incorporating both text embedding and completion approaches based on recently introduced pre-trained large language models (LLMs) such as GPT-3.5/4. We design and implement a web-based system so that students and instructors can easily leverage autograding for learning and teaching. Finally, we conduct an extensive evaluation of our automated grading approaches. We use a popular public dataset in the computing education domain and a new software engineering dataset of our own. The results demonstrate the effectiveness of our approach, and provide useful insights for further research in this area of AI-enabled education.

In machine learning, hyper-parameter optimization (HPO) aims to tune the set of parameters that controls the learning process. HPO could be time-consuming and resource-intensive due to the huge parameter search space and the complexity of models such as deep neural networks. Many of the existing HPO techniques tend to be variants of Bayesian optimization methods; each of which has been applied successfully for model tuning in different application domains. However, these Bayesian optimization methods have not been systematically evaluated against each other in the context of deep learning based malware detection. In this paper, we report a large-scale empirical study comparing popular HPO techniques on the performance of deep learning based malware classifiers. We use a diverse collection of seven datasets covering the most typical features used in malware detection. We conduct our experiments with Ray Tune, a distributed tuning platform, and popular optimization libraries such as Optuna, HyperOpt, Nevergrad, etc., across a wide range of computing platforms including AWS EC2, high-performance workstation, and laptop computers. Our extensive experiments provide useful insights into the application of different HPO techniques in deep learning based malware detection.

Drones are increasingly popular and getting used in a variety of missions such as area surveillance, pipeline inspection, cinematography, etc. While the drone is conducting a mission, anomalies such as sensor fault, actuator fault, configuration errors, bugs in controller program, remote cyberattack, etc., may affect the drone’s physical stability and cause serious safety violations such as crashing into the public. During a flight mission, drones typically log flight status and state units such as GPS coordinates, actuator outputs, accelerator readings, gyroscopic readings, etc. These log data may reflect the abovementioned anomalies. In this paper, we propose a novel, deep learning-based log analysis approach for detecting anomalies in the drone log that could lead to physical instabilities. We train a LSTM-based deep learning model on the normal flight logs produced by a baseline drone. Essentially, the model learns the sequential patterns of flight state units and correlations among them. The model can then be used to detect anomalies in the state units as the log entries are being recorded by the drone’s control program at runtime. In our experiments, we built detection models based on several logs produced by 3 different drone control programs, namely DJI, ArduPilot and PX4, and used them to detect anomalies in the logs. On average, our approach achieves 0.968 recall and 0.963 precision, and it can detect anomalies during runtime within a few milliseconds.

In Android malware classification, the distribution of training data among classes is often imbalanced. This causes the learning algorithm to bias towards the dominant classes, resulting in mis-classification of minority classes. One effective way to improve the performance of classifiers is the synthetic generation of minority instances. One pioneer technique in this area is Synthetic Minority Oversampling Technique (SMOTE) and since its publication in 2002, several variants of SMOTE have been proposed and evaluated on various imbalanced datasets. However, these techniques have not been evaluated in the context of Android malware detection. Studies have shown that the performance of SMOTE and its variants can vary across different application domains. In this paper, we conduct a large scale empirical evaluation of SMOTE and its variants on six different datasets that reflect six types of features commonly used in Android malware detection. The datasets are extracted from a benchmark of 4,572 benign apps and 2,399 malicious Android apps, used in our previous study. Through extensive experiments, we set a new baseline in the field of Android malware detection, and provide guidance to practitioners on the application of different SMOTE variants to Android malware detection.

Cloud gaming enables people playing graphically intensive games from their less powerful, or even outdated computing devices. It is challenging to realize cloud gaming as it requires minimal latency in server-side processing, rendering and streaming, which are expensive in terms of resource requirements, e.g., powerful GPU servers. Commercial gaming providers, e.g., Google Stadia, Amazon Luna, etc., hardly disclose any information on how they optimize gaming performance and cloud cost. In this work, we aim to investigate resource cost optimization for such cloud gaming systems. In contrast to previous work which have been focusing more on theoretical approaches, we deliver a fully functional, cost-optimized cloud-native gaming system, called CloudNPlay, implemented entirely on AWS Lambda, EC2 and other AWS services. We have conducted extensive evaluations on the gaming performance including latency and frame rates; as well as cloud resource cost reduction. The results indicate that games hosted on CloudNPlay are highly playable. More importantly, CloudNPlay reduces the resource cost by around 24% compared to a non-optimized deployment. We plan to open-source CloudNPlay to facilitate further research in cloud gaming.

As IoT ecosystem has been fast-growing recently, there have been various security concerns of this new computing paradigm. Malicious IoT apps gaining access to IoT devices and capabilities to execute sensitive operations (sinks), e.g., controlling door locks and switches, may cause serious security and safety issues. Unlike traditional mobile/web apps, IoT apps highly interact with a wide variety of physical IoT devices and respond to environmental events, in addition to user inputs. It is therefore important to conduct comprehensive testing of IoT apps to identify possible anomalous behaviours. On the other hand, it is also important to optimize the number of test cases generated, considering that there may be many possible ways in which apps, devices, environmental events, and user inputs interact. Existing works investigating security in IoT apps have been using ad-hoc testing approaches, in which test cases are usually designed to test some particular aspects of apps or devices. In this work, we develop an automated, smart fuzzing approach, called SmartFuzz, for testing Samsung SmartThings IoT apps. More specifically, SmartFuzz combines combinatorial test generation with light-weight program analysis, and aims to improve test coverage of sinks in an efficient, automated manner. We have implemented and evaluated our approach using a publicly available dataset of 60 SmartApps. The results have demonstrated the effectiveness and efficiency of SmartFuzz. In particular, SmartFuzz improved coverage of sinks by 184%, while generating and executing 20% fewer test cases as compared to ad-hoc testing.

Combining video streaming and online retailing (V2R) has been a growing trend recently. In this paper, we provide practitioners and researchers in multimedia with a cloud-based platform named Hysia for easy development and deployment of V2R applications. The system consists of: 1) a back-end infrastructure providing optimized V2R related services including data engine, model repository, model serving and content matching; and 2) an application layer which enables rapid V2R application prototyping. Hysia addresses industry and academic needs in large-scale multimedia by: 1) seamlessly integrating state-of-the-art libraries including NVIDIA video SDK, Facebook faiss, and gRPC; 2) efficiently utilizing GPU computation; and 3) allowing developers to bind new models easily to meet the rapidly changing deep learning (DL) techniques. On top of that, we implement an orchestrator for further optimizing DL model serving performance. Hysia has been released as an open source project on GitHub, and attracted considerable attention. We have published Hysia to DockerHub as an official image for seamless integration and deployment in current cloud environments.

This paper proposes and analyzes a new virtual machine (VM) placement technique called Group Instance to deal with co-location attacks in public Infrastructure-as-a-Service (IaaS) clouds. Specifically, Group Instance organizes cloud users into groups with pre-determined sizes set by the cloud provider. Our empirical results obtained via experiments with real-world data sets containing million of VM requests have demonstrated the effectiveness of the new technique. In particular, the advantages of Group Instance are three-fold: 1) it is simple and highly configurable to suit the financial and security needs of cloud providers, 2) it produces better or at least similar performance compared to more complicated, state-of-the-art algorithms in terms of resource utilization and co-location security, and 3) it does not require any modifications to the underlying infrastructures of existing public cloud services.

Recent studies showed that single-machine graph processing systems can be as highly competitive as cluster-based approaches on large-scale problems. While several out-of-core graph processing systems and computation models have been proposed, the high disk I/O overhead could significantly reduce performance in many practical cases. In this paper, we propose GraphMP to tackle big graph analytics on a single machine. GraphMP achieves low disk I/O overhead with three techniques. First, we design a vertex-centric sliding window (VSW) computation model to avoid reading and writing vertices on disk. Second, we propose a selective scheduling method to skip loading and processing unnecessary edge shards on disk. Third, we use a compressed edge cache mechanism to fully utilize the available memory of a machine to reduce the amount of disk accesses for edges. Extensive evaluations have shown that GraphMP could outperform existing single-machine out-of-core systems such as GraphChi, X-Stream and GridGraph by up to 30, and can be as highly competitive as distributed graph engines like Pregel+, PowerGraph and Chaos.

Co-resident attacks on public clouds could extract sensitive information like encryption keys, memory content or workload patterns from cloud-based applications. Most existing defense mechanisms have been targeting cloud providers. They usually require substantial changes to the underlying hardware, cloud infrastructure or VM placement strategies, therefore making their deployment difficult. In this work, we investigate secure VM provisioning from the user perspective. Our approach leverages existing secure resource allocation methods provided by public cloud providers such as EC2 Dedicated Instances. We consider several different VM provisioning algorithms which are secure, cost-effective and at the same time immediately deployable for cloud users without any changes required from cloud providers' part. Extensive experiments using real workload traces and EC2 resource pricing confirm the effectiveness of our approach. To the best of our knowledge, this work is the first to consider secure and cost-effective VM provisioning from the user perspective.

Due to increasing number of avenues for conducting cross-virtual machine (VM) side-channel attacks, the security of public IaaS cloud data centers is a growing concern. These attacks allow an adversary to steal private information from a target user whose VM instance is co-located with that of the adversary. To reduce the probability of malicious co-location, we propose a novel VM placement algorithm called “Previously Co-Located Users First”. We perform a theoretical and empirical analysis of our proposed algorithm to evaluate its resource efficiency and security. Our results, obtained using real-world cloud traces containing millions of VM requests and thousands of actual users, indicate that the proposed algorithm provides a significant increase in the cloud's co-location resistance with little compromise in resource utilization, compared to existing approaches.

In large-scale distributed file systems, efficient metadata operations are critical since most file operations have to interact with metadata servers first. In existing distributed hash table (DHT) based metadata management systems, the lookup service could be a performance bottleneck due to its significant CPU overhead. Our investigations showed that the lookup service could reduce system throughput by up to 70 percent, and increase system latency by a factor of up to 8 compared to ideal scenarios. In this paper, we present MetaFlow, a scalable metadata lookup service utilizing software-defined networking (SDN) techniques to distribute lookup workload over network components. MetaFlow tackles the lookup bottleneck problem by leveraging B-tree, which is constructed over the physical topology, to manage flow tables for SDN-enabled switches. Therefore, metadata requests can be forwarded to appropriate servers using only switches. Extensive performance evaluations in both simulations and testbed showed that MetaFlow increases system throughput by a factor of up to 3.2, and reduce system latency by a factor of up to 5 compared to DHT-based systems. We also deployed MetaFlow in a distributed file system, and demonstrated significant performance improvement.

It is common for real-world applications to analyze big graphs using distributed graph processing systems. Popular in-memory systems require an enormous amount of resources to handle big graphs. While several out-of-core approaches have been proposed for processing big graphs on disk, the high disk I/O overhead could significantly reduce performance. In this paper, we propose GraphH to enable high-performance big graph analytics in small clusters. Specifically, we design a two-stage graph partition scheme to evenly divide the input graph into partitions, and propose a GAB (Gather-Apply-Broadcast) computation model to make each worker process a partition in memory at a time. We use an edge cache mechanism to reduce the disk I/O overhead, and design a hybrid strategy to improve the communication performance. GraphH can efficiently process big graphs in small clusters or even a single commodity server. Extensive evaluations have shown that GraphH could be up to 7.8x faster compared to popular in-memory systems, such as Pregel+ and PowerGraph when processing generic graphs, and more than 100x faster than recently proposed out-of-core systems, such as GraphD and Chaos when processing big graphs.

Recent studies showed that single-machine graph processing systems can be as highly competitive as clusterbased approaches on large-scale problems. While several out-of-core graph processing systems and computation models have been proposed, the high disk I/O overhead could significantly reduce performance in many practical cases. In this paper, we propose GraphMP to tackle big graph analytics on a single machine. GraphMP achieves low disk I/O overhead with three techniques. First, we design a vertex-centric sliding window (VSW) computation model to avoid reading and writing vertices on disk. Second, we propose a selective scheduling method to skip loading and processing unnecessary edge shards on disk. Third, we use a compressed edge cache mechanism to fully utilize the available memory of a machine to reduce the amount of disk accesses for edges. Extensive evaluations have shown that GraphMP could outperform state-of-the-art systems such as GraphChi, X-Stream and GridGraph by 31.6x, 54.5x and 23.1x respectively, when running popular graph applications on a billion-vertex graph.

Many cluster management systems (CMSs) have been proposed to share a single cluster with multiple distributed computing systems. However, none of the existing approaches can handle distributed machine learning (ML) workloads given the following criteria: high resource utilization, fair resource allocation and low sharing overhead. To solve this problem, we propose a new CMS named Dorm, incorporating a dynamically-partitioned cluster management mechanism and an utilization-fairness optimizer. Specifically, Dorm uses the container-based virtualization technique to partition a cluster, runs one application per partition, and can dynamically resize each partition at application runtime for resource efficiency and fairness. Each application directly launches its tasks on the assigned partition without petitioning for resources frequently, so Dorm imposes flat sharing overhead. Extensive performance evaluations showed that Dorm could simultaneously increase the resource utilization by a factor of up to 2.32, reduce the fairness loss by a factor of up to 1.52, and speed up popular distributed ML applications by a factor of up to 2.72, compared to existing approaches. Dorm's sharing overhead is less than 5% in most cases.

Design space exploration refers to the evaluation of implementation alternatives for many engineering and design problems. A popular exploration approach is to run a large number of simulations of the actual system with varying sets of configuration parameters to search for the optimal ones. Due to the potentially huge resource requirements, cloud-based simulation execution strategies should be considered in many cases. In this paper, we look at the issue of running large-scale simulation-based design space exploration problems on commercial Infrastructure-as-a-Service clouds, namely Amazon EC2, Microsoft Azure and Google Compute Engine. To efficiently manage cloud resources used for execution, the key problem would be to accurately predict the running time for each simulation instance in advance. This is not trivial due to the currently wide range of cloud resource types which offer varying levels of performance. In addition, the widespread use of virtualization techniques in most cloud providers often introduces unpredictable performance interference. In this paper, we propose a resource and application-aware (RA2) prediction approach to combat performance variability on clouds. In particular, we employ neural network based techniques coupled with non-intrusive monitoring of resource availability to obtain more accurate predictions. We conducted extensive experiments on commercial cloud platforms using an evacuation planning design problem over a month-long period. The results demonstrate that it is possible to predict simulation execution times in most cases with high accuracy. The experiments also provide some interesting insights on how we should run similar simulation problems on various commercially available clouds.

Many distributed machine learning (ML) systems exhibit high communication overhead when dealing with big data sets. Our investigations showed that popular distributed ML systems could spend about an order of magnitude more time on network communication than computation to train ML models containing millions of parameters. Such high communication overhead is mainly caused by two operations: pulling parameters and pushing gradients. In this paper, we propose an approach called Timed Dataflow (TDF) to deal with this problem via reducing network traffic using three techniques: a timed parameter storage system, a hybrid parameter filter and a hybrid gradient filter. In particular, the timed parameter storage technique and the hybrid parameter filter enable servers to discard unchanged parameters during the pull operation, and the hybrid gradient filter allows servers to drop gradients selectively during the push operation. Therefore, TDF could reduce the network traffic and communication time significantly. Extensive performance evaluations in a real testbed showed that TDF could reduce up to 77% and 79% of network traffic for the pull and push operations, respectively. As a result, TDF could speed up model training by a factor of up to 4 without sacrificing much accuracy for some popular ML models, compared to systems not using TDF.

Although cloud computing is a thriving technology trend in industry and academy, the resource renting cost is still the main obstacle for users to switch to cloud. The existing pricing models are not flexible enough for users. On-demand pricing model does not guarantee resource availability, while reserved pricing model may result in high risk of resource wasting. In this paper, we propose OMTiR: An Open Market for Trading Idle Cloud Resources, enabling users to sell their unused or underutilized resources on negotiable prices. Consequently, users, either as a resource seller or buyer, can reduce the resource renting cost. In addition, the cloud provider can increase revenue by taking arbitrage profit in the market and serving more users using the same amount of resource. A comparative study is conducted using a real world workload trace to show the advantages of the open market model over the existing price models in terms of resource utilization rate and task waiting time.

High Level Architecture (HLA)-based simulations employing optimistic synchronization allows federates to process event and to advance simulation time freely at the risk of over-optimistic execution and execution rollbacks. In this paper, an adaptive resource provisioning system is proposed to accelerate optimistic HLA-based simulations in Virtual Execution Environment (VEE). A performance monitor is introduced using a middleware approach to measure the performance of individual federates transparently to the simulation application. Based on the performance measurements, a resource manager distributes the available computational resources to the federates, making them advance simulation time with comparable speeds. Our proposed approach is evaluated using a real-world simulation model with various workload inputs and different parameter settings. The experimental results show that, compared with distributing resources evenly among federates, our proposed approach can accelerate the simulation execution significantly using the same amount of computational resources.

Recently, application service providers have been employing Infrastructure-as-a-Service (IaaS) clouds such as Amazon EC2 to scale their computing resources on-demand to adapt to dynamic workloads. Existing research has been focusing more on cloud resource scaling in batch processing, non latency-sensitive applications. In this paper, we consider the problem of revenue-cost optimization in cloud-based application service providers with stringent QoS requirements, e.g., online gaming services. We propose an integrated approach which combines resource provisioning algorithms and request scheduling disciplines. The main goal is to maximize the service provider's revenue via satisfying pre-defined QoS requirements, and at the same time, to minimize cloud resource cost. We have implemented the proposed resource provisioning algorithms and scheduling disciplines into a cloud scaling framework developed in our previous work. Extensive experiments have been conducted with a fully functional implementation and realistic workloads modeled after real traces of popular online game servers. The results demonstrated the effectiveness of our proposed approach.

In this paper, we presented the design, implementation, and evaluation of a distributed system to manage the parallelized analytics for Aircraft Spare parts Management and Optimizations (SMO), which is a well-known problem in logistics industry. Our proposed solution is able to solve the resource-intensive SMO problem using distributed computing infrastructures (e.g., private or public clouds) in a scalable manner. We designed and fine-tuned a parallel met heuristics based on a fine-grained flow control workflow model which enables flow controls of running parallel meta-heuristics in multiple processors and achieved significant performance gains. Together with priority based scheduling, the proposed system effectively dispatches submitted SMO jobs over the set of distributed resources to accommodate different classes of users. Extensive experimental studies were conducted to analyze the performance of parallelized SMO job executions in term of execution time, computation and data transmission time, waiting time, memory usage, etc. Insightful lessons have been drawn from the obtained results, and potential areas for further improvements have also been identified.

Maintaining interactivity is one of the key challenges in distributed virtual environments (DVEs). In this paper, we consider a new problem, termed the interactivity-constrained server provisioning problem, whose goal is to minimize the number of distributed servers needed to achieve a prespecified level of interactivity. We identify and formulate two variants of this new problem and show that they are both NP-hard via reductions to the set covering problem. We then propose several computationally efficient approximation algorithms for solving the problem. The main algorithms exploit dependencies among distributed servers to make provisioning decisions. We conduct extensive experiments to evaluate the performance of the proposed algorithms. Specifically, we use both static Internet latency data available from prior measurements and topology generators, as well as the most recent, dynamic latency data collected via our own large-scale deployment of a DVE performance monitoring system over PlanetLab. The results show that the newly proposed algorithms that take into account interserver dependencies significantly outperform the well-established set covering algorithm for both problem variants.

Infrastructure-as-a-Service (IaaS) cloud computing provides the ability to dynamically acquire extra or release existing computing resources on-demand to adapt to dynamic application workloads. In this paper, we propose an extensible framework for on-demand cloud resource provisioning and adaptation. The core of the framework is a set of resource adaptation algorithms that are capable of making informed provisioning decisions to adapt to workload fluctuations. The framework is designed to manage multiple sets of resources acquired from different cloud providers, and to interact with different local resource managers. We have developed a fully functional web-service based prototype of this framework, and used it for performance evaluation of various resource adaptation algorithms under different realistic settings, e.g. when input data such as jobs' wall times are inaccurate. Extensive experiments have been conducted with both synthetic and real workload traces obtained from the Grid Workload Archives, more specifically the traces from the Large Hadron Collider Computing Grid. The results demonstrate the effectiveness and robustness of our proposed algorithms.

In distributed virtual environments, e.g., online gaming, collaborative designs and distributed military simulations, interactivity is one of the most important requirements. The users may notice serious degradations in quality of service when interacting in the virtual world if the response from the system is much slower than what they have experienced in real life. In this paper, we consider the problem of placing distributed servers in the network to reduce client-server communication latencies, which is termed the server placement problem. We proposed two new network-aware placement algorithms which take into account users' locations in the network and connectivity at the autonomous system level to determine good sites for servers. Extensive experiments with realistic network models showed that these new algorithms significantly outperform existing approaches that require full knowledge of network connectivity at the router-level topologies.

Distributed virtual environments (DVEs) allow many simultaneous human users to interact with each other in shared, 3D virtual worlds. The interactivity of a DVE is of crucial importance, as its success greatly depends on users' perceptions when interacting within the virtual world. In this paper, different from traditional application-centric approaches like dead reckoning, we propose a network-centric approach to enhance the interactivity of DVEs by directly reducing the network latencies in client-server communications. We consider a key problem with this approach, termed the server placement problem. Generally, this problem concerns how to place servers in the network to reduce client-server communication latencies. We then suggest several degree-based server placement approaches. Extensive simulation studies using realistic models have shown that appropriate server placement is very effective in enhancing the interactivity of large-scale DVEs

Distributed virtual environments (DVEs) are distributed systems that allow multiple geographically distributed clients (users) to interact simultaneously in a computer-generated, shared virtual world. Applications of DVEs can be seen in many areas nowadays, such as online games, military simulations, collaborative designs, etc. To support large-scale DVEs with real-time interactions among thousands or more distributed clients, a geographically distributed server architecture (GDSA) is generally needed, and the virtual world can be partitioned into many distinct zones to distribute the load among the servers. Due to the geographic distributions of clients and servers in such architectures, it is essential to efficiently assign the participating clients to servers to enhance users' experience in interacting within the DVE. This problem is termed the client assignment problem. In this paper, we propose a two-phase approach, consisting of an initial assignment phase and a refined assignment phase to address this problem. Both phases are shown to be NP-hard, and several heuristic assignment algorithms are then devised based on this two-phase approach. Via extensive simulation studies with realistic settings, we evaluate these algorithms in terms of their performances in enhancing interactivity of the DVE.