Effective agent coordination is crucial in cooperative Multiagent Reinforcement Learning (MARL). While recent advances have significantly improved cooperation by modeling agent interactions through various graph structures, most existing approaches primarily focus on homogeneous agents. Despite the ubiquity of heterogeneous agents, constructing a comprehensive graph that captures their diverse attributes and relationships from scratch is notoriously labor-intensive for both humans and agents, which makes policy learning extremely challenging. To tackle this difficulty, we propose a novel method that utilizes a fuzzy human attention-guided graph to model inter-agent relationships. Instead of learning the graph entirely from scratch, we incorporate abstract human attention, with its uncertainty captured through fuzzy logic, to guide the graph development process. To further accommodate the varying attributes and objectives of heterogeneous agents while maintaining their learning capabilities, the attention-guided graph is fine-tuned through a hyper-network. Our proposed approach is end-to-end trainable and agnostic to specific MARL methods. Empirical evaluations conducted on challenging heterogeneous scenarios from the StarCraft Multiagent Challenge (SMAC) and SMACv2 validate the effectiveness of the proposed method.

Abstract Due to its exceptional learning ability, multi-agent deep reinforcement learning (MADRL) has garnered widespread research interest. However, since the learning is data-driven and involves sampling from millions of steps, training a large number of agents is inherently challenging and inefficient. Inspired by the human learning process, we aim to transfer knowledge from humans to avoid starting from scratch. Given the growing emphasis on the Human-on-the-Loop concept, this study focuses on addressing the challenges of large-population learning by incorporating suboptimal human knowledge into the cooperative multi-agent environment. To leverage human experience, we integrate human knowledge into the training process of MADRL, representing it in natural language rather than specific action-state pairs. Compared to previous works, we further consider the attributes of transferred knowledge to assess its impact on algorithm scalability. Additionally, we examine several features of knowledge mapping to effectively convert human knowledge to the action space where agent learning occurs. In reaction to the disparity in knowledge construction between humans and agents, our approach allows agents to decide freely which portions of the state space to leverage human knowledge. From the challenging domains of the StarCraft Multi-agent Challenge, our method successfully alleviates the scalability issue in MADRL. Furthermore, we find that, despite individual-type knowledge significantly accelerating the training process, cooperative-type knowledge is more desirable for addressing a large agent population. We hope this study provides valuable insights into applying and mapping human knowledge, ultimately enhancing the interpretability of agent behavior.

Recent advances in deep learning have significantly improved the performance of software vulnerability prediction (SVP). To enhance trustworthiness, the SVP highlights predicted lines of code (LoC) that may be vulnerable. However, providing LoC alone is often insufficient for software practitioners, as it lacks detailed information about the nature of the vulnerability. This paper introduces a novel framework that is built on SVP by offering additional explanatory information based on the suggested LoC. Similar to security reports, our framework comprehensively explains the vulnerability aspects, such as Root Cause, Impact, Attack Vector, and Vulnerability Type. The proposed framework is powered by transformer architectures. Specifically, we leverage pre-trained language models for code to fine-tune on two practical datasets: BigVul and Vulnerability Key Aspect, ensuring our framework’s applicability to real-world scenarios. Experiments using the ROUGE and BLEU scores as evaluation metrics show that our framework achieves better performance with CodeT5+, statistically outperforming a baseline study in generating key vulnerability aspects. Additionally, we conducted a small-scale user study with experienced software practitioners to assess the effectiveness of the framework. The results show that 72% of the participants found our framework helpful in accepting the SVP results, and 68% rated the additional explanations as moderately to extremely useful. Editor’s note: Open Science material was validated by the Journal of Systems and Software Open Science Board . • A novel framework generates an explanation from the predicted vulnerable lines. • Comprehensive investigations of factors influencing the quality of the framework. • We conducted a user study to validate its usefulness.

Most research investigating how social influence affects election results mainly uses diffusion models for binary opinions. However, these diffusion models are progressive and focus on the diffusion of one opinion. In this article, we introduce a general diffusion model for ordinal opinions expressed as linear orderings over a finite set of candidates. We employ agent-based modeling to simulate a nonprogressive diffusion process, allowing multiple types of opinion diffusion about different candidates. The proposed agent-based diffusion model can forecast long-term trends of opinion diffusion in social networks by capturing voters’ personalized features and incorporating dynamic social contexts. Furthermore, we examine the possibility of affecting election outcomes by externally changing the ordinal opinions of certain vertices, i.e., campaigning. Since finding influential voters from the social network is computationally challenging, we propose a heuristic approach, i.e., backward influence rank (BIR). Experimental results demonstrate that the proposed BIR approach is superior to the classic greedy approach for campaigning by achieving a similar margin of victory to that of the greedy approach but running two orders of magnitude faster than the greedy approach did.

A key challenge in sensor-based fall prediction is the fact that a fall event can often occur in various configurations of fall poses together with their own spatio-temporal dependencies. This leads us to define a spatio-temporal model to explicitly characterize these internal configurations of poses. In particular, we introduce a graph neural network with spatio-temporal topological structure to encode such latent relations among poses by capturing representative patterns in fall events. Moreover, a human body orientation estimator is devised to capture human low limbs information, and as a result, separate pose dependencies are globally consistent. Empirical evaluations on two benchmark datasets and one in-house dataset suggest our approach significantly outperforms the state-of-the-art methods.

Due to the exponential growth of agent interactions and the curse of dimensionality, learning efficient coordination from scratch is inherently challenging in large-scale multi-agent systems. While agents' learning is data-driven, sampling from millions of steps, human learning processes are quite different. Inspired by the concept of Human-on-the-Loop and the daily human hierarchical control, we propose a novel knowledge-guided multi-agent reinforcement learning framework (hhk-MARL), which combines human abstract knowledge with hierarchical reinforcement learning to address the learning difficulties among a large number of agents. In this work, fuzzy logic is applied to represent human suboptimal knowledge, and agents are allowed to freely decide how to leverage the proposed prior knowledge. Additionally, a graph-based group controller is built to enhance agent coordination. The proposed framework is end-to-end and compatible with various existing algorithms. We conduct experiments in challenging domains of the StarCraft Multi-agent Challenge combined with three famous algorithms: IQL, QMIX, and Qatten. The results show that our approach can greatly accelerate the training process and improve the final performance, even based on low-performance human prior knowledge.

Introducing LexiFuse+: a semi-supervised model merging lexicon features, BERT transfer learning, and one-class classifiers to detect anomalous content in short texts. It tackles challenges of informal text and imbalanced datasets, excelling in hate speech detection. By leveraging one-class classifiers in a fully deep fine-tuned network trained with unlabeled data, LexiFuse+ surpasses base models.

In multi-agent systems, agents with limited capabilities need to find a cooperation partner to accomplish complex tasks. Evaluating the trustworthiness of potential partners is vital in partner selection. Current approaches are mainly averaged-based, aggregating advisors’ information on partners. These methods have limitations, such as vulnerability to unfair rating attacks, and may be locally convergent that cannot always select the best partner. Therefore, we propose a ranking-based partner selection (RPS) mechanism, which clusters advisors into groups according to their ranking of trustees and gives recommendations based on groups. Besides, RPS is an online-learning method that can adjust model parameters based on feedback and evaluate the stability of advisors’ ranking behaviours. Experiments demonstrate that RPS performs better than state-of-the-art models in dealing with unfair rating attacks, especially when dishonest advisors are the majority.

Hate speech detection for social media posts is considered as a binary classification problem in existing approaches, largely neglecting distinct attributes of hate speeches from other sentimental types such as “aggressive” and “racist”. As these sentimental types constitute a significant major portion of data, the classification performance is compromised. Moreover, those classifiers often do not generalize well across different datasets due to a relatively small number of hate-class samples. In this paper, we adopt a one-class perspective for hate speech detection, where the detection classifier is trained with hate-class samples only. Our model employs a BERT-BiLSTM module for feature extraction and a one-class SVM for classification. A comprehensive evaluation with four benchmarking datasets demonstrates the better performance of our model than existing approaches, as well as the advantage of training our model with a combination of the four datasets.

High-performance data streaming technologies are increasingly adopted in IT companies to support the integration of heterogeneous and possibly distributed applications. Compared with the traditional message queuing middleware, a streaming platform enables the implementation of event-streaming systems (ESS) which include not only complex queues but also pipelines that transform and react to the streams of data. By analysing the centralised data streams, one can evaluate the Quality-of-Service for other systems and components that produce or consume those streams. We consider the exploitation ofprobabilistic model checkingas a performance monitoring technique for ESS systems. Probabilistic model checking is a mature, powerful verification technique with successful application in performance analysis. However, an ESS system may contain quantitative parameters that are determined by event streams observed in a certain period of time. In this paper, we present a novel theoretical framework called QV4M (meaning “quantitative verification for monitoring”) for monitoring ESS systems, which is based on two recent methods of probabilistic model checking. QV4M assumes the parameters in a probabilistic system model as random variables and infers the statistical significance for the probabilistic model checking output. We also present an empirical evaluation of computational time and data cost for QV4M.

Hand gesture recognition can be exploited to benefit ubiquitous applications using sensors. Currently, the inherent complexity of human physical activities makes it difficult to accurately recognize gestures with wearable sensors, especially in real time. To this end, a real-time hand gesture recognition system is presented in this paper. In particular, sliding window technology and y-axis threshold are used to detect intended gestures from a continuous data stream and then the segmented data are classified by applying a stacked Long Short-Term Memory (LSTM) model. After noise is removed, six-axis sensor data from wrist-worn devices are fed into the model without requiring feature engineering. We use twelve common hand gestures to evaluate the performance of our model. The experimental results demonstrate the feasibility of our proposed system with an accuracy of 99.8% on average. Our approach allows for an accurate and nonindividual hand gesture recognition. It holds potential to be integrated into a smart watch or other wearable devices for intuitive human computer interaction.

Verifying that a stochastic system is in a certain state when it has reached equilibrium has important applications. For instance, the probabilistic verification of the long-run behavior of a safety-critical system enables assessors to check whether it accepts a human abort-command at any time with a probability that is sufficiently high. The stochastic system is represented as probabilistic model, a long-run property is asserted and a probabilistic verifier checks the model against the property.

SQL Injection Vulnerability (SQLIV) has been the top-ranked threat to the Web security consistently for many years. Penetration tests, which are a most widely adopted technique to detect SQLIV, are usually affected by testing inaccuracy. This problem is even worse in inferencebased, blind penetration tests for online Web sites, where Web page variations (such as those caused by inbuilt dynamic modules or user interactions) may lead to a large number of False Positives (FP). We present a novel approach called Inferential Metamorphic Testing (IMT) to reduce FP in SQLIV penetration tests. First, we define the notion of Inferential Metamorphic Relations (IMR), which is inherited from Mutational Metamorphic Testing (MMT). Second, we present a set of logic operators and mutation operators for generating IMR and deducting the background testing context. Finally, we present an iterative IMT process, which is based on the heuristic IMR generation and the background testing context deduction. Our empirical study demonstrates the effectiveness of our approach by a comparison to three famous SQLIV penetration test tools.

Software systems, especially service-based software systems, need to guarantee runtime performance. If their performance is degraded, some reconfiguration countermeasures should be taken. However, there is usually some latency before the countermeasures take effect. It is thus important not only to monitor the current system status passively but also to predict its future performance proactively. Continuous-time Markov chains (CTMCs) are suitable models to analyze time-bounded performance metrics (e.g., how likely a performance degradation may occur within some future period). One challenge to harness CTMCs is the measurement of model parameters (i.e., transition rates) in CTMCs at runtime. As these parameters may be updated by the system or environment frequently, it is difficult for the model builder to provide precise parameter values. In this paper, we present a framework called ProEva, which extends the conventional technique of time-bounded CTMC model checking by admitting imprecise, interval-valued estimates for transition rates. The core method of ProEva computes asymptotic expressions and bounds for the imprecise model checking output. We also present an evaluation of accuracy and computational overhead for ProEva.

Probabilistic model checking is a formal verification technique that has been applied successfully in a variety of domains, providing identification of system errors through quantitative verification of stochastic system models. One domain that can benefit from probabilistic model checking is cloud computing, which must provide highly reliable and secure computational and storage services to large numbers of mission-critical software systems.

Run-time Quality-of-Service (QoS) assurance is crucial for business-critical systems. Complex behavioral performance metrics (PMs) are useful but often difficult to monitor or measure. Probabilistic model checking, especially parametric model checking, can support the computation of aggregate functions for a broad range of those PMs. In practice, those PMs may be defined with parameters determined by run-time data. In this paper, we address the reliability of QoS evaluation using parametric model checking. Due to the imprecision with the instantiation of parameters, an evaluation outcome may mislead the judgment about requirement violations. Based on a general assumption of run-time data distribution, we present a novel framework that contains light-weight statistical inference methods to analyze the reliability of a parametric model checking output with respect to an intuitive criterion. We also present case studies in which we test the stability and accuracy of our inference methods and describe an application of our framework to a cloud server management problem.

Probabilistic model checking is a verification technique that has been the focus of intensive research for over a decade. One important issue with probabilistic model checking, which is crucial for its practical significance but is overlooked by the state-of-the-art largely, is the potential discrepancy between a stochastic model and the real-world system it represents when the model is built from statistical data. In the worst case, a tiny but nontrivial change to some model quantities might lead to misleading or even invalid verification results. To address this issue, in this paper, we present a mathematical characterization of the consequences of model perturbations on the verification distance. The formal model that we adopt is a parametric variant of discrete-time Markov chains equipped with a vector norm to measure the perturbation. Our main technical contributions include a closed-form formulation of asymptotic perturbation bounds, and computational methods for two arguably most useful forms of those bounds, namely linear bounds and quadratic bounds. We focus on verification of reachability properties but also address automata-based verification of omega-regular properties. We present the results of a selection of case studies that demonstrate that asymptotic perturbation bounds can accurately estimate maximum variations of verification results induced by model perturbations.

Probabilistic model checking is a quantitative verification technology for computer systems and has been the focus of intense research for over a decade. While in many circumstances of probabilistic model checking it is reasonable to anticipate a possible discrepancy between a stochastic model and a real-world system it represents, the state-of-the-art provides little account for the effects of this discrepancy on verification results. To address this problem, we present a perturbation approach in which quantities such as transition probabilities in the stochastic model are allowed to be perturbed from their measured values. We present a rigorous mathematical characterization for variations that can occur to verification results in the presence of model perturbations. The formal treatment is based on the analysis of a parametric variant of discrete-time Markov chains, called parametric Markov chains (PMCs), which are equipped with a metric to measure their perturbed vector variables. We employ an asymptotic method from perturbation theory to compute two forms of perturbation bounds, namely condition numbers and quadratic bounds, for automata-based verification of PMCs. We also evaluate our approach with case studies on variant models for three widely studied systems, the Zeroconf protocol, the Leader Election Protocol and the NAND Multiplexer.