Effective agent coordination is crucial in cooperative Multiagent Reinforcement Learning (MARL). While recent advances have significantly improved cooperation by modeling agent interactions through various graph structures, most existing approaches primarily focus on homogeneous agents. Despite the ubiquity of heterogeneous agents, constructing a comprehensive graph that captures their diverse attributes and relationships from scratch is notoriously labor-intensive for both humans and agents, which makes policy learning extremely challenging. To tackle this difficulty, we propose a novel method that utilizes a fuzzy human attention-guided graph to model inter-agent relationships. Instead of learning the graph entirely from scratch, we incorporate abstract human attention, with its uncertainty captured through fuzzy logic, to guide the graph development process. To further accommodate the varying attributes and objectives of heterogeneous agents while maintaining their learning capabilities, the attention-guided graph is fine-tuned through a hyper-network. Our proposed approach is end-to-end trainable and agnostic to specific MARL methods. Empirical evaluations conducted on challenging heterogeneous scenarios from the StarCraft Multiagent Challenge (SMAC) and SMACv2 validate the effectiveness of the proposed method.

Abstract Due to its exceptional learning ability, multi-agent deep reinforcement learning (MADRL) has garnered widespread research interest. However, since the learning is data-driven and involves sampling from millions of steps, training a large number of agents is inherently challenging and inefficient. Inspired by the human learning process, we aim to transfer knowledge from humans to avoid starting from scratch. Given the growing emphasis on the Human-on-the-Loop concept, this study focuses on addressing the challenges of large-population learning by incorporating suboptimal human knowledge into the cooperative multi-agent environment. To leverage human experience, we integrate human knowledge into the training process of MADRL, representing it in natural language rather than specific action-state pairs. Compared to previous works, we further consider the attributes of transferred knowledge to assess its impact on algorithm scalability. Additionally, we examine several features of knowledge mapping to effectively convert human knowledge to the action space where agent learning occurs. In reaction to the disparity in knowledge construction between humans and agents, our approach allows agents to decide freely which portions of the state space to leverage human knowledge. From the challenging domains of the StarCraft Multi-agent Challenge, our method successfully alleviates the scalability issue in MADRL. Furthermore, we find that, despite individual-type knowledge significantly accelerating the training process, cooperative-type knowledge is more desirable for addressing a large agent population. We hope this study provides valuable insights into applying and mapping human knowledge, ultimately enhancing the interpretability of agent behavior.

Most research investigating how social influence affects election results mainly uses diffusion models for binary opinions. However, these diffusion models are progressive and focus on the diffusion of one opinion. In this article, we introduce a general diffusion model for ordinal opinions expressed as linear orderings over a finite set of candidates. We employ agent-based modeling to simulate a nonprogressive diffusion process, allowing multiple types of opinion diffusion about different candidates. The proposed agent-based diffusion model can forecast long-term trends of opinion diffusion in social networks by capturing voters’ personalized features and incorporating dynamic social contexts. Furthermore, we examine the possibility of affecting election outcomes by externally changing the ordinal opinions of certain vertices, i.e., campaigning. Since finding influential voters from the social network is computationally challenging, we propose a heuristic approach, i.e., backward influence rank (BIR). Experimental results demonstrate that the proposed BIR approach is superior to the classic greedy approach for campaigning by achieving a similar margin of victory to that of the greedy approach but running two orders of magnitude faster than the greedy approach did.

Due to the exponential growth of agent interactions and the curse of dimensionality, learning efficient coordination from scratch is inherently challenging in large-scale multi-agent systems. While agents' learning is data-driven, sampling from millions of steps, human learning processes are quite different. Inspired by the concept of Human-on-the-Loop and the daily human hierarchical control, we propose a novel knowledge-guided multi-agent reinforcement learning framework (hhk-MARL), which combines human abstract knowledge with hierarchical reinforcement learning to address the learning difficulties among a large number of agents. In this work, fuzzy logic is applied to represent human suboptimal knowledge, and agents are allowed to freely decide how to leverage the proposed prior knowledge. Additionally, a graph-based group controller is built to enhance agent coordination. The proposed framework is end-to-end and compatible with various existing algorithms. We conduct experiments in challenging domains of the StarCraft Multi-agent Challenge combined with three famous algorithms: IQL, QMIX, and Qatten. The results show that our approach can greatly accelerate the training process and improve the final performance, even based on low-performance human prior knowledge.

Agent negotiation in multi-agent systems has been extensively studied, focusing on both theoretical and applied research. However, a limited number of studies have considered proposing an offer-generating strategy for agents to propose offers during the negotiation process in the multiple-negotiation situation where interdependency exist between a mixture of discrete issues and continuous issues across different negotiations. Especially, considering the above common real-life situation, there is little work of proposing such a strategy which is able to generate an approximately Pareto optimal solution . To address such challenges, this paper targets at multiple-negotiation scenarios involving interdependency between mixed types of issues across different negotiations. The contributions of this paper are threefold. Firstly, this paper addresses the research gap in mixed-type of issues in multiple negotiations. Secondly, the paper introduces a formalized negotiation model for multiple-negotiation scenarios, addressing both discrete and continuous issues, enabling automatic agents to obtain goal-aligned offers effectively. Thirdly, this paper introduces a Hybrid of PSO (Particle Swarm Optimization) and GA (Genetic Algorithm) Algorithm (i.e., named as HPGA in this paper) as an offer-generating strategy to assist agents in achieving approximately Pareto optimization in multiple-negotiation scenarios. To support those claims, this paper presents an overall modeling framework, introduces the proposed offer-generation strategy, conducts a series of experiments to demonstrate the superiority of the proposed approach in this paper, and presents two realistic case studies . Overall, this research expands upon existing studies in agent-based negotiation by addressing the overlooked aspects of mixed types of issues and issue interdependency across multiple negotiations. The proposed modeling approach and offer-generation strategy contribute to the advancement of negotiation techniques in multi-agent systems.

In multi-agent systems, agents with limited capabilities need to find a cooperation partner to accomplish complex tasks. Evaluating the trustworthiness of potential partners is vital in partner selection. Current approaches are mainly averaged-based, aggregating advisors’ information on partners. These methods have limitations, such as vulnerability to unfair rating attacks, and may be locally convergent that cannot always select the best partner. Therefore, we propose a ranking-based partner selection (RPS) mechanism, which clusters advisors into groups according to their ranking of trustees and gives recommendations based on groups. Besides, RPS is an online-learning method that can adjust model parameters based on feedback and evaluate the stability of advisors’ ranking behaviours. Experiments demonstrate that RPS performs better than state-of-the-art models in dealing with unfair rating attacks, especially when dishonest advisors are the majority.

An optimal train schedule enables the railway system to serve maximal passengers with limited resources. However, inappropriate train services may cause a diversity of adverse impacts such as service delay and long waiting times. This study aims to improve train service with a new schedule compared to the existing train schedule. To develop an alternative train service with a schedule solution, a mixed integer linear programming model is developed. A comparison between the existing train service and an alternative service is conducted based on a subset railway network in New South Wales, Australia. Numerical results proved that the alternative train service outperforms the existing one with regard to delay time and train capacity.

Load forecasting has been deeply studied because of its critical role in Smart Grid. In current Smart Grid, there are various types of customers with different energy consumption patterns. Customer's energy consumption patterns are referred to as customer behaviors. It would significantly benefit load forecasting in a grid if customer behaviors could be taken into account. This paper proposes an innovative method that aggregates different types of customers by their identified behaviors, and then predicts the load of each customer cluster, so as to improve load forecasting accuracy of the whole grid. Sparse Continuous Conditional Random Fields (sCCRF) is proposed to effectively identify different customer behaviors through learning. A hierarchical clustering process is then introduced to aggregate customers according to the identified behaviors. Within each customer cluster, a representative sCCRF is fine-tuned to predict the load of its cluster. The final load of the whole grid is obtained by summing the loads of each cluster. The proposed method for load forecasting in Smart Grid has two major advantages. 1) Learning customer behaviors not only improves the prediction accuracy but also has a low computational cost. 2) sCCRF can effectively model the load forecasting problem of one customer, and simultaneously select key features to identify its energy consumption pattern. Experiments conducted from different perspectives demonstrate the advantages of the proposed load forecasting method. Further discussion is provided, indicating that the approach of learning customer behaviors can be extended as a general framework to facilitate decision making in other market domains.

Predictions of time-series are widely used in different disciplines. We propose CoR, Sparse Gaussian Conditional Random Fields (SGCRF) on top of Recurrent Neural Networks (RNN), for problems of this kind. CoR gains advantages from both RNN and SGCRF. It can not only effectively represent the temporal correlations in observed data, but can also learn the structured information of the output. CoR is challenging to train because it is a hybrid of deep neural networks and densely-connected graphical models. Alternative training can be a tractable way to train CoR, and furthermore, an end-to-end training method is proposed to train CoR more efficiently. CoR is evaluated by both synthetic data and real-world data, and it shows a significant improvement in performance over state-of-the-art methods.

Concurrent multiple negotiation (CMN) provides a mechanism for an agent to simultaneously conduct more than one negotiation. There may exist different interdependency relationships among these negotiations and these interdependency relationships can impact the outcomes of these negotiations. The outcomes of these concurrent negotiations contribute together for the agent to achieve an overall negotiation goal. Handling a CMN while considering interdependency relationships among multiple negotiations is a challenging research problem. This paper: 1) comprehensively highlights research problems of negotiations at concurrent negotiation level; 2) provides a graph-based CMN model with consideration of the interdependency relationships; and 3) proposes a colored Petri net-based negotiation protocol for conducting CMNs. With the proposed protocol, a CMN can be efficiently and concurrently processed and negotiation agreements can be efficiently achieved. Experimental results indicate the effectiveness and efficiency of the proposed protocol in terms of the negotiation success rate, the negotiation time and the negotiation outcome.

A broker acts as a middleman between buyers and sellers in the trading processes to achieve its profit as well as to satisfy buyer's requirements based on seller's offers. This paper proposes a broker-based optimal matching approach in the markets. The major contributions of this paper include (1) an abstract model of a broker agent, that is applicable to a broad range of market types, (2) predicting buyers and sellers' behavior by using Bayes' rule so that a broker can identify an appropriate allocation of items between buyers and sellers, and (3) an objective function and a set of constraints to help a broker to maximize its profit under consideration of buyer and seller's total satisfaction. Experimental results demonstrate the good performance of the proposed approach in terms of satisfying buyer's requirements and maximizing broker's profit.

Multiagent learning (MAL) is a promising technique for agents to learn efficient coordinated behaviors in multiagent systems (MASs). In MAL, concurrent multiple distributed learning processes can make the learning environment nonstationary for each individual learner. Developing an efficient learning approach to coordinate agents' behaviors in this dynamic environment is a difficult problem, especially when agents do not know the domain structure and have only local observability of the environment. In this paper, a coordinated MAL approach is proposed to enable agents to learn efficient coordinated behaviors by exploiting agent independence in loosely coupled MASs. The main feature of the proposed approach is to explicitly quantify and dynamically adapt agent independence during learning so that agents can make a trade-off between a single-agent learning process and a coordinated learning process for an efficient decision making. The proposed approach is employed to solve two-robot navigation problems in different scales of domains. Experimental results show that agents using the proposed approach can learn to act in concert or independently in different areas of the environment, which results in great computational savings and near optimal performance.

Social dilemmas have attracted extensive interest in the research of multiagent systems in order to study the emergence of cooperative behaviors among selfish agents. Understanding how agents can achieve cooperation in social dilemmas through learning from local experience is a critical problem that has motivated researchers for decades. This paper investigates the possibility of exploiting emotions in agent learning in order to facilitate the emergence of cooperation in social dilemmas. In particular, the spatial version of social dilemmas is considered to study the impact of local interactions on the emergence of cooperation in the whole system. A double-layered emotional multiagent reinforcement learning framework is proposed to endow agents with internal cognitive and emotional capabilities that can drive these agents to learn cooperative behaviors. Experimental results reveal that various network topologies and agent heterogeneities have significant impacts on agent learning behaviors in the proposed framework, and under certain circumstances, high levels of cooperation can be achieved among the agents.

SUMMARY Multiagent learning provides a promising paradigm to study how autonomous agents learn to achieve coordinated behavior in multiagent systems. In multiagent learning, the concurrency of multiple distributed learning processes makes the environment nonstationary for each individual learner. Developing an efficient learning approach to coordinate agents’ behavior in this dynamic environment is a difficult problem especially when agents do not know the domain structure and at the same time have only local observability of the environment. In this paper, a coordinated learning approach is proposed to enable agents to learn where and how to coordinate their behavior in loosely coupled multiagent systems where the sparse interactions of agents constrain coordination to some specific parts of the environment. In the proposed approach, an agent first collects statistical information to detect those states where coordination is most necessary by considering not only the potential contributions from all the domain states but also the direct causes of the miscoordination in a conflicting state. The agent then learns to coordinate its behavior with others through its local observability of the environment according to different scenarios of state transitions. To handle the uncertainties caused by agents’ local observability, an optimistic estimation mechanism is introduced to guide the learning process of the agents. Empirical studies show that the proposed approach can achieve a better performance by improving the average agent reward compared with an uncoordinated learning approach and by reducing the computational complexity significantly compared with a centralized learning approach. Copyright © 2012 John Wiley & Sons, Ltd.

Social norms such as social rules and conventions play a pivotal role in sustaining system order by regulating and controlling individual behaviors toward a global consensus in large-scale distributed systems. Systematic studies of efficient mechanisms that can facilitate the emergence of social norms enable us to build and design robust distributed systems, such as electronic institutions and norm-governed sensor networks. This paper studies the emergence of social norms via learning from repeated local interactions in networked multiagent systems. A collective learning framework, which imitates the opinion aggregation process in human decision making, is proposed to study the impact of agent local collective behaviors on the emergence of social norms in a number of different situations. In the framework, each agent interacts repeatedly with all of its neighbors. At each step, an agent first takes a best-response action toward each of its neighbors and then combines all of these actions into a final action using ensemble learning methods. Extensive experiments are carried out to evaluate the framework with respect to different network topologies, learning strategies, numbers of actions, influences of nonlearning agents, and so on. Experimental results reveal some significant insights into the manipulation and control of norm emergence in networked multiagent systems achieved through local collective behaviors.

Electronic Commerce has been a very significant commercial phenomenon in recent years, and autonomous agents are widely adopted by business or individuals in electronic marketplaces to fulfill time consuming tasks in trading. Agent negotiation mechan