Among various value decomposition-based multiagent reinforcement learning (MARL) algorithms, the overall performance of the multiagent system is represented by a scalar global Q value and optimized by minimizing the temporal difference (TD) error with respect to that global Q value. However, the global Q value cannot accurately model the distributed dynamics of the multiagent system, since it is only a simplified representation for different individual Q values of agents. To explicitly consider the correlations between different cooperative agents, in this article, we propose a distributional framework and construct a practical model called distributional multiagent cooperation (DMAC) from a novel distributional perspective. Specifically, in DMAC, we view the individual Q value for the executed action of a random agent as a value distribution, whose expectation can further represent the overall performance. Then, we employ distributional RL to minimize the difference between the estimated distribution and its target for the optimization. The advantage of DMAC is that the distributed dynamics of agents can be explicitly modeled, and this results in better performance. To verify the effectiveness of DMAC, we conduct extensive experiments under nine different scenarios of the StarCraft Multiagent Challenge (SMAC). Experimental results show that the DMAC can significantly outperform the baselines with respect to the average median test win rate.

The RecSys 2023 Challenge involved a conversion prediction task in the online advertising space. The dataset was provided by ShareChat (Mohalla Tech Pvt Ltd). The challenge data represents a sample of ad impressions served to the users over a period of 22 days and the task is for a given ad impression, to predict a conversion (install an app) will happen or not. The challenge ran for 3 months with a public dashboard. There were 519 teams registered and 231 teams made at least one submission. The task setting represents an important research area of modeling ad recommendations under user privacy. We identify interesting themes in feature engineering, addressing sparsity and calibrating across multi-step predictions.

Deep reinforcement learning (DRL) based recommender systems are suitable for user cold-start problems as they can capture user preferences progressively. However, most existing DRL-based recommender systems are suboptimal, since they use the same policy to suit the dynamics of different users. We reformulate recommendation as a multitask Markov Decision Process, where each task represents a set of similar users. Since similar users have closer dynamics, a task-specific policy is more effective than a single universal policy for all users. To make recommendations for cold-start users, we use a default policy to collect some initial interactions to identify the user task, after which a task-specific policy is employed. We use Q-learning to optimize our framework and consider the task uncertainty by the mutual information regarding tasks. Experiments are conducted on three real-world datasets to verify the effectiveness of our proposed framework.

Abstract Real-world sequential decision-making tasks are generally complex, requiring trade-offs between multiple, often conflicting, objectives. Despite this, the majority of research in reinforcement learning and decision-theoretic planning either assumes only a single objective, or that multiple objectives can be adequately handled via a simple linear combination. Such approaches may oversimplify the underlying problem and hence produce suboptimal results. This paper serves as a guide to the application of multi-objective methods to difficult problems, and is aimed at researchers who are already familiar with single-objective reinforcement learning and planning methods who wish to adopt a multi-objective perspective on their research, as well as practitioners who encounter multi-objective decision problems in practice. It identifies the factors that may influence the nature of the desired solution, and illustrates by example how these influence the design of multi-objective decision-making systems for complex problems.

Deep reinforcement learning (DRL)-based recommender systems have recently come into the limelight due to their ability to optimize long-term user engagement. A significant challenge in DRL-based recommender systems is the large action space required to represent a variety of items. The large action space weakens the sampling efficiency and thereby, affects the recommendation accuracy. In this article, we propose a DRL-based method called deep hierarchical category-based recommender system (DHCRS) to handle the large action space problem. In DHCRS, categories of items are used to reconstruct the original flat action space into a two-level category-item hierarchy. DHCRS uses two deep Q -networks (DQNs): 1) a high-level DQN for selecting a category and 2) a low-level DQN to choose an item in this category for the recommendation. Hence, the action space of each DQN is significantly reduced. Furthermore, the categorization of items helps capture the users' preferences more effectively. We also propose a bidirectional category selection (BCS) technique, which explicitly considers the category-item relationships. The experiments show that DHCRS can significantly outperform state-of-the-art methods in terms of hit rate and normalized discounted cumulative gain for long-term recommendations.

In recent years, Reinforcement Learning (RL), has become a popular field of study as well as a tool for enterprises working on cutting-edge artificial intelligence research. To this end, many researchers have built RL frameworks such as openAI Gym, and KerasRL for ease of use. While these works have made great strides towards bringing down the barrier of entry for those new to RL, we propose a much simpler framework called EasyRL, by providing an interactive graphical user interface for users to train and evaluate RL agents. As it is entirely graphical, EasyRL does not require programming knowledge for training and testing simple built-in RL agents. EasyRL also supports custom RL agents and environments, which can be highly beneficial for RL researchers in evaluating and comparing their RL models.

In an ad hoc teamwork setting, the team needs to coordinate their activities to perform a task without prior agreement on how to achieve it. The ad hoc agent cannot communicate with its teammates but it can observe their behaviour and plan accordingly. To do so, the existing approaches rely on the teammates' behaviour models. However, the models may not be accurate, which can compromise teamwork. For this reason, we present Ad Hoc Teamwork by Sub-task Inference and Selection (ATSIS) algorithm that uses a sub-task inference without relying on teammates' models. First, the ad hoc agent observes its teammates to infer which sub-tasks they are handling. Based on that, it selects its own sub-task using a partially observable Markov decision process that handles the uncertainty of the sub-task inference. Last, the ad hoc agent uses the Monte Carlo tree search to find the set of actions to perform the sub-task. Our experiments show the benefits of ATSIS for robust teamwork.

Online reviews have become a vital source of information in purchasing a service (product). Opinion spammers manipulate reviews, affecting the overall perception of the service. A key challenge in detecting opinion spam is obtaining ground truth. Though there exists a large set of reviews, only a few of them have been labeled spam or non-spam. We propose spamGAN, a generative adversarial network which relies on limited labeled data as well as unlabeled data for opinion spam detection. spamGAN improves the state-of-the-art GAN based techniques for text classification. Experiments on TripAdvisor data show that spamGAN outperforms existing techniques when labeled data is limited. spamGAN can also generate reviews with reasonable perplexity.

Malicious vehicle agents broadcast fake information about traffic events and thereby undermine the benefits of vehicle-to-vehicle communication in vehicular ad-hoc networks (VANETs). Trust management schemes addressing this issue do not focus on effective/fast decision making in reacting to traffic events. We propose a Partially Observable Markov Decision Process (POMDP) based approach to balance the trade-off between information gathering and exploiting actions resulting in faster responses. Our model copes with malicious behavior by maintaining it as part of a small state space, thus is scalable for large VANETs. We also propose an algorithm to learn model parameters in a dynamic behavior setting. Experimental results demonstrate that our model can effectively balance the decision quality and response time while still being robust to sophisticated malicious attacks.

In multiagent e-marketplaces, buying agents need to select good sellers by querying other buyers (called advisors). Partially Observable Markov Decision Processes (POMDPs) have shown to be an effective framework for optimally selecting sellers by selectively querying advisors. However, current solution methods do not scale to hundreds or even tens of agents operating in the e-market. In this paper, we propose the Mixture of POMDP Experts (MOPE) technique, which exploits the inherent structure of trust-based domains, such as the seller selection problem in e-markets, by aggregating the solutions of smaller sub-POMDPs. We propose a number of variants of the MOPE approach that we analyze theoretically and empirically. Experiments show that MOPE can scale up to a hundred agents thereby leveraging the presence of more advisors to significantly improve buyer satisfaction.

The performance of trust models highly depend on the characteristics of the environments where they are applied. Thus, it becomes challenging to choose a suitable trust model for a given e-marketplace environment, especially when ground truth about the agent (buyer and seller) behavior is unknown (called unknown environment). We propose a case-based reasoning framework to choose suitable trust models for unknown environments, based on the intuition that if a trust model performs well in one environment, it will do so in another similar environment. Firstly, we build a case base with a number of simulated environments (with known ground truth) along with the trust models most suitable for each of them. Given an unknown environment, case-based retrieval algorithms retrieve the most similar case(s), and the trust model of the most similar case(s) is chosen as the most suitable model for the unknown environment. Evaluation results confirm the effectiveness of our framework in choosing suitable trust models for different e-marketplace environments.

Watchdog technique is a fundamental building block to many trust systems that are designed for securing wireless sensor networks (WSNs). Unfortunately, this kind of technique consumes much energy and hence largely limits the lifespan of WSN. Although the state-of-the-art studies have realized the importance of trust systems' efficiency in WSNs and proposed several preliminary solutions, they have overlooked to optimize the watchdog technique, which is perhaps among the top energy-consuming units. In this paper, we reveal the inefficient use of watchdog technique in existing trust systems, and thereby propose a suite of optimization methods to minimize the energy cost of watchdog usage, while keeping the system's security in a sufficient level. Our contributions consist of theoretical analyses and practical algorithms, which can efficiently and effectively schedule the watchdog tasks depending on the sensor nodes' locations and the target nodes' trustworthiness. We have evaluated our algorithms through experiments on top of a WSNET simulation platform and an in-door WSN testbed in our collaborative lab. The results have successfully confirmed that our watchdog optimization techniques can save at least 39.44% energy without sacrificing much security (<;0.06 in terms of trust accuracy and robustness), even in some cases enhance the protection against certain attacks.

Reputation systems are highly prone to unfair rating attacks. Though many approaches for detecting unfair ratings have been proposed so far, their performance is often affected by the environment where they are applied. For a given unknown real environment, it is difficult to choose the most suitable approach for detecting unfair ratings as the ground truth data necessary to evaluate the accuracy of the detection approaches remains unknown. In this paper, we propose a novel Context-AwaRE (CARE) framework, to choose the most suitable unfair rating detection approach for a given unknown real environment. The framework first identifies simulated environments, closely similar to that of the unknown environment. The detection approaches performing well in the most similar simulated environments are then chosen as the suitable ones for the unknown real environment. Detailed experiments illustrate that the CARE framework can choose the most suitable detection approach to accurately distinguish fair and unfair ratings for any given unknown environment.