Negotiating in uncertain environments, where user preferences are only partially known, poses a challenge for traditional negotiation models that rely on rigid, pre-defined strategies. These models struggle to adapt to changing conditions or transfer knowledge across different negotiation contexts, making them ineffective in dynamic environments. To address this research gap, we propose a novel negotiation model that uses deep reinforcement learning (DRL) to enable agents learn adaptable, generalizable strategies through the notion of “strategy templates”. These templates include (a) choice parameters to select tactics, (b) time parameters to control when tactics are activated, and (c) attribute-value parameters to guide acceptance and inform bidding decisions. As a result, we enable negotiation agents dynamically adapt their strategies, through pre-training on teacher strategies and refining them via online learning in diverse environments. Our agents also derive a user model to approximate partially specified user preferences, thus handling preference uncertainty more effectively. We developed a proof-of-concept prototype using an actor-critic architecture based on DRL, supplemented by stochastic search techniques for the estimation of user model and multi-objective optimization for making mutually beneficial offers. Experimental evaluations show that our model outperforms state-of-the-art approaches in terms of both individual and social-welfare utilities, demonstrating its ability to transfer experience across domains and excel in previously unseen scenarios. This work provides a robust framework for dynamic, adaptable strategy formation, bridging the gap in current negotiation models by addressing uncertainty in user preferences and strategy flexibility.

We study the problem of how an agent that negotiates over multiple issues with an opponent can make offers given that it has incomplete information about the user it represents and the opponent it plays against. To tackle this problem, we take a multi-objective optimization stance, where the negotiating agent estimates the preferences of both user and opponent to generate bids that are (near) Pareto-optimal. However, since the negotiating agent needs to approximate the actual preferences of two parties, uncertainty is involved. To handle this uncertainty, we propose a fuzzy approach consisting of a two-phase Pareto-bid generation step where Phase-I generates the non-dominated solutions using a fuzzy multi-objective evolutionary algorithm, and Phase II ranks them to find the best bid to offer the opponent using a fuzzy multiple-criteria decision-making method. Rigorous experimentation shows that the hybrid fuzzy approach of generating the (near) Pareto-optimal bids reduces the average distance to the Pareto curve and increases the average joint or social welfare utility of the agents leading to “win-win” situations.

Abstract We present a novel negotiation model that allows an agent to learn how to negotiate during concurrent bilateral negotiations in unknown and dynamic e-markets. The agent uses an actor-critic architecture with model-free reinforcement learning to learn a strategy expressed as a deep neural network. We pre-train the strategy by supervision from synthetic market data, thereby decreasing the exploration time required for learning during negotiation. As a result, we can build automated agents for concurrent negotiations that can adapt to different e-market settings without the need to be pre-programmed. Our experimental evaluation shows that our deep reinforcement learning based agents outperform two existing well-known negotiation strategies in one-to-many concurrent bilateral negotiations for a range of e-market settings.

We present a novel negotiation model that allows an agent to learn how to negotiate during concurrent bilateral negotiations in unknown and dynamic e-markets. The agent uses an actor-critic architecture with model-free reinforcement learning to learn a strategy expressed as a deep neural network. We pre-train the strategy by supervision from synthetic market data, thereby decreasing the exploration time required for learning during negotiation. As a result, we can build automated agents for concurrent negotiations that can adapt to different e-market settings without the need to be pre-programmed. Our experimental evaluation shows that our deep reinforcement learning based agents outperform two existing well-known negotiation strategies in one-to-many concurrent bilateral negotiations for a range of e-market settings.