The goal of the Inverse reinforcement learning (IRL) task is to identify the underlying reward function and the corresponding optimal policy from a set of expert demonstrations. While most IRL algorithms’ theoretical guarantees rely on a linear reward structure, we aim to extend the theoretical understanding of IRL to scenarios where the reward function is parameterized by neural networks. Meanwhile, conventional IRL algorithms usually adopt a nested structure, leading to computational inefficiency, especially in high-dimensional settings. To address this problem, we propose the first two-timescale single-loop IRL algorithm under neural network parameterized reward and provide a non-asymptotic convergence analysis under overparameterization. Although prior optimality results for linear rewards do not apply, we show that our algorithm can identify the globally optimal reward and policy under certain neural network structures. This is the first IRL algorithm with a non-asymptotic convergence guarantee that provably achieves global optimality in neural network settings.

Aligning to human preferences and/or intentions is an important requirement for contemporary foundation models. To ensure alignment, popular approaches such as reinforcement learning with human feedback (RLHF) break down the task into three stages: (i) a model is computed with supervised fine-tuning (SFT) based upon large demonstrations data, (ii) a reward model (RM) is estimated based upon human feedback data, and (iii) reinforcement learning (RL) is used to further refine the SFT model by optimizing the estimated reward model. Demonstrations and human feedback data reflect human user preferences in different ways. As a result, the reward model estimate obtained from only human feedback data is likely not as accurate as a reward model estimate obtained from both demonstration and human feedback data. A policy model that optimizes the reward model estimate obtained from both demonstration and human feedback data will likely exhibit better alignment performance. We introduce a tractable algorithm for finding the reward and policy models and provide a finite-time performance guarantee. Additionally, we demonstrate the efficiency of the proposed solution with extensive experiments including alignment problems in LLMs and robotic control problems in MuJoCo. We observe that the proposed solutions outperform the existing alignment algorithm by large margins, especially when the amounts of demonstration and preference data are unbalanced.

In this paper we introduce a general estimation methodology for learning a model of human perception and control in a sensorimotor control task based upon a finite set of demonstrations. The model’s structure consists of (i) the agent’s internal representation of how the environment and associated observations evolve as a result of control actions and (ii) the agent’s preferences over observable outcomes. We consider a model’s structure specification consistent with active inference, a theory of human perception and behavior from cognitive science. According to active inference, the agent acts upon the world so as to minimize surprise defined as a measure of the extent to which an agent’s current sensory observations differ from its preferred sensory observations. We propose a bi-level optimization approach to estimation which relies on a structural assumption on prior distributions that parameterize the statistical accuracy of the human agent’s model of the environment. To illustrate the proposed methodology, we present the estimation of a model for car-following behavior based upon a naturalistic dataset. Overall, the results indicate that learning active inference models of human perception and control from data is a promising alternative to closed-box models of driving.

We consider a Distributed Federated multitask learning scheme that accounts for multiple linear model estimation tasks with heterogeneous and/or correlated data streams. We assume that nodes can be partitioned into groups corresponding to different learning tasks and communicate according to a directed network topology. Each node estimates a linear model asynchronously and is subject to local (within-group) regularization and global (across groups) regularization terms targeting noise reduction and generalization performance improvement, respectively. We provide a finite-time characterization of the convergence of the estimators and task relation and illustrate the scheme’s general applicability of random field temperature estimation.

Aligning human preference and value is an important requirement for contemporary foundation models. State-of-the-art techniques such as Reinforcement Learning from Human Feedback (RLHF) often consist of two stages: 1) supervised fine-tuning (SFT), where the model is fine-tuned by learning from human demonstration data; 2) Preference learning, where preference data is used to learn a reward model, which is in turn used by a reinforcement learning (RL) step to fine-tune the model. Such reward model serves as a proxy to human preference, and it is critical to guide the RL step towards improving the model quality. In this work, we argue that the SFT stage significantly benefits from learning a reward model as well. Instead of using the human demonstration data directly via supervised learning, we propose to leverage an Inverse Reinforcement Learning (IRL) technique to {\it simultaneously} build an reward model and a policy model. This approach leads to new SFT algorithms that are not only efficient to implement, but are robust to the presence of low-quality supervised learning data. Moreover, we discover a connection between the proposed IRL based approach, and a recent line of works called Self-Play Fine-tune (SPIN, \cite{chen2024self}). Theoretically, we show that the proposed algorithms converge to the stationary solutions of the IRL problem. Empirically, we align 1B and 7B models using proposed methods and evaluate them on a reward benchmark model and the HuggingFace Open LLM Leaderboard. The proposed methods show significant performance improvement over existing SFT approaches. Our results indicate that it is beneficial to leverage reward learning throughout the entire alignment process. Our code is available at \url{https://github.com/JasonJiaxiangLi/Reward_learning_SFT}.

Offline inverse reinforcement learning (Offline IRL) aims to recover the structure of rewards and environment dynamics that underlie observed actions in a fixed, finite set of demonstrations from an expert agent. Accurate models of expertise in executing a task has applications in safety-sensitive applications such as clinical decision making and autonomous driving. However, the structure of an expert's preferences implicit in observed actions is closely linked to the expert's model of the environment dynamics (i.e. the ``world''). Thus, inaccurate models of the world obtained from finite data with limited coverage could compound inaccuracy in estimated rewards. To address this issue, we propose a bi-level optimization formulation of the estimation task wherein the upper level is likelihood maximization based upon a conservative model of the expert's policy (lower level). The policy model is conservative in that it maximizes reward subject to a penalty that is increasing in the uncertainty of the estimated model of the world. We propose a new algorithmic framework to solve the bi-level optimization problem formulation and provide statistical and computational guarantees of performance for the associated optimal reward estimator. Finally, we demonstrate that the proposed algorithm outperforms the state-of-the-art offline IRL and imitation learning benchmarks by a large margin, over the continuous control tasks in MuJoCo and different datasets in the D4RL benchmark.

Inverse reinforcement learning (IRL) aims to recover the reward function and the associated optimal policy that best fits observed sequences of states and actions implemented by an expert. Many algorithms for IRL have an inherent nested structure: the inner loop finds the optimal policy given parametrized rewards while the outer loop updates the estimates towards optimizing a measure of fit. For high dimensional environments such nested-loop structure entails a significant computational burden. To reduce the computational burden of a nested loop, novel methods such as SQIL \cite{reddy2019sqil} and IQ-Learn \cite{garg2021iq} emphasize policy estimation at the expense of reward estimation accuracy. However, without accurate estimated rewards, it is not possible to do counterfactual analysis such as predicting the optimal policy under different environment dynamics and/or learning new tasks. In this paper we develop a novel {\em single-loop} algorithm for IRL that does not compromise reward estimation accuracy. In the proposed algorithm, each policy improvement step is followed by a stochastic gradient step for likelihood maximization. We show that the proposed algorithm provably converges to a stationary solution with a finite-time guarantee. If the reward is parameterized linearly we show the identified solution corresponds to the solution of the maximum entropy IRL problem. Finally, by using robotics control problems in Mujoco and their transfer settings, we show that the proposed algorithm achieves superior performance compared with other IRL and imitation learning benchmarks.

Automated vehicle (AV) technologies promise to improve traffic safety and reduce driver workload in the near future. However, most current implementations have limited capabilities and require transition of control between the vehicle and the human driver during automation failures. For this reason, models of driver behavior have been widely studied to assist the design and development of AV technologies. Recent works have shown that driver behavior models grounded in human cognitive information processing achieve better generalization than prior methods. In this work, we applied active inference, a framework of human perception, cognition, and behavior based on the predictive processing theory, to model driver emergency braking responses to automation failures. We estimated the model parameters from experimental data and examined the model parameters using a factor analysis. We verified the model’s braking response prediction capability in counterfactual scenarios. Our results show that the model effectively captured braking reaction times and provided insight on the correspondence between the variations in driver parameters and behavior.

We consider a distributed non-convex optimization where a network of agents aims at minimizing a global function over the Stiefel manifold. The global function is represented as a finite sum of smooth local functions, where each local function is associated with one agent and agents communicate with each other over an undirected connected graph. The problem is non-convex as local functions are possibly non-convex (but smooth) and the Steifel manifold is a non-convex set. We present a decentralized Riemannian stochastic gradient method (DRSGD) with the convergence rate of $\mathcal{O}(1/\sqrt{K})$ to a stationary point. To have exact convergence with constant stepsize, we also propose a decentralized Riemannian gradient tracking algorithm (DRGTA) with the convergence rate of $\mathcal{O}(1/K)$ to a stationary point. We use multi-step consensus to preserve the iteration in the local (consensus) region. DRGTA is the first decentralized algorithm with exact convergence for distributed optimization on Stiefel manifold.

Aeronautical communications are expected to increasingly demand spectrum in the next years. Given the endemic scarcity of spectrum, both the opportunistic use of licensed bands and carrier aggregation enable aeronautical systems to accommodate higher data rates and more aircrafts. We study dynamic spectrum allocation for air-to-ground (A2G) transmissions when carriers can be aggregated to form larger channels with complete and incomplete information: the aircrafts might not reveal their available channels for security reasons as it would expose their location. The single and multiple ground station scenarios are analyzed. Semi-distributed algorithms are proposed to deal with the incomplete information case for both scenarios, exhibiting a very good tradeoff between performance and complexity.

This paper revisits the distributed adaptive control problem for synchronization of multiagent systems where the dynamics of the agents are nonlinear, nonidentical, unknown, and subject to external disturbances. Two communication topologies, represented, respectively, by a fixed strongly-connected directed graph and by a switching connected undirected graph, are considered. Under both of these communication topologies, we use distributed neural networks to approximate the uncertain dynamics. Decentralized adaptive control protocols are then constructed to solve the cooperative tracker problem, the problem of synchronization of all follower agents to a leader agent. In particular, we show that, under the proposed decentralized control protocols, the synchronization errors are ultimately bounded, and their ultimate bounds can be reduced arbitrarily by choosing the control parameter appropriately. Simulation study verifies the effectiveness of our proposed protocols.

We consider the downlink coordinated beamforming problem in a cellular network in which the base stations (BSs) are equipped with multiple antennas and each user is equipped with a single antenna. The BSs cooperate in sharing their local interference information, and they aim to maximize the sum-rate of the users in the network. A decentralized interference pricing beamforming (IPBF) algorithm is proposed to identify the coordinated beamformer, where a BS is penalized according to the interference it creates to its peers. We show that the decentralized pricing mechanism converges to an interference equilibrium, which is a KKT point of the sum-rate maximization problem. The proofs rely on the identification of rank-1 solutions of each BSs' interference-penalized rate maximization problem. Numerical results show that the proposed iterative mechanism reduces significantly the exchanged information with respect to other state-of-the-art beamforming algorithms with very little sum-rate loss. The version of the algorithm that limits the coordination to a cluster of base stations (IPBF-L) is shown to have very small sum-rate loss with respect to the full coordinated algorithm with much less backhaul information exchange.

This paper introduces Truthful Multichannel Auction (TMCA), an auction design for the allocation of wireless channels to several bidders with private information about their channel valuation. Channel allocations are subject to interference constraints in the form of a conflict graph. In contrast to other channel auctions, TMCA allows for variable (instead of fixed) marginal valuations. In TMCA, it is a dominant strategy for bidders to truthfully reveal their channel valuations, which in turn guarantees the implementation of highly efficient allocations in polynomial time. This paper also shows the expected revenue of the auctioneer can be maximized by imposing a reserve price.

We consider a resource management problem in a multi-cell downlink OFDMA network whereby the goal is to find the optimal combination of (i) assignment of users to base stations and (ii) resource allocation strategies at each base station. Efficient resource management protocols must rely on users truthfully reporting privately held information such as downlink channel states. However, individual users can manipulate the resulting resource allocation (by misreporting their private information) if by doing so they can improve their payoff. Therefore, it is of interest to design efficient resource management protocols that are strategy-proof, i.e. it is in the users' best interests to truthfully report their private information. Unfortunately, we show that the implementation of any protocol that is efficient and strategy-proof is NP-hard. Thus, we propose a computationally tractable strategy-proof mechanism that is approximately efficient, i.e. the solution obtained yields at least 1/2 of the optimal throughput. Simulations are provided to illustrate the effectiveness of the proposed mechanism.

Spectrum management has been identified as a crucial step towards enabling the technology of the cognitive radio network (CRN). Most of the current works dealing with spectrum management in the CRN focus on a single task of the problem, e.g., spectrum sensing, spectrum decision, spectrum sharing or spectrum mobility. In this work, we argue that for certain network configurations, jointly performing several tasks of the spectrum management improves the spectrum efficiency. Specifically, we study the uplink resource management problem in a CRN where there exist multiple cognitive users (CUs) and access points (APs), with each AP operates on a set of non-overlapping channels. The CUs, in order to maximize their uplink transmission rates, have to associate to a suitable AP (spectrum decision), and to share the channels belong to this AP with other CUs (spectrum sharing). These tasks are clearly interdependent, and the problem of how they should be carried out efficiently and distributedly is still open in the literature. In this work we formulate this joint spectrum decision and spectrum sharing problem into a non-cooperative game, in which the feasible strategy of a player contains a discrete variable and a continuous vector. The structure of the game is hence very different from most non-cooperative spectrum management game proposed in the literature. We provide characterization of the Nash Equilibrium (NE) of this game, and present a set of novel algorithms that allow the CUs to distributively and efficiently select the suitable AP and share the channels with other CUs. Finally, we study the properties of the proposed algorithms as well as their performance via extensive simulations.

In this paper, we consider a market in which a finite number of firms compete in prices for the incoming demand for service. Upon every customer arrival, an independent auctioneer gathers bids from each one of the competing queuing systems and assigns the incoming customer to the system that submitted the lowest bid. We provide a simple characterization of Markov Perfect equilibrium in terms of ldquoindifference prices,rdquo i.e., price levels at which players are indifferent between committing available capacity or withholding it. We identify sufficient conditions for socially efficient performance in equilibrium.

In this paper, we outline the basis for a new distributed Bayesian search scheme in which all Bayesian decision makers recognize the same performance objective but do not possess the ability to communicate with each other. Coordination among the players is achieved indirectly by tracking search performance. This scheme is ideal for unmanned vehicles searching in extreme environments where extensive bilateral communication between actuators is not feasible. Preliminary results suggest our new search scheme is scalable and can easily be adapted for tracking moving targets.