The rapidly evolving landscape of products, surfaces, policies, and regulations poses significant challenges for deploying state-of-the-art recommendation models at industry scale, primarily due to data fragmentation across domains and escalating infrastructure costs that hinder sustained quality improvements.

Autonomous systems in VEC-enabled ICVs face many challenges, such as self-organization, privacy breach risks, vehicle selection, and resource allocation. As a distributed training framework, Federated Meta-Learning (FML) provides a powerful tool for adaptive and efficient processing of vehicular tasks while securing vehicle data privacy in VEC-enabled ICVs. However, the high-speed mobility of vehicles leads to higher latency and communication interruptions. This article investigates the vehicle selection and resource allocation scheme, subject to the constraints on the number and the residence time of vehicles, the maximum transmission energy consumption, and the ratio of bandwidth resource allocation. It is proved to be a challenging mixed-integer nonlinear programming problem, and we formulate it as a Markov decision process (MDP). We proposed an adaptive Sum Tree-Deep Recurrent Q-network algorithm (ST-DRQN) to solve the optimal resource allocation. ST-DRQN employs an enhanced empirical selection rule and a proportional priority sampling method to address the problems of inefficient model training and slow convergence. Finally, we conducted experiments using intelligent cars equipped with Raspberry Pi to show the effectiveness of the proposed methodology. Experimental results demonstrate that ST-DRQN achieves adaptability and credibility among ICVs while reducing latency and energy costs incurred by long-term training of FML.

Equivariant neural networks have shown great success in reinforcement learning, improving sample efficiency and generalization when there is symmetry in the task. However, in many problems, only approximate symmetry is present, which makes imposing exact symmetry inappropriate. Recently, approximately equivariant networks have been proposed for supervised classification and modeling physical systems. In this work, we develop approximately equivariant algorithms in reinforcement learning (RL). We define approximately equivariant MDPs and theoretically characterize the effect of approximate equivariance on the optimal Q function. We propose novel RL architectures using relaxed group and steerable convolutions and experiment on several continuous control domains and stock trading with real financial data. Our results demonstrate that the approximately equivariant network performs on par with exactly equivariant networks when exact symmetries are present, and outperforms them when the domains exhibit approximate symmetry. As an added byproduct of these techniques, we observe increased robustness to noise at test time. Our code is available at \url{https://github.com/jypark0/approx_equiv_rl.}

We consider discrete-time stationary mean field games (MFG) with unknown dynamics and design algorithms for finding the equilibrium with finite-time complexity guarantees. Prior solutions to the problem assume either the contraction of a mean field optimality-consistency operator or strict weak monotonicity, which may be overly restrictive. In this work, we introduce a new class of solvable MFGs, named the "fully herding class", which expands the known solvable class of MFGs and for the first time includes problems with multiple equilibria. We propose a direct policy optimization method, Accelerated Single-loop Actor Critic Algorithm for Mean Field Games (ASAC-MFG), that provably finds a global equilibrium for MFGs within this class, under suitable access to a single trajectory of Markovian samples. Different from the prior methods, ASAC-MFG is single-loop and single-sample-path. We establish the finite-time and finite-sample convergence of ASAC-MFG to a mean field equilibrium via new techniques that we develop for multi-time-scale stochastic approximation. We support the theoretical results with illustrative numerical simulations. When the mean field does not affect the transition and reward, a MFG reduces to a Markov decision process (MDP) and ASAC-MFG becomes an actor-critic algorithm for finding the optimal policy in average-reward MDPs, with a sample complexity matching the state-of-the-art. Previous works derive the complexity assuming a contraction on the Bellman operator, which is invalid for average-reward MDPs. We match the rate while removing the untenable assumption through an improved Lyapunov function.

The standard contextual bandit framework assumes fully observable and actionable contexts. In this work, we consider a new bandit setting with partially observable and correlated contexts, with linear payoffs, motivated by the applications in finance where decision making is based on market information that typically displays temporal correlation and is not fully observed. We make the following contributions marrying ideas from statistical signal processing with bandits: (i) We propose an algorithmic pipeline named EMKF-Bandit, which integrates system identification, filtering, and classic contextual bandit algorithms into an iterative method alternating between latent parameter estimation and decision making. (ii) We analyze EMKF-Bandit when we select Thompson sampling as the bandit algorithm and show that it incurs a sub-linear regret under conditions on filtering. (iii) We conduct numerical simulations that demonstrate the benefits and practical applicability of the proposed pipeline.

We study policy optimization in Stackelberg mean field games (MFGs), a hierarchical framework for modeling the strategic interaction between a single leader and an infinitely large population of homogeneous followers. The objective can be formulated as a structured bi-level optimization problem, in which the leader needs to learn a policy maximizing its reward, anticipating the response of the followers. Existing methods for solving these (and related) problems often rely on restrictive independence assumptions between the leader’s and followers’ objectives, use samples inefficiently due to nested-loop algorithm structure, and lack finite-time convergence guarantees. To address these limitations, we propose AC-SMFG, a single-loop actor-critic algorithm that operates on continuously generated Markovian samples. The algorithm alternates between (semi-)gradient updates for the leader, a representative follower, and the mean field, and is simple to implement in practice. We establish the finite-time and finite-sample convergence of the algorithm to a stationary point of the Stackelberg objective. To our knowledge, this is the first Stackelberg MFG algorithm with non-asymptotic convergence guarantees. Our key assumption is a "gradient alignment" condition, which requires that the full policy gradient of the leader can be approximated by a partial component of it, relaxing the existing leader-follower independence assumption. Simulation results in a range of well-established economics environments demonstrate that AC-SMFG outperforms existing multi-agent and MFG learning baselines in policy quality and convergence speed.

The development of novel applications causes increased demands on the computational capabilities of Vehicular Edge Computing (VEC). Current works have introduced Unmanned Aerial Vehicles (UAVs) into VEC to solve the resource-constrained problem. However, given the limited storage of UAVs, the key question is to design the offloading strategy and determine which service programs should be cached. In this article, we propose a three-stage game model that aims at providing a precise analysis of the interaction among the Base Station (BS), the UAV, and the User Vehicle (UV). In stage I, the BS is responsible for determining the cache strategy of the UAV and communicating the price strategy to the UVs. In stage II, the UAV communicates the price strategy to the UVs. In stage III, each UV determines its offloading decision based on the price strategy, to minimize the task execution delay and cost. Compared with current approaches, we cache the frequently requested services in the UAV to satisfy the real-time requirements and use game theory to solve the decision-making, which achieves the effect of reducing the delay and cost. The experiment results are performed to assess the convergence and effectiveness of the proposed algorithm.

Two-time-scale optimization is a framework introduced in Zeng et al. (2024) that abstracts a range of policy evaluation and policy optimization problems in reinforcement learning (RL). Akin to bi-level optimization under a particular type of stochastic oracle, the two-time-scale optimization framework has an upper level objective whose gradient evaluation depends on the solution of a lower level problem, which is to find the root of a strongly monotone operator. In this work, we propose a new method for solving two-time-scale optimization that achieves significantly faster convergence than the prior arts. The key idea of our approach is to leverage an averaging step to improve the estimates of the operators in both lower and upper levels before using them to update the decision variables. These additional averaging steps eliminate the direct coupling between the main variables, enabling the accelerated performance of our algorithm. We characterize the finite-time convergence rates of the proposed algorithm under various conditions of the underlying objective function, including strong convexity, convexity, Polyak-Lojasiewicz condition, and general non-convexity. These rates significantly improve over the best-known complexity of the standard two-time-scale stochastic approximation algorithm. When applied to RL, we show how the proposed algorithm specializes to novel online sample-based methods that surpass or match the performance of the existing state of the art. Finally, we support our theoretical results with numerical simulations in RL.

The aim of this paper is to improve the understanding of the optimization landscape for policy optimization problems in reinforcement learning. Specifically, we show that the superlevel set of the objective function with respect to the policy parameter is always a connected set both in the tabular setting and under policies represented by a class of neural networks. In addition, we show that the optimization objective as a function of the policy parameter and reward satisfies a stronger “equiconnectedness” property. To our best knowledge, these are novel and previously unknown discoveries. We present an application of the connectedness of these superlevel sets to the derivation of minimax theorems for robust reinforcement learning. We show that any minimax optimization program which is convex on one side and is equiconnected on the other side observes the minimax equality (i.e. has a Nash equilibrium). We find that this exact structure is exhibited by an interesting class of robust reinforcement learning problems under an adversarial reward attack, and the validity of its minimax equality immediately follows. This is the first time such a result is established in the literature.

We study the problem of finding the Nash equilibrium in a two-player zero-sum Markov game. Due to its formulation as a minimax optimization program, a natural approach to solve the problem is to perform gradient descent/ascent with respect to each player in an alternating fashion. However, due to the non-convexity/non-concavity of the underlying objective function, theoretical understandings of this method are limited. In our paper, we consider solving an entropy-regularized variant of the Markov game. The regularization introduces structures into the optimization landscape that make the solutions more identifiable and allow the problem to be solved more efficiently. Our main contribution is to show that under proper choices of the regularization parameter, the gradient descent ascent algorithm converges to the Nash equilibrium of the original unregularized problem. We explicitly characterize the finite-time performance of the last iterate of our algorithm, which vastly improves over the existing convergence bound of the gradient descent ascent algorithm without regularization. Finally, we complement the analysis with numerical simulations that illustrate the accelerated convergence of the algorithm.

We develop a mathematical framework for solving multi-task reinforcement learning (MTRL) problems based on a type of policy gradient method. The goal in MTRL is to learn a common policy that operates effectively in different environments; these environments have similar (or overlapping) state spaces, but have different rewards and dynamics. We highlight two fundamental challenges in MTRL that are not present in its single task counterpart, and illustrate them with simple examples. We then develop a decentralized entropyregularized policy gradient method for solving the MTRL problem, and study its finite-time convergence rate. We demonstrate the effectiveness of the proposed method using a series of numerical experiments. These experiments range from small-scale "GridWorld" problems that readily demonstrate the trade-offs involved in multi-task learning to large-scale problems, where common policies are learned to navigate an airborne drone in multiple (simulated) environments.

In the modern information society, analysis of human mobility becomes increasingly essential in various areas such as city planning and resource management. With users' historical trajectories, the inherent patterns of their movements can be extracted and utilized to accurately predict the future movements. Plenty of previous work adopted traditional Markov model, which suffers when the trajectory becomes sparse or it shows distinct mobility patterns in different time of day. In this paper, based on an app-collected dataset of 100,000 individuals' actively uploaded location information, we comprehensively analyze the mobility and predictability of each user. To approach the theoretical predictability and overcome the shortcomings of traditional Markov model, we propose a time-variant Markov model based on Gibbs sampling for mobility prediction. Specifically, we model human mobility as several interconnected Markov chains, each chain corresponds to a movement pattern of a period of time. Then, we adopt Gibbs sampling method to simultaneously recover the missing part of trajectories and train the Markov chains, in order to solve the unevenly distribution and the high missing rate. Results show that our prediction algorithm can achieve 11.2 percent higher prediction accuracy than the benchmark method, especially on sparse trajectories. In addition, we discover a high correlation between prediction accuracy and predictability, with correlation coefficient reaching 0.81. Finally, we investigate various factors including spatial and temporal resolution, orders of Markov models, and radius of gyration, in order to further explore the predictability under different circumstances.

Large-scale micro-aerial vehicle (MAV) swarms provide promising solutions for situational awareness in applications such as environmental monitoring, urban surveillance, search and rescue, and so on. However, these scenarios do not provide localization infrastructure and limit cost and size of on-board capabilities of individual nodes, which makes it challenging for nodes to autonomously navigate to suitable preassigned locations. In this article, we present H-DrunkWalk , a collaborative and adaptive technique for heterogeneous MAV swarm navigation in environments not formerly preconditioned for operation. Working with heterogeneous MAV swarm, the H-DrunkWalk achieves high accuracy through collaboration but still maintains a low cost of the entire swarm. The heterogeneous MAV swarm consists of two types of nodes: (1) basic MAVs with limited sensing, communication, computing capabilities and (2) advanced MAVs with premium sensing, communication, computing capabilities. The key focus behind this networked MAV swarm research is to (1) rely on collaboration to overcome limitations of individual nodes and efficiently achieve system-wide sensing objectives and (2) fully take advantage of advanced MAVs to help basic MAVs improve their performance. The evaluations based on real MAV testbed experiments and large-scale physical-feature-based simulations show that compared to the traditional non-collaborative and non-adaptive method (dead reckoning with map bias), our system achieves up to 6× reductions in location estimation errors, and as much as 3× improvements in navigation success rate under the given time and accuracy constraints. In addition, by comprehensively considering the environment, heterogeneous structure, and quality of location estimation, our H-DrunkWalk brings 2× performance improvement (on average) as that of a hardware upgrade.

Brain-inspired computing is a novel computing technology based on neural morphological engineering, which draws lessons from methods of human brain information processing and storage. Combining with the high-performance computing (HPC) platform, they constitute the foundation of general artificial intelligence. However, current brain HPC platforms generally suffer from slow speed, poor scalability, and high energy consumption, which severely restrain its potential and circumscribe the development of general artificial intelligence. The dataflow model was first proposed in the 1970s, providing a novel idea for the development of HPC. In addition, the dataflow model shares similar information processing mechanisms with human's neural system, which makes dataflow models naturally suit the emulation of brain-inspired computing. Based on the contemporary progress of the dataflow model, the Codelet model was proposed. Through a fine-grained asynchronous program execution and resource allocation, the Codelet model successfully realized the distributed computing on the heterogeneous system, effectively improved the computing power and speed, and open up a new path to overcome the shortcomings of the existing high-performance computing technology. We propose a dataflow-based emulation platform, aiming at providing high-performance computing technology support for general brain-inspired intelligent system, as well as using characteristics of dataflow models to fully explore the potential of brain-inspired intelligence. As an example, we will select a convolutional neural network (LeNet5) that already has a spectacular user base to initially verify the superiority and feasibility of our proposal.

Deep learning models have significantly improved the visual quality and accuracy on compressive sensing recovery. In this paper, we propose an algorithm for signal reconstruction from compressed measurements with image priors captured by a generative model. We search and constrain on latent variable space to make the method stable when the number of compressed measurements is extremely limited. We show that, by exploiting certain structures of the latent variables, the proposed method produces improved reconstruction accuracy and preserves realistic and non-smooth features in the image. Our algorithm achieves high computation speed by projecting between the original signal space and the latent variable space in an alternating fashion.

In the modern information society, accurate prediction of human mobility becomes increasingly essential in various areas such as city planning and resource management. With users' historical trajectories, the inherent patterns of their movements can be extracted and utilized to accurately predict the future movements. In this paper, based on a dataset of 100,000 individuals' actively uploaded location information collected by apps, we discover the average theoretical limits of the predictability to be as high as 93%. Since the app-collected data contains the physical context of the location, we implement a clustering method based on the contextual information that cluster the locations into three divisions, street, district and region. In order to solve the unevenly distribution and the high missing rate of the application collected location data, we firstly use the Gibbs sampling algorithm to complete the missing data of the trajectory and then employ a high-order Markov chain model to predict the most likely locations visited by each user. Result shows that our prediction algorithm can achieve accuracy as high as 67%, 78%, 87% for the three context-based divisions respectively, which are 10% higher on average than the divisions without context. In addition, the correlation coefficient between prediction accuracy and predictability reaches as high as 0.86. Finally, we investigate various factors including spatial and temporal resolution, orders of Markov models, radius of gyration, in order to explore the predictability under different circumstances.