Ensuring that AI systems make strategic decisions aligned with the specified preferences in adversarial sequential interactions is a critical challenge for developing trustworthy AI systems, especially when the environment is stochastic and players' incomplete preferences leave some outcomes unranked. We study the problem of synthesizing preference-satisfying strategies in two-player stochastic games on graphs where players have opposite (possibly incomplete) preferences over a set of temporal goals. We represent these goals using linear temporal logic over finite traces (LTLf), which enables modeling the nuances of human preferences where temporal goals need not be mutually exclusive and comparison between some goals may be unspecified. We introduce a solution concept of non-dominated almost-sure winning, which guarantees to achieve a most preferred outcome aligned with specified preferences while maintaining robustness against the adversarial behaviors of the opponent. Our results show that strategy profiles based on this concept are Nash equilibria in the game where players are risk-averse, thus providing a practical framework for evaluating and ensuring stable, preference-aligned outcomes in the game. Using a drone delivery example, we demonstrate that our contributions offer valuable insights not only for synthesizing rational behavior under incomplete preferences but also for designing games that motivate the desired behavior from the players in adversarial conditions.

The problem of reward design examines the interaction between a leader and a follower, where the leader aims to shape the follower’s behavior to maximize the leader’s payoff by modifying the follower’s reward function. Current approaches to reward design rely on an accurate model of how the follower responds to reward modifications, which can be sensitive to modeling inaccuracies. To address this issue of sensitivity, we present a solution that offers robustness against uncertainties in modeling the follower, including 1) how the follower breaks ties in the presence of nonunique best responses, 2) inexact knowledge of how the follower perceives reward modifications, and 3) bounded rationality of the follower. Our robust solution is guaranteed to exist under mild conditions and can be obtained numerically by solving a mixed-integer linear program. Numerical experiments on multiple test cases demonstrate that our solution improves robustness compared to the standard approach without incurring significant additional computing costs.

Contact-awareness poses a significant challenge in the locomotion control of soft snake robots. This article is to develop bioinspired contact-aware locomotion controllers, grounded in a novel theory pertaining to the feedback mechanism of the Matsuoka oscillator. This mechanism enables the Matsuoka central pattern generator (CPG) system to function analogously to a “spinal cord” in the entire contact-aware control framework. Specifically, it concurrently integrates stimuli, such as tonic input signals originating from the “brain” (a goal-tracking locomotion controller) and sensory feedback signals from the “reflex arc” (the contact reactive controller), for generating different types of rhythmic signals to orchestrate the movement of the soft snake robot traversing through densely populated obstacles and even narrow aisles. Within the “reflex arc” design, we have designed two distinct types of contact reactive controllers: 1) a reinforcement learning-based sensor regulator that learns to modulate the sensory feedback inputs of the CPG system, and 2) a local reflexive controller that establishes a direct connection between sensor readings and the CPG's feedback inputs, adhering to a specific topological configuration. These two reactive controllers, when combined with the goal-tracking locomotion controller and the Matsuoka CPG system, facilitate the implementation of two contact-aware locomotion control schemes. Both control schemes have been rigorous tested and evaluated in both simulated and real-world soft snake robots, demonstrating commendable performance in contact-aware locomotion tasks. These experimental outcomes further validate the benefits of the modified Matsuoka CPG system, augmented by a novel sensory feedback mechanism, for the design of bioinspired robot controllers.

The recent advancements in wireless technology enable connected autonomous vehicles (CAVs) to gather information about their environment by vehicle-to-vehicle (V2V) communication. In this work, we design an information-sharing-based multi-agent reinforcement learning (MARL) framework for CAVs, to take advantage of the extra information when making decisions to improve traffic efficiency and safety. The safe actor-critic algorithm we propose has two new techniques: the truncated$\mathcal{Q}$-function and safe action mapping. The truncated$\mathcal{Q}$-function utilizes the shared information from neighboring CAVs such that the joint state and action spaces of the$\mathcal{Q}$-function do not grow in our algorithm for a large-scale CAV system. We prove the bound of the approximation error between the truncated-$\mathcal{Q}$and global$Q$-functions. The safe action mapping provides a provable safety guarantee for both the training and execution based on control barrier functions. Using the CARLA simulator for experiments, we show that our approach improves the CAV system’s efficiency in terms of average velocity and comfort under different CAV ratios and different traffic densities. We also show that our approach avoids the execution of unsafe actions and always maintains a safe distance from other vehicles. We construct an obstacle-at-corner scenario to show that the shared vision can help CAVs to observe obstacles earlier and take action to avoid traffic jams. The experiment video is on https://songyanghan.github.io/cavmarl/.

In this paper, we study planning in stochastic systems, modeled as Markov decision processes (MDPs), with preferences over temporally extended goals. Prior work on temporal planning with preferences assumes that the user preferences form a total order, meaning that every pair of outcomes are comparable with each other. In this work, we consider the case where the preferences over possible outcomes are a partial order rather than a total order. We first introduce a variant of deterministic finite automaton, referred to as a preference DFA, for specifying the user's preferences over temporally extended goals. Based on the order theory, we translate the preference DFA to a preference relation over policies for probabilistic planning in a labeled MDP. In this treatment, a most preferred policy induces a weak-stochastic nondominated probability distribution over the finite paths in the MDP. The proposed planning algorithm hinges on the construction of a multi-objective MDP. We prove that a weak-stochastic nondominated policy given the preference specification is Pareto-optimal in the constructed multi-objective MDP, and vice versa. Throughout the paper, we employ a running example to demonstrate the proposed preference specification and solution approaches. We show the efficacy of our algorithm using the example with detailed analysis, and then discuss possible future directions.

This paper studies temporal planning in probabilistic environments, modeled as labeled Markov decision processes (MDPs), with user preferences over multiple temporal goals. Existing works reflect such preferences as a prioritized list of goals. This paper introduces a new specification language, termed prioritized qualitative choice linear temporal logic on finite traces, which augments linear temporal logic on finite traces with prioritized conjunction and ordered disjunction from prioritized qualitative choice logic. This language allows for succinctly specifying temporal objectives with corresponding preferences accomplishing each temporal task. The finite traces that describe the system's behaviors are ranked based on their dissatisfaction scores with respect to the formula. We propose a systematic translation from the new language to a weighted deterministic finite automaton. Utilizing this computational model, we formulate and solve a problem of computing an optimal policy that minimizes the expected score of dissatisfaction given user preferences. We demonstrate the efficacy and applicability of the logic and the algorithm on several case studies with detailed analyses for each.

In this paper, we study the use of deception for strategic planning in adversarial environments. We model the interaction between the agent (player 1) and the adversary (player 2) as a two-player concurrent stochastic game in which the adversary has incomplete information about the agent’s task specification given as a temporal logic formula. During the interaction, the adversary can infer the agent’s intention from observations and adapt its strategy so as to prevent the agent from satisfying the objective. To plan against such an adaptive opponent, the agent must leverage its knowledge about the adversary’s incomplete information to influence the behavior of the opponent, and thereby be deceptive. To synthesize a deceptive strategy, we introduce a class of hypergame models that capture the interaction between the agent and its adversary given asymmetric, incomplete information. We develop a solution concept for this class of hypergames and show that the subjectively rationalizable strategy for the agent is deceptive and maximizes the probability of satisfying the task in temporal logic. Such a deceptive strategy is obtained by modeling the opponent’s evolving perception of the agent’s objective and integrating it into planning. This allows the agent to manipulate the opponent’s perception so as to induce the opponent into taking actions that benefit the agent. We demonstrate the effectiveness of our deceptive planning algorithm using robot motion planning examples with temporal logic objectives and design a detection mechanism to notify the agent of potential errors in modeling the adversary’s behavior. Note to Practitioners—Many security and defense applications employ deception mechanisms for strategic advantages. This work presents a game-theoretic framework for planning deceptive strategies in stochastic environments and shows that the opponent modeling plays a key role in the design of effective deception mechanisms. For applications to cyber-physical security, the practitioners can employ temporal logic for specifying security properties in the system and analyze defense with deception using the proposed methods.

Intelligent control of soft robots is challenging due to the nonlinear and difficult-to-model dynamics. One promising model-free approach for soft robot control is reinforcement learning (RL). However, model-free RL methods tend to be computationally expensive and data-inefficient and may not yield natural and smooth locomotion patterns for soft robots. In this work, we develop a bioinspired design of a learning-based goal-tracking controller for a soft snake robot. The controller is composed of two modules: An RL module for learning goal-tracking behaviors given the unmodeled and stochastic dynamics of the robot, and a central pattern generator (CPG) with the Matsuoka oscillators for generating stable and diverse locomotion patterns. We theoretically investigate the maneuverability of Matsuoka CPG's oscillation bias, frequency, and amplitude for steering control, velocity control, and sim-to-real adaptation of the soft snake robot. Based on this analysis, we proposed a composition of RL and CPG modules such that the RL module regulates the tonic inputs to the CPG system given state feedback from the robot, and the output of the CPG module is then transformed into pressure inputs to pneumatic actuators of the soft snake robot. This design allows the RL agent to naturally learn to entrain the desired locomotion patterns determined by the CPG maneuverability. We validated the optimality and robustness of the control design in both simulation and real experiments, and performed extensive comparisons with state-of-art RL methods to demonstrate the benefit of our bioinspired control design.

Attention control is a key cognitive ability for humans to select information relevant to the current task. This paper develops a computational model of attention and an algorithm for attention-based probabilistic planning in Markov decision processes. In attention-based planning, the robot decides to be in different attention modes. An attention mode corresponds to a subset of state variables monitored by the robot. By switching between different attention modes, the robot actively perceives task-relevant information to reduce the cost of information acquisition and processing, while achieving near-optimal task performance. Though planning with attention-based active perception inevitably introduces partial observations, a partially observable MDP formulation makes the problem computational expensive to solve. Instead, our proposed method employs a hierarchical planning framework in which the robot determines what to pay attention to and for how long the attention should be sustained before shifting to other information sources. During the attention sustaining phase, the robot carries out a sub-policy, computed from an abstraction of the original MDP given the current attention. We use an example where a robot is tasked to capture a set of intruders in a stochastic gridworld. The experimental results show that the proposed method enables information- and computation-efficient optimal planning in stochastic environments.

It is often desirable to capture and map semantic information of an environment during simultaneous localization and mapping (SLAM). Such semantic information can enable a robot to better distinguish places with similar low-level geometric and visual features and perform high-level tasks that use semantic information about objects to be manipulated and environments to be navigated. While semantic SLAM has gained increasing attention, there is little research on semantic-level data association based on semantic objects, i.e., object-level data association. In this paper, we propose a novel object-level data association algorithm based on bag of words algorithm [1], formulated as a maximum weighted bipartite matching problem. With object-level data association solved, we develop a quadratic-programming-based semantic object initialization scheme using dual quadric and introduce additional constraints to improve the success rate of object initialization. The integrated semantic-level SLAM system can achieve high-accuracy object-level data association and real-time semantic mapping as demonstrated in the experiments. The online semantic map building and semantic-level localization capabilities facilitate semantic-level mapping and task planning in a priori unknown environment.

We consider a class of two-player turn-based zero-sum games on graphs with reachability objectives, known as reachability games, where the objective of Player 1 (P1) is to reach a set of goal states, and that of Player 2 (P2) is to prevent this. In particular, we consider the case where the players have asymmetric information about each other's action capabilities: P2 starts with an incomplete information (misperception) about P1's action set, and updates the misperception when P1 uses an action previously unknown to P2. When P1 is made aware of P2's misperception, the key question is whether P1 can control P2's perception so as to deceive P2 into selecting actions to P1's advantage? To answer this question, we introduce a dynamic hypergame model to capture the reachability game with evolving misperception of P2. Then, we present a fixed-point algorithm to compute the deceptive winning region and strategy for P1 under almost-sure winning condition. Finally, we show that the synthesized deceptive winning strategy is at least as powerful as the (non-deceptive) winning strategy in the game in which P1 does not account for P2's misperception. We illustrate our algorithm using a robot motion planning in an adversarial environment.

In this paper, we present a new locomotion control method for soft robot snakes. Inspired by biological snakes, our control architecture is composed of two key modules: A reinforcement learning (RL) module for achieving adaptive goal-tracking behaviors with changing goals, and a central pattern generator (CPG) system with Matsuoka oscillators for generating stable and diverse locomotion patterns. The two modules are interconnected into a closed-loop system: The RL module, analogizing the locomotion region located in the midbrain of vertebrate animals, regulates the input to the CPG system given state feedback from the robot. The output of the CPG system is then translated into pressure inputs to pneumatic actuators of the soft snake robot. Based on the fact that the oscillation frequency and wave amplitude of the Matsuoka oscillator can be independently controlled under different time scales, we further adapt the option-critic framework to improve the learning performance measured by optimality and data efficiency. The performance of the proposed controller is experimentally validated with both simulated and real soft snake robots.

It is important that automotive engineers understand the interactions between active human maneuvering motions and vehicle dynamics, and how vehicle control affects the physical sensations of the human driver. This paper proposes a new system framework, the human model-based active driving system (HuMADS) for simulating human driver-vehicle interactions. HuMADS integrates the vehicle controller with models of vehicle dynamics and human biomechanics. It has an hierarchical closed-loop architecture for driver-vehicle control systems, including structures and contact interfaces of human and vehicle bodies. HuMADS is based on the OpenSim simulation platform. The developed system regulates the human model dynamics, such that the human model can react realistically to vehicle maneuver motions. The usability of the HuMADS is demonstrated through the simulation of coordinated gas/brake pedal operation and wheel-steering in highway driving tasks. The simulated vehicle dynamics and vehicle maneuvers are comparable with previously published experimental data of car-following driving. In addition, the proposed controllers successfully maintain the human body's balance inside the vehicle during vehicle maneuvers. We are convinced that the HuMADS has potential as a tool for the development of intelligent transportation systems and investigation of integrated safety.

In this work we present a framework that is capable of accurately representing soft robotic actuators in a multiphysics environment in real-time. We propose a constraint-based dynamics model of a 1-dimensional pneumatic soft actuator that accounts for internal pressure forces, as well as the effect of actuator latency and damping under inflation and deflation and demonstrate its accuracy a full soft robotic snake with the composition of multiple 1D actuators. We verify our model's accuracy in static deformation and dynamic locomotion open-loop control experiments. To achieve real-time performance we leverage the parallel computation power of GPUs to allow interactive control and feedback.

The fast pace of global urbanization is drastically changing the population distributions over the world, which leads to significant changes in geographical population densities. Such changes in turn alter the underlying geographical power demand over time, and drive power substations to become over-supplied (demand z capacity) or under-supplied (demand ≈ capacity). In this paper, we make the first attempt to investigate the problem of power substation-user assignment by analyzing large-scale power grid data. We develop a Scalable Power User Assignment (SPUA) framework, that takes large-scale spatial power user/substation distribution data and temporal user power consumption data as input, and control the assignments between users and substations, in a manner that minimizes the maximum substation utilization among all substations. To evaluate the performance of our SPUA framework, we conduct evaluations on real power consumption data and user/substation location data collected from a northwestern province in China for 35 days in 2015. The evaluation results demonstrate that our SPUA framework can achieve a 20-65 percent reduction on the maximum substation utilization, and 2 to 3.7 times reduction on total transmission loss over other baseline methods.

We investigate a sampling-based method for optimal control of continuous-time and continuous-state (possibly nonlinear) systems under co-safe linear temporal logic specifications. We express the temporal logic specification as a deterministic, finite automaton (the specification automaton), and link the automaton's discrete transitions to the continuous system state as it passes through specified regions. The optimal hybrid controller is characterized by a set of coupled partial differential equations. Because these equations are difficult to solve exactly in practice in all cases, we propose instead a sampling based technique to solve for an approximate controller through approximate value iteration. We adopt model reference adaptive search---an importance sampling optimization algorithm---to determine the mixing weights of the approximate value function expressed in a finite basis. Under mild technical assumptions, the algorithm converges, with probability one, to an optimal weight that ensures the satisfaction of temporal logic constraints, while minimizing an upper bound for the optimal cost. We demonstrate the correctness and efficiency of the method through numerical experiments, including temporal logic planning for a linear system, and a nonlinear mobile robot.

In this paper, we propose a sampling-based policy iteration for optimal planning under temporal logic constraints. The method integrates approximate optimal control, importance sampling, and formal methods. For a subclass of linear temporal logic, the planning problem is transformed to an optimal control problem for a hybrid system where discrete transitions are triggered by linear time events in temporal logic. Instead of solving the Hamilton-Jacobi-Bellman equation, we use policy function approximation to reduce the problem into a search of an optimal weight vector that parametrizes the near-optimal policy for given bases. Then, we incorporate Model Reference Adaptive Search - an importance sampling-based optimization algorithm to perform a sample-efficient search within the parameter space of policy function approximations. Facing the discontinuity in cost function introduced by temporal logic constraints and system dynamics, we introduce (1) a rank function in formal logic specifications to enable sample-efficient search; (2) specification-guided basis selection. Under mild technical assumptions, the proposed algorithm converges, with probability one, to a global approximate optimal policy that ensures the satisfaction of temporal logic constraints. The correctness and efficiency of the method are demonstrated through numerical experiments including temporal logic planning for a linear system and a nonlinear mobile robot.

The fast pace of global urbanization is drastically changing the population distributions over the world, which leads to significant changes in geographical population densities. Such changes in turn alter the underlying geographical power demand over time, and drive power substations to become over-supplied (demand << capacity) or under-supplied (demand ≈ capacity). In this paper, we make the first attempt to investigate the problem of power substation-user assignment by analyzing large-scale power grid data. We develop a Scalable Power User Assignment (SPUA) framework, that takes large-scale spatial power user/substation distribution data and temporal user power consumption data as input, and assigns users to substations, in a manner that minimizes the maximum substation utilization among all substations. To evaluate the performance of our SPUA framework, we conduct evaluations on real power consumption data and user/substation location data collected from a province in China for 35 days in 2015. The evaluation results demonstrate that our SPUA framework can achieve a 20%--65% reduction on the maximum substation utilization, and 2 to 3.7 times reduction on total transmission loss over other baseline methods.

This paper considers robot motion planning under temporal logic constraints in probabilistic maps obtained by semantic simultaneous localization and mapping (SLAM). The uncertainty in a map distribution presents a great challenge for obtaining correctness guarantees with respect to the linear temporal logic (LTL) specification. We show that the problem can be formulated as an optimal control problem in which both the semantic map and the logic formula evaluation are stochastic. Our first contribution is to reduce the stochastic control problem for a subclass of LTL to a deterministic shortest path problem by introducing a confidence parameter δ. A robot trajectory obtained from the deterministic problem is guaranteed to have minimum cost and to satisfy the logic specification in the true environment with probability δ. Our second contribution is to design an admissible heuristic function that guides the planning in the deterministic problem towards satisfying the temporal logic specification. This allows us to obtain an optimal and very efficient solution using the A* algorithm. The performance and correctness of our approach are demonstrated in a simulated semantic environment using a differential-drive robot.

We propose a synthesis method of switching control policies for a class of shared autonomy systems in which control authority is held by either a human operator or an autonomous controller based on the state of the overall system. The objective is to optimize the system performance measured by the probability of satisfying a system specification in linear temporal logic, while ensuring a reasonable workload for the human operator. The synthesis method builds upon the construction of an abstract model for the given shared autonomy system from a set of components modeled by Markov decision processes, which are capable of capturing the uncertainty in the operator's performance and response to switching control signals under his different cognitive and physiological states. A cost function is then introduced to quantify the human operator's workload. In order to establish quantitative trade-offs between the operator's effort and the system performance, we propose a two-stage policy synthesis algorithm for generating Pareto-optimal switching control policies.

For systems in which control authority is shared by an autonomous controller and a human operator, it is important to find solutions that achieve a desirable system performance with a reasonable workload for the human operator. We formulate a shared autonomy system capable of capturing the interaction and switching control between an autonomous controller and a human operator, as well as the evolution of the operator's cognitive state in the working environment. To trade-off human's effort and the performance level, e.g., measured by the probability of satisfying the underlying temporal logic specification, a two-stage policy synthesis algorithm is proposed for generating Pareto efficient coordination and control policies with respect to user specified weights.