Research in offline reinforcement learning (RL) marks a paradigm shift in RL. However, a critical yet under-investigated aspect of offline RL is determining the subset of the offline dataset, which is used to improve algorithm performance while accelerating algorithm training. Moreover, the size of reduced datasets can uncover the requisite offline data volume essential for addressing analogous challenges. Based on the above considerations, we propose identifying Reduced Datasets for Offline RL (ReDOR) by formulating it as a gradient approximation optimization problem. We prove that the common actor-critic framework in reinforcement learning can be transformed into a submodular objective. This insight enables us to construct a subset by adopting the orthogonal matching pursuit (OMP). Specifically, we have made several critical modifications to OMP to enable successful adaptation with Offline RL algorithms. The experimental results indicate that the data subsets constructed by the ReDOR can significantly improve algorithm performance with low computational complexity.

Subtask decomposition offers a promising approach for achieving and comprehending complex cooperative behaviors in multiagent systems. Nonetheless, existing methods often depend on intricate high-level strategies, which can hinder interpretability and learning efficiency. To tackle these challenges, we propose a novel approach that specializes subtasks for subgroups by employing diverse observation representation encoders within information bottlenecks. Moreover, to enhance the efficiency of subtask specialization while promoting sophisticated cooperation, we introduce diversity in both optimization and neural network architectures. These advancements enable our method to achieve state-of-the-art performance and offer interpretable subtask factorization across various scenarios in Google Research Football (GRF).

Previous literature on policy diversity in reinforcement learning (RL) either focuses on the online setting or ignores the policy performance. In contrast, offline RL, which aims to learn high-quality policies from batched data, has yet to fully leverage the intrinsic diversity of the offline dataset. Addressing this dichotomy and aiming to balance quality and diversity poses a significant challenge to extant methodologies. This paper introduces a novel approach, termed Stylized Offline RL (SORL), which is designed to extract high-performing, stylistically diverse policies from a dataset characterized by distinct behavioral patterns. Drawing inspiration from the venerable Expectation-Maximization (EM) algorithm, SORL innovatively alternates between policy learning and trajectory clustering, a mechanism that promotes policy diversification. To further augment policy performance, we introduce advantage-weighted style learning into the SORL framework. Experimental evaluations across multiple environments demonstrate the significant superiority of SORL over previous methods in extracting high-quality policies with diverse behaviors. A case in point is that SORL successfully learns strong policies with markedly distinct playing patterns from a real-world human dataset of a popular basketball video game "Dunk City Dynasty."

Offline reinforcement learning (RL) is crucial for real-world applications where exploration can be costly or unsafe. However, offline learned policies are often suboptimal, and further online fine-tuning is required. In this paper, we tackle the fundamental dilemma of offline-to-online fine-tuning: if the agent remains pessimistic, it may fail to learn a better policy, while if it becomes optimistic directly, performance may suffer from a sudden drop. We show that Bayesian design principles are crucial in solving such a dilemma. Instead of adopting optimistic or pessimistic policies, the agent should act in a way that matches its belief in optimal policies. Such a probability-matching agent can avoid a sudden performance drop while still being guaranteed to find the optimal policy. Based on our theoretical findings, we introduce a novel algorithm that outperforms existing methods on various benchmarks, demonstrating the efficacy of our approach. Overall, the proposed approach provides a new perspective on offline-to-online RL that has the potential to enable more effective learning from offline data.

The surge in volumes of video data offers unprecedented opportunities for advancing reinforcement learning (RL). This growth has motivated the development of passive RL, seeking to convert passive observations into actionable insights. This paper explores the prerequisites and mechanisms through which passive data can be utilized to improve online RL. We show that, in identifiable dynamics, where action impact can be distinguished from stochasticity, learning on passive data is statistically beneficial. Building upon the theoretical insights, we propose a novel algorithm named Multiscale State-Centric Planners (MSCP) that leverages two planners at distinct scales to offer guidance across varying levels of abstraction. The algorithm’s fast planner targets immediate objectives, while the slow planner focuses on achieving longer-term goals. Notably, the fast planner incorporates pessimistic regularization to address the distributional shift between offline and online data. MSCP effectively handles the practical challenges involving imperfect pretraining and limited dataset coverage. Our empirical evaluations across multiple benchmarks demonstrate that MSCP significantly outperforms existing approaches, underscoring its proficiency in addressing complex, long-horizon tasks through the strategic use of passive data.

Prior research on policy adaptation in multi-agent games has often relied on online interaction with the target agent in training, which can be expensive and impractical in real-world scenarios. Inspired by recent progress in offline reinforcement learn- ing, this paper studies offline policy adaptation, which aims to utilize the target agent’s behavior data to exploit its weakness or enable effective cooperation. We investigate its distinct challenges of distributional shift and risk-free deviation, and propose a novel learning objective, conservative offline adaptation, that optimizes the worst-case performance against any dataset consistent proxy models. We pro- pose an efficient algorithm called Constrained Self-Play (CSP) that incorporates dataset information into regularized policy learning. We prove that CSP learns a near-optimal risk-free offline adaptation policy upon convergence. Empirical results demonstrate that CSP outperforms non-conservative baselines in various environments, including Maze, predator-prey, MuJoCo, and Google Football.

In zero-sum games, an NE strategy tends to be overly conservative confronted with opponents of limited rationality, because it does not actively exploit their weaknesses. From another perspective, best responding to an estimated opponent model is vulnerable to estimation errors and lacks safety guarantees. Inspired by the recent success of real-time search algorithms in developing superhuman AI, we investigate the dilemma of safety and opponent exploitation and present a novel real-time search framework, called Safe Exploitation Search (SES), which continuously interpolates between the two extremes of online strategy refinement. We provide SES with a theoretically upper-bounded exploitability and a lower-bounded evaluation performance. Additionally, SES enables computationally efficient online adaptation to a possibly updating opponent model, while previous safe exploitation methods have to recompute for the whole game. Empirical results show that SES significantly outperforms NE baselines and previous algorithms while keeping exploitability low at the same time.

Recently, deep multi-agent reinforcement learning (MARL) has shown the promise to solve complex cooperative tasks. Its success is partly because of parameter sharing among agents. However, such sharing may lead agents to behave similarly and limit their coordination capacity. In this paper, we aim to introduce diversity in both optimization and representation of shared multi-agent reinforcement learning. Specifically, we propose an information-theoretical regularization to maximize the mutual information between agents' identities and their trajectories, encouraging extensive exploration and diverse individualized behaviors. In representation, we incorporate agent-specific modules in the shared neural network architecture, which are regularized by L1-norm to promote learning sharing among agents while keeping necessary diversity. Empirical results show that our method achieves state-of-the-art performance on Google Research Football and super hard StarCraft II micromanagement tasks.

Three-dimensional (3D) convolutional neural network (CNN) is widely used for action recognition. However, it performs same on time and space, which dose not match asymmetry of time. To overcome this problem, we propose the time-asymmetric 3D CNN based on the hypothesis that the early frames persist. Our time-asymmetric 3D CNN performs better on Kinetics-400 than normal 3D CNN with less parameters. And it can get comparable results on UCF101 among leading methods, which indicates that our time-asymmetric 3D CNN is a good choice for action recognition.

Wireless sensor–actuator networks (WSANs) offer an appealing communication technology for process automation applications to incorporate the Internet of Things (IoT). In contrast to other IoT applications, process automation poses unique challenges for industrial WSAN due to its critical demands on reliable and real-time communication. While industrial WSANs have received increasing attention in the research community recently, most published results to date have focused on the theoretical aspects and were evaluated based on simulations. There is a critical need for experimental research on this important class of WSANs. We developed an experimental testbed by implementing several key network protocols of WirelessHART, an open standard for WSANs that has been widely adopted in the process industries based on the HART. We then performed a series of empirical studies showing that graph routing leads to significant improvement over source routing in terms of worst-case reliability, but at the cost of longer latency and higher energy consumption. It is therefore important to employ graph routing algorithms specifically designed to optimize latency and energy efficiency. Our studies also suggest that channel hopping can mitigate the burstiness of transmission failures; a larger channel distance can reduce consecutive transmission failures over links sharing a common receiver. Based on these insights, we developed a novel channel hopping algorithm that utilizes far away channels for transmissions. Furthermore, it prevents links sharing the same destination from using channels with strong correlations. Our experimental results demonstrate that our algorithm can significantly improve network reliability and energy efficiency.

With recent adoption of wireless sensor-actuator networks (WSANs) in industrial automation, industrial wireless control systems have emerged as a frontier of cyber-physical systems. Despite their success in industrial monitoring applications, existing WSAN technologies face significant challenges in supporting control systems due to their lack of real-time performance and dynamic wireless conditions in industrial plants. This article reviews a series of recent advances in real-time WSANs for industrial control systems: 1) real-time scheduling algorithms and analyses for WSANs; 2) implementation and experimentation of industrial WSAN protocols; 3) cyber-physical codesign of wireless control systems that integrate wireless and control designs; and 4) a wireless cyber-physical simulator for codesign and evaluation of wireless control systems. This article concludes by highlighting research directions in industrial cyber-physical systems.

Wireless sensor-actuator networks are gaining ground as the communication infrastructure for process monitoring and control. Industrial applications demand a high degree of reliability and real-time guarantees in communication. Because wireless communication is susceptible to transmission failures in industrial environments, industrial wireless standards such as WirelessHART adopt reliable graph routing to handle transmission failures through retransmissions and route diversity. While these mechanisms are critical for reliable communication, they introduce substantial challenges in analyzing the schedulability of real-time flows. This paper presents the first worst-case end-to-end delay analysis for periodic real-time flows under reliable graph routing. The proposed analysis can be used to quickly assess the schedulability of real-time flows with stringent requirements on both reliability and latency. We have evaluated our schedulability analysis against experimental results on a wireless testbed of 69 nodes as well as simulations. Both experimental results and simulations show that our delay bounds are safe and enable effective schedulability tests under reliable graph routing.

The largest source of energy consumption in buildings is heating, ventilation, and air conditioning (HVAC). For an HVAC system to provide comfort and minimize energy consumption, it is crucial to understand the spatiotemporal thermal dynamics, especially in large open spaces. To optimize HVAC control, it is important to establish accurate dynamic thermal models. For this purpose, we constructed a real-world test bed by instrumenting an HVAC-controller auditorium using multiple types of sensors. Based on the dataset, we develop and evaluate a novel data-driven approach to model the complex thermal dynamics in a large space through a combination of data clustering and system identification techniques. Real-world data shows that our approach achieves low estimation errors. Our modeling approach therefore provides a practical foundation for HVAC control and optimization for large open spaces.

Industry is adopting Wireless Sensor-Actuator Networks (WSANs) as the communication infrastructure for process control applications. To meet the stringent real-time performance requirements of control systems, there is a critical need for fast end-to-end delay analysis for real-time flows that can be used for online admission control. This paper presents a new end-to-end delay analysis for periodic flows whose transmissions are scheduled based on the Earliest Deadline First (EDF) policy. Our analysis comprises novel techniques to bound the communication delays caused by channel contention and transmission conflicts in a WSAN. Furthermore, we propose a technique to reduce the pessimism in admission control by iteratively tightening the delay bounds for flows with short deadlines. Experiments on a WSAN testbed and simulations demonstrate the effectiveness of our analysis for online admission control of real-time flows.

With the advent of industrial standards such as WirelessHART, process industries are now gravitating towards wireless control systems. Due to limited bandwidth in a wireless network shared by multiple control loops, it is critical to optimize the overall control performance. In this article, we address the scheduling-control co-design problem of determining the optimal sampling rates of feedback control loops sharing a WirelessHART network. The objective is to minimize the overall control cost while ensuring that all data flows meet their end-to-end deadlines. The resulting constrained optimization based on existing delay bounds for WirelessHART networks is challenging since it is nondifferentiable, nonlinear, and not in closed-form. We propose four methods to solve this problem. First, we present a subgradient method for rate selection. Second, we propose a greedy heuristic that usually achieves low control cost while significantly reducing the execution time. Third, we propose a global constrained optimization algorithm using a simulated annealing (SA) based penalty method. We study SA method under both constant factor penalty and adaptive penalty. Finally, we formulate rate selection as a differentiable convex optimization problem that provides a quick solution through a convex optimization technique. This is based on a new delay bound that is convex and differentiable, and hence simplifies the optimization problem. We study both the gradient descent method and the interior point method to solve it. We evaluate all methods through simulations based on topologies of a 74-node wireless sensor network testbed. The subgradient method is disposed to incur the longest execution time as well as the highest control cost among all methods. Among the SA-based constant penalty method, the greedy heuristic, and the gradient descent method, the first two represent the opposite ends of the tradeoff between control cost and execution time, while the third one hits the balance between the two. We further observe that the SA based adaptive penalty method is superior to the constant penalty method, and that the interior point method is superior to the gradient method. Thus, the interior point method and the SA-based adaptive penalty method are the two most effective approaches for rate selection. While both methods are competitive against each other in terms of control cost, the interior point method is significantly faster than the penalty method. As a result, the interior point method upon convex relaxation is more suitable for online rate adaptation than the SA based adaptive penalty method due to their significant difference in run-time efficiency.

In this paper, a utility maximization-based resource scheduling and sharing (UM-RSS) scheme is proposed for downlink multiuser multiple-input-multiple-output orthogonal frequency-division multiple access (MU-MIMO-OFDMA) systems. Before performing the UM-RSS scheduling scheme, we first allocate the best antenna sequence for every served user by a suboptimal multiuser antenna selection (MAS) algorithm according to the channel state information (CSI). To balance efficiency and fairness, the integrated scheduling algorithm of UM-RSS is responsible for assigning subcarriers to different users, as well as distributing the assigned subcarriers among multiple services for the same user. For a MU-MIMO-OFDMA system, the joint spatial and frequency scheduling may improve the system spectrum efficiency by exploiting the multiuser diversity gain in both frequency and spatial domain. Finally, numeric result simulated show that the UM-RSS scheduling scheme outperforms traditional scheduling schemes in terms of system throughput, system spectral efficiency, and fairness criterion.

Wireless sensor networks are evolving from single-application platforms towards an integrated infrastructure shared by multiple applications. Given the resource constraints of sensor nodes, it is important to optimize the allocation of applications to maximize the overall Quality of Monitoring (QoM). Recent solutions to this challenging application allocation problem are centralized in nature, limiting their scalability and robustness against network failures and dynamics. This paper presents a distributed game-theoretic approach to application allocation in shared sensor networks. We first transform the optimal application allocation problem to a submodular game and then develop a decentralized algorithm that only employs localized interactions among neighboring nodes. We prove that the network can converge to a pure strategy Nash equilibrium with an approximation bound of 1=2. Simulations based on three real-world datasets demonstrate that our algorithm is competitive against a state-of-the-art centralized algorithm in terms of QoM.

With the advent of industrial standards such as Wireless Hart, process industries are now gravitating towards wireless control systems. Due to limited bandwidth in a wireless network shared by multiple control loops, it is critical to optimize the overall control performance. In this paper, we address the scheduling-control co-design problem of determining the optimal sampling rates of feedback control loops sharing a Wireless Hart network. The objective is to minimize the overall control cost while ensuring that all data flows meet their end-to-end deadlines. The resulting constrained optimization based on existing delay bounds for Wireless Hart networks is challenging since it is non-differentiable, non-linear, and not in closed-form. We propose four methods to solve this problem. First, we present a sub gradient method for rate selection. Second, we propose a greedy heuristic that usually achieves low control cost while significantly reducing the execution time. Third, we propose a global constrained optimization algorithm using a simulated annealing (SA) based penalty method. Finally, we formulate rate selection as a differentiable convex optimization problem that provides a closed-form solution through a gradient descent method. This is based on a new delay bound that is convex and differentiable, and hence simplifies the optimization problem. We evaluate all methods through simulations based on topologies of a 74-node wireless sensor network testbed. Surprisingly, the sub gradient method is disposed to incur the longest execution time as well as the highest control cost among all methods. SA and the greedy heuristic represent the opposite ends of the trade off between control cost and execution time, while the gradient descent method hits the balance between the two.

With the emergence of wireless sensor networks, an enabling communication technology for distributed real-time systems, we face the critical challenge of meeting the end-to-end deadlines of real-time flows. This paper presents Real-time Flow Scheduling (RFS), a novel conflict-free real-time transmission scheduling approach for periodic real-time flows in wireless sensor networks. In contrast to existing transmission scheduling algorithms that ignore interference between transmissions or prevent spatial reuse within the same channel, RFS supports spatial reuse through a novel interference-aware transmission scheduling. While recent work on conflict-free transmission scheduling focused on specialized communication patterns such as queries and converge cast, RFS is designed for peer-to-peer real-time flows with arbitrary inter-flow interference. Moreover, RFS has three salient that make it particularly suitable for real-time systems: First, RFS includes a real-time schedulability analysis that accounts for interference between real-time flows. Second, RFS improves reliability by incorporating retransmissions in a flexible scheduling scheme. Finally, RFS enhances scalability by dividing the network into neighborhoods and provides real-time performance for flows crossing multiple neighborhoods through a novel application of the Release Guard protocol. RFS was evaluated through simulations based on the traces collected from an indoor wireless sensor network test bed. Compared to a traditional TDMA protocol, RFS reduces flow latencies by up to 2.5 times, while improving the real-time capacity by as much as 3.9 times.

Wireless sensor networks are often composed of resource-constrained sensor nodes with limited memory space, computational capacity and communication range. The links in WSN are often lossy and unreliable. In order to make fluent and reliable data transport on memory-constrained sensor nodes, we propose a new transport layer protocol reliable transport with memory consideration (RTMC), which provides both hop-by-hop retransmission and congestion control. We have implemented RTMC on MICA2 with only 4 K bytes RAM. Experiment, analysis and simulation results show that RTMC can use channel resource effectively and enable all of the segments to be received by the sink with low transport time and low memory cost.

Balancing energy consumption and prolonging network lifetime are open challenges in Wireless Sensor Networks. In this paper, we design a novel load balanced routing protocol to maximize lifetime of Wireless Sensor Networks (WSNs). In order to balance the energy consumption among sensor nodes, we deploy multiple sinks simultaneously which are connected though wired or wireless infrastructure. We introduce a potential model and propose a routing scheme in which sensor nodes construct routes based on local topology information and the state information from sinks' broadcast messages. Sinks monitor their traffic load and adjust their own parameters to balance the traffic load in the network. Theoretical analysis and simulation results show that our protocol can significantly improve the performance of the system in following aspects: robustness, lifetime and reliability.