Many real-world applications involve a team of agents who must coordinate each other's policies in real-time to achieve a shared goal. Previous studies mainly focus on decentralized control to maximize common rewards, with little consideration of coordination between control policies, which is critical in dynamic and complicated environments. Viewing this issue, we propose a novel adaptive graph coordination strategy that factorizes the joint policy into an adaptive graph generator and a graph-based coordinated policy. We employ a difference aware module to control when to generate graphs and an encoder-decoder module to acquire the underlying decision graph structure. Moreover, we introduce DAGness- and DAG depth constrained optimization to adjust the graph structure and strike a balance between efficiency and performance. We also present a graph-based coordinated policy to make asynchronous decisions based on the inter-agent coordination dependencies implied in the generated graph. Empirical evaluations on some cooperative multi-agent environments demonstrate the superiority of the proposed method, with faster convergence and more efficient coordinated policies.

Advanced large language models (LLMs) frequently reflect in reasoning chain-of-thoughts (CoTs), where they self-verify the correctness of current solutions and explore alternatives. However, given recent findings that LLMs detect limited errors in CoTs, how reflection contributes to empirical improvements remains unclear. To analyze this issue, in this paper, we present a minimalistic reasoning framework to support basic self-verifying reflection for small transformers without natural language, which ensures analytic clarity and reduces the cost of comprehensive experiments. Theoretically, we prove that self-verifying reflection guarantees improvements if verification errors are properly bounded. Experimentally, we show that tiny transformers, with only a few million parameters, benefit from self-verification in both training and reflective execution, reaching remarkable LLM-level performance in integer multiplication and Sudoku. Similar to LLM results, we find that reinforcement learning (RL) improves in-distribution performance and incentivizes frequent reflection for tiny transformers, yet RL mainly optimizes shallow statistical patterns without faithfully reducing verification errors. In conclusion, integrating generative transformers with discriminative verification inherently facilitates CoT reasoning, regardless of scaling and natural language.

Causal dynamics models (CDMs) have demonstrated significant potential in addressing various challenges in reinforcement learning. To learn CDMs, recent studies have performed causal discovery to capture the causal dependencies among environmental variables. However, the learning of CDMs is still confined to small-scale environments due to computational complexity and sample efficiency constraints. This paper aims to extend CDMs to large-scale object-oriented environments, which consist of a multitude of objects classified into different categories. We introduce the Object-Oriented CDM (OOCDM) that shares causalities and parameters among objects belonging to the same class. Furthermore, we propose a learning method for OOCDM that enables it to adapt to a varying number of objects. Experiments on large-scale tasks indicate that OOCDM outperforms existing CDMs in terms of causal discovery, prediction accuracy, generalization, and computational efficiency.

Generating explanations for reinforcement learning (RL) is challenging as actions may produce long-term effects on the future. In this paper, we develop a novel framework for explainable RL by learning a causal world model without prior knowledge of the causal structure of the environment. The model captures the influence of actions, allowing us to interpret the long-term effects of actions through causal chains, which present how actions influence environmental variables and finally lead to rewards. Different from most explanatory models which suffer from low accuracy, our model remains accurate while improving explainability, making it applicable in model-based learning. As a result, we demonstrate that our causal model can serve as the bridge between explainability and learning.

A conservativeness-reduced design of a gain scheduled output feedback H/sub /spl infin// controller for an n-joint rigid robotic manipulator, which integrates the varying-parameter rate without their feedback, is proposed. The robotic system is reduced to be a linear parameter varying (LPV) form, which depends affinely on the varying-parameter. The design of a controller, which satisfies the closed-loop H/sub /spl infin// performance, is reduced to be a solution of the parameterized linear matrix inequalities (LMIs) of parameter matrices. With a use of the concept of "multi-convexity", the solution of the infinite LMIs in the varying-parameter and its rate space is reduced to be a solution of the finite LMIs for the vertex set. The proposed controller eliminates the feedback of the varying-parameter rate and prescribes its upper boundary so that the conservativeness of the controller design is reduced. The experiment results verify the effectiveness of the proposed design.

A new approach to the design of a fuzzy continuous gain scheduled H/sub /spl infin// controller based on Taylor series fitting for n-joint rigid robotic manipulators, which guarantees satisfactory dynamic characteristics in the whole movement range, is presented. This scheme combines the gain scheduled H/sub /spl infin// theory with LMI approach to design a continuous gain scheduled H/sub /spl infin// controller, which is applicable to systems with fast state variations so that the deficiency of the conventional gain scheduled controllers is overcome, with the use of Taylor series fitting. Fuzzy control is then incorporated so chat the designed controller possesses the characteristics of a fast response for large errors and a well-damped response for small errors. The system thus always has a good dynamic performance along with the variations of the system states. Simulations and experiments demonstrate the effectiveness of the designed controller.