In the realm of digital twin systems, establishing simulation environments for creating and testing virtual agents has garnered substantial attention across various applications. The obtained control policies endow virtual agents with more realistic behaviors and interactive capabilities, finding applications in both computer animation and robotic control. While rigid-body simulators are widely used for virtual agents, achieving similar feats in fluid environments presents formidable challenges due to high complexity and exorbitant costs. One major reason is that most fluid simulators feature a fixed domain, which struggles to enable agents to freely navigate in an unbounded, obstacle-filled space, especially when computational resources are limited, thus restricting their wide utility for creating virtual agents. In this paper, we introduce a novel fluid-solid interaction simulator grounded in an efficient lattice Boltzmann solver. A key feature of this simulator is a dynamically moving local domain that encircles the agent, offering greater flexibility for obtaining control policy while maintaining efficiency in simulation. Previous methods, which anchored a square moving local domain along with the agent, suffered from severe spurious flows when the agent underwent rapid acceleration especially when the domain had to rotate, such as during a U-turn. This led to inaccurate results and instability. Conversely, we propose a novel domain-tracking method that harnesses optimal control techniques to address this issue. Our approach not only bolsters local-domain simulation stability, but also improves efficiency by employing a slender domain, which broadens the application scope of direct fluid-solid interactions for virtual agents. We validate our method by comparing simulations to physical phenomena and obtaining control policies for various virtual agents to accomplish challenging tasks. This effort culminates in a series of animations that vividly demonstrate the efficacy of the entire framework potentially used in both computer animation and robotics.

Unmanned aerial vehicles (UAVs) have demonstrated remarkable efficacy across diverse fields. Nevertheless, developing flight controllers tailored to a specific UAV design, particularly in environments with strong fluid-interactive dynamics, remains challenging. Conventional controller design experiences often fall short in such cases, rendering it infeasible to apply time-tested practices. Consequently, a simulation test bed becomes indispensable for controller design and evaluation prior to its actual implementation on the physical UAV. This platform should allow for meticulous adjustment of controllers and should be able to transfer to real-world systems without significant performance degradation. Existing simulators predominantly hinge on empirical models due to high efficiency, often overlooking the dynamic interplay between the UAV and the surrounding airflow. This makes it difficult to mimic more complex flight maneuvers, such as an abrupt midair halt inside narrow channels, in which the UAV may experience strong fluid-structure interactions. On the other hand, simulators considering the complex surrounding airflow are extremely slow and inadequate to support the design and evaluation of flight controllers. In this paper, we present a novel remedy for highly-efficient UAV flight simulations, which entails a hybrid modeling that deftly combines our novel far-field adaptive block-based fluid simulator with parametric empirical models situated near the boundary of the UAV, with the model parameters automatically calibrated. With this newly devised simulator, a broader spectrum of flight scenarios can be explored for controller design and assessment, encompassing those influenced by potent close-proximity effects, or situations where multiple UAVs operate in close quarters. The practical worth of our simulator has been authenticated through comparisons with actual UAV flight data. We further showcase its utility in designing flight controllers for fixed-wing, multi-rotor, and hybrid UAVs, and even exemplify its application when multiple UAVs are involved, underlining the unique value of our system for flight controllers.

In the context of smart cities, autonomous vehicles, such as unmanned delivery vehicles and taxis are gradually gaining acceptance. However, their application scenarios remain significantly fragmented. Typically, an Autonomous Multi-Functional Vehicle (AMFV) is not engaged in other scenarios when idle in a specific one. Currently, a unified system capable of coordinating and using these resources efficiently is lacking. Moreover, there is an absence of an advanced navigation algorithm for facilitating coordinated navigation among Heterogeneous Vehicles (HVs). To address these issues, we propose the LLM-driven Multi-vehicle Dispatching and navigation (LiMeda) framework. It comprises an LLM-driven scheduling module that facilitates efficient allocation considering task scenarios and vehicle information, which addresses the issue of incompatible vehicle resources across various smart city scenarios. And the other is a navigation module, founded on the Heterogeneous Agent Reinforcement Learning (HARL) framework we previously proposed, which can effectively perform cooperative navigation tasks among heterogeneous agents, assisting the cooperative task completion by HVs in a smart city. Experimental results show our method outperforms both traditional scheduling algorithms and Reinforcement Learning navigation algorithms in metric terms. Additionally, it shows remarkable scalability and generalization under varying city scales, vehicle numbers, and task numbers.

The socially-aware navigation system has evolved to adeptly avoid various obstacles while performing multiple tasks, such as point-to-point navigation, human-following, and -guiding. However, a prominent gap persists: in Human-Robot Interaction (HRI), the procedure of communicating commands to robots demands intricate mathematical formulations. Furthermore, the transition between tasks does not quite possess the intuitive control and user-centric interactivity that one would desire. In this work, we propose an LLM-driven interactive multimodal multitask robot navigation framework, termed LIM2N, to solve the above new challenge in the navigation field. We achieve this by first introducing a multimodal interaction framework where language and hand-drawn inputs can serve as navigation constraints and control objectives. Next, a reinforcement learning agent is built to handle multiple tasks with the received information. Crucially, LIM2N creates smooth cooperation among the reasoning of multimodal input, multitask planning, and adaptation and processing of the intelligent sensing modules in the complicated system. Detailed experiments are conducted in both simulation and the real world demonstrating that LIM2N has solid user needs understanding, alongside an enhanced interactive experience.

Reward shaping (RS) is a powerful method in reinforcement learning (RL) for overcoming the problem of sparse or uninformative rewards. However, RS typically relies on manually engineered shaping-reward functions whose construc- tion is time-consuming and error-prone. It also requires domain knowledge which runs contrary to the goal of autonomous learning. We introduce Reinforcement Learning Optimising Shaping Algorithm (ROSA), an automated reward shaping framework in which the shaping-reward function is constructed in a Markov game between two agents. A reward-shaping agent (Shaper) uses switching controls to determine which states to add shaping rewards for more efficient learning while the other agent (Controller) learns the optimal policy for the task using these shaped rewards. We prove that ROSA, which adopts existing RL algorithms, learns to construct a shaping-reward function that is beneficial to the task thus ensuring efficient convergence to high performance policies. We demonstrate ROSA’s properties in three didactic experiments and show its superior performance against state-of-the-art RS algorithms in challenging sparse reward environments.

The study of motion control for the fish-like robots in complex fluid fields is of great importance in improving the performance of underwater vehicles, due to its strong maneuverability, propulsion efficiency, and deceptive visual appearance. In this article, a novel learning-based control framework is first proposed to autonomously explore efficient control policies that are capable of performing motion control tasks in non-quiescent and unknown background flows. First, we utilize a high-fidelity simulation system, named FishGym, to generate various uniform flows. Next, a DRL-based algorithm is incorporated with the FishGym to train the fish-like robot to control its motion to optimally complete a delicately designed task (Approaching Target and Stay) in both quiescent and uniform flow. Then, the obtained control policy together with an online estimator is directly applied to a Path-Following Task. The proposed framework well balances the simulation accuracy and the computational efficiency, which is of crucial importance for effective coupling with the learning algorithm. The simulation results indicate that, via the proposed learning framework, the robot successfully acquired a swimming strategy that can be used to adapt to different background flows and tasks. Furthermore, we also observe some adaptation behavior of the robot, such as rheotaxis, that is similar to the fish in nature, which gains us more insight into the mechanism underlying the adaptation behavior of fish in a complex environment.

Numerical Partial Differential Equation (PDE) solvers often require discretizing the physical domain by using a mesh. Mesh movement methods provide the capability to improve the accuracy of the numerical solution without introducing extra computational burden to the PDE solver, by increasing mesh resolution where the solution is not well-resolved, whilst reducing unnecessary resolution elsewhere. However, sophisticated mesh movement methods, such as the Monge-Ampère method, generally require the solution of auxiliary equations. These solutions can be extremely expensive to compute when the mesh needs to be adapted frequently. In this paper, we propose to the best of our knowledge the first learning-based end-to-end mesh movement framework for PDE solvers. Key requirements of learning-based mesh movement methods are: alleviating mesh tangling, boundary consistency, and generalization to mesh with different resolutions. To achieve these goals, we introduce the neural spline model and the graph attention network (GAT) into our models respectively. While the Neural-Spline based model provides more flexibility for large mesh deformation, the GAT based model can handle domains with more complicated shapes and is better at performing delicate local deformation. We validate our methods on stationary and time-dependent, linear and non-linear equations, as well as regularly and irregularly shaped domains. Compared to the traditional Monge-Ampère method, our approach can greatly accelerate the mesh adaptation process by three to four orders of magnitude, whilst achieving comparable numerical error reduction.

With rapid population growth and urbanization, municipal solid waste (MSW) generation rates are rising around the world. Incineration is considered as an effective way to deal with the growing demand for MSW disposal. The prediction of concentrations of pollutants generated from MSW incineration plants is a powerful support for waste incineration process control, which aims to reduce pollutant emissions. This paper proposes a two-stage prediction method based on the long short-term memory (LSTM) network to forecast typical flue gas pollutants of an MSW incineration plant in South China. In the first stage, an LSTM-based classification model is utilized to determine whether the pollutants are at a low level, and over-sampling strategy is applied to deal with the class-imbalance problem. At the second stage, an LSTM-based prediction model is built to further estimate the amounts of pollutants. Besides, the sliding average method is used to process the multi-scale raw data. Experiment results proved the effectiveness of the proposed method and indicated how different inputs affect the forecasting of pollutant emissions.

Feedback mechanism has been widely used in wireless communication such as channel equalization and resource allocation. In recent years, deep learning (DL) has made great progress in the field of wireless communication. There is now some work that attempts to introduce plain feedback mechanisms into DL algorithm to solve wireless communication problems. However, the improvement of plain feedback DL methods is limited in complex situations due to those methods lack sufficient learning ability on feedback information. In this paper, we propose a Multi-Agent Feedback Enabled Neural Network (MAFENN) equalizer, which consists of a specific learnable feedback agent and two feed-forward agents. Three fully cooperative intelligent agents help the system improve the ability to remove wireless inter-symbol interference (ISI) in receiving ends. We further formulate it into a three-player Stackelberg Game, which helps us to optimize and train this model more efficiently. To verify the feasibility of our proposed MAFENN system and the Stackelberg Game optimization, we conduct a series of experiments to compare the symbol error rate (SER) performance of the MAFENN equalizer and the other methods which utilizes quadrature phase-shift keying (QPSK) modulation scheme. Our performance outperforms that of the other equalizers at different signal-to-noise ratio (SNR) settings for both linear and nonlinear channels.

Benefiting from the breakthrough development of the fifth generation (5G), beyond 5G (B5G) wireless communication networks and Artificial Intelligence (AI) in recent years, the artificial intelligence of things (AIoT) is a new trend in the future. AIoT devices often have high-quality wireless video transmission requirements. However, the propagating signals at millimeter wave suffer from high propagation loss and sensitivity to blockage, resulting in the received video is vulnerable to be interfered. Due to the ability of Deep Learning (DL) to discover and learn good representations, some DL methods have achieved breakthrough performance in video recovery. However, most of these methods cannot exploit information from the higher to lower level to refine themselves. In this paper, we propose a novel retrospective thinking based multi-agent (ReTMA) system to solve the interference problem experienced on wireless channels. Compared with other plain feedback models, we add a retrospective agent on the feedback loop, which makes the entire system have stronger capabilities to learn good representative features. We further formulate it as a Stackelberg game to analyze the dependency relationship between the agents and facilitate the complex training issue of the multiple agents. To verify the feasibility of ReTMA system, we randomly add masks to simulate the severe interference received by the video frames in wireless transmissions. Experimental results show that the performances of similarity index measure (SSIM), peak signal-to-noise ratio (PSNR) and classification accuracy all achieve significant gains compared with those of other plain feedback models at different mask ratios.

This paper presents a problem in power networks that creates an exciting and yet challenging real-world scenario for application of multi-agent reinforcement learning (MARL). The emerging trend of decarbonisation is placing excessive stress on power distribution networks. Active voltage control is seen as a promising solution to relieve power congestion and improve voltage quality without extra hardware investment, taking advantage of the controllable apparatuses in the network, such as roof-top photovoltaics (PVs) and static var compensators (SVCs). These controllable apparatuses appear in a vast number and are distributed in a wide geographic area, making MARL a natural candidate. This paper formulates the active voltage control problem in the framework of Dec-POMDP and establishes an open-source environment. It aims to bridge the gap between the power community and the MARL community and be a drive force towards real-world applications of MARL algorithms. Finally, we analyse the special characteristics of the active voltage control problems that cause challenges (e.g. interpretability) for state-of-the-art MARL approaches, and summarise the potential directions.

This paper presents the experiences gained from ongoing research collaboration between the School of Computer Engineering at Nanyang Technological University and the Southampton e-Science centre at the University of Southampton using a service-oriented approach for complex engineering design optimisation. The service-oriented approach enables programmatic collaboration to be realized while maintaining the autonomy of individual codes at the different institutes and organizations. In the current work, a genetic algorithm optimisation logic implemented as a grid service at Southampton is used to drive the design search process, while the aerodynamic analysis code located in Singapore is used to evaluate the objective function of the design points. Experience gathered from the current study on airfoil shape optimisation is valuable for establishing effective, efficient and customised programmatic links between institutions to solve complicated engineering design problems.