This paper explores using large language models (LLMs) as low-level action planners for embodied tasks. While LLMs excel as the robot’s “brain” for high-level planning, they face challenges in directly controlling the “body” by generating precise low-level actions. This limitation arises from LLMs’ strength in high-level conceptual understanding but their inability to handle spatial perception effectively, restricting their potential in embodied tasks. To address this, we bridge the gap by enabling LLMs to not only comprehend complex instructions but also produce actionable, low-level plans. We introduce Room to Chessboard (R2C), a novel semantic representation that maps environmental states onto a grid-based chessboard, empowering LLMs to generate specific low-level coordinates and guide the robot in a manner akin to playing a game of chess. To further enhance decision-making, we propose the Chain-of-Thought Decision (CoT-D) paradigm, which improves LLMs’ interpretability and context-awareness in spatial reasoning. By jointly training LLMs for high-level task decomposition and low-level action generation, we create a unified "brain-body" system capable of handling complex, free-form instructions while producing precise low-level actions, allowing the robot to flexibly control its movements and adapt to varying tasks. We validate R2C using both fine-tuned open-source LLMs and GPT-4, demonstrating effectiveness on the challenging ALFRED benchmark. Results show that with our R2C framework, LLMs can effectively act as low-level planners, generalizing across diverse settings and open-vocabulary robotic tasks. The code and demonstrations are available at: https://vipl-vsu.github.io/Room2Chessboard.

This paper proposes a coordinated optimization of a geothermal-solar-wind multi-energy microgrid (MEM) considering energy quality. In this MEM, geothermal, solar, and wind energy are converted and conditioned for electricity, thermal energy, and gas productions, in which multi-energy complementarities are fully exploited based on electrolytic thermos-electrochemical effects. Exergy is introduced to understand and quantify the quality of energy carriers, which is formulated as an objective function to reduce the exergy loss rate. The aforementioned issue is thus addressed through the implementation of coordinated optimization to achieve a trade-off between two conflicting objectives. Simulation results on a high-renewable MEM reveal the effectiveness and superiority on flexibility and overall economic enhancements.

Aiming at the renewable energy access problems and the three-phase power imbalance of high-speed railway traction power supply system, the most promising multi-port railway power conditioner (MRPC) is studied and a hybrid optimal power flow control method is proposed to comprehensively consider the maximum renewable energy access and the tolerance for negative sequence currents. Firstly, as the number of locomotives and renewable energy generation are volatile and uncertain, the possibility of power flow should be analyzed and the capacity design of MRPC is accomplished. Secondly, to take full advantage of power converters’ capacity of MRPC, a hybrid optimal power-flow control method is proposed by ac power-flow optimization to minimize negative sequence currents and DC power-flow optimization to maximum renewable energy access. Finally, Simulations are accomplished to verify the effectiveness of the hybrid optimal power flow compensation control to renewable energy access and power quality improvements.

This article proposes a cascaded multilevel battery energy storage based parallel dynamic voltage compensator (DVC) for medium voltage industrial distribution systems. In this parallel DVC, fast switch is series-connected for voltage sag detection, and cascaded multilevel battery energy storage is parallel-connected for voltage support during the voltage sag period. According to the grid-side voltage sags or not, corresponding circuits and three-layer control strategies are developed. While the feedforward proportional–integral control method is adopted in the outer layer to track the ideal compensation voltage when the grid-side voltage sags, the ideal compensation current when the grid-side voltage does not sag is obtained through battery charging current as well as reactive power, harmonics, and negative sequence load current components. In order to cope with parameter differences in cascaded multilevel battery energy storage, the middle layer interphase and the inner layer intermodular state of charge equalization control are adopted in two scenarios. Furthermore, the parallel DVC is quantitatively compared with series DVC in the aspect of required battery capacity, and their advantages and disadvantages are systematically analyzed. Case studies are performed to verify the effectiveness and superiority of the proposed methodology on voltage support.

Video question answering (VideoQA) is challenging since it requires the model to extract and combine multi-level visual concepts from local objects to global actions from complex events for compositional reasoning. Existing works represent the video with fixed-duration clip features that make the model struggle in capturing the crucial concepts in multiple granularities. To overcome this shortcoming, we propose to represent the video with an Event Graph in a hierarchical structure whose nodes correspond to visual concepts of different levels (object, relation, scene and action) and edges indicate their spatial-temporal relationships. We further propose a H ierarchical S patial- T emporal T ransformer (HSTT) which takes nodes from the graph as visual input to realize compositional reasoning guided by the event graph. To fully exploit the spatial-temporal context delivered from the graph structure, on the one hand, we encode the nodes in the order of their semantic hierarchy (depth) and occurrence time (breadth) with our improved graph search algorithm; On the other hand, we introduce edge-guided attention to combine the spatial-temporal context among nodes according to their edge connections. HSTT then performs QA by cross-modal interactions guaranteed by the hierarchical correspondence between the multi-level event graph and the cross-level question. Experiments on the recent challenging AGQA and STAR datasets show that the proposed method clearly outperforms the existing VideoQA models by a large margin, including those pre-trained with large-scale external data. Our code is available at https://github.com/ByZ0e/HSTT.

Video Question Answering (VideoQA) has emerged as a vital tool to evaluate agents’ ability to understand human daily behaviors. Despite the recent success of large vision language models in many multi-modal tasks, complex situation reasoning over videos involving multiple human-object interaction events still remains challenging. In contrast, humans can easily tackle it by using a series of episode memories as anchors to quickly locate question-related key moments for reasoning. To mimic this effective reasoning strategy, we propose the Glance- Focus model. One simple way is to apply an action detection model to predict a set of actions as key memories. However, these actions within a closed set vocabulary are hard to generalize to various video domains. Instead of that, we train an Encoder-Decoder to generate a set of dynamic event memories at the glancing stage. Apart from using supervised bipartite matching to obtain the event memories, we further design an unsupervised memory generation method to get rid of dependence on event annotations. Next, at the focusing stage, these event memories act as a bridge to establish the correlation between the questions with high-level event concepts and low-level lengthy video content. Given the question, the model first focuses on the generated key event memory, then focuses on the most relevant moment for reasoning through our designed multi-level cross- attention mechanism. We conduct extensive experiments on four Multi-Event VideoQA benchmarks including STAR, EgoTaskQA, AGQA, and NExT-QA. Our proposed model achieves state-of-the-art results, surpassing current large models in various challenging reasoning tasks. The code and models are available at https://github.com/ByZ0e/Glance-Focus.

Set-based Face Recognition is widely applied in scenarios like law enforcement and online media data management. Compared with face recognition using a single image, the faces in the set often contain abundant appearance changes. Therefore, how to make full use of the rich information from the set and integrate them into a unified set representation become the key to set-based face recognition. Inspired by the fact that humans usually complete this fine-grained task through integrating the information from the congruent local regions (e.g. an eye to an eye) of multiple faces in a set, we propose a novel method called Local Feature Enhancement Network (LFENet), which can automatically enhance the local feature through transferring the local information across the images. Specifically, we retain the spatial semantic information of the feature maps and apply different relational functions to establish the correlation among the local features. The contained local information will be transferred to the relevant local features to enhance their discriminability. By doing so, the valuable local information carried in some local features can complement those with incomplete information. Besides, the various local information is aligned across faces under different conditions to help the model learn intra-set-compact face representations. Our method achieves state-of-the-art performances on two mainstream set-based face recognition benchmarks: IJB-A and IJB-C, which fully reflects the rationality and effectiveness of our local feature enhancement mechanism.

Visual understanding goes well beyond the study of images or videos on the web. To achieve complex tasks in volatile situations, the human can deeply understand the environment, quickly perceive events happening around, and continuously track objects’ state changes, which are still challenging for current AI systems. To equip AI system with the ability to understand dynamic ENVironments, we build a video Question Answering dataset named Env-QA. Env-QA contains 23K egocentric videos, where each video is composed of a series of events about exploring and interacting in the environment. It also provides 85K questions to evaluate the ability of understanding the composition, layout, and state changes of the environment presented by the events in videos. Moreover, we propose a video QA model, Temporal Segmentation and Event Attention network (TSEA), which introduces event-level video representation and corresponding attention mechanisms to better extract environment information and answer questions. Comprehensive experiments demonstrate the effectiveness of our framework and show the formidable challenges of Env-QA in terms of long-term state tracking, multi-event temporal reasoning and event counting, etc.