Affordance grounding focuses on predicting the specific regions of objects that are associated with the actions to be performed by robots. It plays a vital role in the fields of human-robot interaction, human-object interaction, embodied manipulation, and embodied perception. Existing models often neglect the affordance shared among different objects because they lack the Chain-of-Thought(CoT) reasoning abilities, limiting their out-of-domain generalization and explicit reasoning capabilities. To address these challenges, we propose Affordance-R1, the first unified affordance grounding framework that integrates cognitive CoT guided Group Relative Policy Optimization (GRPO) within a reinforcement learning paradigm. Specifically, we designed a sophisticated affordance function, which contains format, perception, and cognition rewards to effectively guide optimization directions. Furthermore, we constructed a high-quality affordance-centric reasoning dataset, ReasonAff, to support training. Trained exclusively via reinforcement learning with GRPO and without explicit reasoning data, Affordance-R1 achieves robust zero-shot generalization and exhibits emergent test-time reasoning capabilities. Comprehensive experiments demonstrate that our model outperforms well-established methods and exhibits open-world generalization.

Integrating energy storage systems (ESSs) into the railway power flow controller (RPFC) offers a promising path to enhance the interaction capability and connection compatibility between the railway power systems (RPSs) and the utility. Despite its technical advantages, full consideration must be given to the media-type & configuration of ESS, the rating of RPFC, and the control framework that can compatibly integrate the control & planning for cost-efficiency improvement. For this problem, a novel concept that takes the real-time control and off-line configuration approach in a unified framework is proposed for hybrid ESS-integrated RPFC (HESS-RPFC) in this paper, in which the comprehensive techno-economic performances of the whole system are taken into account. First, a unified analytical power regulation framework is established for HESS-RPFC, in which the functions of energy management (EM) and power quality control are seamlessly integrated. To realize EM, improve RPS’s grid-connection compatibility, and reduce the rating of RPFC, a double-layer power regulation strategy of HESS-RPFC is developed on the proposed framework. Subsequently, a lifetime net benefit-oriented optimization model is formulated and solved, through which, the optimal rating configuration of the whole system and HESS’s mode-triggering thresholds can be obtained. Finally, a comprehensive performance evaluation based on real-measured data is conducted, revealing that HESS-RPFC with the proposed method provides superior economic benefits over single-media ESS schemes, ensures satisfactory grid-connection compatibility for the RPS, and reduces the RPFC’s rating by nearly 46%. Additionally, the real-time execution feasibility of the proposal is demonstrated by hardware-in-the-loop tests.

This paper proposes an effective and novel multi-agent deep reinforcement learning (MADRL)-based method for solving the joint virtual network function (VNF) placement and routing (P&R), where multiple service requests with differentiated demands are delivered at the same time. The differentiated demands of the service requests are reflected by their delay- and cost-sensitive factors. We first construct a VNF P&R problem to jointly minimize a weighted sum of service delay and resource consumption cost, which is NP-complete. Then, the joint VNF P&R problem is decoupled into two iterative subtasks: placement subtask and routing subtask. Each subtask consists of multiple concurrent parallel sequential decision processes. By invoking the deep deterministic policy gradient method and multi-agent technique, an MADRL-P&R framework is designed to perform the two subtasks. The newjoint reward and internal rewardsmechanism is proposed to match the goals and constraints of the placement and routing subtasks. We also propose the parameter migration-based model-retraining method to deal with changing network topologies. Corroborated by experiments, the proposed MADRL-P&R framework is superior to its alternatives in terms of service cost and delay, and offers higher flexibility for personalized service demands. The parameter migration-based model-retraining method can efficiently accelerate convergence under moderate network topology changes.

This paper presents a novel and effective deep reinforcement learning (DRL)-based approach to addressing joint resource management (JRM) in a practical multi-carrier non-orthogonal multiple access (MC-NOMA) system, where hardware sensitivity and imperfect successive interference cancellation (SIC) are considered. We first formulate the JRM problem to maximize the weighted-sum system throughput. Then, the JRM problem is decoupled into two iterative subtasks: subcarrier assignment (SA, including user grouping) and power allocation (PA). Each subtask is a sequential decision process. Invoking a deep deterministic policy gradient algorithm, our proposed DRL-based JRM (DRL-JRM) approach jointly performs the two subtasks, where the optimization objective and constraints of the subtasks are addressed by a new joint reward and internal reward mechanism. A multi-agent structure and a convolutional neural network are adopted to reduce the complexity of the PA subtask. We also tailor the neural network structure for the stability and convergence of DRL-JRM. Corroborated by extensive experiments, the proposed DRL-JRM scheme is superior to existing alternatives in terms of system throughput and resistance to interference, especially in the presence of many users and strong inter-cell interference. DRL-JRM can flexibly meet individual service requirements of users.

In 5G and beyond wireless communication systems, providing satisfactory service to users in high-mobility scenarios becomes very essential. Unstable service supply caused by frequent handover under high-mobility compromises the service experience of users. Moreover, the demanding requirements for processing delays and non-immediate information processing errors are particularly prominent. This paper proposes a novel learning-based approach to solving the dynamic multichannel access (DMCA) problem under fast time-varying channel arising from high-mobility. Specifically, we first propose the subjective experience-based quality of service, and formulate the corresponding optimization problem based on the designed access criterion. Invoking our proposed prediction-based deep deterministic policy gradient algorithm and incremental learning-based online channel prediction model, a novel DMCA scheme, which combines the recurrent neural network with deep reinforcement learning, is proposed. Corroborated by experiments built in real channel data, the performance of proposed learning-based DMCA scheme approaches that derived from the exhaustive search method when making a decision at each time-slot, and is superior to the exhaustive search method when making a decision for every few time-slots. Furthermore, our scheme significantly reduces processing delays and effectively alleviates errors caused by the non-immediacy of information acquisition and processing.

This paper deals with the problem of the dynamic multichannel access (MCA) based on the LTE-WLAN aggregation in dynamic heterogeneous networks. To ensure the users' personalized requirements, the minimization after satisfied (MAS) criterion is firstly proposed. Then, a prediction-based deep deterministic policy gradient (P-DDPG) algorithm is presented, achieving continuous processing. Finally, a new reward function is designed according to the MAS criteria. Meanwhile, the virtual user approach and the base station-centric strategy are proposed to design the action spaces so that the number of actions is independent of the number of users. Simulation results demonstrate the effectiveness of the P-DDPG algorithm for solving the dynamic MCA problem and corroborate that the proposed MAS criterion is superior to the existing criteria.