Traditional reinforcement learning (RL) algorithms face significant limitations in handling long-term tasks with sparse rewards. Recent advancements have leveraged large language models (LLMs) to enhance RL by utilizing their world knowledge for task planning and reward generation. However, planning-based approaches often depend on pre-defined skill libraries and fail to optimize low-level control policies, while reward-based methods require extensive human feedback or exhaustive searching due to the complexity of tasks. In this paper, we propose the Progress Reward Model for RL (PRM4RL), a novel framework that integrates task planning and dense reward to enhance RL. For high-level planning, a complex task is decomposed into a series of simple manageable subtasks, with a subtask-oriented, fine-grained progress function designed to monitor task execution progress. For low-level reward generation, inspired by potential-based reward shaping, we use the progress function to construct a Progress Reward Model (PRM), providing theoretically grounded optimality and convergence guarantees, thereby enabling effective policy optimization. Experimental results on robotics control tasks demonstrate that our approach outperforms both LLM-based planning and reward methods, achieving state-of-the-art performance.

ABSTRACT Renal CT images contain complex spatial volumetric information, and deep learning models have difficulty learning consistent features from images with different voxel spacing, resulting in suboptimal segmentation performance for kidney tumors. We have introduced a modified deep learning model called RAScU‐Net to address these challenges. The RAScU‐Net adopts a coarse‐to‐fine segmentation strategy and a novel voxel spacing unification method to segment renal regions while preserving CT image data. The residual asymmetric self‐calibration (RASc) and residual self‐calibration (ResSc) modules are designed to capture feature information from CT images with different voxel spacings, thereby establishing long‐range dependencies. Finally, the innovative Channel Attention Gate (CAG) module generates attention maps from multiple perspectives, facilitating the integration of high‐ and low‐level features. The experimental results demonstrate that the RAScU‐Net significantly surpasses existing kidney tumor segmentation networks, achieving Dice scores of 0.882 and 0.877 for tumor segmentation of KiTS19 and KiTS21, respectively.