As integrated circuits continue to grow in both scale and complexity, macro placement plays a critical role in physical design, directly affecting chip-level performance, power, and area (PPA). Traditional macro placement methods, such as simulated annealing, analytical optimization, and reinforcement learning, face limitations including slow convergence, heavy dependence on large datasets, and over-reliance on intermediate PPA indicators rather than final PPA. Large language models (LLMs) offer strong generative power and semantic reasoning that can potentially automate macro layout tasks while addressing the aforementioned problems in traditional methods, but their limited understanding of layout rules and lack of iterative, feedback-driven refinement make direct application challenging. To address this, we propose CHIP-MAP, a macro placement framework based on multi-agent collaboration and feedback-driven optimization. Furthermore, we introduce two innovative tools: the Module Link Weight Analyzer (MWA) and the Standard Cell Usability Score (SCUS), which are designed to guide fine-grained layout refinement. We evaluate CHIP-MAP on five benchmarks ranging from low-power cores to large multi-core processors implemented at 130nm and 45nm technology nodes. Results show that it achieves up to 1.5% area reduction and an average repair of 61.6% of total negative slack (TNS), while also reducing wirelength and improving timing.

Human intelligence is adept at absorbing valuable insights from external knowledge. This capability is equally crucial for artificial intelligence. In contrast, classical reinforcement learning agents lack such capabilities and often resort to extensive trial and error to explore the environment. This paper introduces $\textbf{PAE}$: $\textbf{P}$lanner-$\textbf{A}$ctor-$\textbf{E}$valuator, a novel framework for teaching agents to $\textit{learn to absorb external knowledge}$. PAE integrates the Planner's knowledge-state alignment mechanism, the Actor's mutual information skill control, and the Evaluator's adaptive intrinsic exploration reward to achieve 1) effective cross-modal information fusion, 2) enhanced linkage between knowledge and state, and 3) hierarchical mastery of complex tasks. Comprehensive experiments across 11 challenging tasks from the BabyAI and MiniHack environment suites demonstrate PAE's superior exploration efficiency with good interpretability.

On-policy reinforcement learning methods, like Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO), often demand extensive data per update, leading to sample inefficiency. This paper introduces Reflective Policy Optimization (RPO), a novel on-policy extension that amalgamates past and future state-action information for policy optimization. This approach empowers the agent for introspection, allowing modifications to its actions within the current state. Theoretical analysis confirms that policy performance is monotonically improved and contracts the solution space, consequently expediting the convergence procedure. Empirical results demonstrate RPO's feasibility and efficacy in two reinforcement learning benchmarks, culminating in superior sample efficiency. The source code of this work is available at https://github.com/Edgargan/RPO.

Leveraging external knowledge and extracting valuable insights are efficient human practices when handling various tasks. In contrast, current artificial intelligence lacks this capability. Recent research aims to teach Reinforcement Learning (RL) agents to incorporate external knowledge in the form of natural language to accelerate exploration. A common assumption in many of these approaches is that all introduced external knowledge is inherently valuable. To eliminate this assumption, we introduce the Knowledge Extraction Exploration Framework (KEEF). KEEF comprises two key components: 1) a knowledge extractor, designed to filter useful external knowledge based on three dimensions, task relevance, environment relevance, and achievement difficulty, through a prediction network and a policy network; and 2) a policy executor, which is a knowledge-conditioned network facilitating joint reasoning between the useful knowledge extracted by the knowledge extractor and the current state of the environment. In eight challenging sparse reward BabyAI environments, KEEF has consistently demonstrated superior sampling efficiency compared to knowledge-based and traditional RL methods.

Conventional reinforcement learning (RL) algorithms often necessitate millions of environment interactions to ascertain an efficacious policy. In stark contrast, humans, leveraging their curiosity mechanisms, can develop proficient policies with minimal effort. Drawing inspiration from this observation, we introduce the Autoencoder Reconstruction Model(ARM), a curiosity-driven RL model that significantly reduces interactions while enhancing policy effectiveness. ARM employs an autoencoder module, utilizing a deep neural network to learn feature representations from the environment. ARM utilizes its Curiosity Measurement Module to motivate RL agents for effective exploration, particularly in environments with sparse rewards. ARM also introduces an innovative mechanism to balance the exploration-exploitation dilemma. Theoretical analyses reveal that the reward shaping introduced by the ARM aligns with the potential-based reward shaping paradigm, thereby preserving the optimality of reinforcement learning. We will release the source code and trained models to facilitate further studies in this research direction.

Reinforcement learning for hard-exploration tasks remains challenging due to the long-term dependence and sparse-and-delay rewards in complex environments. In these challenging tasks, intrinsic motivation has become a dominant paradigm to enable the agent to explore the environment when no external reward feedback is available. In this work, inspired by studies from the human memory mechanism, we present a mnemonic dictionary learning (MDL) model for intrinsic motivation in reinforcement learning. The MDL model leverages sparse dictionary learning to incremental abstract the exploration histories into a compact memory-like dictionary, providing an excellent intrinsic motivation model. This mnemonic dictionary model not only drives the agent to explore novel stats in the environments indicated by the memory reconstruction error but also helps the agent to remember the key states and structure of the environments using its learned bases and reconstruction coefficients. The proposed MDL model can serve as a generative module for existing exploration methods. Extensive experimental results on typical sparse-reward tasks demonstrate its effectiveness and applicability over several competing algorithms. We will release the source code and trained models to facilitate further studies in this research direction.

Heads-up no-limit Texas hold’em (HUNL) is the quintessential game with imperfect information. Representative priorworks like DeepStack and Libratus heavily rely on counter-factual regret minimization (CFR) and its variants to tackleHUNL. However, the prohibitive computation cost of CFRiteration makes it difficult for subsequent researchers to learnthe CFR model in HUNL and apply it in other practical applications. In this work, we present AlphaHoldem, a high-performance and lightweight HUNL AI obtained with an end-to-end self-play reinforcement learning framework. The proposed framework adopts a pseudo-siamese architecture to directly learn from the input state information to the output actions by competing the learned model with its different historical versions. The main technical contributions include anovel state representation of card and betting information, amultitask self-play training loss function, and a new modelevaluation and selection metric to generate the final model.In a study involving 100,000 hands of poker, AlphaHoldemdefeats Slumbot and DeepStack using only one PC with threedays training. At the same time, AlphaHoldem only takes 2.9milliseconds for each decision-making using only a singleGPU, more than 1,000 times faster than DeepStack. We release the history data among among AlphaHoldem, Slumbot,and top human professionals in the author’s GitHub repository to facilitate further studies in this direction.

This work considers the problem of deep reinforcement learning (RL) with long time dependencies and sparse rewards, as are found in many hard exploration games. A graph-based representation is proposed to allow an agent to perform self-navigation for environmental exploration. The graph representation not only effectively models the environment structure, but also efficiently traces the agent state changes and the corresponding actions. By encouraging the agent to earn a new influence-based curiosity reward for new game observations, the whole exploration task is divided into sub-tasks, which are effectively solved using a unified deep RL model. Experimental evaluations on hard exploration Atari Games demonstrate the effectiveness of the proposed method. The source code and learned models will be released to facilitate further studies on this problem.