Deep reinforcement learning aims to learn deep neural network policies to solve large-scale decision-making problems. However, approximating policies using deep neural networks makes it difficult to interpret the learned decision-making process. To address this issue, prior works (Trivedi et al., 2021; Liu et al., 2023; Carvalho et al., 2024) proposed to use human-readable programs as policies to increase the interpretability of the decision-making pipeline. Nevertheless, programmatic policies generated by these methods struggle to effectively solve long and repetitive RL tasks and cannot generalize to even longer horizons during testing. To solve these problems, we propose the Hierarchical Programmatic Option framework (HIPO), which aims to solve long and repetitive RL problems with human-readable programs as options (low-level policies). Specifically, we propose a method that retrieves a set of effective, diverse, and compatible programs as options. Then, we learn a high-level policy to effectively reuse these programmatic options to solve reoccurring subtasks. Our proposed framework outperforms programmatic RL and deep RL baselines on various tasks. Ablation studies justify the effectiveness of our proposed search algorithm for retrieving a set of programmatic options.

Summary In this paper, we propose a deep reinforcement‐based approach to generate an animation of an agent using a broom‐like tool to transport a target object. The tool is attached to the agent. So when the agent moves, the tool moves as well.The challenge is to control the agent to move and use the tool to push the target while avoiding obstacles. We propose a direction sensor to guide the agent's movement direction in environments with static obstacles. Furthermore, different rewards and a curriculum learning are implemented to make the agent efficiently learn skills for manipulating the tool. Experimental results show that the agent can naturally control the tool with different shapes to transport target objects. The result of ablation tests revealed the impacts of the rewards and some state components.

Timing macro modeling has been widely employed to enhance the efficiency and accuracy of parallel and hierarchical timing analysis. However, existing studies primarily focused on generating an accurate and compact timing macro model for single-corner libraries, making it difficult to adapt these approaches to multi-corner situations. This either incurs substantial engineering effort or results in significant performance degradation. To tackle this challenge, we offer a fresh perspective on the timing macro modeling problem by drawing inspiration from recommendation systems and formulating it as a matrix completion task. We propose a neural collaborative filtering-based framework capable of capturing the convoluted relationships between circuit pins and timing corners. This framework enables the precise identification of timing variant regions across different corners. Additionally, we design several training features and implement various training techniques to enhance precision. Experimental results show that our framework reduces model sizes by more than 10% compared to state-of-the-art single-corner approaches, while maintaining competitive timing accuracy and exhibiting significant runtime improvements. Furthermore, when applied to unseen corners, our framework consistently delivers superior performance, demonstrating its potential for use in off-corner chiplets in a heterogeneous integration system.

Aiming to produce reinforcement learning (RL) policies that are human-interpretable and can generalize better to novel scenarios, Trivedi et al. (2021) present a method (LEAPS) that first learns a program embedding space to continuously parameterize diverse programs from a pre-generated program dataset, and then searches for a task-solving program in the learned program embedding space when given a task. Despite the encouraging results, the program policies that LEAPS can produce are limited by the distribution of the program dataset. Furthermore, during searching, LEAPS evaluates each candidate program solely based on its return, failing to precisely reward correct parts of programs and penalize incorrect parts. To address these issues, we propose to learn a meta-policy that composes a series of programs sampled from the learned program embedding space. By learning to compose programs, our proposed hierarchical programmatic reinforcement learning (HPRL) framework can produce program policies that describe out-of-distributionally complex behaviors and directly assign credits to programs that induce desired behaviors. The experimental results in the Karel domain show that our proposed framework outperforms baselines. The ablation studies confirm the limitations of LEAPS and justify our design choices.

As modern photolithography feature sizes continue to shrink, sub-resolution assist feature (SRAF) generation has become a key resolution enhancement technique to improve the manufacturing process window. State-of-the-art works resort to machine learning to overcome the deficiencies of model-based and rule-based approaches. Nevertheless, these machine learning-based methods do not consider or implicitly consider the optical interference between SRAFs, and highly rely on post-processing to satisfy SRAF mask manufacturing rules. In this paper, we are the first to generate SRAFs using reinforcement learning to address SRAF interference and produce mask-rule-compliant results directly. In this way, our two-phase learning enables us to emulate the style of model-based SRAFs while further improving the process variation (PV) band. A state alignment and action transformation mechanism is proposed to achieve orientation equivariance while expediting the training process. We also propose a transfer learning framework, allowing SRAF generation under different light sources without retraining the model. Compared with state-of-the-art works, our method improves the solution quality in terms of PV band and edge placement error (EPE) while reducing the overall runtime.

Abstract In this article, we propose a framework to generate cart‐pulling animation using deep learning in a multiagent environment. Mainly, two workers pull a cart using ropes and interact with crowd agents which exhibit following, wandering, and evasion behaviors. The main idea is to train a policy to learn an individual behavior of the workers and crowd agents. Furthermore, the challenge is that as the ropes are flexible, rewards are designed carefully so that the workers pull the ropes in a collaborative and consistent manner. Hence, the workers can pull the cart while avoiding collision with the crowd agents and surrounding static objects. In the stage of animation generation, we assign the policies deliberately to the workers and the crowd agents so that they interact with each other naturally. We conducted experiments on animal characters and the system could produce animations of characters with diverse behaviors.

Ultra-low quiescent current ($I_{Q}$) low-dropout regulator is the only solution for compact size Internet of Things (IoT) electronic devices. This paper presents an ultra-low$I_{Q}$low dropout regulator, including low$I_{Q}$error amplifier (EA) compensated by the adaptive current control (ACC), low leakage feedback network, low$I_{Q}$current comparator, analog transient enhancement (ATE), and digital transient enhancement (DTE). The chip was fabricated in a standard$0.5~\mu \text{m}$CMOS process. Measurement results show the current peak efficiency of the LDO is as high as 99.99%. Besides, owing to ATE and DTE circuits, when the load current changes from 1mA to 50mA with a 10 ns edge time, the measured undershoot and overshoot voltages are 75 mV and 50 mV, respectively, with the recovery time ($T_{R}$) of 60 ns and 80 ns, respectively, where the best 0.003 ps FoM is achieved.

Abstract This paper proposes a framework that integrates reinforcement learning and blend‐trees to generate animation of multiple agents for object transportation. The main idea is that in the learning stage, policies are learned to control agents to perform specific skills, including navigation, pushing, and orientation adjustment. The policies determine the blending parameters of the blend‐trees to achieve locomotion control of the agents. In the simulation stage, the policies are combined to control the agents to navigate, push objects, and adjust orientation of the objects. We demonstrated several examples to show that the framework is capable of generating animation of multiple agents in different scenarios.