Millimeter wave (mmWave) and terahertz (THz) MIMO systems typically rely on predefined beamforming codebooks for initial access and data transmission. However, these codebooks are often unoptimized for specific conditions, leading to large sizes and significant beam training overhead, thereby complicating support for highly mobile applications. This paper introduces a reinforcement learning framework that optimizes beam patterns using only receive power measurements, adapting to the environment, user distribution, and hardware constraints without prior channel knowledge. The framework explores three reinforcement learning algorithms: Deep Deterministic Policy Gradient (DDPG), Twin Delayed Deep Deterministic Policy Gradient (TD3), and Soft Actor-Critic (SAC). While reinforcement learning has shown promise for beamforming, a comprehensive comparative analysis of advanced RL algorithms under a combination of realistic challenges, such as Non-Line-of-Sight (NLoS) conditions and hardware impairments for adaptive beam codebook design in mmWave/THz systems has been largely unexplored. This paper presents the first such in-depth comparative study. Simulation results demonstrate the superiority of the SAC algorithm, achieving higher beamforming gain and faster convergence compared to DDPG and TD3 in various scenarios, including LoS and NLoS conditions, even with hardware impairments.