Hierarchical Federated Learning (HFL) has emerged as a popular federated learning method by introducing additional aggregation levels using intermediate edge servers. The performance of HFL approaches is contingent upon the aggregation frequency and the number of intermediate aggregation rounds. Existing approaches mainly focus on optimizing the aggregation frequency only, neglecting the impact of intermediate aggregation rounds on training performance. On the other hand, these methods also fail to consider the multidimensional effects of aggregation frequency, consequently focusing only on the performance accuracy while overlooking the detrimental effects on both the communication costs and training latency. This paper introduces HFL-PBRL, a novel HFL framework that employs a bi-level reinforcement learning (RL)-based algorithm to jointly optimize aggregation frequencies and rounds across edge servers. This algorithm is accompanied by a Pareto-efficient multi-objective optimization approach to strike an optimal trade-off among model accuracy, communication cost and convergence time. The varied aggregation frequencies and rounds might introduce inconsistencies; therefore, we employ a hierarchical pullback mechanism that iteratively pulls the client models toward a synchronized anchor model, ensuring effective divergence control. Furthermore, we devise a harmonic weight assignment strategy that dynamically adjusts the aggregation weights of each model based on their current, historical, and anticipated divergence, addressing the model fluctuations and asynchrony. Extensive evaluations demonstrate that HFL-PBRL consistently achieves high model accuracy and faster convergence with minimal communication costs compared to baselines and SOTA.