Achieving safe and efficient cooperative maneuvering in mixed highway traffic constitutes a significant engineering challenge, primarily due to the complex spatiotemporal coupling of multi-vehicle interactions and the inherent conflict between risk suppression and operational efficiency. This paper proposes a hierarchical cooperative control framework that formulates the overtaking task as a team-level constrained multi-agent Markov game with explicit safety budgets. Departing from heuristic penalty tuning or traditional safety layers, we introduce a Lagrangian-based policy optimization mechanism that models safety as a decoupled cost constraint and employs adaptive dual updates to regulate the safety–efficiency trade-off dynamically. A coordination-oriented reward and cost scheme is constructed to guide multiple agents in learning reciprocal yielding and efficient passing behaviors during complex interactions. Structurally, the framework integrates a hierarchical execution interface that maps high-level semantic maneuvers to low-level kinematic references. Under stochastic mixed traffic in highway-env, the proposed method limits the collision rate to 0.28% in the Extended traffic suite while maintaining improved TTC-based safety margins and robust time-gap performance. In addition, zero-shot transfer to the CARLA simulator achieves a 100% success rate with zero collisions under continuous vehicle dynamics and closed-loop control, demonstrating that the learned cooperative behavior remains executable beyond the original training environment. This work provides a scalable and reproducible solution for autonomous vehicle coordination under mixed traffic.