Heterogeneous network is a dedicated approach to leveraging the potential benefits of all existing wireless networks by integrating diverse radio access technologies with varying specifications to fulfill the demands of mobile users. Two primary factors that influence a mobile user to switch network connections are their mobility and changing network conditions. An efficient vertical handover solution can enable the seamless transfer of an ongoing user's connection to a better-suited network. This paper proposes an intelligent, optimized vertical handover decision algorithm with a two-phase approach. First, a feasible network category (i.e., small cell or macro cell) is determined based on user speed. This phase ensures fast-moving users do not experience handover abnormalities during communication. Then, an optimal target node is selected from this feasible set using a hybrid multi-criteria decision-making approach. Second, a dynamic$\epsilon$-greedy-based Q-learning technique is employed to learn the near-optimal handover triggering point, ensuring timely and successful handover execution. Crucially, the$\epsilon$value is not rigidly fixed; instead, it adapts dynamically, driven by rewards from the environment. The conclusions drawn from network simulations illustrate that the proposed algorithm outperforms other existing algorithms by minimizing handover delays, ping-pong effects, failure rates, and packet loss rates.