Continuous-time dynamic graphs (CTDG) composed of chronological events, arise in many real-world complex network applications with high dynamics. Memory-based temporal graph neural networks, due to capacity to update and retain historical node states, excel at capturing dynamic changes, handling long-term dependencies with better interpretability compared to memory-less methods. Despite the effectiveness, memory-based methods suffer from two fatal fundamental flaws (i.e. long-standing staleness and memory offline update time issues), which are usually overlooked by constraints of rigid assumptions in recent works. To address the above flaws, in this work, we propose a novel Dynamic Fresh Memory Vigorous Preserver named DyFMVP, a well-designed state-traceable dual memory architecture for efficient anti-staling representation learning. To eliminate staleness within memory, a continuous memory neural ordinary differential equation process (NDP) is proposed to model the underlying memory transition distribution, rejuvenating the memory to a fresh state. In addition, a history-augmented temporal graph attention is proposed to enhance neighbor aggregation in a history-aware manner. Furthermore, a time-aligning RNN-ODE (TARO) restarter is proposed to be offline updating-free and restart at any future timestamps flexibly. Extensive experiments on various downstream tasks demonstrate the primary model DyFMVP and restarter TARO significantly outperform other state-of-the-art CTDG methods by a large margin, while also exhibiting strong robustness against extreme sparsity and long-term interval. Our code has been available on GitHub11https://github.com/CrescentMoon3/icde2025DyFMVP.