Predicting the future popularity of individual information cascades has attracted much attention in various application fields. It is significantly important for online advertising, viral marketing, rumor detection, and social recommendation. Most approaches target modeling forwarding path or learning important information from discrete static graph. These methods either extract complicated hand-crafted features that rely on domain knowledge and have lower generality, or devote to modeling the arriving intensity function of each message and cannot be optimized for the final popularity. Despite some approaches trying to utilize the underlying structural information in discrete snapshots, they neglect to model the temporal information that implicitly underlying abundant diffusion patterns. Meanwhile, they ignore the inherent dependencies among forwarding behaviors of users. In this paper, we propose a novel learning framework for popularity prediction via modeling temporal dependencies on dynamic evolution process, called TEDDY. Our framework not only models the temporal evolution in a separate snapshot via multiple sequences temporal encoder, but also captures the inherent temporal dependencies among different snapshots. We have conducted extensive experiments on two real-world datasets, i.e. Sina Weibo and American Physical Society. Experimental results demonstrate that our proposed TEDDY significantly improves the prediction accuracy and is superior to the state-of-the-art approaches.