Real-world dynamic systems often exhibit time-varying behavior. While state space models (SSMs) have shown great potential for sequence modeling, especially in capturing long-range dependencies, most previous studies have been limited to time-invariant dynamics. To overcome this limitation, we propose a neural network architecture based on time-varying SSMs with dynamics that evolve over time, called dynamic SSMs. To enhance scalability and efficiency, several techniques are introduced, including sparsification strategy via diagonalization and fast tensor convolution with quasi-linear complexity in sequence length. Extensive experiments on both synthetic and real-world datasets show that the proposed model consistently outperforms existing state-of-the-art methods. Moreover, the model achieves significantly lower time and space complexity compared to architectures such as Transformer and LSTM. This work advances the theoretical foundation of SSMs-based neural networks in deep learning and promotes their further development. Code is available at:https://github.com/leonty1/dssmhttps://github.com/leonty1/dssm.