Recently, text-driven video generation has achieved tremendous progress. However, existing methods neglect the contexts of long short-term frames in the video, thereby compromising temporal consistency. They also encounter challenges of heavy memory costs due to the use of the standard temporal attention mechanism and misalignment between training videos and captions. Additionally, previous approaches for long video generation are flawed because they are hard to ensure content diversity and consistency. To alleviate these issues, we propose a novel Long Short-term Temporal Diffusion (LSTD) model to generate videos with superior temporal consistency. We introduce two novel temporal modules,i.e., the Short-term Temporal Convolution and the Long-term Temporal Attention. The former can learn short-term features with a shallow structure, and the latter concentrates on long-term information of complex motion with a new memory-efficient attention mechanism. The combination of the two modules can ensure the temporal consistency of the generated videos. Furthermore, a novel inference method for long video generation is also proposed, which can iteratively generate hundreds of video frames. Experimental results on UCF-101, MSR-VTT, and two long video benchmarks prove that our method achieves superior zero-shot inference performance even when the size of the training data is reduced by 26.5 times.