Due to their continuous electromechanical deformation, rate-dependent viscoelasticity, and complex mechanical vibration, dynamic modeling and high-speed tracking control of dielectric elastomer actuators (DEAs) remain elusive, significantly limiting their working bandwidth. In this work, we propose a Physics-Informed Token Prediction (PITP) that enables accurate modeling of DEA dynamics and high-speed feedforward tracking control. The PITP framework consists of two key components: a physics-informed encoder and a dynamic decoder. The physics-informed encoder is designed based on a simplified equivalent linear model and trained through the hierarchical optimization training method, which embeds the global dynamic characteristics into tokens, minimizing the need for extensive data and training. Then, the dynamic decoder is developed by using these tokens as state-dependent parameters, capable of describing complex dynamic responses through the autoregressive solution. Finally, by taking advantage of the model's reversibility, a direct inverse compensator is established to linearize the input-output relationship. Experimental results of several DEAs with different configurations and payloads demonstrate that, based on our PITP framework, the complex nonlinear dynamic responses of all DEAs can be precisely described and eliminated within their natural frequency, validating its generality and versatility. By leveraging fast modeling ($< $30 minutes) and high-speed feedforward tracking control, our PITP framework may accelerate DEAs' practical applications.