This study introduces a novel approach for the estimation of seismic source parameters using a multi-task learning network that incorporates a Fourier Transformer architecture. The Fourier Transformer is designed to extract information from both the time and frequency domains, which reduces the time complexity by utilizing Fast Fourier Transform (FFT) in place of the traditional attention mechanism in the Transformer encoder. The network consists of a shared encoder for general feature extraction and four task-specific decoders for parameter estimation. The model is both lightweight and accurate, capable of simultaneously estimating magnitude, epicentral distance, p travel time, and depth based on a 30-second single-station waveform. The proposed approach was validated using the Stanford Earthquake dataset (STEAD) and compared with the state-of-the-art techniques. The results show standard deviations of 0.19 for magnitude, 3.77 km for epicentral distance, 0.46 s for p travel time, and 5.77 km for depth, with a lower error and a faster response compared to the existing prediction framework. Code is available at https://github.com/KG-TSI-Civil/MFTnet.