Transformer architectures have shown strong potential for profiling side-channel analysis (SCA), but their high computational complexity and substantial parameter overhead remain major obstacles to practical deployment in resource-constrained scenarios, especially under severe leakage-trace desynchronization. To address this challenge, we propose TmsNet, an efficient Transformer-based architecture tailored for desynchronized side-channel leakage analysis. TmsNet revisits the Transformer backbone by coupling multi-scale leakage representation with shift-aware modeling, thereby reducing model complexity while maintaining strong key-recovery capability under pronounced temporal misalignment. TmsNet is characterized by three key design components. First, it employs a multi-scale feature extraction module with temporal convolution kernels of different receptive fields to jointly capture fine-grained local leakage patterns and longer-range temporal dependencies, thereby reducing the sensitivity of vanilla Transformers to local misalignment. Second, it introduces a residual fusion mechanism to align and aggregate cross-scale features, yielding more robust feature representations under temporal perturbations. Third, it integrates relative positional encoding with Softmax-based normalization, replacing conventional LayerNorm to better preserve subtle leakage contrasts that may be weakened by mean-variance normalization, while mitigating the impact of countermeasure-induced random delays and clock jitter. Extensive evaluations on the desynchronized ASCADf, ASCADr, and TinyPower benchmark datasets show that TmsNet reduces parameter count by up to 78.5% relative to baseline models while maintaining competitive attack performance. Notably, it recovers the secret key using only 48, 42, and 28 attack traces on the three datasets, respectively, demonstrating an effective balance between architectural efficiency and practical profiling SCA capability.