In multivariate long-term time series forecasting, the success of self-attention is commonly attributed to the attention matrix that encodes token interactions.In this paper, we provide evidence that challenges this view.Through extensive experiments on three classic and three latest Transformer models, we find that dotproduct attention can be replaced by elementwise operations without token interaction, such as the addition and Hadamard product, while maintaining or even improving accuracy.This motivates our central hypothesis: the effectiveness of self-attention in this task arises not from the dynamic attention matrix, but from the multi-branch feature extraction enabled by the parallel Query, Key, and Value projections and their fusion.To validate this hypothesis, we construct a minimalist multi-branch MLP that isolates the 'multi-branch mapping with element-wise operation' structure from the Transformer and show that it achieves competitive performance.Our findings indicate that the source of performance in self-attention is often misinterpreted, as its actual advantage stems from the architectural principle of multi-branch mapping and fusion, rather than the attention matrix.

Time series forecasting is always important, with multivariate long-term series forecasting being its most challenging task. Here, the existing methods typically learn only in a single domain and focus on optimizing model structures, leading to incomplete information mining and imprecise predictions. To address this, we propose a generalized Mixture of Multi-Domain Experts (MoME) for multivariate long-term series forecasting. Unlike most existing methods, MoME focuses on multi-perspective information mining and fusing. To this end, MoME transforms time series into the frequency and spatial domains to learn their respective representations. MoME regards variates information as embedded features and applies fast Fourier transform to the time dimension. Then it learns embedded features in the frequency domain. In spatial domain learning, MoME applies self-attention mechanism on the variates dimension to efficiently capture dependencies among multiple variates. Finally, MoME fuses the outputs from all domains, reinterprets and integrates information across multiple domains, and predicts future time series. Extensive experiments prove that MoME outperforms state-of-the-art (SOTA) methods. Code is available at: https://github.com/lxy-PhD2022/MoME

The existing monocular methods face huge challenges in reconstructing credible details of non-visible areas in large pose images. Due to the fact that facial details are lost in non-visible areas of large pose images, existing methods lose basis when reconstructing details, resulting in unreliable results. Even if the generative model is used to repair the image first, the reconstruction process is very complex and costly. To this end, we propose an end-to-end and self-supervised RGB to depth method that equates small pose to large pose in UV space to obtain labels for non-visible areas. Then, we infer the depth values of non-visible areas and reconstruct a detailed 3D face. Finally, we render it into a face image and use it alongside the labels for self-supervised training of the network. During inference for large pose image, our method could reconstruct credible details of non-visible areas with a basis, rather than blindly. In addition, coarse and detailed reconstruction have mutually exclusive requirements for training images while coarse reconstruction is often not met, resulting in limited reconstruction accuracy. We propose a mutual exclusion elimination method to solve it, improving its accuracy. Extensive experiments demonstrate that our method could reconstruct precise 3D faces with credible details from large pose images. Our supplementary material is published on: https://github.com/lxy-nxu/ICASSP2025/tree/main

Multi-scale analysis, a classical and crucial methodology, is extensively employed in multivariate long-term time series forecasting. However, the existing methods struggle to model arbitrary scales, thus limiting their ability to delve into complex patterns, which in turn limits the forecasting precision. Therefore, we propose the HMSformer, a multi-scale Transformer that includes a shifted stacked embedding mechanism and a unified hierarchical framework, solving this problem in terms of the in-depth and comprehensiveness of multi-scale analysis. The mechanism conducts multi-scale modeling by sampling at different positions and sizes. This sampling reflects the arbitrariness of scales and allows for a thorough scan of any scale, supporting a deeper analysis of temporal relationships. Additionally, it converts input series into structured embeddings optimized for Transformer input, allowing the self-attention mechanism to effectively learn intricate cross-scale relationships. Moreover, the unified hierarchical framework integrates multi-scale modeling across different levels of abstraction by unifying global structures, local segments, and fine details into a consistent representation. Together, these two innovations ensure that HMSformer achieves a more comprehensive and in-depth analysis of temporal relationships. HMSformer demonstrates its effectiveness with a significant improvement, achieving an average mean squared error (MSE) that is 9.925% lower than the baseline across nine authoritative datasets. Source code is available at: https://github.com/lxy-PhD2022/HMSformer

Facing the key challenge in multivariate long-term series prediction, namely effectively modeling the long-term dependencies among high-dimensional variables, we propose an innovative forest network (ForeNet). Firstly, we construct a polytree, which takes variable as leaf node, convolution as edge, and progressive fusion as the root node. Polytree models the dependencies among variables from the bottom up, adopting a local-to-global progressive learning strategy. Moreover, polytree embeds the entire sequence as input channels for convolution, allowing interactions across arbitrary time steps between variables. Thereby, polytree could model long-term dependencies. Then, we construct multiple polytrees with varying branching factors, utilizing the self-attention mechanism to assign weights and combine polytrees into a forest. Forest adopts an ensemble learning strategy to capture complex patterns hidden under high-dimensional variables. Combining progressive and ensemble strategies, ForeNet could effectively address the key challenge mentioned at the beginning. Extensive experiments show that ForeNet could reduce the average MSE by up to 11.53% compared to the baseline, which verifies the effectiveness of ForeNet. Source code is available at: https://github.com/lxy-PhD2022/ForeNet

3D face alignment from monocular images in challenging scenarios such as large poses and occlusions presents a huge challenge. To overcome this challenge, we propose a Multi-granularity Perturbation Relearning Network (MPRN), utilizing relearning attention to capture crucial features. Specifically, MPRN employs an attention mechanism to highlight effective features and further conducts relearning for attention to refine its accuracy. However, in extreme scenarios, the loss of key 3D facial information hampers the effective functioning of relearning attention. To this end, we construct multi-granularity perturbation graphs to infer the missing key 3D facial information and correspondingly guide the multiple times learning of attention module using perturbation graphs at various granularities. By doing this, our MPRN could effectively capture crucial 3D facial features in extreme scenarios, thereby achieving precise 3D face alignment. Experiments on the AFLW 2000-3D and AFLW datasets demonstrate the effectiveness of our MPRN.