Abstract State-space models (SSMs), exemplified by S4, have introduced a novel context modeling method by integrating state-space techniques into deep learning. Despite their effectiveness, SSMs struggle with global context modeling due to data-independent matrices. The Mamba model addresses this with data-dependent variants enabled by the S6 selective-scan algorithm, enhancing context modeling, especially for long sequences. However, Mamba-based architectures face significant parameter scalability challenges, limiting their utility in vision applications. This paper tackles the scalability issue of large SSMs for image classification and action recognition without relying on additional techniques like knowledge distillation. We analyze the distinct characteristics of Mamba-based and Attention-based models, proposing a Mamba-Attention interleaved architecture that enhances scalability, robustness, and performance. We demonstrate that the stable and efficient interleaved architecture resolves the scalability issue of Mamba-based architectures and increases robustness to common corruption artifacts. Our thorough evaluation on the ImageNet-1K, Kinetics-400, and Something-Something-v2 benchmarks demonstrates that our approach improves the accuracy of state-of-the-art Mamba-based architectures by up to $$+1.7$$ + 1.7 %.

We present MixANT, a novel architecture for stochastic long-term dense anticipation of human activities. While recent State Space Models (SSMs) like Mamba have shown promise through input-dependent selectivity on three key parameters, the critical forget-gate ($\textbf{A}$ matrix) controlling temporal memory remains static. We address this limitation by introducing a mixture of experts approach that dynamically selects contextually relevant $\textbf{A}$ matrices based on input features, enhancing representational capacity without sacrificing computational efficiency. Extensive experiments on the 50Salads, Breakfast, and Assembly101 datasets demonstrate that MixANT consistently outperforms state-of-the-art methods across all evaluation settings. Our results highlight the importance of input-dependent forget-gate mechanisms for reliable prediction of human behavior in diverse real-world scenarios.

Stereo images are used in various applications, such as autonomous driving, surveillance, robotics, and 3D-TV. Those images are captured by two horizontally adjacent cameras, capturing a scene from two different points of view. In this work, we propose an end-to-end trainable recurrent neural network (RNN) for stereo image compression, which we call RNNSC. The RNN allows variable compression rates without retraining of the network due to the iterative nature of the recurrent units. The proposed method makes use of the redundancies, to reduce the overall bit rate. Each image in the stereo pair has its separate encoder and decoder network similar to [1]. We propose to share the mutual information between the stereo pair networks by warping the hidden states of one codec network to the other with the help of disparity information that is coded and transmitted independently via JPEG2000. Moreover, we also improve the quality of the shared mutual information by eliminating wrong information by estimating and applying occlusion maps which are computed with a convolutional neural network without direct supervision. The proposed method outperforms all tested image codecs on MS-SSIM, a perceptual metric capturing the structural quality of an image, as shown in Table 1.

In this work, we propose an end-to-end trainable recurrent neural network for stereo image compression. The recurrent neural network allows variable compression rates without retraining the network due to the iterative nature of the recurrent units. The proposed method leverages the fact that stereo images have overlapping fields of view, i.e., mutual information, to reduce the overall bit rate. Each image in the stereo pair has its separate encoder and decoder network. We propose to share the mutual information between the stereo pair networks by warping the hidden states of one of the stereo image network’s recurrent layers to the other stereo image network’s recurrent layers. Moreover, we also improve the quality of the shared mutual information by eliminating the wrong information by estimating occlusion maps using a convolutional neural network. The proposed method results show significant bit rate savings compared to the single image compression baseline model and traditional codecs.