Though Transformer-based image super-resolution (SR) has made remarkable progress, the burdensome computation complexity hinders its applications in memory-limited devices. Existing efficient Transformer-based image SR methods mainly focus on designing efficient local window self-attention mechanisms to improve computational efficiency. However, the limited receptive field of local windows often fails to capture global contextual information effectively. Recently, the Selective State Space Model, e.g., Mamba, has shown powerful potential for long-range dependencies modeling with linear complexity. In this work, we propose a selective state space model for efficient image SR, dubbed S3SR. Specifically, we design the Local-then-Global Fusion Block as the core component, which employs different convolution and a 2D cross scan mechanism to take advantage of local patch texture and global relevance. Extensive experiments have demonstrated the superiority of our S3SR, which even outperforms the efficient Transformer-based SR methods, using less computational cost but with a larger global receptive field.

Efficient transformer-based models have made remarkable progress in image super-resolution (SR). Most of these works mainly design elaborate structures to accelerate the inference of the transformer, where all feature tokens are propagated equally. However, they ignore the underlying characteristic of image content, i.e., various image regions have distinct restoration difficulties, especially for large images (2K-8K), failing to achieve adaptive inference. In this work, we propose an adaptive token sparsification transformer (AdaFormer) to speed up the model inference for image SR. Specifically, a texture-relevant sparse attention block with parallel global and local branches is introduced, aiming to integrate informative tokens from the global view instead of only in fixed local windows. Then, an early-exit strategy is designed to progressively halt tokens according to the token importance. To estimate the plausibility of each token, we adopt a lightweight confidence estimator, which is constrained by an uncertainty-guided loss to obtain a binary halting mask about the tokens. Experiments on large images have illustrated that our proposal reduces nearly 90% latency against SwinIR on Test8K, while maintaining a comparable performance.

Deep neural networks have revealed enormous potential in video super-resolution (VSR), yet the expensive computational expense limits their deployment on resource-limited devices and actual scenarios, especially for restoring multiple frames simultaneously. Existing VSR models contain considerable redundant filters, which drag down the inference efficiency. To accelerate the inference of VSR models, we propose a scalable method based on adaptive patch routing to achieve practical speedup. Specifically, we design a confidence estimator to predict the aggregation performance of each block for adjacent patch information. It learns to dynamically perform block skipping, i.e., choose which basic blocks of the VSR network to execute during inference so as to reduce total computation to the maximum extent without degrading reconstruction accuracy dramatically. However, we observe that skipping error would be amplified as the hidden states propagate along with recurrent networks. To alleviate the issue, we design temporal feature alignment to guarantee the performance. This proposal essentially proposes an adaptive routing scheme for each patch. Extensive experiments demonstrate that our method can not only accelerate inference but also provide strong quantitative and qualitative results. Built upon the BasicVSR model, our method achieves a speedup of 20% on average, going as high as 50% for some images, while even maintaining competitive performance on REDS4.

Stereo image super-resolution (Stereo SR) has been a newly rising and challenging problem with the popular application of dual cameras, which can be used to promote the SR performance by adding auxiliary information from another viewpoint. Most of the existing excellent works have concentrated on leveraging the intrinsic feature correlation of two view images via exploring the non-local attention mechanism. However, they only perform interaction once for feature registration and fusion accompanied by the complex view transition constraint, which cannot fully take advantage of the information in the stereo image pairs. In this paper, we propose a joint feature aggregation network for Stereo SR to calibrate single-view features and integrate cross-view knowledge effectively. Specifically, we introduce a self-calibrated feature extractor to excavate multi-scale and multi-direction features within the single-view image. What’s more, we design an adaptive fusion module with the cross-view attention mechanism, which is utilized to mine and fuse the long-range dependencies between the stereo image pairs so as to get rid of the inflexible cycle constraints. Extensive experimental results demonstrate that our proposal successfully achieves superior performance against the state-of-the-art methods on four datasets.