Video super-resolution (SR) reconstruction technology aims at obtaining high quality reconstruction of high-resolution (HR) video sequences by inferring the lost detailed information from their low-resolution (LR) counterparts. However, this technology is an ill-posed problem because significant detailed information is lost in the process of video degrading. The existing learning-based SR reconstruction methods can be adapted to a larger super-resolution factor, but it cannot be guaranteed that any low-resolution image block can find its corresponding high-resolution block matching in a limited-scale training set. Some noise and over smooth phenomenon usually exist while dealing with some unique features that rarely appear in a given training data set. The self-similarity based SR methods do not rely on accurate sub-pixel motion estimation and thus can be adapted to complex motion patterns. However, under conditions of insufficient internal similar blocks, some visual flaws are usually produced due to the mismatched internal instances.

To address the problems in the existing video super-resolution methods, such as noise, over smooth and visual artifacts, which are caused by the reliance on limited external training or mismatch of internal similarity patch instances, this study proposes a novel video super-resolution reconstruction algorithm based on deep learning and spatio-temporal feature similarity (DLSS-VSR). The video super-resolution reconstruction mechanism with the joint internal and external constraints is established utilizing the complementary advantages of both external deep correlation mapping learning and internal spatio-temporal nonlocal self-similarity prior constraint. A deep learning model based on deep convolutional neural network is constructed to learn the nonlinear correlation mapping between low-resolution and high-resolution video frame patches. A novel spatio-temporal feature similarity calculation method is proposed, which considers both internal video spatio-temporal self-similarity and external clean nonlocal similarity. For the internal spatio-temporal feature self-similarity, we improve the accuracy and robustness of similarity matching by proposing a similarity measure strategy based on spatio-temporal moment feature similarity and structural similarity. The external nonlocal similarity prior constraint is learned by the patch group-based Gaussian mixture model. The time efficiency for spatio-temporal similarity matching is further improved based on saliency detection and region correlation judgment strategy, which achieves a better tradeoff between super-resolution accuracy and speed. Experimental results demonstrate that the DLSS-VSR algorithm achieves competitive super-resolution quality compared to other state-of-the-art algorithms in both subjective and objective evaluations.