Online action detection, which aims to identify an ongoing action from a streaming video, is an important subject in real-world applications. For this task, previous methods use recurrent neural networks for modeling temporal relations in an input sequence. However, these methods overlook the fact that the input image sequence includes not only the action of interest but background and irrelevant actions. This would induce recurrent units to accumulate unnecessary information for encoding features on the action of interest. To overcome this problem, we propose a novel recurrent unit, named Information Discrimination Unit (IDU), which explicitly discriminates the information relevancy between an ongoing action and others to decide whether to accumulate the input information. This enables learning more discriminative representations for identifying an ongoing action. In this paper, we further present a new recurrent unit, called Information Integration Unit (IIU), for action anticipation. Our IIU exploits the outputs from IDN as pseudo action labels as well as RGB frames to learn enriched features of observed actions effectively. In experiments on TVSeries and THUMOS-14, the proposed methods outperform state-of-the-art methods by a significant margin in online action detection and action anticipation. Moreover, we demonstrate the effectiveness of the proposed units by conducting comprehensive ablation studies.

From a streaming video, online action detection aims to identify actions in the present. For this task, previous methods use recurrent networks to model the temporal sequence of current action frames. However, these methods overlook the fact that an input image sequence includes background and irrelevant actions as well as the action of interest. For online action detection, in this paper, we propose a novel recurrent unit to explicitly discriminate the information relevant to an ongoing action from others. Our unit, named Information Discrimination Unit (IDU), decides whether to accumulate input information based on its relevance to the current action. This enables our recurrent network with IDU to learn a more discriminative representation for identifying ongoing actions. In experiments on two benchmark datasets, TVSeries and THUMOS-14, the proposed method outperforms state-of-the-art methods by a significant margin. Moreover, we demonstrate the effectiveness of our recurrent unit by conducting comprehensive ablation studies.

State-of-the-art super-resolution (SR) methods commonly assume that the downscaling kernel (the point spread function of the camera) is a Gaussian kernel. Therefore, the methods are particularly vulnerable to solving the problem of blind SR, which deals with a real low-resolution (LR) image that does not follow the assumption. Recently, to address this issue, several internal learning-based methods, which train an image-specific network using a single input image, have been introduced. In this approach, the blind SR is modeled as a two-stage optimization problem, which conducts downscaling kernel estimation followed by SR network training with the estimated kernel. In this letter, we assume that not only the estimated kernel can be employed for SR network training, but also the super-resolved image can contribute to downscaling kernel estimation. To that end, we propose a unified internal learning-based blind SR method that jointly trains two image-specific networks for 1) downscaling together with kernel estimation and 2) SR. More specifically, we train the two networks to minimize a dual back-projection loss; the SR network is trained to reconstruct a given input image from the LR image generated by the downscaling network, and the downscaling network is trained to downscale the high-resolution image generated by the SR network to be as close as possible to the given input image. Through the complementary training, the downscaling kernel estimation becomes more accurate, resulting in better SR performance. In the experiment, we show that the proposed method outperforms previous two-stage internal learning-based methods in terms of both SR performance and efficiency.

Recent temporal action proposal generation approaches have suggested integrating segment- and snippet score-based methodologies to produce proposals with high recall and accurate boundaries. In this paper, different from such a hybrid strategy, we focus on the potential of the snippet score-based approach. Specifically, we propose a new snippet score-based method, named Snippet Relatedness-based Generator (SRG), with a novel concept of “snippet relatedness”. Snippet relatedness represents which snippets are related to a specific action instance. To effectively learn this snippet relatedness, we present “pyramid non-local operations” for locally and globally capturing long-range dependencies among snippets. By employing these components, SRG first produces a 2D relatedness score map that enables the generation of various temporal intervals reliably covering most action instances with high overlap. Then, SRG evaluates the action confidence scores of these temporal intervals and refines their boundaries to obtain temporal action proposals. On THUMOS-14 and ActivityNet-1.3 datasets, SRG outperforms state-of-the-art methods for temporal action proposal generation. Furthermore, compared to competing proposal generators, SRG leads to significant improvements in temporal action detection.

In this paper, we propose a blind text image deblurring algorithm by using a text-specific hybrid dictionary. After careful analysis, we find that the text-specific hybrid dictionary has the great ability of providing powerful contextual information for text image deblurring. Here, it is worth noting that our proposed method is inspired by our observation that an intermediate latent image contains not only sharp regions, but also multiple types of small blurred regions. Based upon our discovery, we propose a prior for text images based on sparse representation, which models the relationship between an intermediate latent image and a desired sharp image. To this end, we carefully collect three different image patch pairs, which are 1) Gaussian blur-sharp, 2) motion blur-sharp, and 3) sharp-sharp, in order to construct the text-specific hybrid dictionary. We also propose a new optimization framework suitable for the task of text image deblurring in this paper. Extensive experiments have been conducted on a challenging dataset of synthetic and real-world text images. Our results demonstrate that the proposed method outperforms the state-of-the-art image deblurring methods both quantitatively and qualitatively.

In this paper, we present a novel grid encoding model for content-aware image retargeting. In contrast to previous approaches such as vertex-based and axis-aligned grid encoding models, our approach takes each horizontal/vertical distance between two adjacent vertices as an optimization variable. Upon this difference-based encoding scheme, every vertex position of a target grid is subsequently determined after optimizing the one-dimensional values. Our quad edge-based grid model has two major advantages for image retargeting. First, the model enables a grid optimization problem to be developed in a simple quadratic program while ensuring the global convexity of objective functions. Second, due to the independency of variables, spatial regularizations can be applied in a locally adaptive manner to preserve structural components. Based on this model, we propose three quadratic objective functions. Note that, in our work, their linear combination guides a grid deformation process to obtain a visually comfortable retargeting result by preserving salient regions and structural components of an input image. Comparative evaluations have been conducted with ten existing state-of-the-art image retargeting methods, and the results show that our method built upon the quad edge-based model consistently outperforms other previous methods both on qualitative and quantitative perspectives.

In recent years, computer vision applications dealing with foreground objects are becoming more important with an increasing demand of advanced intelligent systems. Most of these applications assume that an image contains one or more objects, which often produce undesired results when noticeable objects do not appear in the image. In this letter, we address the problem of ascertaining the existence of objects in an image. In the first step, the input image is partitioned into nonoverlapping local patches, then the patches are categorized into three classes, namely natural, man-made, and object to estimate object candidates. Then a Bayesian methodology is employed to produce more reliable results by eliminating false positives. To boost the object patch detection performance, we exploit the difference between coarse and fine segmentation results. To demonstrate the effectiveness of the proposed method, extensive experiments have been conducted on several benchmark image databases. Furthermore, we have shown the usefulness of our approach by applying it to a real application (i.e., image thumbnailing).

In this paper, a visual attention model is incorporated for efficient saliency detection, and the salient regions are employed as object seeds for our automatic object segmentation system. In contrast with existing interactive segmentation approaches that require considerable user interaction, the proposed method does not require it, i.e., the segmentation task is fulfilled in a fully automatic manner. First, we introduce a novel unified spectral-domain approach for saliency detection. Our visual attention model originates from a well-known property of the human visual system that the human visual perception is highly adaptive and sensitive to structural information in images rather than nonstructural information. Then, based on the saliency map, we propose an iterative self-adaptive segmentation framework for more accurate object segmentation. Extensive tests on a variety of cluttered natural images show that the proposed algorithm is an efficient indicator for characterizing the human perception and it can provide satisfying segmentation performance.

This paper presents a novel method for detecting salient regions in both images and videos based on a discriminant center-surround hypothesis that the salient region stands out from its surroundings. To this end, our spatiotemporal approach combines the spatial saliency by computing distances between ordinal signatures of edge and color orientations obtained from the center and the surrounding regions and the temporal saliency by simply computing the sum of absolute difference between temporal gradients of the center and the surrounding regions. Our proposed method is computationally efficient, reliable, and simple to implement and thus it can be easily extended to various applications such as image retargeting and moving object extraction. The proposed method has been extensively tested and the results show that the proposed scheme is effective in detecting saliency compared to various state-of-the-art methods.

There have been several interactive approaches to extracting objects from still images, since it is significantly difficult to automatically segment objects in complex background. In this paper, we present a novel automatic scheme for extracting salient objects from natural images. To this end, segmentation of salient objects is formulated as a global energy minimization problem in an iterative self-adaptive framework. By employing a saliency detection technique, object and background seeds are inferred automatically. The problem in this step is that the automatically generated seeds may not be reliably positioned. An iterative reversible graph cut method is introduced to overcome the problem inherent in the saliency-based seed extraction method. In the iterative self-adaptive framework, bidirectional state transitions are iteratively involved to reduce the mis-classified pixels. Experimental results show that the proposed segmentation method yields more accurate segmentation results than previous segmentation approaches.

In saliency detection, regions attracting visual attention need to be highlighted while effectively suppressing non-salient regions for the semantic scene understanding. However, most previous methods tend to fail in suppressing highly textured backgrounds and also high contrast edges belonging to the non-salient regions. To address this problem, we propose a method for detecting salient regions based on a self-ordinal resemblance measure (SORM). Our saliency map is defined by using the center-surround computations based on the ordinal signatures obtained from local regions centered at each pixel. It can be regarded as an energy map and thus extended to image retargeting. Our approach is fully automatic and nonparametric. To justify robustness of our approach, the proposed method is compared with the state of the art methods on various images.