Image deraining is a hot research topic, which aims to remove various rain streaks (raindrops) from rainy images and restore the backgrounds. Though image deraining has been extensively studied in recent years, few methods are able to effectively and efficiently derain real-world high-resolution rainy images. In general, existing image deraining methods are restricted by two main factors while processing high-resolution images. First, the computational complexity and memory usage of existing deep learning-based methods are high when it comes to derain high-resolution images. Second, as the image resolution increases, it is difficult to simultaneously extract and aggregate both global and local features for clean rain removal. In this paper, we propose a novel network, called Global-Local Grafting Fusion Network (GLGFN), for deraining real-world high-resolution images. Our GLGFN utilizes a staggered connection structure to achieve deeper sampling depth while maintaining low computational cost. It adopts the Transformer and CNN based encoders (backbones) to extract global and local features, respectively, and then grafts global features into local features to guide the extraction of rain streaks. In addition, for well fusing global and local features, we also propose a Grafting Fusion Module (GFM), which adopts Cross Sparse Attention (CSA) and Selective Kernel Fusion (SK Fusion) to efficiently aggregate global and local features. Extensive experiments conducted on several high-resolution real rainy datasets have demonstrated the effectiveness and efficiency of our proposed GLGFN. We will release our code and dataset.

Image operators can be instrumental to computational imaging and photography. However, many of them are computationally intensive. In this article, we propose an effective yet efficient joint upsampling method to accelerate various image operators. We show that edge-preserving filtering can be facilitated with a downsampling-and-upsampling process. Moreover, when the extent of smoothing is mild, the process is detail preserving, i.e., the fine details lost in the low-resolution (LR) images can be accurately restored in the high-resolution (HR) images. Given an HR input and an LR output of an operator, we downsample the HR input and calculate its affinities to the HR input. By applying the affinities to the LR output, we promote its resolution. Due to the strong detail-preserving property, the HR output derived in the previous step may exhibit aliasing artifacts around the salient edges. We further refine it based on the linear relations in a small neighborhood to rid the artifacts. Experiments on various image operators show that our method achieves superior quality over the state-of-the-art joint upsampling methods. Furthermore, the running time of our method is linear to the number of pixels. Our naive implementation derives 1080P images in real time (24 fps) on an NVIDIA GTX 3070 GPU.

Existing deraining methods focus mainly on a single input image. However, with just a single input image, it is extremely difficult to accurately detect and remove rain streaks, in order to restore a rain-free image. In contrast, a light field image (LFI) embeds abundant 3D structure and texture information of the target scene by recording the direction and position of each incident ray via a plenoptic camera. LFIs are becoming popular in the computer vision and graphics communities. However, making full use of the abundant information available from LFIs, such as 2D array of sub-views and the disparity map of each sub-view, for effective rain removal is still a challenging problem. In this paper, we propose a novel method, 4D-MGP-SRRNet, for rain streak removal from LFIs. Our method takes as input all sub-views of a rainy LFI. To make full use of the LFI, it adopts 4D convolutional layers to simultaneously process all sub-views of the LFI. In the pipeline, the rain detection network, MGPDNet, with a novel Multi-scale Self-guided Gaussian Process (MSGP) module is proposed to detect high-resolution rain streaks from all sub-views of the input LFI at multi-scales. Semi-supervised learning is introduced for MSGP to accurately detect rain streaks by training on both virtual-world rainy LFIs and real-world rainy LFIs at multi-scales via computing pseudo ground truths for real-world rain streaks. We then feed all sub-views subtracting the predicted rain streaks into a 4D convolution-based Depth Estimation Residual Network (DERNet) to estimate the depth maps, which are later converted into fog maps. Finally, all sub-views concatenated with the corresponding rain streaks and fog maps are fed into a powerful rainy LFI restoring model based on the adversarial recurrent neural network to progressively eliminate rain streaks and recover the rain-free LFI. Extensive quantitative and qualitative evaluations conducted on both synthetic LFIs and real-world LFIs demonstrate the effectiveness of our proposed method.

Smoothing images while preserving salient edges is a crucial task in computational photography. Existing edge-preserving filters suffer from various artifacts, such as halos, gradient reversals, and intensity shifts. Observing that various artifacts are strongly related to salient edges with large gradients, we propose a continuous mapping function to process the gradients. The proposed function is literally edge-preserving, i.e., it keeps large gradients intact while attenuating small gradients. We propose an L1-regularized reconstruction model based on the processed gradients for edge-preserving image filtering. The L1-regularization facilitates the edge-preserving property in the reconstructed results. To solve the proposed L1-regularized model, we implement an efficient algorithm based on the alternating direction method of multipliers (ADMM) and Fourier domain optimization. We have conducted qualitative and quantitative experiments to evaluate the proposed filter. The results demonstrate that our filter better handles various artifacts and delivers superior image quality on various applications. The proposed filter is highly efficient, our GPU implementation takes 70ms to process a color image with 1 megapixel on an NVIDIA GTX 1070 GPU.

Depth-of-field (DoF) rendering is an important technique in computational photography that simulates the human visual attention system. Existing DoF rendering methods usually suffer from a high computational cost. The task of DoF rendering can be accelerated by guided upsampling methods. However, the state-of-the-art guided upsampling methods fail to distinguish the focus and defocus areas, resulting in unsatisfying DoF effects. In this paper, we propose a novel deep weighted guided upsampling network (DWGUN) based on a encoder and decoder framework to jointly upsample the low-resolution DoF image under the guidance of the corresponding high-resolution all-in-focus image. Due to the intuitive weight design, the traditional weighted image upsampling is not tailored to DoF image upsampling. We propose a deep refocus-defocus edge-aware module (DREAM) to learn the spatially-varying weights and embed them in the deep weighted guided upsampling block (DWGUB). We have conducted comprehensive experiments to evaluate the proposed method. Rigorous ablation studies are also conducted to validate the rationality of the proposed components.

Edge-preserving image filtering is an essential task in computational photography and imaging. In this paper, we propose a simple yet effective global edge-preserving filter based on soft clustering, and we propose a novel soft clustering algorithm based on a restricted Gaussian mixture model. Given specified parameters, the soft clustering process is firstly performed on the image to derive the partition matrix, from which the affinity matrix is then constructed for filtering. The filtering output is calculated as the weighted average of the pixels in the local window, so the proposed filter could suppress the intensity shift artifacts that impede most global filters. Besides, the weights in the proposed filter are derived by clustering, which properly separates dissimilar pixels, so the proposed filter could handle the halo artifacts that haunt many local filters. Moreover, our filter provides flexible control over the amount of smoothing that is deficient in the deep learning-based filters. Besides the efficacy in smoothing, the proposed filter naturally has low computational complexity. Qualitative and quantitative results suggest that the proposed filter benefits various applications, including edge-preserving smoothing, image enhancing, flash/non-flash fusion, HDR tone mapping, and dehazing.

Origami has provided a potential way to construct 3D curved structures by folding flat sheet materials without cutting or stretching. As a traditional origami, waterbomb tessellation is widely studied from aspects of science and engineering. However, users cannot easily utilize this kind of origami to fit curved target surfaces because the underlying geometric constraints limit the design space. In this study, we propose computational design methods for approximating cylindrical and axisymmetric curved surfaces based on waterbomb tessellations. With consideration of symmetry and periodic repetition, a single strip of the waterbomb tessellation is first modeled and then longitudinally and circumferentially replicated to construct cylindrical and axisymmetric waterbomb tessellations, respectively. To fulfill flat-foldability, an optimization process is introduced for minimizing flat-foldable residuals iteratively and then a regulation process of the crease pattern is applied for further reducing such residuals. In addition, we demonstrate waterbomb-derivative tessellations with quad-paddings to expand the design variations. Furthermore, rigid-folding sequences and several physically engineered origami pieces are presented. The proposed methods can be utilized to facilitate the design of origami-inspired structures for various engineering design purposes, such as foldable shelters, tubular structures, metamaterials, and so on.

Kernel canonical correlation analysis (KCCA) is an efficient dimensionality reduction tool in the application of remote sensing image classification. However, it suffers from the problem of parametric sensitivity since a single kernel is used. In this letter, a KCCA ensemble framework is put forward to improve the robustness of KCCA. Following the philosophy that two heads are better than one, multiple KCCA models are incorporated into the framework. And more importantly, their terms are weighted to adjust their contribution to the result according to their performance. In addition, over-fitting is overcome by introducing a Laplacian regularization term in our framework, hence, the name Laplacian regularized kernel canonical correlation ensemble. Experimental results on NWPU-RESISC45 data set show that our proposed method achieves better classification performances as compared to state-of-the-art methods in both shallow and deep features.

Abstract Content‐based human motion analysis has captured extensive concerns of researchers from the domains of computer animation, human‐machine interaction, entertainment, etc. However, it is a non‐trivial task due to the spatial and temporal variations in the motion data. In this paper, we propose a generalized model (GM)‐based approach to model the variations and accurately recognize motion patterns. We partition the human character model into five parts, and extract the features of the submotions of each specific body part using clustering techniques. These features from the training trials in each class are combined to build the GM. We propose a new penalty based similarity measure for DTW to be used with the GMs for isolated motion recognition. On the other hand, from the GMs five body partition index maps are constructed and used for matching together with a flexible end point detection scheme during continuous motion recognition. In the experiments, we examine the effectiveness and efficiency of the approach in both isolated motion and continuous motion recognition. The results show that our proposed method has good performance compared with other state‐of‐the‐art methods in recognition accuracy and processing speed.

Abstract In this paper, we present an interactive dancing game based on motion capture technology. We address the problem of real‐time recognition of the user's live dance performance in order to determine the interactive motion to be rendered by a virtual dance partner. The real‐time recognition algorithm is based on a human body partition indexing scheme with flexible matching to determine the end of a move as well as to detect unwanted motion. We show that the system can recognize the live dance motions of users with good accuracy and render the interactive dance move of the virtual partner. Copyright © 2011 John Wiley & Sons, Ltd.

In this paper, a novel concept of segmental singular value decomposition (SegSVD) is proposed to represent a motion pattern with a hierarchical structure. The similarity measure based on the SegSVD representation is also proposed. SegSVD is capable of capturing the temporal information of the time series. It is effective in matching patterns in a time series in which the start and end points of the patterns are not known in advance. We evaluate the performance of our method on both isolated motion classification and continuous motion recognition for dance movements. Experiments show that our method outperforms existing work in terms of higher recognition accuracy.