Despite the strong performance of transformers, quadratic computation complexity of self-attention presents challenges in applying them to vision tasks. Linear attention reduces this complexity from quadratic to linear, offering a strong computation-performance tradeoff. To further optimize this, automatic pruning is an effective method to find a structure that maximizes performance within a target resource through training without any heuristic approaches. However, directly applying it to multihead attention is not straightforward due to channel mismatch. In this article, we propose an automatic pruning method to deal with this problem. Different from existing methods that rely solely on training without any prior knowledge, we integrate channel similarity-based weights into the pruning indicator to preserve the more informative channels within each head. Then, we adjust the pruning indicator to enforce that channels are removed evenly across all heads, thereby avoiding any channel mismatch. We incorporate a reweight module to mitigate information loss due to channel removal and introduce an effective pruning indicator initialization for linear attention, based on the attention differences between the original structure and each channel. By applying our pruning method to the FLattenTransformer on ImageNet-1K, which incorporates original and linear attention mechanisms, we achieve a 30% reduction of FLOPs in a near lossless manner. It also has 1.96% of accuracy gain over the DeiT-B model while reducing FLOPs by 37%, and 1.05% accuracy increase over the Swin-B model with a 10% reduction in FLOPs as well. The proposed method outperforms previous state-of-the-art efficient models and the recent pruning methods.

Most pruning methods focus on determining redundant channels from the pre-trained model. However, they overlook the cost of training large networks and the significance of selecting channels for effective reconfiguration. In this paper, we present a "pruning from scratch" framework that considers reconfiguration and expression capacity. Our Shared Pruning Module (SPM) handles a channel alignment problem in residual blocks for lossless reconfiguration after pruning. Moreover, we introduce nuclear norm-based regularization to preserve the representability of large networks during the pruning process. By combining it with MACs-based regularization, we achieve an efficient and powerful pruned network while compressing towards target MACs. The experimental results demonstrate that our method prunes redundant channels effectively to enhance representation capacity of the network. Our approach compresses ResNet50 on ImageNet without requiring additional resources, achieving a top-1 accuracy of 75.25% with only 41% of the original model’s MACs. Code is available at https://github.com/jsleeg98/NuSPM.

Recently, convolution neural networks (CNNs) have achieved state-of-the-art performance in infrared small target detection. However, the limited number of public training data restricts the performance improvement of CNN-based methods. To handle the scarcity of training data, we propose a method that can generate synthetic training data for infrared small target detection. We adopt the generative adversarial network framework where synthetic background images and infrared small targets are generated in two independent processes. In the first stage, we synthesize infrared images by transforming visible images to infrared ones. In the second stage, target masks are implanted on the transformed images. Then, the proposed intensity modulation network synthesizes realistic target objects that can be diversely generated from further image processing. Experimental results on the recent public dataset show that when we train various detection networks using the dataset composed of both real and synthetic images, detection networks yield better performance than using real data only.

We propose an unsupervised approach to predict depth maps from images captured by a wireless endoscopic capsule. Recent advances in deep learning have shown that accurate depth maps can be predicted from a single image, where the deep network is trained via unsupervised or self-supervised learning by using monocular video sequences or stereo image pairs. However, directly applying these techniques to endoscopic images does not yield satisfactory results owing to the inherent difficulties of the wireless capsule imaging such as dim lighting and low-resolution of images, which are different from normal imaging conditions. For that reason, we exploit the environmental characteristics of endoscopic images - there is no external light source except ones attached to the capsule. Based on this condition, we propose the direct attenuation model-based depth map prediction scheme to guide depth prediction and to add meaningful cues to the loss function. We experimentally verify the proposed method with various endoscopic images.

Palette-based image recoloring is one of the popular color manipulation methods by transforming dominant scene colors into corresponding user-specific colors intuitively. Because conventional palette extraction methods are based only on global color distribution, they cannot deal with local distinctive colors, which can cause not only the degraded result of image recoloring, but also the limitation of user selection. In this paper, we propose a new palette extraction method by iterative palette partitioning based on cluster validation. To find candidate colors for palette partitioning, we compute patch uniformity for local patches. For image recoloring with extracted palette, we use a statistics-based color transfer method by considering the pixel-wise weight from palette colors. In the experiments, our automatic or user-specific palette extraction shows more plausible palette representation than previous methods. Furthermore, image recoloring results show the effectiveness of the proposed method in terms of quality of color transfer and user experience.

Accurate stereo matching for depth extraction requires a large memory space, which restricts its use in resource-limited systems. The problem is aggravated by the recent trend of applications requiring significantly high pixel resolution and disparity levels. To alleviate the high memory requirement, we propose to represent the aggregation costs as a Gaussian mixture model (GMM) function. Only a set of GMM parameters is stored and used instead of all the costs for each pixel. We also propose GMM parameter update-based aggregation along multiple paths. To preserve the accuracy of the disparity map, we employ a depth confidence measure and propose an update rule for the slanted surface of an object. Experimental results over the KITTI dataset show that the proposed method reduces the memory requirement to less than 5% of that of semiglobal matching, while the accuracy is maintained at the level of state-of-the-art semiglobal and local methods.

Modelling and analyzing noise in images is a fundamental task in many computer vision systems. Traditionally, noise has been modelled per color channel assuming that the color channels are independent. Although the color channels can be considered as mutually independent in camera RAW images, signals from different color channels get mixed during the imaging process inside the camera due to gamut mapping, tone-mapping, and compression. We show the influence of the in-camera imaging pipeline on noise and propose a new noise model in the 3D RGB space to accounts for the color channel mix-ups. A data-driven approach for determining the parameters of the new noise model is introduced as well as its application to image denoising. The experiments show that our noise model represents the noise in regular JPEG images more accurately compared to the previous models and is advantageous in image denoising.

Object proposals for detecting moving or static video objects need to address issues such as speed, memory complexity and temporal consistency. We propose an efficient Video Object Proposal (VOP) generation method and show its efficacy in learning a better video object detector A deep-learning based video object detector learned using the proposed VOP achieves state-of-the-art detection performance on the Youtube-Objects dataset. We further propose a clustering of VOPs which can efficiently be used for detecting objects in video in a streaming fashion. As opposed to applying per-frame convolutional neural network (CNN) based object detection, our proposed method called Objects in Video Enabler thRough LAbel Propagation (OVERLAP) needs to classify only a small fraction of all candidate proposals in every video frame through streaming clustering of object proposals and class-label propagation. Source code for VOP clustering is available at https://github. com/subtri/streaming_VOP_clustering.

This paper introduces a new color transfer method which is a process of transferring color of an image to match the color of another image of the same scene. The color of a scene may vary from image to image because the photographs are taken at different times, with different cameras, and under different camera settings. To solve for a full nonlinear and nonparametric color mapping in the 3D RGB color space, we propose a scattered point interpolation scheme using moving least squares and strengthen it with a probabilistic modeling of the color transfer in the 3D color space to deal with mis-alignments and noise. Experiments show the effectiveness of our method over previous color transfer methods both quantitatively and qualitatively. In addition, our framework can be applied for various instances of color transfer such as transferring color between different camera models, camera settings, and illumination conditions, as well as for video color transfers.

In this paper, we propose a video stabilization method that takes advantages of both online and offline video stabilization methods in a semi-online framework. Our approach takes the fixed length incoming frames for having the advantages of offline methods with the same length of delayed results. We stabilize input frames by warping to the keyframe to increase the visual stability. For preserving user intent camera motion correctly, we determine the next keyframe update by measuring inconsistency between a current keyframe and incoming frames. Moreover, inter-frame motion smoothing by quadratic fitting bridges the keyframes smoothly for pleasant viewing experiences. Our algorithm not only handles rapid camera motion changes, but also stabilizes the input camera path smoothly in real-time. Experimental results show that the proposed algorithm is comparable to the state-of-the-art offline video stabilization methods with only fixed length of incoming frames.

Despite recent advances in video segmentation, many opportunities remain to improve it using a variety of low and mid-level visual cues. We propose improvements to the leading streaming graph-based hierarchical video segmentation (streamGBH) method based on early and mid level visual processing. The extensive experimental analysis of our approach validates the improvement of hierarchical supervoxel representation by incorporating motion and color with effective filtering. We also pose and illuminate some open questions towards intermediate level video analysis as further extension to streamGBH. We exploit the supervoxels as an initialization towards estimation of dominant affine motion regions, followed by merging of such motion regions in order to hierarchically segment a video in a novel motion-segmentation framework which aims at subsequent applications such as foreground recognition.

In this paper, we present a particle-filter-based localization framework with decoupled visual measurements (image features) for process and measurement models. Thus our approach enables using camera-based motion estimation while achieving the independence between the process noise and the measurement noise in the Bayesian filtering framework. In addition, we alternately perform sequential and global localization on the basis of the marginal likelihood in order to avoid severe errors caused by incorrect data association.

By the laws of quantum physics, pixel intensity does not have a true value, but should be a random variable. Contrary to the conventional assumptions, the distribution of intensity may not be an additive Gaussian. We propose to directly model the intensity difference and show its validity by an experimental comparison to the conventional additive model. As a model of the intensity difference, we present a Skellam distribution derived from the Poisson photon noise model. This modeling induces a linear relationship between intensity and Skellam parameters, while conventional variance computation methods do not yield any significant relationship between these parameters under natural illumination. The intensity-Skellam line is invariant to scene, illumination, and even most of camera parameters. We also propose practical methods to obtain the line using a color pattern and an arbitrary image under natural illumination. Because the Skellam parameters that can be obtained from this linearity determine a noise distribution for each intensity value, we can statistically determine whether any intensity difference is caused by an underlying signal difference or by noise. We demonstrate the effectiveness of this new noise model by applying it to practical applications of background subtraction and edge detection.

A global pose estimation method of an Unmanned Aerial Vehicle (UAV) by matching forward-looking aerial images from the UAV flying at low altitude with down-looking images from a satellite is proposed. To overcome the limitation of significantly different camera viewpoints and characteristics, we use buildings as a cue of matching. We extract buildings from aerial images and construct a 3D model of buildings, using the fundamental matrix. We estimate the global pose of the vehicle by matching 3D structure of buildings with satellite images, using a particle filter. Experimental results show that the proposed approach is a promising method to the global pose estimation of the UAV with forward-looking vision data.

In this paper, we introduce the Skellam distribution as a sensor noise model for CCD or CMOS cameras. This is derived from the Poisson distribution of photons that determine the sensor response. We show that the Skellam distribution can be used to measure the intensity difference of pixels in the spatial domain, as well as in the temporal domain. In addition, we show that Skellam parameters are linearly related to the intensity of the pixels. This property means that the brighter pixels tolerate greater variation of intensity than the darker pixels. This enables us to decide automatically whether two pixels have different colors. We apply this modeling to detect the edges in color images. The resulting algorithm requires only a confidence interval for a hypothesis test, because it uses the distribution of image noise directly. More importantly, we demonstrate that without conventional Gaussian smoothing the noise model-based approach can automatically extract the fine details of image structures, such as edges and corners, independent of camera setting.

The CCD camera and the 2D laser range finder are widely used for motion estimation and 3D reconstruction. With their own strengths and weaknesses, low-level fusion of these two sensors complements each other. We combine these two sensors to perform motion estimation and 3D reconstruction simultaneously and precisely. We develop a motion estimation scheme appropriate for this sensor system. In the proposed method, the motion between two frames is estimated using three points among the scan data, and refined by nonlinear optimization. We validate the accuracy of the proposed method using real images. The results show that the proposed system is a practical solution for motion estimation as well as for 3D reconstruction.

Most change detection methods are based on gray-level images. A gray-level image is regarded as a 1-D projection of three channels of color images. Therefore, more precise change detection results are expected by utilizing color information. We previously developed a change detection scheme using color images. In this paper, we determine which color space should be selected for accurate change detection based on our previous detection scheme. Our method can be applied to various color spaces, including gray-level images. Then we can measure the expected number of error pixels in order to select an appropriate color space which gives the best result among various color spaces. The experiments show that selecting a color space based on measurements results in the fewest error pixels.

We present a novel change detection method using a statistical model of the image noise. Most change detection methods are based on gray-level images. However, color images can provide much richer scene information. One major problem to use the color images in change detection is how to combine three components in color space as a detection cue. We use the Euclidean color distance of three channels to measure the difference between two consecutive images. Specifically, we present a new noise model for each color channel. Through this modeling we can estimate the distribution of the Euclidean color distance for unchanged regions. We can find the optimal threshold to detect changes using this estimated distribution. Although we use the optimal threshold, inevitably there may be false classifications. To reject these erroneous cases, we adopt the graph cuts method that efficiently minimizes the global energy, which takes into account the effect of neighboring pixels.