Different humans perceive a scene through distinct visual attention shifts that can be represented by scanpaths of different varieties. Approaches that predict multiple visual scanpaths on an image must thus consider producing scanpaths of distinct varieties for human-like generation, which has been mostly overlooked in the existing literature. In this paper, we introduce ScanFormer, a framework to predict diverse visual scanpaths of different varieties on an image employing a meshed-memory transformer. The memory-augmented encoder of the transformer generates multi-level contextual features that capture the relationships among image regions and embed learned biases towards them. The meshed decoder of the transformer models inter-fixation dependencies to successively predict the fixations of the output scanpath by taking the features from the encoder, previous fixations, and a condition representing the variety of the scanpath as the inputs. The generation of multiple diverse visual scanpaths on an image is facilitated by learning to embed scanpath variety as a representation related to a scanpath’s uniqueness among multiple scanpaths. We evaluate the proposed approach on three standard datasets in terms of five types of established quantitative measures. Saccade amplitude and orientation density plots are also considered in the performance analysis. The experimental results demonstrate the superiority of ScanFormer over state-of-the-art methods in generating multiple diverse human-like visual scanpaths on images. Further, an ablation study is provided to empirically establish the significance of the various components of our framework. Code link: https://github.com/ashishverma03/ScanFormer .

Reconstruction of intricate local patterns and large missing regions during 3D shape completion has the contradictory requirements of computation over a wider context and operations for finer detail restoration. To this end, we propose a multi-resolution spectral encoding based 3D shape completion approach to work on truncated Signed Distance Field (SDF) based shape representations. Our novelty lies in judiciously integrating multi-resolution 3D convolutional blocks that encode the input shape and a spectral module (SM) that captures the shape-wide context, thus addressing the contradictory requirements. SM acts on the features extracted from both partial input scans and shape priors using the multi-resolution convolutional blocks. Our SM contains a 3D convolutional block placed between fast Fourier transform (FFT) and inverse FFT operations, which results in the expansion of the receptive field for the appropriate context computation. Our approach has an attention-based encoder-decoder architecture, where the encoding of a partial scan is acted upon by shape prior encodings to produce attention maps. These attention maps are lever-aged differently in pretraining, and in the later training and inference stages of our approach to produce the reconstructed 3D shape. A surface gradient-based loss function is used in addition to the L1 loss, both in the pretraining and training stages for emphasizing the differences in minute details. These along with an attention refinement operation often leads to complete reconstruction while restoring finer details. Experiments using standard synthetic and real datasets demonstrate the superiority of our approach over the state-of-the-art.

Use of depth information is often beneficial for salient object detection (SOD) in color images, and hence, RGB-D SOD has gained popularity. However, the availability of depth data related to a color image can not be assumed, which is when RGB SOD is needed. RGB SOD can still leverage the benefits of depth information if the depth is estimated from the RGB data and suitably employed. To this end, we propose a novel estimated depth based interactive (EDINet) framework that performs SOD in images using only RGB data as its input. EDINet consists of a depth attentive feature extractor (DAFE) to capture features from depth-attentive regions and an RGB-estimated depth interaction module (RGB-EDI) that fuses hierarchical features from DAFE and the input RGB data. The salient object map is obtained after its progressive enhancement through its iterative use as a feedback into RGB-EDI. Quantitative and qualitative performance comparison with the state-of-the-art (SOTA) on benchmark RGB datasets shows the effectiveness of our framework. Considering RGB-D benchmark datasets and RGB-D SOD SOTA, the usefulness of EDINet in RGB-D SOD is also demonstrated by employing the actual depth instead of the estimated one.

The objective of 3-dimensional point cloud completion is to estimate a plausible complete shape from a given partial point cloud. Most of the data-driven point cloud completion approaches have been proposed in a supervised manner needing one-to-one correspondence between the partial and complete shapes. A promising way to solve the paired data dependency is to use the mapping capability of a pre-trained point cloud generation network to the best possible matching latent vector. However, recovering the composite structural details and complex geometry of a 3D shape is often difficult using a single latent vector alone. In this paper, we propose to employ multiple latent vectors, each of which generates individual feature maps, which are then combined to reconstruct a faithful complete 3D shape corresponding to an available partial shape. Deploying more than one latent vector enables the pre-trained generative network to increase its fidelity by using multiple combinations of feature representations learned by each single latent. Experimental results show that our algorithm performs well compared to the other existing shape completion methods. We also study the completion performance with a varying number of latent codes and the role of each latent vector in the final complete shape generation.

Anomaly detection in videos is a challenging task as anomalies in different videos are of different kinds. Therefore, a promising way to approach video anomaly detection is by learning the non-anomalous nature of the video at hand. To this end, we propose a one-class few-shot learning driven transformer based approach for anomaly detection in videos that is self-context aware. Features from the first few consecutive non-anomalous frames in a video are used to train the transformer in predicting the non-anomalous feature of the subsequent frame. This takes place under the attention of a self-context learned from the input features themselves. After the learning, given a few previous frames, the video-specific transformer is used to infer if a frame is anomalous or not by comparing the feature predicted by it with the actual. The effectiveness of the proposed method with respect to the state-of-the-art is demonstrated through qualitative and quantitative results on different standard datasets. We also study the positive effect of the self-context used in our approach.

Abstract This paper proposes a multiple salient region detection and localization approach for unstructured industrial robot work environments with arbitrarily located and orientated objects. Different from the existing, the authors' novel technique to detect multiple salient regions performs locally adaptive center‐surround operations on proto‐object partitions obtained through color consistency and spatial proximity analysis. The multi‐scale center‐surround operations are done by masks that are local structure‐aware yielding regions with precise and accurate boundaries as required for robotic manipulation. First, experiments to evaluate the multiple salient region detection performance are carried out using four standard databases having images with multiple salient objects. Quantitative result analysis using F‐measure, shuffled F‐measure, shuffled AUC and MAE, and subjective result inspection suggests that the proposed approach is in general better at collectively detecting multiple salient regions than the state‐of‐the‐art, including those based on deep learning. Then, real‐life experiments involving robotic manipulation are carried out to demonstrate the utility of the multiple salient region detection method. For robotic manipulation, object localization is improved after salient region detection by employing a fast shadow detection algorithm proposed based on hue analysis, and recognition through existing matching techniques is applied only at the localized salient regions. The benefit of the novel multiple salient region detection approach in the robotic manipulation system is shown using localization and pose estimation accuracy, rates of detection and recognition, positional and angular errors, and processing speed.

Most videos, including those captured through aerial remote sensing, are usually nonstationary in nature having time-varying feature statistics. Although sophisticated reconstruction and prediction models exist for video anomaly detection (VAD), effective handling of nonstationarity has seldom been considered explicitly. In this letter, we propose to perform prediction using a time-recursive differencing network followed by autoregressive moving average estimation for VAD. The differencing network is employed to effectively handle nonstationarity in video data during the anomaly detection. Focusing on the prediction process, the effectiveness of the proposed approach is demonstrated considering a simple optical flow-based video feature, and by generating qualitative and quantitative results on three aerial video data sets and two standard anomaly detection video data sets. Equal error rate (EER), area under curve (AUC), and ROC curve-based comparison with several existing methods including the state-of-the-art reveal the superiority of the proposed approach.

A noise-suppressing low-light image enhancement approach is proposed in this paper based on the extent of exposedness at each image pixel. To this end, a progressive, structure-aware exposedness estimation procedure is presented that quantifies local and global exposedness. These exposedness values are leveraged to produce a locally smooth pixel-level map that signifies the required degrees of enhancement at image pixels. This map is subsequently used in an enhancement function, which satisfies a few important properties, to generate the enhanced image. Before the enhancement, inherent noise in the low-light image is diminished employing a detail-preserving, low gradient magnitude suppression method. Subjective and quantitative analysis of results on a wide variety of natural and synthetically generated low-light images from standard databases using PSNR, iRSE, SSIM, and measures of perceptual quality, natural image statistics and brightness preservation suggests that our approach in general outperforms the state-of-the-art. Ablation studies and further experiments show the importance of a few components of our approach, and that our approach is computationally fast.

Real-world image degradation due to light scattering can be described based on the Koschmieder’s model. Training deep models to restore such degraded images is challenging as real-world paired data is scarcely available and synthetic paired data may suffer from domain-shift issues. In this paper, a zero-shot single real-world image restoration model is proposed leveraging a theoretically deduced property of degradation through the Koschmieder’s model. Our zero-shot network estimates the parameters of the Koschmieder’s model, which describes the degradation in the input image, to perform image restoration. We show that a suitable degradation of the input image amounts to a controlled perturbation of the Koschmieder’s model that describes the image’s formation. The optimization of the zero-shot network is achieved by seeking to maintain the relation between its estimates of Koschmieder’s model parameters before and after the controlled perturbation, along with the use of a few no-reference losses. Image dehazing and underwater image restoration are carried out using the proposed zero-shot framework, which in general outperforms the state-of-the-art quantitatively and subjectively on multiple standard real-world image datasets. Additionally, the application of our zero-shot framework for low-light image enhancement is also demonstrated.

Hazy images suffer from low visibility since the light gets scattered as it passes through various atmospheric particles. Moreover, such images are prone to color distortion, particularly in real weather conditions like sandstorms. In this letter, an effective dehazing technique is proposed using weighted least squares filtering on dark channel prior and color correction that involves automatic detection of color cast images. We show that the spread of the hue in a hazy image can differentiate a color cast image from a non-cast one. We propose a measure using the same for categorizing hazy images as cast and non-cast ones. Our novel color correction is performed by color balancing using a non-linear transformation followed by a cast-adaptive airlight refinement. Subjective and quantitative evaluations show that our method outperforms the state-of-the-art. It removes cast satisfactorily and reduces haze substantially while maintaining the naturalness of the image. Moreover, it produces visually pleasing images without halo artifacts.

Dynamic video summarisation (video skimming) is a process of generating a shorter video (video skim) as a summary of a given video, which helps in its easier and quicker comprehension. In this study, an efficient dynamic summarisation approach for user videos is proposed using vector ordering for ranking video units (frames/shots). User videos are casually shot unscripted videos, where skimming involves the selection of its interesting part(s) ignoring many uninteresting ones. The concept of R‐ordering of vectors is employed to find a representative frame, which is used to perform relative ranking of the video frames. It is theoretically shown that significance is given to each element of a frame's feature vector while computing the importance scores that lead to the frame ranks used for skimming. Furthermore, the allocation of different weights to the features involved is also achieved using linear and Gaussian process regressions. Through extensive experiments considering several standard datasets with human‐labelled ground truth, the proposed approach is demonstrated to be efficient and to perform better than the relevant state‐of‐the‐art.

Salient object detection is evaluated using binary ground truth (GT) with the labels being salient object class and background. In this study, the authors corroborate based on three subjective experiments on a novel image dataset that objects in natural images are inherently perceived to have varying levels of importance. The authors' dataset, named SalMoN (saliency in multi‐object natural images), has 588 images containing multiple objects. The subjective experiments performed record spontaneous attention and perception through eye fixation duration, point clicking and rectangle drawing. As object saliency in a multi‐object image is inherently multi‐level, they propose that salient object detection must be evaluated for the capability to detect all multi‐level salient objects apart from the salient object class detection capability. For this purpose, they generate multi‐level maps as GT corresponding to all the dataset images using the results of the subjective experiments, with the labels being multi‐level salient objects and background. They then propose the use of mean absolute error, Kendall's rank correlation and average area under precision–recall curve to evaluate existing salient object detection methods on their multi‐level saliency GT dataset. Approaches that represent saliency detection on images as local‐global hierarchical processing of a graph perform well in their dataset.

Recently pixel pairing and pixel sorting/selection have been used in prediction-error expansion based reversible data hiding schemes to generate low entropy prediction-error histograms (PEH) necessary for achieving high fidelity. Such schemes generally use the four-neighbor average rhombus predictor as it allows pixel sorting and flexible pixel pairing. In this paper, we propose the maximally separated averages (MSA) predictor that uses the four-neighborhood context. It can replace the rhombus predictor in pixel pairing and sorting based schemes for lowering PEH entropy further to achieve higher performance. At each pixel location, we choose the two maximally separated average values and decide either on using one of them as the predicted value or on avoiding prediction at the pixel location. This is based on the observation that the prediction-error sequence entropy decreases with the increase in the separation between the two average values. Experimental results demonstrate that the state-of-the-art schemes achieve considerable performance improvement by using the proposed MSA predictor.

A Kalman filter‐based spatio‐temporal seam determination scheme is proposed in this study for video retargeting through seam carving. In probably a first, the authors’ retargeting approach is designed to be hardware friendly, and to achieve spatial and temporal coherence through an optimal solution with explicit mechanisms to reduce jitter and structural distortion. In a video frame, while spatially coherent seam is determined by the popular approach of seam energy minimisation, temporally coherent seam is determined using Kalman prediction and updation processes. Further, these spatial and temporal seams are combined judiciously to obtain the spatio‐temporal seam to be removed/repeated for decreasing/increasing the frame size. The authors show that the proposed Kalman filter‐based approach has less theoretical complexity compared to the existing. Extensive experimental results show that the proposed approach consistently outperforms the state‐of‐the‐art, both qualitatively and quantitatively in performance, and in computational time.

Usually, limited information is available for enhancing low light images. We propose that presence of interactions between superposed luminance levels be exploited to draw reliable information for the enhancement. Grover's algorithm implements the interaction resulting in information passing between luminance levels, following which a noise-resilient transformation is employed. Qualitative and quantitative results depict that our proposal is capable of enhancing low light images better than the state-of-the-art.

Video skimming is a process of generating a shorter yet fully comprehensible version of a given video as its dynamic summary. A generic skimming system involves division of the video into segments and selecting the segments based on their suitability. The suitability is often obtained considering various features of the video and combining their individual contributions. Suggesting that the combination causes loss of information, we propose collective representation of the individual contributions in the form of a vector and use vector reduced (R)-ordering to judge the suitability. R-ordering based tree-structured organization and similarity levels of the video segments are employed to determine the suitability. Comparing with user generated summaries, we show that a video summary generated by a general skimming approach using R-ordering will be more effective in covering the important parts of a given video than when a feature combination is used.

This paper presents a prediction based spatio-temporal seam carving scheme for video retargeting. It resizes the video maintaining appropriate balance between spatial and temporal coherence. In a video frame, the proposed approach finds a ‘temporal’ seam by using Kalman filter estimation and then modifies it with the help of ‘spatial’ seam considering both spatial and temporal coherency. Unlike image retargeting, it is of utmost importance in retargeting a video frame to consider temporal coherency along with spatial coherency to remove or replicate unimportant background portion. This will ensure that insignificant amount of motion artifacts are introduced during resizing. The proposed Kalman filter based approach not only predicts a spatio-temporal seam to mark a portion of the frame where there is more possibility of having spatially and temporally coherent seam, but also has low time complexity. The proposed approach outperforms other state-of-the-art video retargeting methods which is illustrated by experimental results.

Histogram equalization, which aims at information maximization, is widely used in different ways to perform contrast enhancement in images. In this paper, an automatic exact histogram specification technique is proposed and used for global and local contrast enhancement of images. The desired histogram is obtained by first subjecting the image histogram to a modification process and then by maximizing a measure that represents increase in information and decrease in ambiguity. A new method of measuring image contrast based upon local band-limited approach and center-surround retinal receptive field model is also devised in this paper. This method works at multiple scales (frequency bands) and combines the contrast measures obtained at different scales using L(p)-norm. In comparison to a few existing methods, the effectiveness of the proposed automatic exact histogram specification technique in enhancing contrasts of images is demonstrated through qualitative analysis and the proposed image contrast measure based quantitative analysis.

Feature space based approaches have been the most popular ones among those used to perform image segmentation. In this paper, a density modification framework is proposed in order to aid feature space based segmentation in images. The framework embeds a position-dependent property associated with each sample in the feature space of an image into the corresponding density map and hence modifies it. The property association and embedding operations in the framework is implemented using a fuzzy set theory based system devised with cue from beam theory of solid mechanics and the appropriateness of this approach is established. Experimental results of segmentation in images are given to demonstrate the effectiveness of the proposed framework.

This paper presents a novel histogram thresholding methodology using fuzzy and rough set theories. The strength of the proposed methodology lies in the fact that it does not make any prior assumptions about the histogram unlike many existing techniques. For bilevel thresholding, every element of the histogram is associated with one of the two regions by comparing the corresponding errors of association. The regions are considered ambiguous in nature, and, hence, the error measures are based on the fuzziness or roughness of the regions. Multilevel thresholding is carried out using the proposed bilevel thresholding method in a tree structured algorithm. Segmentation, object/background separation, and edge extraction are performed using the proposed methodology. A quantitative index to evaluate image segmentation performance is also proposed using the median of absolute deviation from median measure, which is a robust estimator of scale. Extensive experimental results are given to demonstrate the effectiveness of the proposed methods in terms of both qualitative and quantitative measures.

Quantifying ambiguities in images using fuzzy set theory has been of utmost interest to researchers in the field of image processing. In this paper, we present the use of rough set theory and its certain generalizations for quantifying ambiguities in images and compare it to the use of fuzzy set theory. We propose classes of entropy measures based on rough set theory and its certain generalizations, and perform rigorous theoretical analysis to provide some properties which they satisfy. Grayness and spatial ambiguities in images are then quantified using the proposed entropy measures. We demonstrate the utility and effectiveness of the proposed entropy measures by considering some elementary image processing applications. We also propose a new measure called average image ambiguity in this context.