With the rise of smart home applications, WiFi-based posture estimation has become a research focus. Existing methods often emphasize end-to-end feature extraction while neglecting high-resolution intermediate spatial features. We observe that horizontal and vertical antennas capture electromagnetic wave distributions in different directions, while multiple subcarrier frequencies inherently enhance multipath resolution. By analyzing signal correlations among subcarriers, we achieve equivalence to virtual antennas, enabling the generation of high-resolution spatial correlation maps. Using a cascaded network with UNet, we extract human spatial transformation features and a steering vector to obtain a spatial spectrum, mapping keypoints with high precision. Tests in through-wall environments achieve an average mPCK of 0.90 at 20 pixels and 0.98 at 50 pixels, significantly improving detection accuracy.

Abstract Referring image segmentation aims to segment object in an image based on a referring expression. Its difficulty lies in aligning expression semantics with visual instances. The existing methods based on semantic reasoning are limited by the performance of external syntax parser and do not explicitly explore the relationships between visual instances. This article proposes an end-to-end method for referring image segmentation by aligning ’linguistic relationship’ with ’visual relationships’. This method does not rely on external syntax parser for expression parsing. In this paper, the expression is adaptively and structurally parsed into three components: ’subject’, ’object’, and ’linguistic relationship’ by the Semantic Component Parser (SCP) in a learnable manner. Instances Activation Map Module (IAM) locates multiple visual instances based on the subject and object. In addition, the Relationship Based Visual Localization Module (RBVL) firstly enables each instance of the image to learn global knowledge, then decodes the visual relationships between these visual instances, and finally aligns the visual relationships with the linguistic relationships to further accurately locate the target object. The experimental results show that the proposed method improves performance by 4– 9% compared with baseline method on multiple referring image segmentation datasets.

Most existing image deblurring methods construct statistical prior to describe the difference between blur and clear image. They discard the position information and ignore pixel feature changing in deblurring, which results in inferior restoration performance for images unsatisfying corresponding assumptions. Intuitively, fuzziness of pixel belonging to different image regions will reduce along with image deblurring. This phenomenon could intrinsically describe the pixel characteristic. To this end, we analyze fuzziness of pixels and objects in a blurry image, and utilize the similarity between two fuzzy objects on image pixels to depict the blur degree of an image, which is inspired by overlap functions and overlap indices. To minimize the similarity between fuzzy objects, we introduce the non-parameters model to construct an integer programming problem. Energy minimization could significantly reduce the similarity between two fuzzy objects. Experimental results show that the proposed method can achieve better performance than the state-of-the-art blind deblurring methods on benchmark datasets and natural images.

We present an effective blind image deblurring algorithm based on superpixel segmentation prior (SSP). The motivation of this work is an interesting observation that the more rough the segmentation boundaries are, the clearer the image will be. Intuitively, the blurry images have less image details, which results in more smooth segmentation boundaries. The clear images have more vivid textural details and obtain more rough boundaries. The segmentation roughness could be defined as the length of image segmentation boundaries. However, the segmentation boundary length is not differentiable, making it difficult to integrate into existing joint optimization framework. Therefore, we transform the segmentation boundary length into the segmentation entropy to guide the process of image deblurring. With the image becomes clearer, its boundary becomes more rough, while the segmentation entropy is much smaller. The analysis of relationship between segmentation entropy and segmentation boundaries is detailed. Benefiting from the convexity of segmentation entropy, we propose a novel algorithm by integrating half-quadratic split and gradient descent to alternately minimize energy function. Extensive experiments show that the proposed method achieves best performance with the state-of-the-art blind deblurring methods on natural and face image deblurring.

Existing unsupervised video object segmentation generates object information from the whole video, which ignores analysis of the local clips. However, we observe that local clips and their relationships are also useful for the video object segmentation. For example, the simple background clips can be used to improve the segmentation of complex background clips. In this paper, we propose a novel unsupervised segmentation framework to segment the primary object based on two aspects, i.e., the complexity awareness of video clips and their segmentation propagation. The first one is used to select the simple clips with smooth backgrounds and the second one generates an object prior from the simple clips and propagates the object prior to help and improve the segmentation of the complex clips. A complexity awareness method using the static cues and the dynamic cues are proposed to evaluate the complexity of the video frames. A new object prior learning model based on the local part structure is designed and a local part-based prior propagation is proposed for the complex clip segmentation. To verify our method, we collect a new challenging video segmentation data set, in which each video contains diverse backgrounds. Experimental results demonstrate that our method outperforms several state-of-the-art methods both on a classical data set and our new data set.

This paper proposes an edge-based superpixel similarity measurement, which globally evaluates the similarity between superpixels by binary edge maps. The basic idea is to assess whether the superpixels are surrounded by the same edges. To this end, we first describe the edge spatial distributions by directional regions and then use the directional regions to represent the surrounding relationships of superpixels and edges by their traverse relationships, which form the histogram feature. Finally, the similarity is simply calculated by the distances between the features. To verify the proposed similarity measurement, we use our global similarity measurement to perform superpixel clustering. Two clustering methods, the directed graph clustering (DGC) and spectral clustering (ultrametric contour map) are combined to achieve the clustering process. The combination of our global similarity measurement and DGC to form a new three-layer-based superpixel generation method, which can quickly generate the superpixel from edge maps, is highlighted. We verify the global similarity measurement by the BSDS500 dataset. The experimental results demonstrate that the proposed global similarity measurement can improve the clustering accuracy in terms of larger intersection-over-union-criterion-based values. The code can be downloaded from https://github.com/FanmanMeng/Superpixel-Similarity-Measurement.

Weakly supervised local part segmentation is challenging, due to the difficulty of modeling multiple local parts from image level prior. In this paper, we propose a new weakly supervised local part proposal segmentation method based on the observation that local parts will keep fixed along the object pose variations. Hence, the local part can be segmented by capturing object pose variations. Based on such observation, a new local part proposal segmentation model is proposed. Three aspects, such as shape similarity-based cosegmentation, shape matching-based part detection and segmentation, and graph matching-based part assignment are considered. A part segmentation energy function is first proposed. Four terms, such as MRF-based single image segmentation term, shape feature-based foreground consistency term, NCuts-based part segmentation term, and two-order graphs matching based part consistency term, are contained. Then, a three sub-minimization-based energy minimization method is proposed to accomplish approximation solution. Finally, we verify our method based on three image data sets (PASCAL VOC 2008 Part data set, UCB Bird data set, and Cat-Dog data set), and one video data set (UCF Sports) data set. The experimental results demonstrate a better segmentation performance compared with the existing object cosegmentation and part proposal generation methods.

Fine-grained categorization is a challenging task mainly due to two factors: first, objects share similar appearances between different categories; second, objects present significant pose variation within the same category. To address these challenges, we propose a method to automatically detect discriminative and pose-invariant regions, which is referred to as a polygon-based classifier (PBC). In the first stage, we generate a set of polygons that are composed of multiple parts. For each polygon, a classifier is trained based on deep features of a convolutional network. Then, a greedy algorithm is employed to select the discriminative and complementary polygon-based classifiers that deliver highest classification accuracy for fine-grained object categories. In the second stage, the confusing classes of the first stage are selected and employed to train the polygon-based classifiers. Then, a greedy algorithm is employed to select discriminative classifiers. For the test images, we use the classifiers trained in the first stage to obtain a coarse result. Then, the classifiers of the second stage are adopted to distinguish the confusing classes of the coarse result. In our experiments, the proposed approach is evaluated on three well-known fine-grained datasets. The experiments show that our approach outperforms the state-of-the-art methods.

In this paper, we propose a video object segmentation method via global consistency aware query strategy. The aim is to obtain higher segmentation accuracy with less user annotation. Intuitively, we hope to annotate some frames to obtain better segmentation performance than to annotate other frames, which can be modeled by active learning framework. Specifically, we first generate a sample space of potential annotation regions via an object proposals method for each frame. Then, the annotation likelihood for the region is calculated in terms of annotation history and global consistency for the object in the video. Third, the segmentation result of the annotation region can be obtained by minimizing an MRF energy function. Fourth, the algorithm will provide the user with the most valuable frame to annotate, which has high annotation likelihood and large segmentation result change. Finally, the annotation is added to the framework to begin the next iteration. Experiments on a number of video sequences demonstrate that the proposed method can reduce the user effort and obtain the higher segmentation accuracy compared with the state-of-the-art methods.

Blind image quality assessment (BIQA) aims to estimate the subjective quality of a query image without access to the reference image. Existing learning-based methods typically train a regression function by minimizing the average error between subjective opinion scores and model predictions. However, minimizing average error does not necessarily lead to correct quality rank-orders between the test images, which is a highly desirable property of image quality models. In this paper, we propose a novel rank-order regularized regression model to address this problem. The key idea is to introduce a pairwise rank-order constraint into the maximum margin regression framework, aiming to better preserve the correct perceptual preference. To the best of our knowledge, this is the first attempt to incorporate rank-order constraints into margin-based quality regression model. By combing with a new local spatial structure feature, we achieve highly consistent quality prediction with human perception. Experimental results show that the proposed method outperforms many state-of-the-art BIQA metrics on popular publicly available IQA databases (i.e., LIVE-II, TID2013, VCL@FER, LIVEMD, and ChallengeDB).

Segment propagation transfers object priors among images, which is an important prior generation manner in image segmentation. The existing propagation methods focus on object foreground propagation, while the detailed part propagation is deficiency, which is caused by the challenges that not only the multiple part regions, but also their relationships need to be transferred. In this paper, a part propagation method is proposed. Two level propagations such as object level propagation, and part level propagation are successively used for the part propagation. The object level propagation is to transfer global shape information among images, which is formulated as graph matching based edge fragments matching problem, with dynamic programming solution. The part level propagation is to transfer the more detailed part labels, which is formulated as pixel level structure matching problem, and is efficiently solved by traditional dense pixel matching methods. The proposed method is verified on 15 challenging classes selected from PASCAL 2010 dataset, Bird dataset and Cat-Dog dataset. The experimental results demonstrate the effectiveness of the proposed method.

In this paper, we propose an efficient blind image quality assessment (BIQA) algorithm, which is characterized by a new feature fusion scheme and a k-nearest-neighbor (KNN)-based quality prediction model. Our goal is to predict the perceptual quality of an image without any prior information of its reference image and distortion type. Since the reference image is inaccessible in many applications, the BIQA is quite desirable in this context. In our method, a new feature fusion scheme is first introduced by combining an image's statistical information from multiple domains (i.e., discrete cosine transform, wavelet, and spatial domains) and multiple color channels (i.e., Y, Cb, and Cr). Then, the predicted image quality is generated from a nonparametric model, which is referred to as the label transfer (LT). Based on the assumption that similar images share similar perceptual qualities, we implement the LT with an image retrieval procedure, where a query image's KNNs are searched for from some annotated images. The weighted average of the KNN labels (e.g., difference mean opinion score or mean opinion score) is used as the predicted quality score. The proposed method is straightforward and computationally appealing. Experimental results on three publicly available databases (i.e., LIVE II, TID2008, and CSIQ) show that the proposed method is highly consistent with human perception and outperforms many representative BIQA metrics.

Most existing video segmentation methods are focused on extracting the primary objects in test video sequences. They assumed that only one object appeared through the whole video sequences, which is impractical in many applications. In this paper, we focus on the object segmentation from the long video sequences which consist of many different scenes, shot cuts and various motion patterns, etc. In order to solve this problem, we propose a framework to segment the objects in relative video shots, while discarding the irrelative video shots. A graph is constructed to model the video object detection and final segmentation is obtained by getting the superpixels in the detection boxes. We also introduce a new long video segmentation dataset which corresponds to the pixel-wise ground truth. The experiments demonstrate that our proposed method can deal with the object segmentation in long video sequence.

This paper proposes a new constrained directed graph clustering (DGC) method and segmentation propagation method for the multiple foreground cosegmentation. We solve the multiple object cosegmentation with the perspective of classification and propagation, where the classification is used to obtain the object prior of each class and the propagation is used to propagate the prior to all images. In our method, the DGC method is designed for the classification step, which adds clustering constraints in cosegmentation to prevent the clustering of the noise data. A new clustering criterion such as the strongly connected component search on the graph is introduced. Moreover, a linear time strongly connected component search algorithm is proposed for the fast clustering performance. Then, we extract the object priors from the clusters, and propagate these priors to all the images to obtain the foreground maps, which are used to achieve the final multiple objects extraction. We verify our method on both the cosegmentation and clustering tasks. The experimental results show that the proposed method can achieve larger accuracy compared with both the existing cosegmentation methods and clustering methods.

This paper proposes a novel saliency object detection method by using the mid-level and high-level visual cues. In the mid-level objectness evaluation, we generate three complementary saliency maps, such as the multi-scale segmentation cue, the background cue and the spatial color distribution cue. The first cue is used to highlight the objects via the local region segment. The second cue uses the background priors to detect the saliency information. The third cue is to capture the spatial color distribution. For the high-level visual cue, we propose an objectness evaluation model to distinguish the object and the background. All the saliency cues are finally combined to achieve the saliency detection. The experimental results show that the proposed method outperforms the state-of-the-art saliency object detection methods.

In this paper, we propose a framework to discover and segment favorite object from the natural images. The main idea is to first generate the shape based common template of the favorite object using the images collected from the web. Then, the common template is used to extract the favorite object from the original images. In the common template generation, co-segmentation is used to provide the initial segments. The median graph theory is employed to construct the common template. We also propose a new shape descriptor namely directional shape representation to handle shape variations. We test our method on the images collected from image datasets and web. Experimental results demonstrate the effectiveness of the proposed method.

This letter presents a noise-robust descriptor by exploring a set of local contrast patterns (LCPs) via global measures for texture classification. To handle image noise, the directed and undirected difference masks are designed to calculate three types of local intensity contrasts: directed, undirected, and maximum difference responses. To describe pixel-wise features, these responses are separately quantized and encoded into specific patterns based on different global measures. These resulting patterns (i.e., LCPs) are jointly encoded to form our final texture representation. Experiments are conducted on the well-known Outex and CUReT databases in the presence of high levels of noise. Compared to many state-of-the-art methods, the proposed descriptor achieves superior texture classification performance while enjoying a compact feature representation.

The problem of unsupervised segmentation of multi-class regions can be significantly boosted when they irregularly recur in multiple images. The existing segmentation methods are either weakly supervised, such as tagging images with object classes, or are limited by the assumption that each image contains all the object instances. In this paper, we propose a new method to cosegment multiclass regions from a group of images without the assumption about object configurations. The key idea is to discover the unknown object-like proposals via a robust ensemble clustering scheme. The proposals are then used to derive unary and pairwise energy potentials across all the images, which can be minimized with the α-expansion. Experimental evaluation on a number of image groups demonstrates the good performance of the proposed method on the multiclass region cosegmentation.

In this paper, we improve co-segmentation performance by repairing bad segments based on their quality evaluation and segment propagation. Starting from co-segmentation results of the existing co-segmentation method, we first perform co-segmentation quality evaluation to score each segment. Good segments can be filter out based on the scores. Then, a propagation method is designed to transfer good segments to the rest bad ones so as to repair the bad segmentation. In our method, the quality evaluation is implemented by the measurements of foreground consistency and segment completeness. Two propagation methods such as global propagation and local region propagation are then defined to achieve the more accurate propagation. We verify the proposed method using four state-of-the-arts co-segmentation methods and two public datasets such as ICoseg dataset and MSRC dataset. The experimental results demonstrate the effectiveness of the proposed quality evaluation method. Furthermore, the proposed method can significantly improve the performance of existing methods with larger intersection-over-union score values.

In this paper, we develop a new algorithm to segment multiple common objects from a group of images. Our method consists of two aspects: directed graph clustering and prior propagation. The clustering is used to cluster the local regions of the original images and generate the foreground priors from these clusterings. The second step propagates the prior of each class and locates the common objects from the images in terms of foreground map. Finally, we use the foreground map as the unary term of Markov random field segmentation and segment the common objects by graph-cuts algorithm. We test our method on FlickrMFC and ICoseg datasets. The experimental results show that the proposed method can achieve larger accuracy compared with several state-of-arts co-segmentation methods.

In this paper, we present a robust texture representation by exploring an ensemble of binary codes. The proposed method, called Locally Enhanced Binary Coding (LEBC), is training-free and needs no costly data-to-cluster assignments. Given an input image, a set of features that describe different pixel-wise properties, is first extracted so as to be robust to rotation and illumination changes. Then, these features are binarized and jointly encoded into specific pixel labels. Meanwhile, the Local Binary Pattern (LBP) operator is utilized to encode the neighboring relationship. Finally, based on the statistics of these pixel labels and LBP labels, a joint histogram is built and used for texture representation. Extensive experiments have been conducted on the Outex, CUReT and UIUC texture databases. Impressive classification results have been achieved compared with state-of-the-art LBP-based and even learning-based algorithms.