Image segmentation is a difficult and challenging task because of the complex object appearance and diverse object categories. Traditional methods directly use visual features for segmentation but ignore the correlation between objects. We introduce a knowledge reasoning module (KRM) for external knowledge aggregation and leverage a graphic neural network to aggregate the knowledge feature, which is concatenated with a visual feature for semantic segmentation. To this end, we use word embedding of category names as semantic feature and establish the relationship between categories. Through iteration, the aggregated features can be enriched. In experiments, three well known semantic segmentation methods are used as baseline. Our experiment results outperform the baseline methods on the food dataset Food-Seg103 and Cityscapes, and demonstrate the effectiveness of our proposed method.

Abstract Action recognition has been a popular area of computer vision research in recent years. The goal of this task is to recognise human actions in video frames. Most existing methods often depend on the visual features and their relationships inside the videos. The extracted features only represent the visual information of the current video itself and cannot represent the general knowledge of particular actions beyond the video. Thus, there are some deviations in these features, and the recognition performance still requires improvement. In this sudy, we present a novel spatio‐temporal knowledge module (STKM) to endow the current methods with commonsense knowledge. To this end, we first collect hybrid external knowledge from universal fields, which contains both visual and semantic information. Then graph convolution networks (GCN) are used to represent and aggregate this knowledge. The GCNs involve (i) a spatial graph to capture spatial relations and (ii) a temporal graph to capture serial occurrence relations among actions. By integrating knowledge and visual features, we can get better recognition results. Experiments on AVA, UCF101‐24 and JHMDB datasets show the robustness and generalisation ability of STKM. The results report a new state‐of‐the‐art 32.0 mAP on AVA v2.1. On UCF101‐24 and JHMDB datasets, our method also improves by 1.5 AP and 2.6 AP, respectively, over the baseline method.

The mainstream object detection algorithms rely on recognizing object instances individually, but do not consider the high-level relationship among objects in context. This will inevitably lead to biased detection results, due to the lack of commonsense knowledge that humans often use to assist the task for object identification. In this paper, we present a novel reasoning module to endow the current detection systems with the power of commonsense knowledge. Specifically, we use graph attention network (GAT) to represent the knowledge among objects. The knowledge covers visual and semantic relations. Through the iterative update of GAT, the object features can be enriched. Experiments on the COCO detection benchmark indicate that our knowledge-based reasoning network has achieved consistent improvements upon various CNN detectors. We achieved 1.9 and 1.8 points higher Average Precision (AP) than Faster-RCNN and Mask-RCNN respectively, when using ResNet50-FPN as backbone.

Automatic assessment of breast cancer metastases plays an important role to help pathologist reduce the time-consuming work in histopathological whole-slide image diagnosis. From the utilization of knowledge point of view, the low-magnification level and high-magnification level are carefully checked by the pathologists for tumor pattern and cell tumor characteristic. In this paper, we propose a novel automatic patient-level tumor segmentation and classification method, which makes full use of the diagnosis knowledge clues from pathologists. For tumor segmentation, a multi-level view DeepLabV3+ (MLV-DeepLabV3+) is designed to explore the distinguishing features of cell characteristics between tumor and normal tissue. Furthermore, the expert segmentation models are selected and integrated by Pareto-front optimization to imitate the expert consultation to get perfect diagnosis. For wholeslide classification, multi-level magnifications are adaptive checked to focus on the effective features in different magnification. The experimental results demonstrate that our pathologist knowledge-based automatic assessment of whileslide image is effective and robust on the public benchmark dataset.

Model efficiency for object detection has become more and more important recently, especially when intelligent mobile devices get more and more popular. Current lightweight object detection model is either migrated from lightweight classification models, or pruned directly from complex object detection models. These pipelines can not match the performance requirements of edge devices. In this work, we propose a neural architecture search (NAS) method to build a detection model automatically that can perform well on edge devices. Specifically, the proposed method supports the search of not only multiscale feature network, but also backbone network. This enables us to search out a global optimal model. To this end, we've made a special design that the backbone and feature network can share the same search space. This method greatly reduces the search time while ensuring the search accuracy. It can find a good architecture in 14GPU days. Additionally, we add latency information into the main objective during performance estimation. Therefore, we can control between the speed and accuracy to better adapt to the edge environment. Experiments on the PASCAL VOC benchmark indicate that the searched architecture (named NAS-EOD) can get good accuracy even if training from scratch. When using a pre-training scheme, our model is superior to state-of-the-art small object detection models.

Object detection is an important task in very high-resolution remote sensing image analysis. Traditional detection approaches are often not sufficiently robust in dealing with the variations of targets and sometimes suffer from limited training samples. In this paper, we tackle these two problems by proposing a novel method for object detection based on structural feature description and query expansion. The feature description combines both local and global information of objects. After initial feature extraction from a query image and representative samples, these descriptors are updated through an augmentation process to better describe the object of interest. The object detection step is implemented using a ranking support vector machine (SVM), which converts the detection task to a ranking query task. The ranking SVM is first trained on a small subset of training data with samples automatically ranked based on similarities to the query image. Then, a novel query expansion method is introduced to update the initial object model by active learning with human inputs on ranking of image pairs. Once the query expansion process is completed, which is determined by measuring entropy changes, the model is then applied to the whole target data set in which objects in different classes shall be detected. We evaluate the proposed method on high-resolution satellite images and demonstrate its clear advantages over several other object detection methods.

Segmentation and classification are important tasks in remote sensing image analysis. Recent research shows that images can be described in hierarchical structure or regions. Such hierarchies can produce the state-of-the-art segmentations and can be used in the classification. However, they often contain more levels and regions than required for an efficient image description, which may cause increased computational complexity. In this letter, we propose a new hierarchical segmentation method that applies graph Laplacian energy as a generic measure for segmentation. It reduces the redundancy in the hierarchy by an order of magnitude with little or no loss of performance. In the classification stage, we apply local self-similarity feature to capture the internal geometric layouts of regions in an image. By incorporating advantages from both semantic hierarchical segmentation and local geometric region description, we have achieved better performance than those from the methods being compared. In the experimental section, we validate the effectiveness of our method by showing results on QuickBird and GeoEye-1 image data sets.

In this paper, we propose an approach for object detection via structural feature selection and part-based shape model. It automatically learns a shape model from cluttered training images without need to explicitly use bounding boxes on objects. Our approach first builds a class-specific codebook of local contour features, and then generates structural feature descriptors by combining context shape information. These descriptors are robust to both within-class variations and scale changes. Through exploring pairwise image matching using fast earth mover's distance, feature weights can be iteratively updated. Those discriminative foreground features are assigned high weights and then selected to build a part-based shape model. Finally, object detection is performed by matching each testing image with this model. Experiments show that the proposed method is very effective. It has achieved comparable performance to the state-of-the-art shape-based detection methods, but requires much less training information.

In order to detect the objects of interest, many different approaches have been proposed. One kind of popular approaches are based on template matching, which use a template of the object class to match the image at different positions. The matching can be computed using similarity measures such as the correlation coefficient. These approaches, although easy and robust, has the limitation of not containing to much variability of the object class, especially for shape information. Despite the statistical variation in each kind of object, collecting enough training samples is another problem which is time consuming. Inspire by template matching and incremental learning, a new object-oriented object detection methodology for very high resolution remote sensing images is proposed in this paper. We obtain the first initial query results via the bag-of-visual-words method. Then we introduce two query expansion baseline expansion and PAS expansion to obtain a new incremental model for re-query. In the experiment part, we compare and evaluate the performance of our proposed methods.