Given a language query, temporal grounding task is to localize temporal boundaries of the described event in an untrimmed video. There is a long-standing challenge that multiple moments may be associated with one same video-query pair, termed label uncertainty. However, existing methods struggle to localize diverse moments due to the lack of multi-label annotations. In this paper, we propose a novel Diverse Temporal Grounding framework (DTG) to achieve diverse moment localization with only single-label annotations. By delving into the label uncertainty, we find the diverse moments retrieved tend to involve similar actions/objects, driving us to perceive these interest moments. Specifically, we construct soft multi-label through semantic similarity of multiple video-query pairs. These soft labels reveal whether multiple moments in the intra-videos contain similar verbs/nouns, thereby guiding interest moment generation. Meanwhile, we put forward a diverse moment regression network (DMRNet) to achieve multiple predictions in a single pass, where plausible moments are dynamically picked out from the interest moments for joint optimization. Moreover, we introduce new metrics that better reveal multi-output performance. Extensive experiments conducted on Charades-STA and ActivityNet Captions show that our method achieves state-of-the-art performance in terms of both standard and new metrics.

Temporal grounding aims to localize temporal boundaries within untrimmed videos by language queries, but it faces the challenge of two types of inevitable human uncertainties: query uncertainty and label uncertainty. The two uncertainties stem from human subjectivity, leading to limited generalization ability of temporal grounding. In this work, we propose a novel DeNet (Decoupling and Debias) to embrace human uncertainty: Decoupling — We explicitly disentangle each query into a relation feature and a modified feature. The relation feature, which is mainly based on skeleton-like words (including nouns and verbs), aims to extract basic and consistent information in the presence of query uncertainty. Meanwhile, modified feature assigned with style-like words (including adjectives, adverbs, etc) represents the subjective information, and thus brings personalized predictions; De-bias — We propose a de-bias mechanism to generate diverse predictions, aim to alleviate the bias caused by single-style annotations in the presence of label uncertainty. Moreover, we put forward new multi-label metrics to diversify the performance evaluation. Extensive experiments show that our approach is more effective and robust than state-of-the-arts on Charades-STA and ActivityNet Captions datasets.

Visual relationship detection, as a challenging task used to find and distinguish interactions between object-pairs in one image, has received much attention recently. In this work, we devise a unified visual relationship detection framework with two types of correlation exploitation to address the combination explosion problem in the object-pairs proposing stage and the non-exclusive label problem in the predicate recognition stage. In the object-pairs proposing stage, with the exploitation of relative location correlation between two objects in one pair, one location-embedded rating module (LRM) is developed to effectively select plausible proposals. In the predicate recognition stage, one label-correlation graph module (LGM) is introduced to measure the implicit semantic correlation among predicates; and then assign discrete distributed labels to predicates to improve the precision of top-n recall. Experiments on the two widely used VRD and VG datasets show that our proposed method outperforms current state-of-the-art methods.

Crowdsourcing is a process of acquisition, integration, and analysis of big and heterogeneous data generated by a diversity of sources in urban spaces, such as sensors, devices, vehicles, buildings, and human. Especially, nowadays, no countries, no communities, and no person are immune to urban emergency events. Detection about urban emergency events, e.g., fires, storms, traffic jams is of great importance to protect the security of humans. Recently, social media feeds are rapidly emerging as a novel platform for providing and dissemination of information that is often geographic. The content from social media usually includes references to urban emergency events occurring at, or affecting specific locations. In this paper, in order to detect and describe the real time urban emergency event, the 5W (What, Where, When, Who, and Why) model is proposed. Firstly, users of social media are set as the target of crowd sourcing. Secondly, the spatial and temporal information from the social media are extracted to detect the real time event. Thirdly, a GIS based annotation of the detected urban emergency event is shown. The proposed method is evaluated with extensive case studies based on real urban emergency events. The results show the accuracy and efficiency of the proposed method.

Hashing has been widely used in large-scale image retrieval based on approximate nearest neighbor search. Most learning-to-hashing methods adopt a two-stage algorithm to generate binary codes. First, original images are mapped into continuous visual features. Then, binary codes are generated by quantization step or separate projection. Nevertheless, these methods are sensitive to quantization operation, i.e., thresholding. To explicitly address this issue, this study proposes a novel feature quantization scheme with a loopy recurrent neural network, called loopy residual hashing, for the purpose of high accuracy in image retrieval. Instead of one-off thresholding-based feature binarization, the proposed approach performs an iterative threshold-then-approximate operation, which calculates the quantization residual after each thresholding step and then imitates another round of binarization to further approximate the coding residual. The resulting sequences of binary codes possess higher representation accuracy and extensive experiments on image retrieval demonstrate its superior discriminative capability over the prior art. In the meantime, theoretical approximation error analysis is given.

Visual relationship recognition, as a challenging task used to distinguish the interactions between object pairs, has received much attention recently. Considering the fact that most visual relationships are semantic concepts defined by human beings, there are many human knowledge, or priors, hidden in them, which haven't been fully exploited by existing methods. In this work, we propose a novel visual relationship recognition model using language and position guided attention: language and position information are exploited and vectored firstly, and then both of them are used to guide the generation of attention maps. With the guided attention, the hidden human knowledge can be made better use to enhance the selection of spatial and channel features. Experiments on VRD [2] and VGR [1] show that, with language and position guided attention module, our proposed model achieves state-of-the-art performance.

Visual relationship detection, as a challenging task used to find and distinguish the interactions between object pairs in one image, has received much attention recently. In this work, we propose a novel visual relationship detection framework by deeply mining and utilizing relative location of object-pair in every stage of the procedure. In both the stages, relative location information of each object-pair is abstracted and encoded as auxiliary feature to improve the distinguishing capability of object-pairs proposing and predicate recognition, respectively; Moreover, one Gated Graph Neural Network(GGNN) is introduced to mine and measure the relevance of predicates using relative location. With the location-based GGNN, those non-exclusive predicates with similar spatial position can be clustered firstly and then be smoothed with close classification scores, thus the accuracy of top n recall can be increased further. Experiments on two widely used datasets VRD and VG show that, with the deeply mining and exploiting of relative location information, our proposed model significantly outperforms the current state-of-the-art.

A public safety event is a danger and urgent event that need early detection, quick response, and accuracy recover. The efficient method for responding to a happening public safety event is to collect and describe the related data. Besides the surveillance cameras from the physical space, the social media data can also be used to collect and describe the related data of a public safety event. In this work, the proposed method focuses on the step for describing public safety events. Given a public safety event, videos from the surveillance cameras and social messages from social sensors are collected. The multi-modal information including texts, images, videos, and spatial-temporal data is mined to give a description precisely and concisely. First, the social sensors are associated to surveillance cameras by the spatial and temporal information. In the second stage, the social messages are associated to surveillance cameras by the semantic information. In the third stage, the social messages are associated to surveillance cameras by the visual feature. Besides the text, social sensors may upload images or videos. Finally, the multi-modal description step is given based on the three different associations. The experiments on the real data demonstrate the superiority of the proposed framework. Case studies on the real public safety event show the proposed model has good performance and high effectiveness.

Hashing is a widely adopted method based on an approximate nearest neighbor search and is used in large-scale image retrieval tasks. Conventional learning-based hashing algorithms employ end-to-end representation learning, which is a one-off technique. Because of the tradeoff between efficiency and performance, conventional learning-based hashing methods must sacrifice code length to improve performance, which increases their computational complexity. To improve the efficiency of binary codes, motivated by the “nonsalient-to-salient” attention scheme of humans, we propose a recursive hashing mechanism that maps progressively expanded salient regions to a series of binary codes. These salient regions are generated by a conventional saliency model based on bottom-up saliency-driven attention and a semantic-guided saliency model based on top-down task-driven attention. After obtaining a series of salient regions, we perform long-range temporal modeling of salient regions using a graph-based recurrent deep network to obtain more refined representative features. The later output nodes inherit aggregated information from all previous nodes and extract discriminative features from more salient regions. Therefore, this network possesses more significant information and satisfactory scalability. The proposed recursive hashing neural network, optimized by a triplet ranking loss, is end-to-end trainable. Extensive experimental results from several image retrieval benchmarks show the scalability of our method and demonstrate its strong performance compared with state-of-the-art methods.

An emergency event is a sudden, urgent, usually unexpected incident or occurrence that requires an immediate reaction or assistance for emergency situations, which plays an increasingly important role in the global economy and in our daily lives. Recently, the web is becoming an important event information provider and repository due to its real-time, open, and dynamic features. In this paper, web resources based states detecting algorithm of an event is developed in order to let the people know of an emergency event clearly and help the social group or government process the emergency events effectively. The relationship between web and emergency events is first introduced, which is the foundation of using web resources to detect the state of emergency events imaged on the web. Second, five temporal features of emergency events are developed to provide the basis for state detection. Moreover, the outbreak power and the fluctuation power are presented to integrate the above temporal features for measuring the different states of an emergency event. Using these two powers, an automatic state detecting algorithm for emergency events is proposed. In addition, heuristic rules for detecting the states of emergency event on the web are discussed. Our evaluations using real-world data sets demonstrate the utility of the proposed algorithm, in terms of performance and effectiveness in the analysis of emergency events.

Intelligent video and image analysis technology has been paid much attention recently. But how to effectively evaluate the performance of intelligent video and image analysis methods remains a meaningful and challenging task, which involves many aspects, such as constructing reasonable datasets, developing efficient evaluation tools, designing effective evaluation metrics. We focus on the area of public security and build an Intelligent Video and Image Analysis Evaluation Platform for Public Security (IVIAEPPS). This paper introduces some existing works on building this platform, as well as a video and image challenge based on it. We conclude the paper with some discussions on future works.

This paper proposes a novel recursive hashing scheme, in contrast to conventional "one-off" based hashing algorithms. Inspired by human's "nonsalient-to-salient" perception path, the proposed hashing scheme generates a series of binary codes based on progressively expanded salient regions. Built on a recurrent deep network, i.e., LSTM structure, the binary codes generated from later output nodes naturally inherit information aggregated from previously codes while explore novel information from the extended salient region, and therefore it possesses good scalability property. The proposed deep hashing network is trained via minimizing a triplet ranking loss, which is end-to-end trainable. Extensive experimental results on several image retrieval benchmarks demonstrate good performance gain over state-of-the-art image retrieval methods and its scalability property.

Association link network (ALN) can organize massive Web information to provide many intelligent services in our big data society. Effective semantic layered technologies not only can provide theoretical support for knowledge discovery in Web resources, but also can improve the searching efficiency of related information systems such as Web information system and industrial information system. How to realize the layer division of association semantic by the hierarchy analysis of ALN is an important research topic. To solve this problem, this paper proposes a hierarchy-cutting model of association semantic. First, experiments of four types of keywords with different linking roles are conducted to discover the possible distribution law. Experimental results show that these keywords with association role reveal previous power-law distribution. Then, based on the discovered power-law distribution, up-cutting and down-cutting points are presented to divide the association semantic into three layers. At the same time, theories of the hierarchy-cutting model are presented. Finally, examples of current core topic and permanent topics belonging to a domain are given. The experiments show that hierarchy-cutting points have high accuracy. The multilayer theory of association semantic can provide a theoretical support for knowledge recommendation with different particle sizes on ALNs.

Summary An emergency event is an unexceptional event that exceeds the capacity of normal resources and organization to cope and a situation that poses an immediate risk to health, life, property, or environment. Crowdsourcing connects unobtrusive and ubiquitous sensing technologies, advanced data management and analytics models, and novel visualization methods, to create solutions that improve urban environment, human life quality, and city operation systems. The crowdsourcing on social media can be used to detect and analyze urban emergency events. In this paper, in order to detect and describe the real‐time urban emergency event, the knowledge base model is proposed. The crowdsourcing‐based knowledge base model is firstly introduced, which uses the information from social media. Secondly, the basic definition of the proposed knowledge base model including keywords, patterns, positive sentences, and knowledge graph is given. Thirdly, the temporal information is added to the proposed knowledge base model. The case study on real data sets shows that the proposed algorithm has good performance and high effectiveness in the analysis and detection of emergency events. Copyright © 2016 John Wiley & Sons, Ltd.

Color, as a notable and stable attribute of vehicles, can serve as a useful and reliable cue in a variety of applications in intelligent transportation systems. Therefore, vehicle color recognition in natural scenes has become an important research topic in this area. In this paper, we propose a deep-learning-based algorithm for automatic vehicle color recognition. Different from conventional methods, which usually adopt manually designed features, the proposed algorithm is able to adaptively learn representation that is more effective for the task of vehicle color recognition, which leads to higher recognition accuracy and avoids preprocessing. Moreover, we combine the widely used spatial pyramid strategy with the original convolutional neural network architecture, which further boosts the recognition accuracy. To the best of our knowledge, this is the first work that employs deep learning in the context of vehicle color recognition. The experiments demonstrate that the proposed approach achieves superior performance over conventional methods.

In this paper, we study the problem of recognizing car brands in surveillance videos, cast it as an image classification problem, and propose a novel multiple instance learning method, named Spatially Coherent Discriminative Pattern Learning, to discover the most discriminative patterns in car images. The learned discriminative patterns can effectively distinguish cars of different brands with high accuracy and efficiency. The experimental results demonstrate that our method is significantly superior to recent image classification methods on this problem. The proposed method is able to deliver an end-to-end real-time car recognition system for video surveillance. Moreover, we construct a large and challenging car image data set, consisting of 37 195 real-world car images from 30 brands, which could serve as a standard benchmark in this field and be used in various related research communities.

SUMMARY Association relations between concepts are a class of simple but powerful regularities in binary data, which play important roles in enterprises and organizations with huge amounts of data. However, although there can be easily large number of association relation mined from databases, since existing objective and subjective methods scarcely take semantics into consideration, it has been recognized early in the knowledge discovery literature that most of them are of no interest to the user. In this paper, the semantic discrimination capability (SDC) of association relation is measured based on discrimination value model first. The formula of SDC integrating both statistical and graph features is proposed from five different strategies. The high correlation coefficient of the proposed method against discrimination value shows that the proposed SDC measure is accuracy. Moreover, an application using SDC on document clustering is carried out, which shows that SDC has broad prospects on data‐related task such as document clustering. Copyright 2013 John Wiley © Sons, Ltd.

Radio Signal Strength (RSS) based ranging is attractive by the low cost and easy deployment. In real environments, its accuracy is severely affected by the multipath effect and the external radio interference. The well-known fingerprint approaches can deal with the issues but introduce too much overhead in dynamic environments. In this paper, we attempt to address the issue along a completely different direction. We propose a new ranging framework called Fredi that exploits the frequency diversity to overcome the multi-path effect solely based on RSS measurements. We design a Discrete Fourier Transformation based algorithm and prove that it has the optimal solution under ideal cases. We further revise the algorithm to be robust to the measurement noises in practice. We implement Fredi on top of the USRP-2 platform and conduct extensive real environments in indoor environments. Experimental results show the superiority performance compared with the traditional methods.

Topology control is an effective method to improve the energy-efficiency and increase the communication capacity of Wireless Sensor Networks (WSNs). Traditional topology control algorithms are based on deterministic model that fails to consider lossy links which provide only probabilistic connectivity. Noticing this fact, we propose a novel probabilistic network model. We meter the network connectivity using network reachability. It is defined as the minimal of the upper limit of the end-to-end delivery ratio between any pair of nodes in the network. We attempt to find a minimal transmission power for each node while the network reachability is above a given application-specified threshold. The whole procedure is called probabilistic topology control (PTC). We prove that PTC is NP-hard and propose a fully distributed algorithm called BRASP. We prove that BRASP has the guaranteed performance and the communication overhead is O(|E| + |V|). The experimental results show that the network energy-efficiency can be improved by up to 250% and the average node degree is reduced by 50%.