Deep reinforcement learning (DRL) frameworks have shown their remarkable effectiveness in learning navigation policy for the mobile robot navigating in a human crowded environment. Moreover, attention mechanisms coupled with DRL allows the robot to identify neighbors with different level of influence and incorporate them into the robot’s decision. However, as the crowd density increases, attention mechanisms may fail to identify critical neighbors which can lead to significant drops in navigation efficiency. In this work, we aim to address this limitation by encoding both human-human and human-robot interaction using a special class of Graph Convolutional Networks (GCN) known as Message-Passing GCN (MP-GCN). In contrast to existing methods, where attention between robot and humans are encoded uniformly, the proposed approach named MP-GatedGCN-RL encodes asymmetric interactions using the combination of novel message-passing function and edge-wise gating mechanisms. We evaluate our approach on the simulated environments of ETH/UCY pedestrians datasets consisting of different scenarios like collision avoidance, group forming, diverging, crossing, and so on. Experimental results demonstrate that our proposed method outperforms the conventional benchmark dynamic avoidance method ORCA with a 20.6% increase in success rate and a 9.1% reduction in navigation time. Moreover, we also achieve a 5.5% enhancement in success rate compared to other state-of-the-art DRL-based methods without any additional labeled expert data nor prior supervised learning.

Collision-free path planning is challenging for mobile robot navigation tasks. Recently, the deep reinforcement learning method provided a more effective way to derive safe and efficient velocity commands directly from raw sensor information. Twin Delayed Deep Deterministic policy gradient (TD3) is an efficient approach for DRL navigation. However, original TD3 exists issues such as inefficiency learning and slow convergence speed which may influence the model to derive an ideal action for mobile robot navigation. To address these issues, we proposed a novel dueling architecture model, dueling deep deterministic policy gradient (Dueling-TD3), we compose the dueling network architecture into the critic network to increase the Q-value estimate precision. The results demonstrate that our proposed model outperforms the original model in terms of route planning capabilities.

The recent years have seen the increasing importance of cognitive models for improved robot navigation. In this paper, a novel cognitive navigation package, which consists of topometric map representation and a three-level path planner, is proposed. The topometric maps are built from architectural floor plans with additional features within a limited number of selected regions. The inherent discrepancies between floor plans and the robot’s actual sensory readings are handled by the three-level path planner. The unique feature of this approach is that accurate localization is only required at the selected regions. At the other regions the robot will rely on the guiding directions towards the goal rather than on its accurate position on the map for navigation. Experiments show that our approach can endow a robot with capability for semantic interpretation and localization in unseen and dynamic environments.

Vision-based vehicle detection under bad weather conditions is still a challenging problem. Adhesive raindrops on windshield have been known to diffract light and distort parts of the scene behind them. In this paper, we propose a Vehicle-Aware Generative Adversarial Networks (VAGAN) to improve vehicle detection from rain images. We train a Generative Adversarial Network (GAN) on image pairs, each comprising of an original rain image and one that is manually labelled with colored bounding boxes of the vehicles therein. The latter represents a fake version of the original image emphasizing the regions of interest. To further enhance vehicle awareness, we exploit the fact that the vehicle rear lights are usually turned on during rainy conditions to compute a saliency map of image, and use it formulate a background preserving constrain on the learning vehicle loss function. We show that this novel adversarial framework is able to generate new images with colored regions overlay over vehicles, hence effectively learning to differentiate image background from vehicles. The final vehicle detection in the generated images is not affected by image translation noise because we can simply use color segmentation to localize the vehicles. Experimental results on a large dataset show that our approach is an effective way to solve vehicle detection in rain images, achieving state-of-the-art performance.

While much effort has been devoted to deep learning object detection, relatively limited attention has been paid to object detection in bad weather, e.g. rain, snow or haze. In heavy rain, the raindrop on the front windshield can make it difficult to detect object from an in-car camera. The conventional way to cope with this has been to use radar as the main detection sensor. However, radar is highly susceptible to false positives. Furthermore, many entry level radar sensors only return the centroid of each detected object, rather than its size and extent. In addition, due to lack of texture input, radar cannot discriminate a vehicle from a non-vehicle object, e.g. roadside pole. This motivates us to detect vehicle by fusing radar and vision. In this paper, we first calibrate the radar and camera with respect to the ground plane. The radar detections are then projected to the camera image for target width estimation. Empirical evaluation on a large database shows that there is a natural synergy in both sensors, as the image based estimation is found to be greatly facilitated by the accuracy of the radar detection.

Spatial verification is a key step to remove outliers for accurate feature matching in visual place recognition. In this paper, we propose a novel method for outlier detection using a hierarchical spatial verification scheme. Given a set of putative correspondences between a pair of images, we convert the matching problem into a 4D transformation space and identify promising similarity transformations using Hough voting. In the hierarchical scheme, we first use a hypothesize-and-verify technique to identify groups of correspondences according to each similarity transformation. Second, the group with the largest number of correspondences serves as a standard to subsequently remove outliers in other groups by explicit geometric consistency checking. We have compared the proposed method with the state-of-the-art solutions on five popular public datasets to show that our method has better performance in place recognition and loop closure detection.

Humans are adept at tool use. We can intuitively and immediately improvise and use unknown objects in our environment as tools, to assist us in performing tasks. In this study, we provide similar cognition and capabilities to robots. Neuroscientific studies on tool use have suggested that human dexterity with tools is enabled by the embodiment of the tools, which in effect, allows humans to immediately transfer prior skills acquired without tools, onto tasks requiring tool use. Here, utilizing the theoretical results from our investigations on embodiment and tool use in humans over the last years, we propose a concept and algorithm to enable similar skill transfer by robots. Our algorithm enables a robot that has had no prior learning with tools, to automatically recognize an object (seen for the first time) in its environment as a potential tool for an otherwise unattainable task, and use the tool to perform the task thereafter.

Human robot interaction is increasingly becoming an important subject of research as robots become more pervasive in human spaces. In this paper, we describe generating navigation paths that are more socially acceptable. A human user first divides the operating environment into segments, and then connects all adjacent segments by defining topological paths. These paths form the preferred routes through which robots should navigate, injecting a form of social norm such as not using certain private areas as intermediate waypoints and maintaining a keep-left/right behavior in walkways. Given arbitrary start and destination points, we compute appropriate entry and exit locations into any of the fixed paths and move there by using existing agility based planning methods. Once the robot gets onto the prescribed paths, it stays on these fixed routes but is allowed to make slight deviations to avoid obstacles. Collectively, our navigation framework exploits the best of route based navigation and agile based motion planning. We empirically show that our navigation method produces efficient movement in a hospital ward and hotel that is also socially acceptable.

This paper describes the gesture recognition system developed by the Institute for Infocomm Research (I2R) for the 2013 ICMI CHALEARN Multi-modal Gesture Recognition Challenge. The proposed system adopts a multi-modal approach for detecting as well as recognizing the gestures. Automated gesture detection is performed using both audio signals and information about hand joints obtained from the Kinect sensor to segment a sample into individual gestures. Once the gestures are detected and segmented, features extracted from three different modalities, namely, audio, 2-dimensional video (RGB), and skeletal joints (Kinect) are used to classify a given sequence of frames into one of the 20 known gestures or an unrecognized gesture. Mel frequency cepstral coefficients (MFCC) are extracted from the audio signals and a Hidden Markov Model (HMM) is used for classification. While Space-Time Interest Points (STIP) are used to represent the RGB modality, a covariance descriptor is extracted from the skeletal joint data. In the case of both RGB and Kinect modalities, Support Vector Machines (SVM) are used for gesture classification. Finally, a fusion scheme is applied to accumulate evidence from all the three modalities and predict the sequence of gestures in each test sample. The proposed gesture recognition system is able to achieve an average edit distance of 0.2074 over the 275 test samples containing 2,742 unlabeled gestures. While the proposed system is able to recognize the known gestures with high accuracy, most of the errors are caused due to insertion, which occurs when an unrecognized gesture is misclassified as one of the 20 known gestures.

This paper summarizes our contribution to the Technicolor Rich Multimedia Retrieval from Input Videos Grand Challenge. We hold the view that semantic analysis of a given news video is best performed in the text domain. Starting with a noisy text obtained from applying Automatic Speech Recognition (ASR), a graph-based approach is then used to enrich the text by propagating labels from visually similar videos culled from parallel (YouTube) News sources. From the enriched text, we next extract salient keywords to form a query to a news video search engine, retrieving a larger corpus of related news video. Compared to a baseline method that only uses the ASR text, significant improvement in precision has been obtained, indicating that retrieval has benefited from the ingestion of the external labels. Capitalizing on the enriched metadata, we find that videos are more amenable to the Wikipedia-based Explicit Semantic Analysis (ESA), resulting in better support for subtopic news video retrieval. We apply our methods to an in-house live news search portal, and report on several best practices.

We introduce a novel query-to-modality mapping approach to the TRECVid 2010 known-Item video search (KIS) task. To search for a specific target video, a KIS query is verbose with many multi-modal attributes. Issuing all search terms to a retrieval engine will confuse the search criteria in different modalities and result in "topic drift". We propose decomposing a KIS query into a set of short uni-modal subqueries and issue them to the search index of the corresponding modality features, such as text-based metadata, visualbased high-level features. To do so, we introduce novel syntactic query features and cast the query-to-modality mapping as a classification problem. Retrieval results on the TRECVid 2010 KIS dataset shows that our approach outperforms existing methods by a significant margin.

We propose an attribute centric approach for visual object recognition. The attributes of an object are the observable visual properties that help to uniquely describe it. We present methods for identifying and learning these object attributes. To identify suitable object attributes, we process the corresponding Wikipedia pages to select terms that not only have high occurrence frequency, the images of these concepts must also be visually consistent. To learn object attributes, we assume prior knowledge of the object class-specific distributions of patches over the attributes, and introduce a novel algorithm that iteratively refines these distributions by a nearest-neighbor attribute classifier. Given an unseen image, its attribute vector is first formed by the distribution of patches over the attributes, and its final class is then determined by the attribute representation. We report efficacy of the proposed framework on an animal data set of ten classes, where the test set consists of images collected from the web.

Because of the inherent ambiguity in user queries, an important task of modern retrieval systems is faceted topic retrieval (FTR), which relates to the goal of returning diverse or novel information elucidating the wide range of topics or facets of the query need. We introduce a generative model for hypothesizing facets in the (news) video domain by combining the complementary information in the visual keyframes and the speech transcripts. We evaluate the efficacy of our multimodal model on the standard TRECVID-2005 video corpus annotated with facets. We find that: (1) the joint modeling of the visual and text (speech transcripts) information can achieve significant F-score improvement over a text-alone system; (2) our model compares favorably with standard diverse ranking algorithms such as the MMR. Our FTR model has been implemented on a news search prototype that is undergoing commercial trial.

The problem of polysemy in keyword-based image search arises mainly from the inherent ambiguity in user queries. We propose a latent model based approach that resolves user search ambiguity by allowing sense specific diversity in search results. Given a query keyword and the images retrieved by issuing the query to an image search engine, we first learn a latent visual sense model of these polysemous images. Next, we use Wikipedia to disambiguate the word sense of the original query, and issue these Wiki-senses as new queries to retrieve sense specific images. A sense-specific image classifier is then learnt by combining information from the latent visual sense model, and used to cluster and re-rank the polysemous images from the original query keyword into its specific senses. Results on a ground truth of 17K image set returned by 10 keyword searches and their 62 word senses provides empirical indications that our method can improve upon existing keyword based search engines. Our method learns the visual word sense models in a totally unsupervised manner, effectively filters out irrelevant images, and is able to mine the long tail of image search.

A trend in recent news video retrieval systems is to exploit contextual cues to improve search precision. Compared to using information only at the shot-level, the inclusion of story-level contextual cues can intuitively broaden query coverage and facilitate multimodal retrieval. This paper examines further the utility of story-level context in two aspects. Firstly, by apply a story-level matching against a parallel news corpus, better retrieval precision is achieved in both text-based query and document expansion. Secondly, a story-level semantic concept model is better apt at modeling visual similarity, resulting in improved retrieval precision. We quantify the case for story-level processing by its positive impact on the TRECVID-2005 dataset.

A novel Temporal Neighboring Pattern Similarity (TNPS) measure is proposed to recognize the time of a digital clock overlay. TNPS detects the presence of a clock overlay by monitoring the periodic changes of the clock digit, and infers the clock time by its natural transition cycle. Compared to traditional methods such as OCR, this method is faster and more reliable because it converts a pattern recognition problem to a pattern change detection problem. Experiments show the recognition result is promising and accurate. One of the applications for this method is to detect the start time of soccer game for our real time live soccer video highlights and event alerts.

To be viable advertising platforms, methods for in-program content placement in sports video must balance against clutter. We propose viewer relevance (VR) measures of video frames in the temporal and spatial domain. Video sub-segments with low temporal VR are first selected, within which actual content is emplaced on regions with low spatial VR. We compute VR measures using color, motion, texture and domain features, upon which spatio-temporal techniques are used to segment spatial regions for actual content placement. Results from preliminary subjective viewing trials on soccer and tennis video indicate that our approach is promising

Event detection is essential for sports video summarization, indexing and retrieval and extensive research efforts have been devoted to this area. However, the previous approaches are heavily relying on video content itself and require the whole video content for event detection. Due to the semantic gap between low-level features and high-level events, it is difficult to come up with a generic framework to achieve a high accuracy of event detection. In addition, the dynamic structures from different sports domains further complicate the analysis and impede the implementation of live event detection systems. In this paper, we present a novel approach for event detection from the live sports game using web-casting text and broadcast video. Web-casting text is a text broadcast source for sports game and can be live captured from the web. Incorporating web-casting text into sports video analysis significantly improves the event detection accuracy. Compared with previous approaches, the proposed approach is able to: (1) detect live event only based on the partial content captured from the web and TV; (2) extract detailed event semantics and detect exact event boundary, which are very difficult or impossible to be handled by previous approaches; and (3) create personalized summary related to certain event, player or team according to user's preference. We present the framework of our approach and details of text analysis, video analysis and text/video alignment. We conducted experiments on both live games and recorded games. The results are encouraging and comparable to the manually detected events. We also give scenarios to illustrate how to apply the proposed solution to professional and consumer services.

We report on our development of a real-time system to deliver sports video highlights of a live game to mobile videophones over existing GPRS networks. To facilitate real-time analysis, a circular buffer receives live video data from which simple audio/visual features are computed to detect for highlight-worthiness according to a priori decision scheme. A separate module runs algorithms to insert content into the highlight for mobile advertising. The system is now under trial over new 3G networks.

In this paper, we propose a novel method to detect baseball video scene for commercial advertisement insertion. The method uses the criteria based on better advertisement effect to generate a set of rules. From these rules, proper timing (starting time and ending time) and location are identified to insert the advertisement in the broadcast baseball video automatically. The proper timing is detected based on the consistent existence of simple background objects in the video for a period of time. The proper location is identified by less-informative-region detection in the video. The rules make sure that the critical information of the video is not blocked by the advertisement and the advertisement is always stable, clear and viewable on the stationary background of the video. The experimental results are encouraging.

In this paper, we propose a novel automatic approach for personalized music sports video generation. Two research challenges, semantic sports video content selection and automatic video composition, are addressed. For the first challenge, we propose to use multi-modal (audio, video and text) feature analysis and alignment to detect the semantic of events in sports video. For the second challenge, we propose video-centric and music-centric music video composition schemes to automatically generate personalized music sports video based on user's preference. The experimental results and user evaluations are promising and show that our system's generated music sports video is comparable to manually generated ones. The proposed approach greatly facilitates the automatic music sports video generation for both professionals and amateurs.

We propose a twice-iterated composite Fourier transform (TICFT) technique to detect the singing voice boundaries from acoustical polyphonic music signals. We show that the cumulative TICFT energy in the lower coefficients is capable of differentiating the harmonic structures of vocal and instrumental music in higher octaves. The musical signal is first segmented into frames based on quarter-notes. Then TICFT is used to measure the harmonic structure of each frame. Finally, the vocal and instrumental frames are classified by applying music domain knowledge. Experimental results show over 80% frame level accuracy can be achieved.

We describe soccer highlight generation from only the audio stream in the video. A novel audio feature is used to detect parts of the commentary corresponding to dominant and excited speech. It is computed by a twice-iterated composite Fourier transform (CFT) on short-time windows, wherein the magnitude spectrum of the first transform is input to a second transform. Dominant speech portions are found to be robustly characterized by increased density in the peak profile. We verify the robustness of CFT via large scale empirical testing and explain its working based on a pulse train postulate of dominant speech signals. Our audio-only approach results in a compute-efficient system deployable on current generation set-top-boxes and digital video recording devices.

We present a two dimensional timeline to replace the conventional 1D timeline used in video annotation systems. The extra dimension is augmented to represent the text annotations from user input and link them to the corresponding video time. A histogram of user annotations is developed over time. Not only does it highlight the history of prior user interaction with the video annotation system, it also gives a bird's eye view of how annotations are distributed along the video timeline. Users can then quickly find the annotation hotspots. Using this unique 2D timeline, we have developed a video annotation system prototype, Conversant Media (CM). Experimental trials conducted at two educational institutes on presentation skill training yield encouraging feedback, one of which concluded that CM can become a potential alternative to classroom visits for trainee teachers

While most current approaches for sports video analysis are based on broadcast video, in this paper, we present a novel approach for highlight detection and automatic replay generation for soccer videos taken by the main camera. This research is important as current soccer highlight detection and replay generation from a live game is a labor-intensive process. A robust multi-level, multi-model event detection framework is proposed to detect the event and event boundaries from the video taken by the main camera. This framework explores the possible analysis cues, using a mid-level representation to bridge the gap between low-level features and high-level events. The event detection results and mid-level representation are used to generate replays which are automatically inserted into the video. Experimental results are promising and found to be comparable with those generated by broadcast professionals.

Soccer video content-based analysis remains a challenging problem due to the lack of structure in a soccer game. To automate game and tactic analysis, we need to detect and track important activities such as ball possession in a soccer video that is highly correlated to the camera's field-view. In this paper, we present a system that tracks the camera's field-view in a soccer video in real-time. It utilizes a host of content-based visual cues that are obtained by independent threads running in parallel. The result is visualized as an active rectangular bounding box that approximates the camera's field of view superimposed on a virtual soccer field. Experimental results show that the system can reliably track the camera field-view as the game progresses.

Despite a lot of research efforts in sports video analysis, soccer video indexing remains a challenging task due to the lack of structure in a soccer game that could help in structure analysis. In particular, little work was done in detecting and tracking the ball whose trajectory could play a crucial role for detecting key events. We propose a novel framework for accurately detecting the ball for broadcast soccer video. Our framework combines both direct and indirect insights to identify the ball rather than conventional simple template matching methods. It has three key components. First we infer the ball size range from the player size. Next non-ball objects are removed to reduce the possible ball candidates. Last but not least, a Kalman filer-based procedure mines candidate trajectories in candidate feature images. Then, a procedure selects the reliable ball trajectories from them. The experimental results on two 1000-frame sequences confirm that the proposed framework is very effective and obtain a better result than existing methods.

We report our work in real-time detection of goal-mouth appearances in MPEG soccer video. Processing on sub-optimal quality images after MPEG-decoding, the system constrains the Hough Transform-based line-mark detection to only the dominant green regions typically seen in soccer video. The vertical goal-posts and horizontal goal-bar are then isolated by color-based region (pole)-growing. We demonstrate its application for quick video browsing and virtual content insertion. Extensive test over a large data set of about 15 hours of MPEG-1 soccer video @1.15Mbps, CIF-resolution, shows the robustness of our method.

In this paper, we describe a working system that detects and segments goal-mouth appearances of soccer video in real-time. Processing on sub-optimal quality images after MPEG-decoding, the system constrains the Hough Transform-based line-mark detection to only the dominant green regions. The vertical goal-posts and horizontal goal-bar are then isolated by color-based region (pole)-growing. We demonstrate its application for quick video browsing and virtual content insertion.

This paper first presents an improved trajectory-based algorithm for automatically detecting and tracking the ball in broadcast soccer video. Unlike the object-based algorithms, our algorithm does not evaluate whether a sole object is a ball. Instead, it evaluates whether a candidate trajectory, which is generated from the candidate feature image by a candidate verification procedure based on Kalman filter,, which is generated from the candidate feature image by a candidate verification procedure based on Kalman filter, is a ball trajectory. Secondly, a new approach for automatically analyzing broadcast soccer video is proposed, which is based on the ball trajectory. The algorithms in this approach not only improve play-break analysis and high-level semantic event detection, but also detect the basic actions and analyze team ball possession, which may not be analyzed based only on the low-level feature. Moreover, experimental results show that our ball detection and tracking algorithm can achieve above 96% accuracy for the video segments with the soccer field. Compared with the existing methods, a higher accuracy is achieved on goal detection and play-break segmentation. To the best of our knowledge, we present the first solution in detecting the basic actions such as touching and passing, and analyzing the team ball possession in broadcast soccer video.