In this paper, a multimodal classification model is constructed to predict the pineapple quality. We compile a pineapple dataset consisting of 500 pineapples with two modalities: one is tapping a pineapple to record the sounds, and the other is taking pictures by cameras. Three classification models, including audio model, visual model, and audio-visual model, are built based on the deep learning architectures for the corresponding feature representations. The experimental evaluation demonstrates that the audio-visual model, which combines both audio and visual representations, can yield the best accuracy than the other two unimodal models.

In the task of approximate nearest neighbor search, the conventional lookup-table indexing calculates the distances (or similarities) between the query and codewords, and then re-ranks the data points associated with the nearest (or the most similar) codewords. To address the codeword quantization loss problem exhibited in the conventional method, the probability-based indexing leverages the data distribution among codewords learned by neural networks to locate the nearest neighbor [8]. In this paper, we present a multitasking model to improve the probability-based indexing method. The model is formulated by two objectives of NN distribution probabilities and data retrieval quantity. The NN distribution probabilities are an estimation to determine the possible codewords where the nearest neighbor may be associated. The candidate retrieval quantity specifies the prediction for the least number of codewords to be re-ranked for capturing the nearest neighbor. The proposed model is then trained by minimizing triplet loss, probability loss, and quantity loss. By learning these tasks in parallel, we find the predictions for both data distribution probability and data retrieval quantity are more accurate, so that search accuracy and computation efficiency can be improved together. We experiment on two billion-scale benchmark datasets to evaluate the proposed method and compare with several approximate nearest neighbor search methods, and the results demonstrate the outperformance of the proposed method.

In this study, we present a novel clustering-based collaborative filtering (CF) method for recommender systems. Clustering-based CF methods can effectively deal with data sparsity and scalability problems. However, most of them are applied to a single representation space, which might not characterize complex user–item interactions well. We argue that the user–item interactions should be observed from multiple views and characterized in an adaptive way. To address this issue, we leveraged the global and local properties to construct multiple representation spaces by learning various training datasets and loss functions. An attention network was built to generate a blended representation according to the relative importance of the representation spaces for each user–item pair, providing a flexible way to characterize diverse user–item interactions. Substantial experiments were evaluated on four popular benchmark datasets. The results show that the proposed method is superior to several CF methods where only one representation space is considered.

In this study, we present a novel ranking model based on learning neighborhood relationships embedded in the index space. Given a query point, conventional approximate nearest neighbor search calculates the distances to the cluster centroids, before ranking the clusters from near to far based on the distances. The data indexed in the top-ranked clusters are retrieved and treated as the nearest neighbor candidates for the query. However, the loss of quantization between the data and cluster centroids will inevitably harm the search accuracy. To address this problem, the proposed model ranks clusters based on their nearest neighbor probabilities rather than the query-centroid distances. The nearest neighbor probabilities are estimated by employing neural networks to characterize the neighborhood relationships, i.e., the density function of nearest neighbors with respect to the query. The proposed probability-based ranking can replace the conventional distance-based ranking for finding candidate clusters, and the predicted probability can be used to determine the data quantity to be retrieved from the candidate cluster. Our experimental results demonstrated that the proposed ranking model could boost the search performance effectively in billion-scale datasets.

Cross-modal hashing, which searches nearest neighbors across different modalities in the Hamming space, has become a popular technique to overcome the storage and computation barrier in multimedia retrieval recently. Although dozens of cross-modal hashing algorithms are proposed to yield compact binary code representation, applying exhaustive search in a large-scale dataset is impractical for the real-time purpose, and the Hamming distance computation suffers inaccurate results. In this paper, we propose a novel index scheme over binary hash codes in cross-modal retrieval. The proposed indexing scheme exploits a few binary bits of the hash code as the index code. Based on the index code representation, we construct an inverted index structure to accelerate the retrieval efficiency and train a neural network to improve the indexing accuracy. Experiments are performed on two benchmark datasets for retrieval across image and text modalities, where hash codes are generated by three cross-modal hashing methods. Results show the proposed method effectively boosts the performance over the benchmark datasets and hash methods.

We propose a novel method to merge convolutional neural-nets for the inference stage. Given two well-trained networks that may have different architectures that handle different tasks, our method aligns the layers of the original networks and merges them into a unified model by sharing the representative codes of weights. The shared weights are further re-trained to fine-tune the performance of the merged model. The proposed method effectively produces a compact model that may run original tasks simultaneously on resource-limited devices. As it preserves the general architectures and leverages the co-used weights of well-trained networks, a substantial training overhead can be reduced to shorten the system development time. Experimental results demonstrate a satisfactory performance and validate the effectiveness of the method.

In this paper, we propose a fast search method for asymmetric distance computation in large-scale binary codes. Asymmetric distances take advantage of less information loss at the query side. However, calculating asymmetric distances with the linear scan approach is prohibitive in a large-scale dataset. We present a novel algorithm called multi-index voting, which integrates the multi-index hashing technique with a voting mechanism, to select appropriate candidates and calculate their asymmetric distances. Substantial experimental evaluations are given to demonstrate that, guided by the voting mechanism, the proposed method can yield an approximate accuracy to the linear scan approach while accelerating the run time with a significant speedup. For example, one result shows that in a dataset of one billion 256-bit binary codes, examining only 0.5% of the dataset can reach 95~99% close accuracy to the linear scan approach but can accelerate over 73~128 times.

In this article, we propose a method of approximate asymmetric nearest-neighbor search for binary embedding codes. The asymmetric distance takes advantage of less information loss at the query side. However, calculating asymmetric distances through exhaustive search is prohibitive in a large-scale dataset. We present a novel method, called multi-index voting, that integrates the multi-index hashing technique with a voting mechanism to select appropriate candidates and calculate their asymmetric distances. We show that the candidate selection scheme can be formulated as the tail of the binomial distribution function. In addition, a binary feature selection method based on minimal quantization error is proposed to address the memory insufficiency issue and improve the search accuracy. Substantial experimental evaluations were made to demonstrate that the proposed method can yield an approximate accuracy to the exhaustive search method while significantly accelerating the runtime. For example, one result shows that in a dataset of one billion 256-bit binary codes, examining only 0.5% of the dataset, can reach 95--99% close accuracy to the exhaustive search method and accelerate the search by 73--128 times. It also demonstrates an excellent tradeoff between the search accuracy and time efficiency compared to the state-of-the-art nearest-neighbor search methods. Moreover, the proposed feature selection method shows its effectiveness and improves the accuracy up to 8.35% compared with other feature selection methods.

In this paper, we study the subsequence matching problem of near-duplicate video detection. In particular, we address the application of monitoring a continuous stream with a large video dataset. To achieve real-time response and high accuracy, we propose a novel framework containing two characteristics . First, the subsequence matching is transformed to a 2-D Hough space projection of pairwise frame similarities between two subsequences. We present an approximate Hough transform that replaces the 2-D Hough space with a 1-D Hough space. The near-duplicate subsequence detection can be deemed to be the voting and searching in the 1-D Hough space with a lower time complexity. Second, a coarse-to-fine matching strategy is incorporated in the proposed framework. The coarse level matching selects the candidate videos based on the time-decay hit frequency between the query stream and dataset videos. The fine level matching applies the approximate Hough transform to detect the near-duplicate subsequences coexisting within the query stream and candidate videos. Several state-of-the-art methods are implemented for comparison. Experimental results show our framework outperforms in terms of accuracy and efficiency.

In this article, we study the efficiency problem of video stream near-duplicate monitoring in a large-scale repository. Existing stream monitoring methods are mainly designed for a short video to scan over a query stream; they have difficulty being scalable for a large number of long videos. We present a simple but effective algorithm called incremental similarity update to address the problem. That is, a similarity upper bound between two videos can be calculated incrementally by leveraging the prior knowledge of the previous calculation. The similarity upper bound takes a lightweight computation to filter out unnecessary time-consuming computation for the actual similarity between two videos, making the search process more efficient. We integrate the algorithm with inverted indexing to obtain a candidate list from the repository for the given query stream. Meanwhile, the algorithm is applied to scan each candidate for locating exact near-duplicate subsequences. We implement several state-of-the-art methods for comparison in terms of accuracy, execution time, and memory consumption. Experimental results demonstrate the proposed algorithm yields comparable accuracy, compact memory size, and more efficient execution time.

Sports video annotation, an active research area in the field of multimedia content understanding, is an essential process in applications, such as summarization, highlight extraction, event detection, and retrieval. This paper considers the issue in relation to the annotation of baseball videos. Conventional baseball video annotation frameworks are based primarily on video content analysis, such as scoreboard recognition and machine learning techniques, which require a substantial amount of human input to collect and organize training data. The performance of such frameworks might become unstable if they encounter audiovisual patterns not included in the training data. To address the issue, we propose a novel framework for baseball video annotation that aligns high-level webcast text with low-level video content. Several cues, which are derived from the video content and webcast text, are utilized for alignment by leveraging hierarchical agglomerative clustering and genetic algorithm optimization. In addition, we develop an unsupervised method to learn the pitch segment properties of baseball videos by Markov random walk, and thereby reduce the need for human intervention substantially. Our experiments demonstrate that the proposed framework yields a robust result against a variety of video content and enhances the automaticity in baseball video annotation.

In this paper, we present a novel near-duplicate video detection approach based on video query reformulation to expedite the video subsequence search process. The proposed video query reformulation method addresses two key issues: 1) how to efficiently skip unnecessary subsequence matches and 2) how to effectively increase the skip probability. First, we present an incremental update mechanism that rapidly estimates the similarity between two video subsequences to skip unnecessary matches. Second, we formulate an optimization problem of subsequence partition to increase the skip probability; a trust-region-based gradient descent algorithm is applied to solve the optimization problem. Extensive experiments cover various feature representations, subsequence granularities, and baseline methods; the results demonstrate that the proposed query reformulation method is robust and efficient to deal with a variety of near-duplicates in a large-scale video dataset.

A novel framework for background music identification is proposed in this paper. Given a piece of audio signals that mixes background music with speech/noise, we identify the music part with source music data. Conventional methods that take the whole audio signals for identification are inappropriate in terms of efficiency and accuracy. In our framework, the audio content is filtered through speech center cancellation and noise removal to extract clear music segments. To identify these music segments, we use a compact feature representation and efficient similarity measurement based on the min-hash theory. The results of experiments on the RWC music database show a promising direction.

Pitching is the starting point of an event in baseball games. Hence, locating pitching shots is a critical step in content analysis of a baseball game video. However, pitching frames vary with innings and games. Existing methods that require a great deal of effort to construct empirical rules or label training data do not capture the characteristics of various pitching frames very well. In this paper, we present an unsupervised method for pitching frame detection by using Markov random walk. A video stream is first divided into content-homogeneous shots, and these shots are merged into states through hierarchical agglomerative clustering. Then, the state with the highest visit probability according to the Markov random walk theory is deemed the set of pitching frames. Finally, a model trained on the pitching frames in the pitching state is further used to detect the remaining potential pitching frames in other states. Our experiments demonstrate that the proposed method yields satisfactory results in a variety of MLB games.

This paper presents a novel time-series linear search (TLS) method for detecting video copies. The method utilizes a sliding window to locate window sequences that are near-duplicates of a given query sequence. We address two issues of the conventional TLS method in order to strengthen its video copy detection capability. First, to accelerate the TLS process, we use a sequence-level signature as a compact representation of a video sequence based on the min-hash theory, and develop an efficient heap manipulation technique for fast generation of each window sequence's signature. Second, to improve the robustness of the TLS method, we use two techniques, namely, window length estimation and threshold transform, to resolve the containment relation problem caused by various types of video transformation and editing, such as frame cropping and speed change. The results of experiments on the MUSCLE-VCD-2007 dataset demonstrate that the proposed method is efficient and robust against different types of video transformation and editing.

The increase in the number of video copies, both legal and illegal, has become a major problem in the multimedia and Internet era. In this article, we propose a novel method for detecting various video copies in a video sequence. To achieve fast and robust detection, the method fully integrates several components, namely the min-hashing signature to compactly represent a video sequence, a spatio-temporal matching scheme to accurately evaluate video similarity compiled from the spatial and temporal aspects, and some speedup techniques to expedite both min-hashing indexing and spatio-temporal matching. The results of experiments demonstrate that, compared to several baseline methods with different feature descriptors and matching schemes, the proposed method which combines both global and local feature descriptors yields the best performance when encountering a variety of video transformations. The method is very fast, requiring approximately 0.06 seconds to search for copies of a thirty-second video clip in a six-hour video sequence.

In this article, we present a content-based video search service for mobile device users. After taking a photograph, the user can upload it to our server, which will then return videos containing key frames that are similar to the uploaded picture. The proposed service demonstrates that the search application is feasible.

An effective video copy detection framework should be robust against spatial and temporal variations, e.g., changes in brightness and speed. To this end, a content-based approach for video copy detection is proposed. We define the problem as a partial matching problem in a probabilistic model and transform it into a shortest-path problem in a matching graph. To reduce the computation costs of the proposed framework, we introduce some methods that rapidly select key frames and candidate segments from a large amount of video data. The experiment results show that the proposed approach not only handles spatial and temporal variations well, but it also reduces the computation costs substantially.

In this paper, a novel method is presented to detect video copies for a given video query. These copies and the query have identical or near-duplicate content, which might differ in their spatiotemporal structures slightly. To address both the efficient and effective issues, we conduct the bag-of words model for video feature representation, and apply a coarse-to-fine matching scheme to analyze the video spatiotemporal structure. The proposed method can deal with various kinds of video transformations, such as cropping, zooming, speed change, and subsequence insertion/deletion, which are not well addressed in existing methods. Besides, two indexing methods are employed to speed up the matching process. Experimental results show that the proposed method can behave in an efficient and effective manner.

In online resources, such as news and weblogs, authors often extract articles, embed content, and comment on existing articles related to a popular event. Therefore, it is useful if authors can check whether two or more articles share common parts for further analysis, such as cocitation analysis and search result improvement. If articles do have parts in common, we say the content of such articles is event-relevant. Conventional text classification methods classify a complete document into categories, but they cannot represent the semantics precisely or extract meaningful event-relevant content. To resolve these problems, we propose a near-duplicate detection approach for finding event-relevant content in Web documents. The efficiency of the approach and the proposed duplicate set generation algorithms make it suitable for identifying event-relevant content. The experiment results demonstrate the potential of the proposed approach for use in weblogs.

Rapid growth of digital photography in recent years spurred the need of photo management tools. In this study, we propose an automatic organization framework for photo collections based on image content, so that a novel browsing experience is provided for users. For each photograph, human faces, together with corresponding clothes and nearby regions are located. We extract color histograms of these regions as the image content feature. Then a similarity matrix of a photo collection is generated according to temporal and content features of those photographs. We perform hierarchical clustering based on this matrix, and extract duplicate subjects of a cluster by introducing the contrast context histogram (CCH) technique. The experimental results show that the developed framework provides a promising result for photo management

The study shows a human motion retrieval system, HUMOR. The system provides high-level multi-modal query interfaces so that users can choose a convenient modality, including text, stickman, images, and motion clip, to specify their queries. Besides, we propose a novel search method based on a well-structured index mechanism to retrieve human motion efficiently. The usability of the multi-modal user interface and the effectiveness of the proposed system has been demonstrated.

Abstract Human spontaneous motions such as walking and runing are seldom described in detail. However, some well designed motions such as conditioning and rehabilitation exercises are usually described in detail and edited into exercise manuals for training purpose. This paper presents a novel framework for synthesizing new motions based on given motion manuals and corresponding motion capture examples. First, using text analysis method, a set of basic motion texts is extracted from exercise manuals and converted into text normal form. Then we introduce an annotation method to build the mapping table between basic motion texts and their corresponding motion clips in the given examples. To facilitate the search for proper motion clips, a multi‐dimensional index structure based on posture parameters is proposed. Then a new motion with given textual description can be synthesized by converting the description sentences into a sequence of text normal forms and then concatenating the corresponding motion clips to form the desired motion. Moreover, to realize the smooth concatenation, we show that it can be achieved by finding an appropriate path in the proposed multi‐dimensional index space. Several experimental examples are given to demonstrate the proposed method is effective in synthesizing desired motions according to given descriptions. Copyright © 2004 John Wiley & Sons, Ltd.

A fuzzy logic framework is proposed to alleviate two problems in traditional CBIR systems, including the semantic gap and the perception subjectivity. The proposed framework consists of two major parts, including (1) model construction and (2) query comparison. In the model construction part, fuzzy linguistic terms with associated fuzzy membership functions are automatically generated through an unsupervised fuzzy clustering algorithm. The linguistic terms provide a nature way of expressing user's concepts, and the membership functions characterize the mapping between image features and human visual concepts. We also define the syntax and semantics rules of a query description language to unify the query expression of textual descriptions, visual examples, and relevance feedbacks. In the query comparison part, a similarity function is inferred based on user's feedbacks to measure the similarity between the query and each image in the database. The user's preference is also captured and retained in his/her own profile to achieve personalization. Our work provides a unified and comprehensive framework for incorporation a fuzzy approach into CBIR systems. To verify our CBIR framework, we select Tamura features to describe and retrieve texture images. Experimental results show that the proposed framework is indeed effective to alleviate the semantic gap and the perception subjectivity problems.

In this study, we propose a simple and effective approach to synthesize new motions from a given sequence of continuous motion capture data. First an index function, based on posture features of each motion frame, is introduced to segment the given motion capture data into indexed motion clips. Then based on the fact that motion coherence implies index coherence, a new motion with start frame f/sub start/ and end frame fend can be synthesized by finding a smooth path connecting f/sub start/ to f/sub end/ in the multidimensional index space. Moreover, an algorithm for finding smooth paths is presented and a relevance feedback mechanism is provided to refine the results. The merit of the proposed framework is that it can generate fast prototyping of desired motions with only a small amount of preprocessing time. Experimental results are given to shown the effectiveness of the proposed framework.

In this study, we propose a fuzzy logic CBIR system for textures, named LinStar Texture (i.e., Linguistic Star for Textures). The proposed system consists of two major phases, including database creation and query comparison. In the database creation phase, six Tamura features are extracted to describe each texture image in the database. A term set on each Tamura feature is generated through a fuzzy clustering algorithm so that degrees of appearance for the feature can be interpreted as five linguistic terms. In the query comparison phase, a user can pose textual descriptions or visual examples to find the desired textures. Furthermore, the query can be expressed as a logic composition of linguistic terms or Tamura feature values. The final similarity is then computed by aggregating each individual similarity through min-max composition rules. Experimental results reveal the proposed system is indeed effective. The retrieved images are perceptually satisfactory. The retrieval time is very fast.