Video re-localization aims to localize a sub-sequence, called target segment, in an untrimmed reference video that is similar to a given query video. In this work, we propose an attention-based model to accomplish this task in a weakly supervised setting. Namely, we derive our CNN-based model without using the annotated locations of the target segments in reference videos. Our model contains three modules. First, it employs a pre-trained C3D network for feature extraction. Second, we design an attention mechanism to extract multiscale temporal features, which are then used to estimate the similarity between the query video and a reference video. Third, a localization layer detects where the target segment is in the reference video by determining whether each frame in the reference video is consistent with the query video. The resultant CNN model is derived based on the proposed co-attention loss which discriminatively separates the target segment from the reference video. This loss maximizes the similarity between the query video and the target segment while minimizing the similarity between the target segment and the rest of the reference video. Our model can be modified to fully supervised re-localization. Our method is evaluated on a public dataset and achieves the state-of-the-art performance under both weakly supervised and fully supervised settings.

Phonotactic constraints can be employed to distinguish languages by representing a speech utterance as a multinomial distribution or phone events. In the present study, we propose a new learning mechanism based on subspace-based representation, which can extract concealed phonotactic structures from utterances, for language verification and dialect/accent identification. The framework mainly involves two successive parts. The first part involves subspace construction. Specifically, it decodes each utterance into a sequence of vectors filled with phone-posteriors and transforms the vector sequence into a linear orthogonal subspace based on low-rank matrix factorization or dynamic linear modeling. The second part involves subspace learning based on kernel machines, such as support vector machines and the newly developed subspace-based neural networks (SNNs). The input layer of SNNs is specifically designed for the sample represented by subspaces. The topology ensures that the same output can be derived from identical subspaces by modifying the conventional feed-forward pass to fit the mathematical definition of subspace similarity. Evaluated on the “General LR” test of NIST LRE 2007, the proposed method achieved up to 52%, 46%, 56%, and 27% relative reductions in equal error rates over the sequence-based PPR-LM, PPR-VSM, and PPR-IVEC methods and the lattice-based PPR-LM method, respectively. Furthermore, on the dialect/accent identification task of NIST LRE 2009, the SNN-based system performed better than the aforementioned four baseline methods.

Music videos are one of the most popular types of video streaming services, and instrument playing is among the most common scenes in such videos. In order to understand the instrument-playing scenes in the videos, it is important to know what instruments are played, when they are played, and where the playing actions occur in the scene. While audio-based recognition of instruments has been widely studied, the visual aspect of music instrument playing remains largely unaddressed in the literature. One of the main obstacles is the difficulty in collecting annotated data of the action locations for training-based methods. To address this issue, we propose a weakly supervised framework to find when and where the instruments are played in the videos. We propose using two auxiliary models: 1) a sound model and 2) an object model to provide supervision for training the instrument-playing action model. The sound model provides temporal supervisions, while the object model provides spatial supervisions. They together can simultaneously provide temporal and spatial supervisions. The resulting model only needs to analyze the visual part of a music video to deduce which, when, and where instruments are played. We found that the proposed method significantly improves localization accuracy. We evaluate the result of the proposed method temporally and spatially on a small dataset (a total of 5400 frames) that we manually annotated.

This paper presents a learning and scoring framework based on neural networks for speaker verification. The framework employs an autoencoder as its primary structure while three factors are jointly considered in the objective function for speaker discrimination. The first one, relating to the sample reconstruction error, makes the structure essentially a generative model, which benefits to learn most salient and useful properties of the data. Functioning in the middlemost hidden layer, the other two attempt to ensure that utterances spoken by the same speaker are mapped into similar identity codes in the speaker discriminative subspace, where the dispersion of all identity codes are maximized to some extent so as to avoid the effect of over-concentration. Finally, the decision score of each utterance pair is simply computed by cosine similarity of their identity codes. Dealing with utterances represented by i-vectors, the results of experiments conducted on the male portion of the core task in the NIST 2010 Speaker Recognition Evaluation (SRE) significantly demonstrate the merits of our approach over the conventional PLDA method.

We address offensive tactic recognition in broadcast basketball videos. As a crucial component towards basketball video content understanding, tactic recognition is quite challenging because it involves multiple independent players, each of which has respective spatial and temporal variations. Motivated by the observation that most intra-class variations are caused by non-key players, we present an approach that integrates key player detection into tactic recognition. To save the annotation cost, our approach can work on training data with only video-level tactic annotation, instead of key players labeling. Specifically, this task is formulated as an MIL (multiple instance learning) problem where a video is treated as a bag with its instances corresponding to subsets of the five players. We also propose a representation to encode the spatio-temporal interaction among multiple players. It turns out that our approach not only effectively recognizes the tactics but also precisely detects the key players.

Player segmentation in team sports videos is challenging but crucial to video semantic understanding, such as player interaction identification and tactic analysis. We leverage the appearance similarity among players of the same team, and cast this task as a co-segmentation problem. In this way, the extra knowledge shared across players significantly reduces unfavorable uncertainty in segmenting individual players. We are also aware that the performance of co-segmentation highly depends on the used features, and further propose a contrast-based approach to estimate the discriminant power of each feature in an unsupervised manner. It turns out that our approach can properly fuse features by assigning higher weights to discriminant ones, and result in remarkable performance gains. The promising results on segmenting basketball players manifest the effectiveness of our approach.

This study addresses the task of aligning lyrics with accompanied singing recordings. With a vowel-only representation of lyric syllables, our approach evaluates likelihood scores of vowel types with glottal pulse shapes and formant frequencies extracted from a small set of singing examples. The proposed vowel likelihood model is used in conjunction with a prior model of frame-wise syllable sequence in determining an optimal evolution of syllabic position. In lyrics alignment experiments, we optimized numerical parameters on two independent development sets and then tested the optimized system on two other datasets. New objective performance measures are introduced in the evaluation to provide further insight into the quality of alignment. Use of glottal pulse shapes and formant frequencies is shown by a controlled experiment to account for a 0.07 difference in average normalized alignment error. Another controlled experiment demonstrates that, with a difference of 0.03, F0-invariant glottal pulse shape gives a lower average normalized alignment error than does F0-invariant spectrum envelope, the latter being assumed by MFCC-based timbre models.

Unlike conventional omnidirectional antennas, switched beam antennas exploit antenna arrays and signal processing techniques to focus energy in a specific beam-width and orientation. Recent research has shown that access points of a WLAN can exploit such switched beam antennas to increase the overall network capacity. The achievable sum rate of a WLAN with switch beam antennas is however mainly determined by how each AP selects its beam, including the orientation and width, and how each client associates with a proper AP. The goal of this paper is to solve the Joint Beam configuration and Client association (JBC) problem such that the sum rate of all clients in the network can be maximized. We formulate the JBC problem as a mixed integer linear programming model, and propose a 2-approximation algorithm to solve it. Our proposed algorithm has two distinctive properties: 1) it can be realized as a distributed protocol that allows the APs to configure their beams without the help of a central coordinator, and 2) it can be generally applied both in specific scenarios, where exact client locations are known, and in uncertain scenarios, where only geographic client distribution is given. Finally, we adjust the sum-rate maximization algorithm to the throughput maximization algorithm, which further takes medium sharing among clients into account. The simulation results show that the proposed algorithm outperforms both WLANs using omnidirectional antennas and other heuristics using switched beam antennas.

Modeling the association between music and emotion has been considered important for music information retrieval and affective human computer interaction. This paper presents a novel generative model called acoustic emotion Gaussians (AEG) for computational modeling of emotion. Instead of assigning a music excerpt with a deterministic (hard) emotion label, AEG treats the affective content of music as a (soft) probability distribution in the valence-arousal space and parameterizes it with a Gaussian mixture model (GMM). In this way, the subjective nature of emotion perception is explicitly modeled. Specifically, AEG employs two GMMs to characterize the audio and emotion data. The fitting algorithm of the GMM parameters makes the model learning process transparent and interpretable. Based on AEG, a probabilistic graphical structure for predicting the emotion distribution from music audio data is also developed. A comprehensive performance study over two emotion-labeled datasets demonstrates that AEG offers new insights into the relationship between music and emotion (e.g., to assess the “affective diversity” of a corpus) and represents an effective means of emotion modeling. Readers can easily implement AEG via the publicly available codes. As the AEG model is generic, it holds the promise of analyzing any signal that carries affective or other highly subjective information.

This paper presents a novel approach to extraction of vocal melodies from accompanied singing recordings. Central to our approach is a model of vocal fundamental frequency (F0) likelihood that integrates acoustic-phonetic knowledge and real-world data. This model consists of a timbral fitness score and a loudness measure of each F0 candidate. Timbral fitness is measured for the partial amplitudes of an F0 candidate, with respect to a small set of vocal timbre examples. This F0-specific measurement of timbral fitness depends on an acoustic-phonetic F0 modification of each timbre example. In the loudness part of the likelihood model, sinusoids are detected, tracked, and pruned to give loudness values that minimize interference from the accompaniment. A final F0 estimate is determined by a prior model of F0 sequence in addition to the likelihood model. Melody extraction is completed by detecting voiced time positions according to the singing voice loudness variations given by the estimated F0 sequence. The numerical parameters involved in our approach were optimized on three development sets from different sources before the system was evaluated on ten test sets separate from these development sets. Controlled experiments show that use of the timbral fitness score accounts for a 13% difference in overall accuracy.

In this paper, we study the use of two kinds of kernel-based discriminative models, namely support vector machine (SVM) and deep neural network (DNN), for speaker verification. We treat the verification task as a binary classification problem, in which a pair of two utterances, each represented by an i-vector, is assumed to belong to either the “within-speaker” group or the “between-speaker” group. To solve the problem, we employ various binary operations to retain the basic relationship between any pair of i-vectors to form a single vector for training the discriminative models. This study also investigates the correlation of achievable performances with the number of training pairs and the various combinations of basic binary operations, using the SVM and DNN binary classifiers. The experiments are conducted on the male portion of the core task in the NIST 2005 Speaker Recognition Evaluation (SRE), and the results are competitive or even better, in terms of normalized decision cost function (minDCF) and equal error rate (EER), while compared to other non-probabilistic based models, such as the conventional speaker SVMs and the LDA-based cosine distance scoring.

Phonotactics, dealing with permissible phone patterns and their frequencies of occurrence in a specific language, is acknowledged to be related to spoken language recognition (SLR). With the assistance of phone recognizers, each speech utterance can be decoded into an ordered sequence of phone vectors filled with likelihood scores contributed by all possible phone models. In this paper, we propose a novel approach to dig the concealed phonotactic structure out of the phone-likelihood vectors through a kind of multivariate time series analysis: dynamic linear models (DLM). In these models, treating the generation of phone patterns in each utterance as a dynamic system, the relationship between adjacent vectors is linearly and time-invariantly modeled, and unobserved states are introduced to capture a temporal coherence intrinsic in the system. Each utterance expressed by the DLM is further transformed into a fixed-dimensional linear subspace so that well-developed distance measures between two subspaces can be applied to linear discriminant analysis (LDA) in a dissimilarity-based fashion. The results of SLR experiments on the OGI-TS corpus demonstrate that the proposed framework outperforms the well-known vector space modeling (VSM)-based methods and achieves comparable performance to our previous subspace-based method.

Visualizing audio signals during playback has long been a fundamental function of music players. However, most visual effects are generated by audio signal processing directly and render meaningless or incomprehensible displays to users. In this paper, we present an intelligent music player called the Playing with Tagging (PWT) music player. By integrating a real-time music tagger, the PWT player can display dynamic tag distributions via a set of tag bars that move in sync with the music. To synchronize the tag distributions, the music tagger must be able to online recognize the music tags. We utilize a Gaussian mixture model (GMM) as an auditory feature encoding reference and a mixture of tag-based aspect models (TBAMs) to predict the tag distribution for a short sliding chunk of the music played. To evaluate the real-time tagging function, we simulate tag prediction on short music chunks. The results of experiments on the MajorMiner dataset demonstrate the potential and effectiveness of the proposed music tagging method.

We aim to resolve the difficulties of action recognition arising from the large intra-class variations. These unfavorable variations make it infeasible to represent one action instance by other ones of the same action. We hence propose to extract both instance-specific and class-consistent features to facilitate action recognition. Specifically, the instance-specific features explore the self-similarities among frames of each video instance, while class-consistent features summarize within-class similarities. We introduce a generative formulation to combine the two diverse types of features. The experimental results demonstrate the effectiveness of our approach.

This paper presents a novel subspace-based approach for phonotactic language recognition. The whole framework is divided into two parts: the speech feature representation and the subspacebased learning algorithm. First, the phonetic information as well as the contextual relationship, possessed by spoken utterances, are more abundantly retrieved by likelihood computation and feature concatenation through the decoding processed by an automatic speech recognizer. It is assumed that the extracted phone frames reside in a lower dimensional eigen-subspace, in which the structure of data can be approximately captured. Each utterance is further represented by a fixed-dimensional linear subspace. Second, to measure the similarity between two utterances, suitable non-Euclidean metrics are explored and applied to non-linear discriminant analysis in a kernel fashion, followed by a back-end classifier, such as the k-nearest neighbor (K-NN) classifier. The results of experiments on the OGI-TS database demonstrate that the proposed framework outperforms the well-known vector space modeling based method with relative reductions of 38.90% and 27.13% on the 1-to-50-second and 3-second data sets respectively in equal error rate (EER).

One of the most exciting but challenging endeavors in music research is to develop a computational model that comprehends the affective content of music signals and organizes a music collection according to emotion. In this paper, we propose a novel acoustic emotion Gaussians (AEG) model that defines a proper generative process of emotion perception in music. As a generative model, AEG permits easy and straightforward interpretations of the model learning processes. To bridge the acoustic feature space and music emotion space, a set of latent feature classes, which are learned from data, is introduced to perform the end-to-end semantic mappings between the two spaces. Based on the space of latent feature classes, the AEG model is applicable to both automatic music emotion annotation and emotion-based music retrieval. To gain insights into the AEG model, we also provide illustrations of the model learning process. A comprehensive performance study is conducted to demonstrate the superior accuracy of AEG over its predecessors, using two emotion annotated music corpora MER60 and MTurk. Our results show that the AEG model outperforms the state-of-the-art methods in automatic music emotion annotation. Moreover, for the first time a quantitative evaluation of emotion-based music retrieval is reported.

We propose a model-based approach for human body pose recognition from a single-view depth camera. The proposed algorithm applies an articulated cylinder model to detect human pose and track them based on a particle filter without numerous training data or heuristic detectors. To reduce high degrees of freedom, we adopt a hierarchical method that detects torso and limbs successively. Moreover, we take the advantage of a particle filter to track complex human motion and the results show that the proposed system is robust in human motion tracking. The qualitative evaluation shows that our method can deal with self-occlusion problem and ambiguous human motion effectively, and the quantitative evaluation shows that the average tracking error is 0.06 meters with a standard deviation of 0.04 meters. The proposed method tracks human poses successfully at the speed of 18 frames per second on a laptop with Intel Core i3-2100 CPU and without graphic processing unit.

The object occlusion is one serious issue in object tracking, especially when objects merge or split. The tracker may fail if it has no adaptability to such variations. In this paper, we present an improved solution of the appearance-based tracking for occlusion handling. The proposed method first recognizes the motion situations (merging and splitting of the moving objects), and then applies different template finding approach on each motion to get an association between detected blobs and targets. By this way, the accuracy of the data association can be improved when object occlusion occurs. Experiments were conducted to compare our results with those by other popular tracking algorithms. The proposed method is found robust for short time complete occlusion and partial occlusion.

This paper presents a novel content-based music retrieval system that accepts a query containing multiple tags with multiple levels of preference (denoted as an MTML query) to retrieve music from an untagged music database. We select a limited number of popular music tags to form the tag space and design an interface for users to input queries by operating the scroll bars. To effect MTML content-based music retrieval, we introduce a tag-based music aspect model that jointly models the auditory features and tag-based text features of a song. Two indexing methods and their corresponding matching methods, namely pseudo song-based matching and tag co-occurrence pattern-based matching, are incorporated into the pre-learned tag-based music aspect model. Finally, we evaluate the proposed system on the Major Miner dataset. The results demonstrate the potential of using MTML queries to retrieve music from an untagged music database.

The comfort of rides has been identified as one of the top criteria that affects passengers' satisfaction with public transportation systems. Conventional comfort measurement approaches rely on professional measure tools or interviews from passengers, which are costly, time-consuming, and not yet feasible. The concept of Internet of Things (IoT) is a new solution to answer this problem. The idea of IoT is to interconnect state-of-the-art digital products in physical world to provide more powerful applications. Vehicles equipped with GPS devices and wireless access technologies are parts of the IoT elements in traffic networks. We use the GPS data to measure the comfort level of vehicle rides, and provide a detailed comfort statistics as a value added service. Using real data collected from one of the Taipei taxi service providers, we show that over 95% taxi trajectories are viewed as comfortable. In addition, rides without passengers get higher comfort scores than with passengers. We also give the rankings of all taxi drivers according to a number of criteria, such as the comfort score and the number of loads. With the ranking results, we can track back to the trajectories and infer drives' driving behaviors, road conditions, and traffic conditions. We believe that the proposed solution has the potential to provide a representative comfort measurement service for taxi services and additional value-added services for public transportation systems.

This paper presents a novel content-based query-by-tag music search system for an untagged music database. We design a new tag query interface that allows users to input multiple tags with multiple levels of preference (denoted as an MTML query) by colorizing desired tags in a web-based tag cloud interface. When a user clicks and holds the left mouse button (or presses and holds his/her finger on a touch screen) on a desired tag, the color of the tag will change cyclically according to a color map (from dark blue to bright red), which represents the level of preference (from 0 to 1). In this way, the user can easily organize and check the query of multiple tags with multiple levels of preference through the colored tags. To effect the MTML content-based music retrieval, we introduce a probabilistic fusion model (denoted as GMFM), which consists of two mixture models, namely a Gaussian mixture model and a multinomial mixture model. GMFM can jointly model the auditory features and tag labels of a song. Two indexing methods and their corresponding matching methods, namely pseudo song-based matching and tag affinity-based matching, are incorporated into the pre-learned GMFM. We evaluate the proposed system on the MajorMiner and CAL-500 datasets. The experimental results demonstrate the effectiveness of GMFM and the potential of using MTML queries to search music from an untagged music database.

A new approach to automatic music transcription for music with two timbres from a monaural sound source is proposed in this paper. The system is mainly composed of two parts, a fundamental frequency detector and a timbre discriminator. In the fundamental frequency detector, the short time Fourier transform (STFT) and a peak detection algorithm are used. By combining the relative magnitude of the fundamental frequency and its harmonics, a characteristic timbre vector is computed. The timbre discrimination is done by the classification of the timbre vectors with the support vector machine (SVM). Particularly, by designing the training and classification procedure in SVM, the mixed-timbre signal can also be classified in this system. Using a small database of polyphonic music of two timbres as the testing input, a 73% hit rate is achieved by this system.

In surveillance applications, keyframes are usually used to summarize important video content. However, traditional keyframe extracting methods select video frames as keyframes directly from the input video, and the information in these selected video frames is scattered and the users cannot perceive how the events happened in the original video easily. In this paper, we propose a new type of keyframe (called an “augmented 3-D keyframe”) that is a more meaningful and compact keyframe, augmented with representative objects, important contents (human faces, license plates, and so on), motion information (including trajectories and simple movement situations), and some marks of the moving objects extracted from a surveillance video clip captured by a static camera. The innovation is that we include simple 3-D geometric information to generate this augmented 3-D keyframe without reconstructing a complete 3-D scene. Testing results of the proposed augmented 3-D keyframe are presented and compared with results obtained by the traditional keyframe approach.

In this paper, we conduct the window design for single-input single-output (SISO) and multiple-input multiple- output (MIMO) OFDM systems. The key value of this paper is to optimize the receiver SINR when carrier frequency offset (CFO) and phase noise both affecting an OFDM system. To our best knowledge, this is the first research to design window coefficients through leveraging the CFO and phase noise statistics for SISO and MIMO OFDM systems over frequency-selective channels. CFO and phase noise will destroy the orthogonality of OFDM system and lead to inter-carrier interference (ICI). Simulation results show that our proposed method can improve the system performance.

Currently, how to equip machines with the ability for properly recognizing users' felt-emotion during multimedia presentation is a growing issue. In this study we focused on the approach for recognizing music-induced emotional responses from brain activity. A comparative study was conducted to testify the feasibility of using hierarchical binary classifiers to improve the classification performance as compared with nonhierarchical schemes. According to our classification results, we not only found that using one-against-one scheme of hierarchical binary classifier results in an improvement to performance, but also established an alternative solution for emotion recognition by proposed model-based scheme depending on 2D emotion model.

In this technical demonstration, we showcase an interactive content presentation (ICP) system that integrates media-expressed-emotion-based composition, user-perceived preference feedback, and interactive digital art creation. ICP harmonizes the browsing of multimedia contents by presenting them in the form of music videos (photos, blog articles with accompanied music) based on their expressed emotion similarity. ICP facilitates content browsing by automatically and dynamically selecting the media to be played next in real time, responding to user's preference feedback measured from physiological signals. In addition, ICP enhances the enjoyments of content browsing by incorporating interactive digital art creation. ICP achieves these goals by properly integrating recent researches on media-expressed emotion classification,cross-media composition, and physiological signal processing.

An approach to recognize the emotion responses during multimedia presentation using the electroencephalogram (EEG) signals is proposed. The association between EEG signals and music-induced emotion responses was investigated in three factors, including: 1) the types of features, 2) the temporal resolutions of features, and 3) the components of EEG. The results showed that the spectrum power asymmetry index of EEG signal was a sensitive marker to reflect the brain activation related to emotion responses, especially for the low frequency bands of delta, theta and alpha components. Besides, the maximum classification accuracy was obtained around 92.73% by using support vector machine (SVM) based on 60 features derived from all EEG components with the feature temporal resolution of one second. As such, it will be able to provide key clues to develop EEG-inspired multimedia applications, in which multimedia contents could be offered interactively according to the userspsila immediate feedback.

The virtual conduction system (VCS) allows a user to conduct a photo-realistic pseudo orchestra. The VCS includes four modules: (1) gesture recognition module, (2) audio rendering module, (3) video rendering module, and (4) multi-resolution display module. With gesture recognition and tempo adjustment, the user not only can change the playback rate of an audio and video recording, but also can control the volume from different portion of an orchestra in real time. With the video rendering module and multi-resolution display module, the VCS provides a new visual experience with an interactive multi-resolution wall-size display.

As the operation frequency reaches gigahertz in very deep-submicron designs, the effect of on-chip inductance on circuit performance can no longer be neglected. Therefore, it is desired to extract interconnect impedance and inductance accurately. Most previous works on impedance and inductance extraction are based on rectangular discretization which has been shown effective for the classical Manhattan based IC interconnect structures. As technology advances, however, more general IC interconnect structures, such as the X-based interconnect structure, have been in production. Those general interconnect structures allow wires to be routed with non-Manhattan shapes. For the non-Manhattan interconnect structures, rectangular discretization is obviously not sufficient. In this paper, we propose to use the surface integral formulation with triangular discretization to extract impedance and inductance for the general IC interconnect structures. Comparative studies with the well-known FastImp and IE3D show that our approach is flexible and effective.

Octave detection has been an open issue in automatic transcription of polyphonic music over the years. In the literature, pitch detectors for polyphonic music usually fail to detect octaves for lack of information about the timbre of each instrument that appears in the music. To reveal the possibility that this difficulty in automatic music transcription can be overcome, here a music transcription system that is capable of octave detection is proposed. Preceded by a constant-Q time-frequency analysis front end implemented with a fast algorithm for nonorthonormal discrete wavelet transform [1], a classifier using support vector machine technique [2] serves as a novel octave detector in this system.

This paper presents a new method for musical chord recognition based on a model of human perception. We classify the chords directly from the sound without the information of timbres and notes. A wavelet-based transform as well as a self-organized map (SOM) neural network is adopted to imitate human ears and cerebra, respectively. The resultant system can classify chords very well even in a noisy environment.

A new approach for analysis of RF coils in magnetic resonance (MR) experiments is reported. Instead of assuming current distribution in conventional quasi-static algorithm, this approach transforms the coil geometry into an equivalent circuit for complex current calculation. Self and mutual inductance are taken into consideration. Frequency responses of RF coils and transverse magnetic field (B1) maps can be simulated. This approach is especially efficient for phased array coil design for its small matrix size when implemented on computers. Experiments on both single surface coil and phased array coils are consistent with simulation results. Index Terms-Magnetic resonance, method of moment, phased array coil, RF coil.