Cloud data centers (CDCs) are becoming a cost-effective method for processing and storage of multimedia data including images, video, and audio. Since CDCs are physically located in different jurisdictions, and are managed by external parties, data security is a growing concern. Data encryption at CDCs is commonly practiced to improve data security. However, to process the data at CDCs, data must often be decrypted, which raises issues in security. Thus, there is a growing demand for data processing techniques in encrypted domain in such an outsourced environment. In this article, we analyze encrypted domain speech content processing techniques for noise reduction. Noise contaminates speech during transmission or during the acquisition process by recording. As a result, the quality of the speech content is degraded. We apply Shamir’s secret sharing as the cryptosystem to encrypt speech data before uploading it to a CDC. We then propose finite impulse response digital filters to reduce white and wind noise in the speech in the encrypted domain. We prove that our proposed schemes meet the security requirements of efficiency, accuracy, and checkability for both semi-honest and malicious adversarial models. Experimental results show that our proposed filtering techniques for speech noise reduction in the encrypted domain produce similar results when compared to plaintext domain processing.

Sung language recognition relies on both effective feature extraction and acoustic modeling. In this paper, we study rhythm based music segmentation with the frame size being the duration of the smallest note in the music, as opposed to fixed length segmentation in spoken language recognition. It is found that acoustic features extracted from the rhythm based segmentation scheme outperform those from fixed length segmentation. We also study the effectiveness of a musically motivated acoustic feature. Octave scale cepstral coefficients (OSCCs) by comparing with the other acoustic features: Log frequency cepstral coefficients, Linear prediction coefficients (LPC) and LPC-derived cepstral coefficients. Finally, we examine the modeling capabilities of Gaussian mixture models and support vector machines in sung language recognition experiments. Experiments conducted on a corpus of 400 popular songs sung in English, Chinese, German, and Indonesian, showed that the OSCC feature outperforms other features. A sung language recognition accuracy of 64.9% was achieved when Gaussian mixture models were trained on shifted-delta-OSCC acoustic features, extracted via rhythm based music segmentation.

In this paper, we report the influence that classification accuracies have in speech analysis from a clinical dataset by adding acoustic low-level descriptors (LLD) belonging to prosodic (i.e. pitch, formants, energy, jitter, shimmer) and spectral features (i.e. spectral flux, centroid, entropy and roll-off) along with their delta (Δ) and delta-delta (Δ-Δ) coefficients to two baseline features of Mel frequency cepstral coefficients and Teager energy critical-band based autocorrelation envelope. Extracted acoustic low-level descriptors (LLD) that display an increase in accuracy after being added to these baseline features were finally modeled together using Gaussian mixture models and tested. A clinical data set of speech from 139 adolescents, including 68 (49 girls and 19 boys) diagnosed as clinically depressed, was used in the classification experiments. For male subjects, the combination of (TEO-CB-Auto-Env + Δ + Δ-Δ) + F0 + (LogE + Δ + Δ-Δ) + (Shimmer + Δ) + Spectral Flux + Spectral Roll-off gave the highest classification rate of 77.82% while for the female subjects, using TEO-CB-Auto-Env gave an accuracy of 74.74%.

The expectation maximization (EM) algorithm is widely used in the Gaussian mixture model (GMM) as the state-of-art statistical modeling technique. Like the classical EM method, the proposed EM-Information Theoretic algorithm (EM-IT) adapts means, covariances and weights, however this process is not conducted directly on feature vectors but on a smaller set of centroids derived by the information theoretic procedure, which simultaneously minimizes the divergence between the Parzen estimates of the feature vector's distribution within a given Gaussian component and the centroid's distribution within the same Gaussian component. The EM-IT algorithm was applied to the speaker verification problem using NIST 2004 speech corpus and the MFCC with dynamic features. The results showed an improvement of the equal error rate (ERR) by 1.5% over the classical EM approach. The EM-IT also showed higher convergence rates compare to the EM method.

We propose a signal-based approach instead of the commonly used model-based approach, to automatically align vocal music with text lyrics at the word level. In this approach, we use a text-to-speech system to synthesize the singing voice according to the lyrics. In this way, aligning the music signal with the corresponding text lyrics becomes the alignment of two audio signals. This study uses the results of music information modeling and singing voice synthesis. In music information modeling, we study different music representation strategies for music segmentation, music region indexing and region content descriptions; in singing voice synthesis, we generate singing voice by making use of music knowledge to approximate the target vocal line in terms of tempo. The experimental results on a 20-song database show 26.3% and 36.1% word level alignment error rates at eighth note and sixteenth note alignment tolerances respectively. The proposed approach presents an alternative and effective solution to music-lyrics alignment which may require less training dataset.

The introduction of Gaussian mixture models (GMMs) in the field of speaker verification has led to very good results. This paper illustrates an evolution in state-of-the-art speaker verification by highlighting the contribution of recently established information theoretic based vector quantization technique. We explore the novel application of three different vector quantization algorithms, namely K-means, Linde-Buzo-Gray (LBG) and information theoretic vector quantization (ITVQ) for efficient speaker verification. The expectation maximization (EM) algorithm used by GMM requires a prohibitive amount of iterations to converge. In this paper, comparable alternatives to EM including K-means, LBG and ITVQ algorithm were tested. The GMM-ITVQ algorithm was found to be the most efficient alternative for the GMM-EM. It gives correct classification rates at a similar level to that of GMM-EM. Finally, representative performance benchmarks and system behaviour experiments on NIST SRE corpora are presented.

A novel content-adaptive music watermarking technique is proposed in this article. To optimally balance inaudibility and robustness when embedding and extracting watermarks, the embedding scheme is highly related to the music structure and human auditory system (HAS). A note-based segmentation method is proposed and used for music vocal/instrumental boundary detection. A multiple bit hopping and hiding scheme with different embedding parameters is applied to vocal and instrumental frames of the music. The experimental results in inaudibility and robustness are provided to support all novel features in the proposed watermarking scheme.

With the increasing use of audio sensors in surveillance and monitoring applications, event detection using audio streams has emerged as an important research problem. This paper presents a hierarchical approach for audio based event detection for surveillance. The proposed approach first classifies a given audio frame into vocal and nonvocal events, and then performs further classification into normal and excited events. We model the events using a Gaussian mixture model and optimize the parameters for four different audio features ZCR, LPC, LPCC and LFCC. Experiments have been performed to evaluate the effectiveness of the features for detecting various normal and the excited state human activities. The results show that the proposed top-down event detection approach works significantly better than the single level approach

This paper first discusses how the signal segmentation and tonal characteristics of music notes effect in music chord detection. Two approaches, pitch class profile approach and psycho-acoustical approach, which differently represent these tonal characteristics, are examined for chord detection. The analysis of the tonal characteristics reveals that not only the fundamental frequency of music note but also its harmonics and sub-harmonies in different octaves contribute for detecting related music chord. A hierarchical approach, which transforms the music chord tonal characteristics in each octave onto probabilistic space, is then proposed for modeling the music chord. Our experimental results show that detection of chord type, major, minor, diminish, and augmented, and individual chords, 12 chords per chord type, are improved with the proposed hierarchical chord modeling approach. Experimental results also reveal that the tempo proportional signal segmentation is more effective extracting tonal characteristics than using fixed length segmentation

This paper proposes a novel framework for music content indexing and retrieval. The music structure information, i.e., timing, harmony and music region content, is represented by the layers of the music structure pyramid. We begin by extracting this layered structure information. We analyze the rhythm of the music and then segment the signal proportional to the inter-beat intervals. Thus, the timing information is incorporated in the segmentation process, which we call Beat Space Segmentation. To describe Harmony Events, we propose a two-layer hierarchical approach to model the music chords. We also model the progression of instrumental and vocal content as Acoustic Events. After information extraction, we propose a vector space modeling approach which uses these events as the indexing terms. In query-by-example music retrieval, a query is represented by a vector of the statistics of the n-gram events. We then propose two effective retrieval models, a hard-indexing scheme and a soft-indexing scheme. Experiments show that the vector space modeling is effective in representing the layered music information, achieving 82.5% top-5 retrieval accuracy using 15-sec music clips as the queries. The soft-indexing outperforms hard-indexing in general.

In this article, we propose a novel approach for automatic music video summarization. The proposed summarization scheme is different from the current methods used for video summarization. The music video is separated into the music track and video track. For the music track, a music summary is created by analyzing the music content using music features, an adaptive clustering algorithm, and music domain knowledge. Then, shots in the video track are detected and clustered. Finally, the music video summary is created by aligning the music summary and clustered video shots. Subjective studies by experienced users have been conducted to evaluate the quality of music summaries and effectiveness of the proposed summarization approach. Experiments are performed on different genres of music videos and comparisons are made with the summaries generated based on music track, video track, and manually. The evaluation results indicate that summaries generated using the proposed method are effective in helping realize users' expectations.

In this paper, we present a novel approach for music summarization based on music structure analysis. From the audio signal, we first extract the note onset representing the time tempo of the song and the music structure analysis can be performed based on this tempo information. After music content has been structured into different semantic regions such as introduction (intro), verse, chorus, ending (outro), etc., the final music summary can be created with chorus and music phrases which are included anterior or posterior to the selected chorus to get the desired length of the final summary. In this way, we can guarantee that the summaries begin and end at meaningful music phrase boundaries, which is a difficult problem for existing music summarization methods. Experiments show our proposed method can capture the main theme of the music compared to the ideal summaries selected by music experts and user subjective evaluation indicates our proposed method has a good performance.

In this paper, we propose a novel approach for automatic music video summarization based on audio-visual-text analysis and alignment. The music video is separated into the music and video tracks. For the music track, the chorus is detected based on music structure analysis. For the video track, we first segment the shots and classify the shots into close-up face shots and non-face shots, then we extract the lyrics and detect the most repeated lyrics from the shots. The music video summary is generated based on the alignment of boundaries of the detected chorus, shot class and the most repeated lyrics from the music video. The experiments on chorus detection, shot classification, and lyrics detection using 20 English music videos are described. Subjective user studies have been conducted to evaluate the quality and effectiveness of summary. The comparisons with the summaries based on our previous method and the manual method indicate that the results of summarization using the proposed method are better at meeting users' expectations.

Automatic music classification and summarization are very useful to music indexing, content-based music retrieval and on-line music distribution, but it is a challenge to extract the most common and salient themes from unstructured raw music data. In this paper, we propose effective algorithms to automatically classify and summarize music content. Support vector machines are applied to classify music into pure music and vocal music by learning from training data. For pure music and vocal music, a number of features are extracted to characterize the music content, respectively. Based on calculated features, a clustering algorithm is applied to structure the music content. Finally, a music summary is created based on the clustering results and domain knowledge related to pure and vocal music. Support vector machine learning shows a better performance in music classification than traditional Euclidean distance methods and hidden Markov model methods. Listening tests are conducted to evaluate the quality of summarization. The experiments on different genres of pure and vocal music illustrate the results of summarization are significant and effective.

We propose a twice-iterated composite Fourier transform (TICFT) technique to detect the singing voice boundaries from acoustical polyphonic music signals. We show that the cumulative TICFT energy in the lower coefficients is capable of differentiating the harmonic structures of vocal and instrumental music in higher octaves. The musical signal is first segmented into frames based on quarter-notes. Then TICFT is used to measure the harmonic structure of each frame. Finally, the vocal and instrumental frames are classified by applying music domain knowledge. Experimental results show over 80% frame level accuracy can be achieved.

Automatic music summarization is very useful for music indexing, content-based music retrieval and on-line music distribution, but it is a challenge to extract automatically the most common and salient themes from unstructured raw music data. We propose an effective approach to summarize music content automatically. First, a number of features are extracted to characterize the music content. Based on the extracted features, an adaptive clustering algorithm is then applied to structure the music content. Finally, the music summary is created in terms of the clustering results and domain-related music knowledge. A user study is conducted to evaluate the quality of summarization. The experiments on different genres of music illustrate the results of summarization are significant and effective to actual expectation.

In this paper, we present a novel approach for music structure analysis. A new segmentation method, beat space segmentation, is proposed and used for music chord detection and vocal/instrumental boundary detection. The wrongly detected chords in the chord pattern sequence and the misclassified vocal/instrumental frames are corrected using heuristics derived from the domain knowledge of music composition. Melody-based similarity regions are detected by matching sub-chord patterns using dynamic programming. The vocal content of the melody-based similarity regions is further analyzed to detect the content-based similarity regions. Based on melody-based and content-based similarity regions, the music structure is identified. Experimental results are encouraging and indicate that the performance of the proposed approach is superior to that of the existing methods. We believe that music structure analysis can greatly help music semantics understanding which can aid music transcription, summarization, retrieval and streaming.

Automatic musical genre classification is very useful for music indexing and retrieval. In this paper, an efficient and effective automatic musical genre classification approach is presented. A set of features is extracted and used to characterize music content. A multi-layer classifier based on support vector machines is applied to musical genre classification. Support vector machines are used to obtain the optimal class boundaries between different genres of music by learning from training data. Experimental results of multi-layer support vector machines illustrate good performance in musical genre classification and are more advantageous than traditional Euclidean distance based method and other statistic learning methods.

This paper presents a novel framework called audio keywords to assist event detection in soccer video. Audio keyword is a middle-level representation that can bridge the gap between low-level features and high-level semantics. Audio keywords are created from low-level audio features by using support vector machine learning. The created audio keywords can be used to detect semantic events in soccer video by applying a heuristic mapping. Experiments of audio keywords creation and event detection based on audio keywords have illustrated promising results. According to the experimental results, we believe that audio keyword is an effective representation that is able to achieve more intuitionistic result for event detection in sports video compared with the method of event detection directly based on low-level features.