Whistle contour extraction aims to derive animal whistles from time-frequency spectrograms as polylines. For toothed whales, whistle extraction results can serve as the basis for analyzing animal abundance, species identity, and social activities. During the last few decades, as long-term recording systems have become affordable, automated whistle extraction algorithms were proposed to process large volumes of recording data. Recently, a deep learning-based method demonstrated superior performance in extracting whistles under varying noise conditions. However, training such networks requires a large amount of labor-intensive annotation, which is not available for many species. To overcome this limitation, we present a framework of stage-wise generative adversarial networks (GANs), which compile new whistle data suitable for deep model training via three stages: generation of background noise in the spectrogram, generation of whistle contours, and generation of whistle signals. By separating the generation of different components in the samples, our framework composes visually promising whistle data and labels even when few expert annotated data are available. Regardless of the amount of human-annotated data, the proposed data augmentation framework leads to a consistent improvement in performance of the whistle extraction model, with a maximum increase of 1.69 in the whistle extraction mean F1-score. Our stage-wise GAN also surpasses one single GAN in improving whistle extraction models with augmented data.

We present a learning-based method for extracting whistles of toothed whales (Odontoceti) in hydrophone recordings. Our method represents audio signals as time-frequency spectrograms and decomposes each spectrogram into a set of time-frequency patches. A deep neural network learns archetypical patterns (e.g., crossings, frequency modulated sweeps) from the spectrogram patches and predicts time-frequency peaks that are associated with whistles. We also developed a comprehensive method to synthesize training samples from background environments and train the network with minimal human annotation effort. We applied the proposed learn-from-synthesis method to a subset of the public Detection, Classification, Localization, and Density Estimation (DCLDE) 2011 workshop data to extract whistle confidence maps, which we then processed with an existing contour extractor to produce whistle annotations. The F1-score of our best synthesis method was 0.158 greater than our baseline whistle extraction algorithm (~25% improvement) when applied to common dolphin (Delphinus spp.) and bottlenose dolphin (Tursiops truncatus) whistles.

An efficient music representation based on the matching pursuit (MP) technique with content-adaptive dictionaries (CADs) is investigated in this work. The Gabor atoms are commonly adopted in the MP-based signal representation due to their excellent time-frequency localization property. However, the Gabor dictionary may not yield a concise representation for music signals. Music signals have special characteristics which are specified by pitches and durations of music notes. In this work, we exploit music characteristics to create a set of content-adaptive elementary functions called atoms that efficiently capture the inherent structure of music signals. As a result, we are able to project musical signals onto a subspace spanned by atoms from CAD for a concise and efficient representation. The proposed CAD representation technique is applied to music enhancement with noisy background to demonstrate the power of the proposed representation.

We study the problem of automatic musical beat tracking from acoustic data, i.e., finding locations of beats of a music piece by computers on-the-fly, in this work. An on-line musical beat tracking algorithm based on Kalman filtering (KF) with an enhanced probability data association (EPDA) method is proposed. The beat tracking algorithm is built upon a linear dynamic model of beat progression, to which the Kalman filtering technique can be conveniently applied. The beat tracking performance can be seriously degraded by noisy measurements in the Kalman filtering process. Three methods are presented for noisy measurements selection. They are the local maximum (LM) method, the probabilistic data association (PDA) method and the enhanced PDA (EPDA) method. We see that the performance of EPDA outperforms that of LM and PDA significantly.

A modified Kalman filtering approach to on-line music beat tracking is proposed in this work. The proposed algorithm first detects onset signals from the acoustic waveform. Then, it tracks the beat and the tempo using the Kalman filter. Two techniques (i.e., observation smoothing and lock detection) are adopted to improve the tracking performance furthermore. It is shown by experimental results that the proposed modified Kalman filtering approach provides satisfactory beat tracking performance.

A new approach to musical onset detection by detecting the significant change of joint energy and phase features is proposed in this work. We modify the phase deviation method in (Duxbury et al., 2003) to yield a more robust phase feature. To capture the energy deviation of an audio waveform along time, we compute the error of a forward linear prediction filter as the energy feature. Several ways to fuse obtained energy and phase features are discussed and compared. It is demonstrated by experimental results that a proper scheme to integrate energy and phase features can provide excellent onset detection results.

The analysis of audio signals of popular and rock songs of the verse-chorus form to reconstruct its original musical structures is investigated in this work. We first compute the similarity degree between any two measures in a song based on selected features and represent these numbers in a measure-based similarity matrix. Then, we study the similarity across a sequence of consecutive measures, which is revealed by straight segments in parallel with the diagonal line of the similarity matrix. Generally, chorus parts have higher similarity values while verse parts have lower similarity values. As a result, the verse parts are difficult to detect in the presence of the chorus parts. To tackle this problem systematically, the Viterbi Algorithm is adopted to find optimal paths in the lower-triangular similarity matrix, which represent repetitive segments of both choruses and verses. Finally, several post-processing steps are developed to decode the music structure into the verse, the chorus and other non-repetitive parts. Experimental results obtained from several musical audio data are shown to demonstrate the performance of the proposed method.