Audiovisual representation learning typically relies on the correspondence between sight and sound. However, there are often multiple audio tracks that can correspond with a visual scene. Consider, for example, different con-versations on the same crowded street. The effect of such counterfactual pairs on audiovisual representation learning has not been previously explored. To investigate this, we use dubbed versions of movies and television shows to augment cross-modal contrastive learning. Our approach learns to represent alternate audio tracks, differing only in speech, similarly to the same video. Our results, from a comprehensive set of experiments investigating different training strategies, show this general approach improves performance on a range of downstream auditory and audio-visual tasks, without majorly affecting linguistic task per-formance overall. These findings highlight the importance of considering speech variation when learning scene-level audiovisual correspondences and suggest that dubbed au-dio can be a useful augmentation technique for training au-diovisual models toward more robust performance on di-verse downstream tasks.

Research into the prediction and analysis of perceived audio quality is hampered by the scarcity of openly available datasets of audio signals accompanied by corresponding subjective quality scores. To address this problem, we present the Open Dataset of Audio Quality (ODAQ), a new dataset containing the results of a MUSHRA listening test conducted with expert listeners from 2 international laboratories. ODAQ contains 240 audio samples and corresponding quality scores. Each audio sample is rated by 26 listeners. The audio samples are stereo audio signals sampled at 44.1 or 48 kHz and are processed by a total of 6 method classes, each operating at different quality levels. The processing method classes are designed to generate quality degradations possibly encountered during audio coding and source separation, and the quality levels for each method class span the entire quality range. The diversity of the processing methods, the large span of quality levels, the high sampling frequency, and the pool of international listeners make ODAQ particularly suited for further research into subjective and objective audio quality. The dataset is released with permissive licenses, and the software used to conduct the listening test is also made publicly available.

One of the major challenges in Automatic Drum Transcription (ADT) research is the lack of large-scale labeled dataset featuring audio with polyphonic mixtures; this limitation around data availability greatly impedes the progress of data-driven approaches in the context of ADT. To tackle this issue, we propose a semi-automatic way of compiling a labeled dataset using the audio-to-MIDI alignment technique. The resulting dataset consists of 1565 polyphonic mixtures of music with audio-aligned MIDI ground truth. To validate the quality and generality of this dataset, an ADT model based on Convolutional Neural Network (CNN) is trained and evaluated on several publicly available datasets. The evaluation results suggest that our proposed model, which is trained solely on the compiled dataset, compares favorably with the state-of-the-art ADT systems. The result also implies the possibility of leveraging audio-to-MIDI alignment in creating datasets for a broader range of audio related tasks.

A novel vehicle tracking algorithm robust to multi- viewpoint pattern and occlusion is proposed. To improve accuracy, after object detection, the overlapping parts of bounding boxes are removed before block matching. It is helpful for reducing the interference from nearby vehicles or objects. To perform block matching well, in addition to partial similarity and position correlation, many advanced features, including saliency features and several new global features, are adopted. These features can retrieve important information from vehicles and are helpful for tracking. Experiments show that the proposed algorithm achieves favorable performance against state-of-the-art vehicle tracking methods, including rule-based and learning-based methods.

Person re-identification (Re-ID) aims to match images of the same person across distinct camera views. In this paper, we propose the Space-Time Guided Association Learning (STGAL) for unsupervised Re-ID without ground truth identity nor image correspondence observed during training. By exploiting the spatial-temporal information presented in pedestrian data, our STGAL is able to identify positive and negative image pairs for learning Re-ID feature representations. Experiments on a variety of datasets confirm the effectiveness of our approach, which achieves promising performance when comparing to the state-of-the-art methods.

In Western popular music, drums and percussion are an important means to emphasize and shape the rhythm, often defining the musical style. If computers were able to analyze the drum part in recorded music, it would enable a variety of rhythm-related music processing tasks. Especially the detection and classification of drum sound events by computational methods is considered to be an important and challenging research problem in the broader field of music information retrieval. Over the last two decades, several authors have attempted to tackle this problem under the umbrella term automatic drum transcription (ADT). This paper presents a comprehensive review of ADT research, including a thorough discussion of the task-specific challenges, categorization of existing techniques, and evaluation of several state-of-the-art systems. To provide more insights on the practice of ADT systems, we focus on two families of ADT techniques, namely methods based on non-negative matrix factorization and recurrent neural networks. We explain the methods’ technical details and drum-specific variations and evaluate these approaches on publicly available data sets with a consistent experimental setup. Finally, the open issues and underexplored areas in ADT research are identified and discussed, providing future directions in this field.

In this paper, a blind bandwidth extension algorithm for music signals has been proposed. This method applies the K-means algorithm to firstly cluster audio data in the feature space, and constructs multiple envelope predictors for each cluster accordingly using Support Vector Regression (SVR). A set of well-established audio features for Music Information Retrieval (MIR) has been used to characterize the audio content. The resulting system is applied to a variety of music signals without any side information provided. The subjective listening test results show that this method can improve the perceptual quality successfully, but the minor artifacts still leave room for future improvements.

FusionGAN is a novel genre fusion framework for music generation that integrates the strengths of generative adversarial networks and dual learning. In particular, the proposed method offers a dual learning extension that can effectively integrate the styles of the given domains. To efficiently quantify the difference among diverse domains and avoid the vanishing gradient issue, FusionGAN provides a Wasserstein based metric to approximate the distance between the target domain and the existing domains. Adopting the Wasserstein distance, a new domain is created by combining the patterns of the existing domains using adversarial learning. Experimental results on public music datasets demonstrated that our approach could effectively merge two genres.

This paper presents an unsupervised method for systematically identifying anomalies in music datasets. The model integrates categorical regression and robust estimation techniques to infer anomalous scores in music clips. When applied to a music genre recognition dataset, the new method is able to detect corrupted, distorted, or mislabeled audio samples based on commonly used features in music information retrieval. The evaluation results show that the algorithm outperforms other anomaly detection methods and is capable of finding problematic samples identified by human experts. The proposed method introduces a preliminary framework for anomaly detection in music data that can serve as a useful tool to improve data integrity in the future.

An innovative prosthetic hand prototype is presented in this paper. To increase the grasping stability, an underactuated finger is embedded in a traditional gripper. There are a total of three motors employed in the hand system: the gripper, the embedded finger, and the wrist. The preliminary performance shows that this prosthetic hand is capable of performing various daily jobs such as turning a key, pinching a card, grasping tubes of various sizes, etc. The maximum grasping force for the gripper is about 2 kg and that for the embedded finger is about 0.3 kg. The closing time for both the gripper and the embedded finger is about 0.2 sec, which is relative fast among similar prosthetic hands. The performance of this innovative prosthetic hand prototype shows the idea of the embedded finger is promising and further improvements will continue.