One way of expressing an environmental sound is using vocal imitations, which involve the process of replicating or mimicking the rhythm and pitch of sounds by voice. We can effectively express the features of environmental sounds, such as rhythm and pitch, using vocal imitations, which cannot be expressed by conventional input information, such as sound event labels, images, or texts, in an environmental sound synthesis model. In this paper, we propose a framework for environmental sound synthesis from vocal imitations and sound event labels based on a framework of a vector quantized encoder and the Tacotron2 decoder. Using vocal imitations is expected to control the pitch and rhythm of the synthesized sound, which only sound event labels cannot control. Our objective and subjective experimental results show that vocal imitations effectively control the pitch and rhythm of synthesized sounds.

The detection of shouted speech is crucial in audio surveillance and monitoring. Although it is desirable for a security system to be able to identify emergencies, existing corpora provide only a binary label (i.e., shouted or normal) for each speech sample, making it difficult to predict the shout intensity. Furthermore, most corpora comprise only utterances typical of hazardous situations, meaning that classifiers cannot learn to discriminate such utterances from shouts typical of less hazardous situations such as cheers. Thus, this paper presents a novel research source, the RItsumeikan Shout Corpus (RISC), which contains wide variety types of shouted speech samples collected in recording experiments. Each shouted speech sample in RISC has a shout type and is also assigned shout intensity ratings via a crowdsourcing service. We also present a comprehensive performance comparison among deep learning approaches for speech type classification tasks and a shout intensity prediction task. The results show that feature learning based on the spectral and cepstral domains achieves high performance, no matter which network architecture is used. The results also demonstrate that shout type classification and intensity prediction are still challenging tasks, and RISC is expected to contribute to further development in this research area.

In recent years, much research has been into speech emotion recognition (SER) using deep learning to predict emotions conveyed by speech. We studied the method that detected the emotion for the whole utterance using the frame-based SER, which estimates emotions in each frame rather than in a whole utterance. One of the problems with this method is that the emotional label sequence, which is used in training the frame-based SER, does not sufficiently consider phonemic characteristics. To solve this problem, we propose new methods of recognizing the emotion for the whole utterance using frame-based SER that considers the phoneme class attribute such as vowels, voiced consonants, unvoiced consonants, and other symbols in training. As a result, we found that the proposed methods significantly improve the performance of the result for the whole utterance compared to conventional methods.

Some studies have revealed that contexts of scenes (e.g., "home," "office," and "cooking") are advantageous for sound event detection (SED). Mobile devices and sensing technologies give useful information on scenes for SED without the use of acoustic signals. However, conventional methods can employ pre-defined contexts in inference stages but not undefined contexts. This is because one-hot representations of pre-defined scenes are exploited as prior contexts for such conventional methods. To alleviate this problem, we propose scene-informed SED where pre-defined scene-agnostic contexts are available for more accurate SED. In the proposed method, pre-trained large-scale language models are utilized, which enables SED models to employ unseen semantic contexts of scenes in inference stages. Moreover, we investigated the extent to which the semantic representation of scene contexts is useful for SED. Experimental results performed with TUT Sound Events 2016/2017 and TUT Acoustic Scenes 2016/2017 datasets show that the proposed method improves micro and macro F-scores by 4.34 and 3.13 percentage points compared with conventional Conformer- and CNN– BiGRU-based SED, respectively.

In conventional sound event detection (SED) models, two types of events, namely, those that are present and those that do not occur in an acoustic scene, are regarded as the same type of the events. The conventional SED methods cannot effectively exploit the difference between the two types of events. The all time frames of sound events that do not occur in an acoustic scene are easily regarded as inactive in the scene, that is, the events are easy-to-train. The time frames of the events that are present in a scene must be classified as active in addition to inactive in the acoustic scene, that is, the events are difficult-to-train. To take advantage of the training difficulty, we apply curriculum learning into SED, where models are trained from easy- to difficult-to-train events. To utilize the curriculum learning, we propose a new objective function for SED, wherein the events are trained from easy-to difficult-to-train events. Experimental results show that the F-score of the proposed method is improved by 10.09 percentage points compared with that of the conventional binary cross entropy-based SED.

Sound event detection (SED) and acoustic scene classification (ASC) are major tasks in environmental sound analysis. Considering that sound events and scenes are closely related to each other, some works have addressed joint analyses of sound events and acoustic scenes based on multitask learning (MTL), in which the knowledge of sound events and scenes can help in estimating them mutually. The conventional MTL-based methods utilize one-hot scene labels to train the relationship between sound events and scenes; thus, the conventional methods cannot model the extent to which sound events and scenes are related. However, in the real environment, common sound events may occur in some acoustic scenes; on the other hand, some sound events occur only in a limited acoustic scene. In this paper, we thus propose a new method for SED based on MTL of SED and ASC using the soft labels of acoustic scenes, which enable us to model the extent to which sound events and scenes are related. Experiments conducted using TUT Sound Events 2016/2017 and TUT Acoustic Scenes 2016 datasets show that the proposed method improves the SED performance by 3.80% in F-score compared with conventional MTL-based SED.

This paper studies the characteristics of dancecharts on rhythm-based video games while clustering the dancecharts with the track features. In the opinion of expert players, it is considered that the difficulty level of the rhythm-based games depends on multiple features for the song and chart. The difficulty levels for dance games has not well been studied though it has been already done for other genres of games. This paper designed the track features for dance games, which would be varied depending on characteristics of the track. Moreover, we conducted a dance-charts clustering by using the k-means method with the designed features. As a result of the clustering, it was confirmed that the clusters were composed mostly based on the step frequency and the complexity of rhythm. Also, it was found that the scores with the extremely characteristic features were separated from other scores and clustered as one class with themselves. The discussion in this paper indicated a possibility for the objective evaluation of the dance-charts.

Speech enhancement has been widely investigated for several decades, but by modifying only the amplitude spectrum of a speech signal, ignoring the phase spectrum, which has been regarded as an unimportant feature. However, it was recently reported that the phase spectrum plays an important role in speech quality and intelligibility. In this paper, we propose a phase reconstruction method based on harmonic enhancement using the fundamental frequency and phase distortion feature. This feature is known to show fluctuations in the phase spectrum with respect to time and frequency. We estimate the speech phase spectrum by considering the relationship between harmonic phase spectra. Experimental evaluations indicate that the proposed phase reconstruction method improves speech quality in various noisy environments.

Speech enhancement in noisy environments has been widely investigated by modifying only the amplitude spectrum of the speech signal, while the phase spectrum, which is regarded as an unimportant feature, is ignored. However, it has recently been reported that the phase spectrum plays an important role in the intelligibility and quality of speech. We propose a speech-enhancement method with phase reconstruction, which estimates inartificial phase spectrum by using the time-frequency feature called phase distortion, though a conventional phase reconstruction estimates artificial one. The objective experimental results indicate improvement in speech quality with and efficiency of the proposed method.

Desired/undesired speech discrimination is as important as speech/non-speech discrimination to achieve useful applications such as speech interfaces and teleconferencing systems. Conventional methods of voice activity detection (VAD) utilize the directional information of sound sources to distinguish desired from undesired speech. However, these methods have to utilize multiple microphones to estimate the directions of sound sources. Here, we propose a new method to discriminate desired from undesired speech with a single microphone. We assumed that the desired talkers would be close to the microphone, and the proposed method could distinguish close/distant-talking speech from observed signals based on the kurtosis of the linear prediction (LP) residual signals. The experimental results revealed that the proposed method could distinguish close-talking speech from distant-talking speech within a 10% equal error rate (EER) in ordinary reverberant environments with less processing time.

Recently, parametric loudspeakers have been utilized to reproduce an audible sound in a particular area by an ultrasound. However, parametric loudspeakers have lower sound quality than general electrodynamic loudspeakers because it is difficult to reproduce an audible sound at lower frequency. Moreover, the reproduced sound includes harmonic distortions causing deterioration of sound quality. Thus, in this paper, we attempt to improve sound quality of parametric loudspeakers. It depends on a modulation technique for designing the amplitude modulated wave. Therefore, we propose a new modulation method to reproduce audible sound with high quality. The proposed method controls a weight of double sidebands for reducing harmonic distortions at higher frequency and emphasizing power at lower frequency. We carried out evaluation experiments to confirm the effectiveness of the proposed method. The results confirmed that the proposed method could improve sound quality of parametric loudspeakers.

We digitally archived festival music signals (“Ohayashi”) in the Yamahoko parades of Gion festival in Kyoto, Japan. Besides, the festival music, which consists of Japanese traditional drums, flutes, bells, ambient noise and in-float driving noise are needed to reproduce the authentic atmosphere of this festival. To reproduce a high-quality sound field, we recorded the festival music in the presence of ambient noise and float noise by using multi-channel recording. We reproduced the sound field of one of the parades with the recorded sound sources. We employed point-source loudspeakers for reproducing Japanese traditional drums, flutes, and bells with omni-directional radiation characteristics. After that, we built a web-based system linked to a map of the parade route that would produce an acoustic sound field with realistic sensations at different points along the route. This system reproduces the Ohayashi at particular positions along the route when the user clicks circular buttons on the map.

Spoken Document Retrieval (SDR) and Spoken Term Detection (STD) have been two of the most intensively investigated topics in spoken document processing research according to the establishment of the SDR and STD test collections by the Text REtrieval Conference (TREC) and NIST. Because Japanese spoken document processing researchers also requires such test collections for SDR and STD, we have established a working group to develop these collections in Special Interest Group-Spoken Language Processing (SIG-SLP) of the Information Processing Society of Japan. The working group has constructed and made available a test collection for SDR, and is now constructing new test collections for STD that will be open to researchers. The present paper introduces the policies, outline, and schedule of the new test collections. Then, the new test collections are compared with the NIST STD test collections. Index Terms: spoken term detection, test collection 1.

This paper describes a stochastic modeling between an F0contour and linguistic features of a sentence for speech synthesis. The F0contour of a sentence is represented by concatenation of the F0patterns of a Japanese syntactic unit, bunsetsu .A bunsetsu F0pattern is composed of the F0average and the F0shape. The F0average is independently predicted for each bunsetsu by a quantification theory from linguistic features of the bunsetsu. The most probable sequence of bunsetsu F0shapes for a sentence are found in the F0shape database by a probabilistic measure. The probability that an F0contour is observed for a sentence is defined by two kinds of probabilities, the F0 shape production and the F0shape bigram. The latter is a probability of adjacent occurrence of two F0shapes, like a word bigram in speech recognition. Several typical bunsetsu F0shapes are extracted by clustering of training data and stored in the F0shape database. The probability of the F0shape production is computed for each bunsetsu based on the distribution of linguistic features in the cluster.

This paper describes topic identification for Japanese TV news speech based on the keyword spotting technique. Three thousands of nouns are selected as keywords which contribute to topic identification, based on criterion of mutual information and a length of the word. This set of the keywords identified the correct topic for 76.3% of articles from newspaper text data. Further, we performed keyword spotting for TV news speech and identified the topics of the spoken message by calculating possibilities of the topics in terms of an acoustic score of the spotted word and a topic probability of the word. In order to neutralize effect of false alarms, bias of the topics in the keyword set is removed. Topic identification rate is 66.5% assuming that identification is correct if the correct topic is included in the top three topics. The removal of the bias improved the identification rate by 6.1%. 1. INTRODUCTION Enormous speech data, such as TV news, can be stored in the data base as compu...

The paper describes automatic generation of speech synthesis rules which predict the accent component value (stress level) for the bunsetsu in long noun phrases. The rules are inductively inferred from a lot of speech data by using two kinds of tree-based methods, the conventional tree generation and the SBR-tree algorithm. The rule sets automatically generated by the two methods have the almost same performance and decrease the prediction error to about 14 Hz from 23 Hz of the accent component value. The rate of the correct reproduction of the change, that is increase or decrease, for adjacent bunsetsu pairs is also used as a measure of evaluation and the generated rule sets correctly reproduce about 80% of the change. Effectiveness of the rule sets is verified through a listening test. SBR-tree methods generate very compact rules which are easy for human experts to interpret and match with the former studies.>

A concept-to-speech (CTS) conversion system, SOCS (speech output from case structure representation), is described. For CTS systems, how to represent the input to the system is a very important issue. In SOCS, a representation based on case structure and phrase patterns, which is called concept representation, is converted into synthetic speech. SOCS plays a crucial role in a speech output interface which is designed as a general frontend of various intelligent performance systems (IPSs). The message transmission of the concept representation satisfies the requirements both from IPS and the speech synthesis component. SOCS has two new built-in mechanisms for direct control of prosodic parameters by a concept representation. The first is the pause marker which is generated along with words in the sentence generation. The pause marker is used both to insert pauses and to set the boundaries of the prosodic phrase component of F/sub 0/. The second is the prosody modification function (PMF) embedded in the custom templates. PMF controls the prosodic parameters for the prepared sentence pattern. The dialog management in a speech output interface based on SOCS is also discussed.>