We aimed to develop WindVibraTransformer, a foundational model based on Transformers applied to vibration spectrograms, to derive precise and robust feature representations for wind turbine condition monitoring. In anomaly detection for wind turbine equipment, it is common to employ inlier modeling, where models are built solely on data from normal operation, with deviations classified as anomalies. Acquiring data for wind turbines, especially from the specific turbine being monitored, is challenging, often resulting in limited availability. However, this limitation can increase susceptibility to false alarms due to environmental variations. Thus, in this study, we explored the use of a vision Transformer trained with self-supervised learning to predict masked spectra conditioned on unmasked ones from vibration spectrograms, serving as the precise and robust feature extractor. Since this model aims to capture relationships within the context of the surroundings rather than solely the overview of the spectrogram, the resulting feature representations are expected to be complex yet less directly influenced by environmental variations. The complexity of these feature representations is addressed by constructing the normal state model using highly expressive normalizing flows. Experimental comparisons conducted on anomaly detection using real wind turbine vibration signals demonstrated that our approach enables highly reliable detection compared to existing unsupervised auto-encoder-based and supervised wind turbine classifier-based feature extraction methods.

To achieve robust wind turbine anomaly detection, we attempted to incorporate metric learning into the process of learning discriminative feature representations. In the context of anomaly detection based on inlier modeling, where anomalies are identified as inputs deviating from the normal state distribution, the key to building a high-performance and robust system, even with limited training data, lies in eliminating the influence of environmental differences from the inputs and obtaining a compact normal state distribution. To address this, we designed a deep neural network-based feature extractor to mitigate the impact of environmental changes. Additionally, we integrated metric learning into its learning process to obtain a more compact distribution by embedding inputs with similar properties close to one another. We validated the effectiveness of the developed network as a feature extractor for constructing a normal model with limited data through experimental comparisons using vibration data collected from sensors. Specifically, we utilized MobileNet as a discriminative feature extractor to identify 37 wind turbines. By introducing metric learning based on t-SNE into its training process, we achieved a notable 50 % improvement in the AUC. Remarkably, this significant improvement was accomplished using only a small amount of data, approximately 13 minutes, obtained from the monitored turbine.

This paper presents a fast intra mode decision method for H.265/HEVC using outliers of DCT coefficients and neighboring block information. H.265/HEVC requires huge computational load to detect the best intra mode in each intra frame. We propose a novel low-complexity intra mode detection method by restricting the number of mode candidates to be evaluated.

This paper presents a language-model training method for improving automatic transcription of online spoken contents. Unlike previously studied LVCSR tasks such as broadcast news and lectures, large-sized task-specific corpora for training language models cannot be prepared and used in recognition because of the diversity of topics, vocabularies, and speaking styles. To overcome difficulties in preparing such task-specific language models in advance, we propose collaborative training of language models on the basis of wisdom of crowds. On our public web service for LVCSR-based spoken document retrieval PodCastle, over half a million recognition errors were corrected by anonymous users. By leveraging such corrected transcriptions, component language models for various topics can be built and dynamically mixed to generate an appropriate language model for each podcast episode in an unsupervised manner. Experimental results with Japanese podcasts showed that the mixed languages models significantly reduced the word error rate. Index Terms: web service, LVCSR, language modeling, wisdom of crowds, error correction

In this paper, we introduce recent advances of a speech retrieval web service, PodCastle, that collects and amplifies voluntary contributions by anonymous users. Our goal is to provide users with a public web service based on speech recognition and crowdsourcing so that they can experience state-of-the-art speech recognition performance through a useful service. Pod-Castle enables users to find speech data (such as podcasts and YouTube video clips) that include a search term, read full texts of their recognition results, and easily correct recognition errors by simply selecting from a list of candidates. The resulting corrections were used to improve both the speech retrieval and recognition performances. In our experiences from its practical use over the past four years (since December, 2006), over half a million recognition errors in about one hundred thousand speech data were corrected by anonymous users and we confirmed that the speech recognition performance of PodCastle was actually improved by those corrections. Index Terms: speech retrieval, spoken document retrieval, wisdom of crowds, crowdsourcing

In recognizing spontaneous speech, the performance of typical speech recognizers tends to be degraded by filled and silent pauses, which are hesitation phenomena frequently occurred in such speech. In this paper, we present a method for improving the performance of a speech recognizer by detecting and handling both filled pauses (lengthened vowels) and silent (unfilled) pauses. Our method automatically detects these pauses by using a bottom-up acoustical analysis in parallel with a typical speech decoding process, and then incorporates the detected results into the decoding process. From the results of experiments conducted using the CIAIR spontaneous speech corpus, the effectiveness of the proposed method was confirmed.

This paper presents acoustic-model-training techniques for improving automatic transcription of podcasts. A typical approach for acoustic modeling is to create a task-specific corpus including hundreds (or even thousands) of hours of speech data and their accurate transcriptions. This approach, however, is impractical in podcast-transcription task because manual generation of the transcriptions of the large amounts of speech covering all the various types of podcast contents will be too costly and time consuming. To solve this problem, we introduce collaborative training of acoustic models on the basis of wisdom of crowds, i.e., the transcriptions of podcast-speech data are generated by anonymous users on our web service PodCastle. We then describe a podcast-dependent acoustic modeling system by using RSS metadata to deal with the differences of acoustic conditions in podcast speech data. From our experimental results on actual podcast speech data, the effectiveness of the proposed acoustic model training was confirmed. Index Terms: podcast, LVCSR, acoustic model training, wisdom of crowds, error correction

This paper presents a method to detect acoustic events that can be used to find “hot spots ” in podcast programs. We focus on meaningful non-verbal audible reactions which suggest hot spots such as laughter and reactive tokens. In order to detect this kind of short events and segment the counterpart utterances, we need accurate audio segmentation and classification, dealing with various recording environments and background music. Thus, we propose a method for automatically estimating and switching penalty weights for the BIC-based segmentation depending on background environments. Experimental results show significant improvement in detection accuracy by proposed method compared to when using a constant penalty weight. Index Terms: acoustic event detection, laughter detection, podcast, Bayesian Information Criterion

Given podcasts (audio blogs) that are sets of speech files called episodes, this paper describes a method for retrieving episodes that have similar content. Although most previous retrieval methods were based on bibliographic information, tags, or users' playback behaviors without considering spoken content, our method can compute content-based similarity based on speech recognition results of podcast episodes even if the recognition results include some errors. To overcome those errors, it converts intermediate speech-recognition results to a confusion network containing competitive candidates, and then computes the similarity by using keywords extracted from the network. Experimental results with episodes that have different word accuracy and content showed that keywords obtained from competitive candidates were useful in retrieving similar episodes. To show relevant episodes, our method will be incorporated into PodCastle, a public web service that provides full-text searching of podcasts on the basis of speech recognition.

In this paper we present a presentation training system that observes a presentation rehearsal and provides the speaker with recommendations for improving the delivery of the presentation, such as to speak more slowly and to look at the audience. Our system "Presentation Sensei" is equipped with a microphone and camera to analyze a presentation by combining speech and image processing techniques. Based on the results of the analysis, the system gives the speaker instant feedback with respect to the speaking rate, eye contact with the audience, and timing. It also alerts the speaker when some of these indices exceed predefined warning thresholds. After the presentation, the system generates visual summaries of the analysis results for the speaker's self-examinations. Our goal is not to improve the content on a semantic level, but to improve the delivery of it by reducing inappropriate basic behavior patterns. We asked a few test users to try the system and they found it very useful for improving their presentations. We also compared the system's output with the observations of a human evaluator. The result shows that the system successfully detected some inappropriate behavior. The contribution of this work is to introduce a practical recognition-based human training system and to show its feasibility despite the limitations of state-of-the-art speech and video recognition technologies.

In this paper, we describe a public web service, “PodCastle”, that provides full-text searching of Japanese podcasts on the basis of automatic speech recognition. This is an instance of our research approach, “Speech Recognition Research 2.0”, which is aimed at providing users with a web service based on Web 2.0 so that they can experience state-of-the-art speech recognition performance, and at promoting speech recognition technologies in cooperation with anonymous users. PodCastle enables users to find podcasts that include a search term, read full texts of their recognition results, and easily correct recognition errors. The results of the error correction can then be used to improve the performance of both full-text search and speech recognition. Although we know of no state-of-the-art speech recognizer that can successfully transcribe all of the various kinds of podcasts, the mechanism we propose will gradually increase the usefulness and applicability of PodCastle. Index Terms: information retrieval, speech recognition, error correction, wisdom of crowds, Web 2.0

This paper describes speech recognition techniques that enable a Web 2.0 service “PodCastle ” where users can search and read transcribed texts of podcasts, and correct recognition errors in those texts. Most previous speech recognizers had difficulties transcribing podcasts because podcasts include various kinds of contents recorded in different conditions and cover recent topics that tend to have many out-of-vocabulary words. To overcome such difficulties, we continuously improve speech recognizers by using information aggregated on the basis of Web 2.0. For example, a language model is adapted to a topic of the target podcast on the fly, the pronunciations of out-of-vocabulary words are obtained from a Web 2.0 service, and an acoustic model is trained by using the results of the error correction by anonymous users. The experiments we report in this paper show that our techniques produce promising results for podcasts.

It is tedious to handwrite long passages of text by hand. To make this process more efficient, we propose predictive handwriting that provides input predictions when the user writes by hand. A predictive handwriting system presents possible next words as a list and allows the user to select one to skip manual writing. Since it is not clear if people are willing to use prediction, we first run a user study to compare handwriting and selecting from the list. The result shows that, in Japanese, people prefer to select, especially when the expected performance gain from using selection is large. Based on these observations, we designed a multimodal input system, called speech-pen, that assists digital writing during lectures or presentations with background speech and handwriting recognition. The system recognizes speech and handwriting in the background and provides the instructor with predictions for further writing. The speech-pen system also allows the sharing of context information for predictions among the instructor and the audience; the result of the instructor's speech recognition is sent to the audience to support their own note-taking. Our preliminary study shows the effectiveness of this system and the implications for further improvements.

This paper describes a system that can automatically synchronize between polyphonic musical audio signals and corresponding lyrics. Although there were methods that can synchronize between monophonic speech signals and corresponding text transcriptions by using Viterbi alignment techniques, they cannot be applied to vocals in CD recordings because accompaniment sounds often overlap with vocals. To align lyrics with such vocals, we therefore developed three methods: a method for segregating vocals from polyphonic sound mixtures, a method for detecting vocal sections, and a method for adapting a speech-recognizer phone model to segregated vocal signals. Experimental results for 10 Japanese popular-music songs showed that our system can synchronize between music and lyrics with satisfactory accuracy for 8 songs

In this paper, we propose a novel speech input interface function, called “Speech Repair ” in which recognition errors can be easily corrected by selecting candidates. During the speech input, this function displays not only the typical speechrecognition result but also other competitive candidates. Each word in the result is separated by line segments and accompanied by other word candidates. A user who finds a recognition error can simply select the correct word from the candidates for that temporal region. In order to overcome the difficulty of generating appropriate candidates, we adopted a confusion network that can condense a huge internal word graph of a large vocabulary continuous speech recognition (LVCSR) system. In our experiments, almost all recognition errors were corrected and the effectiveness of speech repair was confirmed. 1.

For cooperative work of robots and humans in the real world, a communicative function based on speech is indispensable for robots. To realize such a function in a noisy real environment, it is essential that robots be able to extract target speech spoken by humans from a mixture of sounds by their own resources. We have developed a method of detecting and extracting speech events based on the fusion of audio and video information. In this method, audio information (sound localization using a microphone array) and video information (human tracking using a camera) are fused by a Bayesian network to enable the detection of speech events. The information of detected speech events is then utilized in sound separation using adaptive beam forming. In this paper, some basic investigations for applying the above system to the humanoid robot HRP-2 are reported. Input devices, namely a microphone array and a camera, were mounted on the head of HRP-2, and acoustic characteristics for sound localization/separation performance were investigated. Also, the human tracking system was improved so that it can be used in a dynamic situation. Finally, overall performance of the system was tested via off-line experiments.

In communication learning by second language, three abil-ities have to be improved; listening, speaking and writing abilities. In this sense, it is important to evaluate user’s pro-nunciation ability and to detect mispronunciations in En-glish CALL (Computer-Assisted Language Learning) sys-tems. In this paper, we propose three functions (segmenta-tion, phrasing and pronunciation evaluation) in the CALL system using speech recognition technologies. The system was evaluated by the answer to questionnaires for ten learn-ers. 1.