Text-based audio generation models have limitations as they cannot encompass all the information in audio, leading to restricted controllability when relying solely on text. To address this issue, we propose a novel model that enhances the controllability of existing pre-trained text-to-audio models by incorporating additional conditions including content (timestamp) and style (pitch contour and energy contour) as supplements to the text. This approach achieves fine-grained control over the temporal order, pitch, and energy of generated audio. To preserve the diversity of generation, we employ a trainable control condition encoder that is enhanced by a large language model and a trainable Fusion-Net to encode and fuse the additional conditions while keeping the weights of the pre-trained text-to-audio model frozen. Due to the lack of suitable datasets and evaluation metrics, we consolidate existing datasets into a new dataset comprising the audio and corresponding conditions and use a series of evaluation metrics to evaluate the controllability performance. Experimental results demonstrate that our model successfully achieves fine-grained control to accomplish controllable audio generation.

Learning meaningful frame-wise features on a partially labeled dataset is crucial to semi-supervised sound event detection. Prior works either maintain consistency on frame-level predictions or seek feature-level similarity among neighboring frames, which cannot exploit the potential of unlabeled data. In this work, we design a Local and Global Consistency (LGC) regularization scheme to enhance the model on both label- and feature-level. The audio CutMix is introduced to change the contextual information of clips. Then, the local consistency is adopted to encourage the model to leverage local features for frame-level predictions, and the global consistency is applied to force features to align with global prototypes through a specially designed contrastive loss. Experiments on the DESED dataset indicate the superiority of LGC, surpassing its respective competitors largely under the same settings. Besides, combining LGC with existing methods can obtain further improvements. The code is available at https://github.com/Ming-er/LGC-SED.

The recently proposed Mean Teacher method, which exploits large-scale unlabeled data in a self-ensembling manner, has achieved state-of-the-art results in several semi-supervised learning benchmarks.Spurred by current achievements, this paper proposes an effective Couple Learning method 1 that combines a well-trained model and a Mean Teacher model.The suggested pseudo-labels generated model (PLG) increases strongly-and weakly-labeled data to improve the Mean Teacher method's performance.Moreover, the Mean Teacher's consistency cost reduces the noise impact in the pseudo-labels introduced by detection errors.The experimental results on Task 4 of the DCASE2020 challenge demonstrate the superiority of the proposed method, achieving about 44.25% F1-score on the validation set, significantly outperforming the baseline system's 32.39%.furthermore, this paper also propose a simple and effective experiment called the Variable Order Input (VOI) experiment, which proves the significance of the Couple Learning method.Our developed Couple Learning code is available on GitHub.

In real scenarios such as offices, the fine tuning of indoor temperature is crucial to achieve people’s thermal comfort. Micro-expressions are unconscious and spontaneous movements of subtle facial muscles which typically reflects the real feelings of human beings. In this paper, we put forward an algorithm to detect thermal sensation using micro-expression recognition (ME-TSRN). At first, the spatio-temporal features of facial action units (AUs) are extracted, and an attention module is used to generate weights for each AU, and weight features during feature aggregation. We then establish the first micro-expression dataset for thermal comfort analysis (METC), which combines micro-expression and thermal comfort and including 510 micro-expression video samples, the corresponding thermal sensation label, the AU label, and a variety of environmental information and physical signs information. Our ME-TSRN algorithm achieves 70.37% accuracy and 64.62% Fl-score on the METC dataset.

The accuracy of Automatic Speech Recognition (ASR) is critical to speech-based products, such as subtitling, speech translation, spoken dialogue. We aim to enhance ASR accuracy by correcting errors in ASR hypothesis with sequence-to-sequence (seq2seq) models. In this paper, we propose to boost Transformer-based ASR error correction by fusing the pre-trained BERT [1] in encoder and copying mechanism in decoder, which exploits externally well-learned token representation and copying correct tokens in ASR transcript respectively. In addition, we leverage Text-to-Speech (TTS) synthesized data and ASR 5-best hypotheses to augment the training data and make the data more diverse. We evaluate our approach on two internal test sets and two public ASR test sets. Experimental results show that the proposed approach decreases the average Character Error Rate (CER) from 9.36% to 7.30% compared with ASR hypothesis without correction, and outperforms Transformer-based model by a large margin.

The Recurrent Neural Network Transducer (RNN-T) extends Connectionist Temporal Classification (CTC) by jointly modeling both input-output and output-output dependencies, and it has been successfully applied in end-to-end speech recognition. RNN-T adds language information by expanding its dimensions in this way, but the model training is difficult. And some paths of the model are unreasonable when decoding. Therefore, we present the Alignment RNN Transducer (ART) algorithm, which uses the forward-backward algorithm in CTC to find the best path alignment information as context-related information for training. We not only reduce the training dimensions but also adds context information to the model. And we still use CTC as the loss function to avoid decoding the impossible path in the RNN-T. We verify the algorithm on a Mandarin speech corpus AIShell-1, and it achieves a 13.65% CER, compared with the 16.79% CER given by the RNN-T model.

It is increasingly important to recognize daily activity pattern in order to detect early sign of dementia in the aging society. This paper shows indoor-outdoor activity recognition using only a smartphone. We developed an indoor living activity recognition engine and an outdoor migration activity recognition engine, and combined them into an Android™ application. The former recognizes not only "resting" or "walking" but also user's various living activities such as "vacuuming" and "brushing teeth" by using a built-in accelerometer and a built-in microphone. The latter engine recognizes the user's means of migration, namely, "resting," "walking," "running," and "boarding" by using a built-in accelerometer. It enables users to continuously recognize indoor-outdoor activities by switching between the two engines depending on an acquisition condition of GPS.

It is increasingly important to recognize daily activity pattern in order to detect early sign of dementia in the aging society. This paper shows indoor-outdoor activity recognition using only a smartphone. We developed an indoor living activity recognition engine and an outdoor migration activity recognition engine, and combined them into an Android™ application. The former recognizes not only "resting" or "walking" but also user's various living activities such as "vacuuming" and "brushing teeth" by using a built-in accelerometer and a built-in microphone. The latter engine recognizes the user's means of migration, namely, "resting," "walking," "running," and "boarding" by using a built-in accelerometer. It enables users to continuously recognize indoor-outdoor activities by switching between the two engines depending on an acquisition condition of GPS satellites.

This paper describes a new sleep monitoring system for home use. The basic system consists of a wearable physiological sensor and the PC software for analyzing sleep quality from user's wrist motion and heart rate variability. Different from a conventional sleep monitoring device used in a hospital, the sensor is so small and easy-to-use that a normal person can use it at home. This means that the system is useful not only for a sleep specialist who wants to check his/her patient's daily sleep pattern, but also is useful for self-care. The concepts of the tele-care system and the real-time control of electric appliances for creating better sleep environment are also mentioned.