Memory deficits are prevalent in epilepsy and other brain disorders, significantly affecting quality of life. In particular, patients with mesial temporal lobe epilepsy require ongoing monitoring and repeated memory assessments to track cognitive function, which could benefit from automated transcription tools. We used a classic free recall (FR) verbal memory task, where participants recalled words, to assess their performance by recording, transcribing, and detecting vocalizations of correctly remembered words. Conventional manual speech transcription methods are time-consuming and prone to human error, especially in a noisy hospital environment. To address these limitations, a modified U-Net architecture was employed for noise reduction, resulting in a signal-to-noise ratio (SNR) of 15.8 and a mean squared error (MSE) of 0.0021. We also developed an automated transcription interface utilizing the Whisper speech recognition model, which was fine-tuned for Polish, Czech, and English languages. Dynamic Time Warping (DTW) was applied to provide precise word-level timestamps of vocalization onset and offset. The interface was iteratively refined over eight development cycles, incorporating feedback from target users. Transcription accuracy was evaluated with Word Error Rates (WER) of 10.3% for Czech, 7.1% for Polish, and 5% for English, alongside respective Character Error Rates (CER) of 12%, 10.8%, and 7.5%. Our automated interface outperformed manual transcription, reducing transcription time fourfold and achieving 87.5% user satisfaction. These results demonstrate robust transcription accuracy of the challenging Slavic languages and highlight the potential of automated transcription to streamline speech processing using emerging human-computer interface technologies.

The aim of the research is to analyze the relationship between sound, color, and emotion. For this purpose, a survey application was prepared, enabling the assignment of a color to a given speaker’s/singer’s voice recordings. Subjective tests were then conducted, enabling the respondents to assign colors to voice/singing samples. In addition, a database of voice/singing recordings of people speaking in a natural way and with expressed emotion was prepared, where discrete colors were assigned in subjective tests. These data were used in a machine-learning approach that consisted in searching for the relationship between sound, color, and emotion. Analyses based on correlational analysis and learning algorithms were performed. It occurred that assigning values of naturally sounding and emotionally charged speech/singing parameters to colors (and their parameters) did not enable finding a correlation between the given voice, emotions, and color features. The machine learning model achieved high accuracy in the relation between the generated colors and the colors corresponding to the emotions in the literature and questionnaire results.

The attractiveness of the face plays an important role in everyday life, especially in the modern world where social media and the Internet surround us. In this study, an attempt to assess the attractiveness of a face by machine learning is shown. Attractiveness is determined by three deep models whose sum of predictions is the final score. Two annotated datasets available in the literature are employed for training and testing the algorithms, i.e., a dataset named SCUT-FBP5500 to train the deep learning models to predict facial attractiveness and Face Research Lab London Set designated for the test. The first model pays attention to the dominant background colors in the photo; the second model is based on a pre-trained deep neural network. Finally, for facial proportion assessment, distances between key points on the face are linked with attractiveness ratings, so the last dataset considers face proportions. Several algorithms are trained and tested, including baseline machine learning algorithms, i.e., LinearSVR, SDGRegressor, Lasso, RandomForestRegressor, and deep models, such as Xception VGG19 ResNet50v2, and MobileNetv2. A discussion of the results, as well as some concluding remarks, are also provided. The results from the trained models based on SCUT-FBP5500 show a systematic error for the Face Research Lab London Set database. This is probably caused by a different type of image evaluation in both databases. Although the results obtained show no visible winner among the algorithms employed, the best results are seen for five clusters and five colors fed onto the regressor.

The motivation behind this study lies in adapting acoustic noise monitoring systems for road traffic monitoring for driver's safety. Such a system should recognize a vehicle type and weather-related pavement conditions based on the audio level measurement. The study presents the effectiveness of the selected machine learning algorithms in acoustic-based road traffic monitoring. Bases of the operation of the acoustic road traffic detector are briefly described. Principles of several machine learning algorithms, data acquisition process, and information about the dataset built are explained. The study is conducted using the audio recordings prepared by the authors, registered in several locations and under different meteorological conditions of the road surface. For each recording containing a single-vehicle passage, a vector of 67 parameters extracted from the audio signal is calculated. Fisher Linear Discriminant Analysis and Regression Analysis, the fastest among algorithms employed, return the following values of accuracy: 0.968 and 0.978, precision: 0.919 and 0.853, recall: 0.882 and 0.974, and F1-score: 0.898 and 0.868 for vehicle type classification. In the case of the road pavement conditions, the obtained metrics are as follows: accuracy of 0.933, precision of 0.898, recall of 0.9, and F1-score of 0.884.

The research community has long studied computer-assisted pronunciation training (CAPT) methods in non-native speech. Researchers focused on studying various model architectures, such as Bayesian networks and deep learning methods, as well as on the analysis of different representations of the speech signal. Despite significant progress in recent years, existing CAPT methods are not able to detect pronunciation errors with high accuracy (only 60% precision at 40%–80% recall). One of the key problems is the low availability of mispronounced speech that is needed for the reliable training of pronunciation error detection models. If we had a generative model that could mimic non-native speech and produce any amount of training data, then the task of detecting pronunciation errors would be much easier. We present three innovative techniques based on phoneme-to-phoneme (P2P), text-to-speech (T2S) and speech-to-speech (S2S) conversion to generate correctly pronounced and mispronounced synthetic speech. We show that these techniques not only improve the accuracy of three machine learning models for detecting pronunciation errors, but also help establish a new state-of-the-art in the field. Earlier studies have used simple speech generation techniques such as P2P conversion, but only as an additional mechanism to improve the accuracy of pronunciation error detection. We, on the other hand, consider speech generation to be the first-class method of detecting pronunciation errors. The effectiveness of these techniques is assessed in the tasks of detecting pronunciation and lexical stress errors. Non-native English speech corpora of German, Italian, and Polish speakers are used in the evaluations. The best proposed S2S technique improves the accuracy of detecting pronunciation errors in AUC metric by 41% from 0.528 to 0.749 compared to the state-of-the-art approach.

A common approach to the automatic detection of mispronunciation in language learning is to recognize the phonemes produced by a student and compare it to the expected pronunciation of a native speaker. This approach makes two simplifying assumptions: a) phonemes can be recognized from speech with high accuracy, b) there is a single correct way for a sentence to be pronounced. These assumptions do not always hold, which can result in a significant amount of false mispronunciation alarms. We propose a novel approach to overcome this problem based on two principles: a) taking into account uncertainty in the automatic phoneme recognition step, b) accounting for the fact that there may be multiple valid pronunciations. We evaluate the model on non-native (L2) English speech of German, Italian and Polish speakers, where it is shown to increase the precision of detecting mispronunciations by up to 18% (relative) compared to the common approach.

This paper describes two novel complementary techniques that improve the detection of lexical stress errors in non-native (L2) English speech: attention-based feature extraction and data augmentation based on Neural Text-To-Speech (TTS). In a classical approach, audio features are usually extracted from fixed regions of speech such as the syllable nucleus. We propose an attention-based deep learning model that automatically derives optimal syllable-level representation from frame-level and phoneme-level audio features. Training this model is challenging because of the limited amount of incorrect stress patterns. To solve this problem, we propose to augment the training set with incorrectly stressed words generated with Neural TTS. Combining both techniques achieves 94.8% precision and 49.2% recall for the detection of incorrectly stressed words in L2 English speech of Slavic and Baltic speakers.

We propose a weakly-supervised model for word-level mispronunciation detection in non-native (L2) English speech. To train this model, phonetically transcribed L2 speech is not required and we only need to mark mispronounced words. The lack of phonetic transcriptions for L2 speech means that the model has to learn only from a weak signal of word-level mispronunciations. Because of that and due to the limited amount of mispronounced L2 speech, the model is more likely to overfit. To limit this risk, we train it in a multi-task setup. In the first task, we estimate the probabilities of word-level mispronunciation. For the second task, we use a phoneme recognizer trained on phonetically transcribed L1 speech that is easily accessible and can be automatically annotated. Compared to state-of-the-art approaches, we improve the accuracy of detecting word-level pronunciation errors in AUC metric by 30% on the GUT Isle Corpus of L2 Polish speakers, and by 21.5% on the Isle Corpus of L2 German and Italian speakers.

This paper proposed a novel approach for the detection and reconstruction of dysarthric speech. The encoder-decoder model factorizes speech into a low-dimensional latent space and encoding of the input text. We showed that the latent space conveys interpretable characteristics of dysarthria, such as intelligibility and fluency of speech. MUSHRA perceptual test demonstrated that the adaptation of the latent space let the model generate speech of improved fluency. The multi-task supervised approach for predicting both the probability of dysarthric speech and the mel-spectrogram helps improve the detection of dysarthria with higher accuracy. This is thanks to a low-dimensional latent space of the auto-encoder as opposed to directly predicting dysarthria from a highly dimensional mel-spectrogram.

A method for assessing separability of EEG signals associated with three classes of brain activity is proposed. The EEG signals are acquired from 23 subjects, gathered from a headset consisting of 14 electrodes. Data are processed by applying Discrete Wavelet Transform (DWT) for the signal analysis and an autoencoder neural network for the brain activity separation. Processing involves 74 wavelets from 3 DWT families: Coiflets, Daubechies and Symlets. Euclidean distance between clusters normalized with respect to the standard deviation of the whole set of data are used to separate each task performed by participants. The results of this stage allow for an assessment of separability between subsets of data associated with each activity performed by experiment participants. The speed of convergence of the training process employing deep learning-based clustering is also measured.

This study addresses two issues related to both machine- and subjective-based speech evaluation by investigating five phonological phenomena related to allophone production. Its aim is to use objective parametrization and phonological classification of the recorded allophones. These allophones were selected as specifically difficult for Polish speakers of English: aspiration, final obstruent devoicing, dark lateral /l/, velar nasal and pre-fortis clipping. A set of audio features based on mechanism of each phonological process was created. Recordings of phonetic material prepared by phonology expert were executed. First, several speakers were recorded while reading words from a teleprompter. Then, every word was played back from the previously recorded sample read by a phonology expert and each examined speaker repeated a particular word trying to imitate correct pronunciation. The next step consisted in partitioning by editing two recorded sets of words into allophones, then signals were analyzed and subsequently audio excerpts were parametrized. The comparison of two sets of allophones was reinforced by the phonology expert's assessment of produced speech sounds. Analyses presented in this paper allowed for discovering a set of parameters, which enable to determine whether the target processes were pronounced correctly.

The aim of this article is to investigate whether separating music tracks at the pre-processing phase and extending feature vector by parameters related to the specific musical instruments that are characteristic for the given musical genre allow for efficient automatic musical genre classification in case of database containing thousands of music excerpts and a dozen of genres. Results of extensive experiments show that the approach proposed for music genre classification is promising. Overall, conglomerating parameters derived from both an original audio and a mixture of separated tracks improve classification effectiveness measures, demonstrating that the proposed feature vector and the Support Vector Machine (SVM) with Co-training mechanism are applicable to a large dataset.

A review of available audio-visual speech corpora and a description of a new multimodal corpus of English speech recordings is provided. The new corpus containing 31 hours of recordings was created specifically to assist audio-visual speech recognition systems (AVSR) development. The database related to the corpus includes high-resolution, high-framerate stereoscopic video streams from RGB cameras, depth imaging stream utilizing Time-of-Flight camera accompanied by audio recorded using both: a microphone array and a microphone built in a mobile computer. For the purpose of applications related to AVSR systems training, every utterance was manually labeled, resulting in label files added to the corpus repository. Owing to the inclusion of recordings made in noisy conditions the elaborated corpus can also be used for testing robustness of speech recognition systems in the presence of acoustic background noise. The process of building the corpus, including the recording, labeling and post-processing phases is described in the paper. Results achieved with the developed audio-visual automatic speech recognition (ASR) engine trained and tested with the material contained in the corpus are presented and discussed together with comparative test results employing a state-of-the-art/commercial ASR engine. In order to demonstrate the practical use of the corpus it is made available for the public use.

Mood of music is considered as one of the most intuitive criteria for listeners, thus this work is focused on the emotional content of music and its automatic recognition.The research study presented in this work contains an attempt to music emotion recognition including audio parameterization and rough sets.A music set consisting of 154 excerpts from 10 music genres was evaluated in the listening experiment.This may be treated as a ground truth.The results achieved indicated a strong correlation between subjective results and objective descriptors and on that basis a vector of parameters related to mood of music was created.On the other hand, rough set-based processing was applied to derive reducts containing the most promising features in the context of mood recognition, as well as confusion matrices of the mood recognition.Both approaches indicate strong relationship between objective descriptors and subjective evaluation of mood of music.

This paper aims at presenting a multimedia system providing polysensory training for pupils with educational difficulties. The particularly interesting aspect of the system lies in the sonic interaction with image projection in which sounds generated lead to stimulation of a particular part of the human brain. The system architecture, video processing methods, therapeutic exercises and guidelines for children’s interaction with the system are presented. Results of pupils’ improvements after several weeks of exercising with the system are provided. The outcome of this study suggests that learning and developing through the interactive method helped to improve children’s spatial orientation skills.

The aim of this paper is to assess the robustness of the non-blind audio content watermarking scheme proposed by the authors. The authors present the architecture of the designed system along with the employed workflows for embedding and extracting the watermark followed by the implementation phase description and the analysis of the experimental results. Some possible attack simulations on the embedded watermarks are reviewed, and the robustness of the proposed approach is evaluated in the context of the influence of lossy compression on the watermark degradation. Subjective and objective analyses are performed for the algorithm proposed by the authors and compared with the Audio Watermarking Tools (AWT) encoder. Finally, the advantages and drawbacks of the proposed approach are debated followed by the conclusion section outlining possible improvements to the proposed method.

An approach to the knowledge representation extraction from biomedical signals analysis concerning motor activity of Parkinson disease patients is proposed in this paper. This is done utilizing accelerometers attached to their body as well as exploiting video image of their hand movements. Experiments are carried out employing artificial neural networks and support vector machine to the recognition of characteristic motor activity disorders in patients. Obtained results indicate that it is possible to interpret some selected patient's body movements with a sufficiently high effectiveness.

This study aims to create an algorithm for assessing the degree to which songs belong to genres defined a priori. Such an algorithm is not aimed at providing unambiguous classification-labelling of songs, but at producing a multidimensional description encompassing all of the defined genres. The algorithm utilized data derived from the most relevant examples belonging to a particular genre of music. For this condition to be met, data must be appropriately selected. It is based on the fuzzy logic principles, which will be addressed further. The paper describes all steps of experiments along with examples of analyses and results obtained.

The role of computers in school education is briefly discussed. Multimodal interfaces development history is shortly reviewed. Examples of applications of multimodal interfaces for learners with special educational needs are presented, including interactive electronic whiteboard based on video image analysis, application for controlling computers with facial expression and speech stretching audio interface representing audio modality. Intelligent and adaptive algorithms applications to the developed multimodal interfaces are discussed.

In the paper problems related to the classification of singing voice quality are presented. For this purpose a database consisting of singers' sample recordings is constructed and parameters are extracted from recorded voice of trained and untrained singers. The parameterization process is based on both voice source and formant analysis of a singing voice. These parameters are explained as to their physical interpretation and analyzed statistically in order to diminish their number. The statistical analysis is based on the Fisher statistic. In such a way a feature vector of a singing voice is formed. Decision systems based on neutral networks and rough sets are utilized in the context of the voice type and voice quality classification. Results obtained in the automatic classification performed by both decision systems are compared. A possibility to classify automatically type/quality of voice is judged. The methodology proposed provides means for discerning trained and untrained singers.

Abstract There are many problems in the musical signal domain that have not been solved yet. Among such problems are automatic recognition and editing of musical sound patterns, retrieval of audio material, detection of transient states, and articulation features in sounds. It seems that the key challenge is in building inexpensive browsers to retrieve audio material contained in multimedia bases and Internet sites. This overview shows motivation and intermediate and long‐term goals of the research projects that have conducted for several years in the Multimedia Systems Department, Gdansk University of Technology, Poland.

The aim of this paper is to present solutions related to identifying musical data. These are discussed mainly on the basis of experiments carried out at the Multimedia Systems Department, Gdansk University of Technology, Gdansk, Poland. The topics presented in this paper include automatic recognition of musical instruments and separation of duet sounds. The classification process is shown as a three-layer process consisting of pitch extraction, parametrization, and pattern recognition. These three stages are discussed on the basis of experimental examples. Artificial neural networks (ANNs) are employed as a decision system and they are trained with a set of feature vectors (FVs) extracted from musical sounds recorded at the Multimedia Systems Department. The frequency envelope distribution (FED) algorithm is presented, which was introduced to musical duet separation. For the purpose of checking the efficiency of the FED algorithm, ANNs are also used. They are tested on FVs derived from musical sounds after the separation process is performed. The experimental results are shown and discussed.

The engineered Multimedia Hearing Aid Fitting Expert System is the experimental software that allows one to find the characteristics of a hearing aid matching patients' needs and to choose automatically suitable hearing device characteristics. The key issues related to the engineered application are based on the expert system implementation. This expert system uses both fuzzy logic and rough set processing of analytical data. The principles of the engineered expert media application and some details of the rough set and fuzzy logic implementation will be presented in this article. © 2002 John Wiley & Sons, Inc.

New concepts of echo cancellation and reduction of non-stationary noise affecting audio signals transmitted in telecommunication channels are proposed in the paper. In the both cases, some methods originated form artificial intelligence domain, i.e.: genetic algorithms, neural networks, rough sets are applied. Moreover, in the noise reduction method, some features of the human auditory system are exploited. A number of experiments have been carried out, and a brief discussion on some of them is included.

This paper deals with the problem of receiving of a desired signal in noisy or "cocktail-party" conditions. This problem is vital in many domains, such as communications, multimedia (multimodal interaction), speech recognition, and psychoacoustics (hearing prostheses). It can be partially solved by classical filtering techniques, however these techniques often introduce distortions into the filtered signal. Spatial filtration can be performed based on an artificial neural network (ANN). Such an algorithm is elaborated, and some details concerning its implementation are described. Moreover, results of experiments are presented. These results demonstrate that an ANN-based nonlinear filter increases the signal-to-noise ratio and improves speech intelligibility.

The development of advanced hearing prosthetic aids and cochlear implants allows many hearing impaired people to receive audio signals. A multimedia catalogue of hearing impairments was developed which includes the results of audiology tests, medical description of the subjects with different types of hearing impairments and sound examples illustrating consequences of a hearing loss. Moreover, within the framework of research projects some digital signal processing applications were developed for hearing prostheses, such as a digital waveguide model of hearing aid acoustical parts and a neural network-based beamformer. Another multimedia database was created in order to facilitate the diagnosis of hearing nerve sensitivity on the basis of electrostimulation tests. Some results of the above multimedia signal processing applications are shown.