Prior approaches to lead instrument detection primarily analyze mixture audio, limited to coarse classifications and lacking generalization ability. This paper presents a novel approach to lead instrument detection in multitrack music audio by crafting expertly annotated datasets and designing a novel framework that integrates a self-supervised learning model with a track-wise, frame-level attention-based classifier. This attention mechanism dynamically extracts and aggregates track-specific features based on their auditory importance, enabling precise detection across varied instrument types and combinations. Enhanced by track classification and permutation augmentation, our model substantially outperforms existing SVM and CRNN models, showing robustness on unseen instruments and out-of-domain testing. We believe our exploration provides valuable in-sights for future research on audio content analysis in multitrack music settings.

Independent deeply learned matrix analysis (IDLMA) is a state-of-the-art determined audio source separation method based on pretrained deep neural networks (DNNs). Owing to the excellent expression power of DNNs, IDLMA can handle a wider range of sources than conventional source models such as nonegative matrix factorization (NMF). However, owing to its supervised nature, the separation performance of IDLMA often degrades in the presence of timbral mismatches between the training data and the to-be-separated data. In this paper, we propose two source models that encompass the NMF- and DNN-based source models by constructing a prior distribution of the source power spectrogram (product of priors: PoP) on the basis of the product-of-expert concept. Since the NMF-based source model works well for a fully blind situation, the proposed models can handle the timbral mismatch without losing the expression power of DNNs. By introducing the PoP-based source models into IDLMA, we propose IDLMA extensions (PoP-IDLMAs) and derive their efficient parameter estimation algorithms on the basis of the majorization–minimization algorithm. Experimental results demonstrated the effectiveness of the proposed PoP-IDLMAs and that the proposed models greatly improve the source power estimation in frequency bands above 500 Hz.

A differentiable digital signal processing (DDSP) autoencoder is a musical sound synthesizer that combines a deep neural network (DNN) and spectral modeling synthesis. It allows us to flexibly edit sounds by changing the fundamental frequency, timbre feature, and loudness (synthesis parameters) extracted from an input sound. However, it is designed for a monophonic harmonic sound and cannot handle mixtures of harmonic sounds. In this paper, we propose a model (DDSP mixture model) that represents a mixture as the sum of the outputs of multiple pretrained DDSP autoencoders. By fitting the output of the proposed model to the observed mixture, we can directly estimate the synthesis parameters of each source. Through synthesis parameter extraction experiments, we show that the proposed method has high and stable performance compared with a straightforward method that applies the DDSP autoencoder to the signals separated by an audio source separation method.

Multichannel nonnegative matrix factorization (MNMF) is a common blind source separation technique that employs full-rank spatial covariance matrices (SCMs). The full-rank SCMs can simulate reverberant mixing systems where the sources are spatially spread. In conventional MNMF, spectrograms of observed signals are modeled by some types of distribution, e.g., the Gaussian distribution and Student’s t distribution. However, MNMF based on the sub-Gaussian distribution has not been proposed because its cost function is difficult to minimize. In this paper, we address the statistical model extension of MNMF to the sub-Gaussian distribution to improve the source separation accuracy. In the proposed method, the generalized Gaussian distribution is utilized as the sub-Gaussian model. Moreover, to design an auxiliary function for the proposed cost function, we introduce the joint-diagonalizability constraint to SCMs similarly to FastMNMF. Two types of update rule for the proposed MNMF are derived on the basis of the majorization-minimization (MM) and majorization-equalization (ME) algorithms. Since the optimization speed of each parameter affects the source separation performance, we experimentally analyze the best combination of MM- and ME-algorithm-based update rules in the proposed method. Experiments of blind source separation reveal that the proposed MNMF based on the sub-Gaussian model can outperform conventional methods.

Independent deeply learned matrix analysis (IDLMA) is a fast and high-performance method for multichannel audio source separation. IDLMA utilizes the deep neural network inference of source models and the blind estimation of demixing filters based on source independence. In conventional IDLMA, iterative projection (IP) is exploited to estimate the demixing filters. Although IP is a fast algorithm, it sometimes fails to estimate an appropriate solution. This is because IP updates the demixing filters in a sourcewise manner, where only one source model is used for each update, and the update sometimes becomes unstable owing to the specific low-quality source models. In this paper, we first derive a new numerically stable microphone-wise update algorithm that exploits all source model information simultaneously. The microphone-wise update problem cannot be solved by IP; instead, a new type of vectorwise coordinate descent algorithm is introduced. Next, comparison analysis of the proposed microphone-wise update and IP reveals the tradeoff w.r.t. convergence speed and numerical stability. To resolve this tradeoff problem, we propose the automatic selection of update rules on the basis of the likelihood function of observed signals. Finally, experimental results show the efficacy of the proposed IDLMA with the automatic selection of update rules.

In this paper, we address a convolutive blind source separation (BSS) problem and propose a new extended framework of FastMNMF by introducing prior information for joint diagonalization of the spatial covariance matrix model. Recently, FastMNMF has been proposed as a fast version of multichannel nonnegative matrix factorization under the assumption that the spatial covariance matrices of multiple sources can be jointly diagonalized. However, its source-separation performance was not improved and the physical meaning of the joint-diagonalization process was unclear. To resolve these problems, we first reveal a close relationship between the joint-diagonalization process and the demixing system used in independent low-rank matrix analysis (ILRMA). Next, motivated by this fact, we propose a new regularized FastMNMF supported by ILRMA and derive convergence-guaranteed parameter update rules. From BSS experiments, we show that the proposed method outperforms the conventional FastMNMF in source-separation accuracy with almost the same computation time.

Independent low-rank matrix analysis (ILRMA) is a fast and stable method of blind audio source separation. Conventional ILRMAs assume time-variant (super-)Gaussian source models, which can only represent signals that follow a super-Gaussian distribution. In this article, we focus on ILRMA based on a generalized Gaussian distribution (GGD-ILRMA) and propose a new type of GGD-ILRMA that adopts a time-variant sub-Gaussian distribution for the source model. We propose a new update scheme called generalized iterative projection for homogeneous source models (GIP-HSM) and obtain a convergence-guaranteed update rule for demixing spatial parameters by combining the GIP-HSM scheme and the majorization-minimization (MM) algorithm. Furthermore, a new extension of the MM algorithm is proposed for the convergence acceleration by applying the majorization-equalization algorithm to a multivariate case. In the experimental evaluation, we show the versatility of the proposed method, i.e., the proposed time-variant sub-Gaussian source model can be applied to various types of source signal.

Audio source separation is an important problem for many audio applications. Independent low-rank matrix analysis (ILRMA) is a recently proposed algorithm that employs the statistical independence between sources and the low-rankness of the time-frequency structure in each source. As reported in this paper, we have developed a new framework that enables us to introduce a spatial regularization of the demixing matrix in ILRMA. Since the conventional optimization cannot be applied to this regularized ILRMA, we derive a novel approach based on vectorwise coordinate descent, which does not require a step-size parameter and guarantees convergence. In experiments, ILRMA with beamforming-based regularization is evaluated as an application of the proposed framework.

In this paper, to address problems in multichannel music signal separation, we propose a new hybrid method that combines directional clustering and advanced nonnegative matrix factorization (NMF). The aims of multichannel music signal separation technology is to extract a specific target signal from observed multichannel signals that contain multiple instrumental sounds. In previous studies, various methods using NMF have been proposed, but many problems remain including poor separation accuracy and lack of robustness. To solve these problems, we propose a new supervised NMF (SNMF) with spectrogram restoration and a hybrid method that concatenates the proposed SNMF after directional clustering. Via the extrapolation of supervised spectral bases, the proposed SNMF attempts both target signal separation and reconstruction of the lost target components, which are generated by preceding directional clustering. In addition, we experimentally reveal the trade-off between separation and extrapolation abilities and propose a new scheme for adaptive divergence, where the optimal divergence can be automatically changed in each time frame according to the local spatial conditions. The results of an evaluation experiment show that our proposed hybrid method outperforms the conventional music signal separation methods.

In this paper, we propose a new approach for addressing music signal separation based on the generalized Bayesian estimator with automatic prior adaptation. This method consists of three parts, namely, the generalized MMSE-STSA estimator with a flexible target signal prior, the NMF-based dynamic interference spectrogram estimator, and closed-form parameter estimation for the statistical model of the target signal based on higher-order statistics. The statistical model parameter of the hidden target signal can be detected automatically for optimal Bayesian estimation with online target-signal prior adaptation. Our experimental evaluation can show the efficacy of the proposed method.

Minimally invasive surgery (MIS) is commonly used in pediatric operations. This method greatly benefits patients because of the reduced surgical trauma. To perform such surgery smoothly, doctors must be highly skilled. To reduce operating difficulties, a great deal of research on surgical systems have been carried out. However, in some cases, smaller workspaces limit the application of MIS. For example, the workspace of infant congenital esophageal atresia (ICEA) surgery is only around 30×30×30 mm. Until now, most ICEA surgeries have been manually performed with traditional instruments. This paper presents a smart surgical robot (SSR) for ICEA surgery. The robot is composed of two slave arms, each consisting of a positioning manipulator and a surgical tool manipulator. The positioning manipulator uses a selective compliance assembly robot arm (SCARA) and a screw-pair mechanism to achieve translational movement in 3D space, and the surgical tool manipulator uses a “double screw drive + universal joint” structure to allow an omni directional bending motion. During surgery, the surgeon first creates the workspace manually to explore the target esophagus. The SSR system is then applied to perform operation. The configuration of the SSR means it can perform tissue manipulation under endoscopic view in a small workspace. Experimental results show that the endoscopic view permits the SSR system to be operated intuitively and accurately in the target workspace.

A single-channel dereverberation method with low computational complexity is proposed. We introduce the complementary Wiener filter which can suppress a late reverberation during silence intervals via theoretical analysis and numerical calculation. An implementation represents reductions both of memory consumption and operative calculations compared to a conventional method; each reduction is almost by half. Dereverberation performance is evaluated by an experimental simulation using speech signals and measured room impulse responses. The performance under several hundreds of msec of the reverberation time is similar to the conventional method: 6 [dB] reverberation reduction and 3 [dB] improvement of target-to-interference ratio.

In this paper, we provide a theoretical analysis of the amount of musical noise in iterative spectral subtraction, and its optimization method for the least musical noise generation. To achieve high-quality noise reduction with low musical noise, iterative spectral subtraction, i.e., iteratively applied weak nonlinear signal processing, has been proposed. Although the effectiveness of the method has been reported experimentally, there have been no theoretical studies. Therefore, in this paper, we formulate the generation process of musical noise by tracing the change in kurtosis of noise spectra, and conduct a comparison of the amount of musical noise for different parameter settings but the same achieved level of noise attenuation. Furthermore, we theoretically derive the optimal internal parameters that generate no musical noise. It is clarified that to find a fixed point in kurtosis yields the no-musical-noise property. Comparative experiments with commonly used noise reduction methods show the proposed method's efficacy.

In this paper, we provide a new theoretical analysis of the amount of musical noise generated via generalized spectral subtraction based on higher order statistics. Power spectral subtraction is the most commonly used spectral subtraction method, and in our previous study a musical noise assessment theory limited to the power spectral domain was proposed. In this paper, we propose a generalization of our previous theory on spectral subtraction for arbitrary exponent parameters. We can thus compare the amount of musical noise between any exponent domains from the results of our analysis. We also clarify that less musical noise is generated when we choose a lower exponent spectral domain; this implies that there is no theoretical justification for using power/amplitude spectral subtraction.

In this paper, we propose a musical-noise-controllable algorithm for array signal processing with the aim for high-performance and high-quality noise reduction. Recently, many methods of integrating linear microphone array signal processing and nonlinear signal processing for noise reduction have been studied, but these methods often suffer from the problem of musical noise. In the proposed algorithm, channelwise spectral subtraction is applied before adaptive array signal processing. We also introduce a new automatic control algorithm to obtain the subtraction strength parameter used in the spectral subtraction, which depends on the amount of generated musical noise, measured by higher order statistics. We confirm the effectiveness of the proposed algorithm via objective and subjective evaluations.

In this paper, we propose a new blind speech extraction method consisting of a minimum mean-square error short-time spectral amplitude (MMSE STSA) estimator and noise estimation based on independent component analysis (ICA). First, we perform a computer simulation using the artificial noise whose stationarity could be controlled parametrically, and the obtained results indicate that the proposed method is superior to conventional methods, such as blind spatial subtraction array (BSSA) and the original MMSE STSA estimator under the non-point-source and nonstationary noise condition. Finally, we conduct an experiment in an actual railway-station environment, and objective and subjective evaluations to confirm the advantage of the proposed method in the real world.

In this paper, we conduct a theoretical analysis of the amount of musical noise generated via methods of integrating beamforming and spectral subtraction (SS) based on higher-order statistics under the same noise reduction performance condition. In our previous analysis, we did not consider the effect of flooring technique in SS and the fact that the noise reduction performances of the integration methods are not equivalent. Then, in this study, we analyze the amount of generated musical noise with consideration of such problems. As a result of the analysis, it is clarified that an appropriate structure depends on both the parameters of SS and the statistical characteristics of the input signal. Moreover, it is also revealed that a specific structure is proper to reduce the musical noise for almost all cases.

A new blind source extraction method in widespread noise conditions is proposed, which is based on multiple frequency-domain independent component analysis (FDICA) combining projection back and spectral subtraction. In addition, We implement the proposed method to digital signal processor (DSP) for a more realistic real-time operation, and develop a new blind source extraction (BSE) microphone which can extract a target sound in real-time. In this paper, we illustrate and evaluate the proposed method and BSE microphone. And experimental results reveal that the extraction performance of the proposed method are superior to that of conventional methods, and we show the efficacy of microphone.

In this paper, we describe and review our recent development of hands-free speech dialogue system which is used for railway station guidance. In the application at the real railway station, robustness against reverberation and noise is the most essential issue for the dialogue system. To address the problem, we introduce two key techniques in our proposed hands-free system; (a) speech dialogue system construction with real speech database collection and language/acoustic model improvement, and (b) microphone array preprocessing using blind spatial subtraction array which can solve the reverberation-naiveness problem inherent in conventional microphone arrays. The experimental assessment of the proposed dialogue system reveals that our system can provide the recognition accuracy of more than 80% under realistic railway-station conditions.

In this paper, we propose a new ICA-based BSS algorithm including estimation of sources' probability density functions (PDFs) to adapt the nonlinear activation function to various noise conditions. In the proposed method, closed-form second-order ICA is introduced as a computational-cost-efficient preprocessing to extract sources' PDFs, which is beneficial for real-time application. Compared with various type of conventional ICAs, e.g., fixed activation-function type and ML-based type, our proposed algorithm can give a faster and higher convergence. Based on the proposed source-adaptive ICA, we show a real-time noise reduction results under diffuse noise environment. Also we can demonstrate our recently developed hands-free robot spoken dialogue system via real-time ICA.

In this paper, we conduct an analysis for reduction of musical noise in integration method of microphone array signal processing and nonlinear signal processing. In these days, for better noise reduction, integration methods of microphone array signal processing and nonlinear signal processing have been researched. However, non-linear signal processing causes musical noise. Since such musical noise make users uncomfortable, it is desired that musical noise is mitigated. Moreover, in these days, it is reported that higher-order statistics is strongly related with the amount of generated musical noise. Thus, we analyze the integrated method of microphone array signal processing and nonlinear signal processing, based on higher-order statistics. Also, we propose an architecture for reducing musical noise based on the analysis. The effectiveness of the proposed architecture and analysis correctness are shown via a computer simulation and a subjective evaluation.

In this paper, we reveal new findings about the generated musical noise in minimum mean-square error short-time spectral amplitude (MMSE STSA) processing. Recently we have proposed a objective metric of musical noise based on kurtosis change ratio on spectral subtraction (SS). Also we found an interesting relationship among the degree of generated musical noise, the shapes of signal-s probability density function, the strength parameter of SS processing. This paper is aimed to automatically evaluate the sound quality of various types of noise reduction methods using kurtosis change ratio. We give a mathematical analysis based on higher-order statistics viewpoint, and lead to a valuable relation in that MMSE STSA has a weakness in speech period distortion rather than noise period, and vice versa in SS.

We propose a new blind spatial subtraction array (BSSA) consisting of a noise estimator based on independent component analysis (ICA) for efficient speech enhancement. In this paper, first, we theoretically and experimentally point out that ICA is proficient in noise estimation under a non-point-source noise condition rather than in speech estimation. Therefore, we propose BSSA that utilizes ICA as a noise estimator. In BSSA, speech extraction is achieved by subtracting the power spectrum of noise signals estimated using ICA from the power spectrum of the partly enhanced target speech signal with a delay-and-sum beamformer. This ldquopower-spectrum-domain subtractionrdquo procedure enables better noise reduction than the conventional ICA with estimation-error robustness. Another benefit of BSSA architecture is ldquopermutation robustness". Although the ICA part in BSSA suffers from a source permutation problem, the BSSA architecture can reduce the negative affection when permutation arises. The results of various speech enhancement test reveal that the noise reduction and speech recognition performance of the proposed BSSA are superior to those of conventional methods.

In this paper, we describe development and evaluation of handsfree spoken dialogue system which is used for railway station guidance. In the application at the railway station, noise robustness is the most essential issue for the dialogue system. To address the problem, we introduce two key techniques in our proposed hands-free system; (a) blind spatial subtraction array (BSSA) as a preprocessing, which can efficiently reduce nonstationary and diffuse noises in real-time, and (b) robust voice activity detection (VAD) based on speech decoding for further improvement of speech recognition accuracy. The experimental assessment of the proposed dialogue system reveals that the combination of real-time BSSA and robust VAD can provide the recognition accuracy of more than 80% under adverse railway-station noise conditions.

In this paper, we construct a hands-free robot spoken dialogue system based on the real-time blind spatial subtraction array (BSSA) and evaluate the system. BSSA is the blind source extraction method, and the source extraction in BSSA is carried out by subtracting the power spectrum of the estimated noise signal by the independent component analysis from the power spectrum of the target speech partly enhanced signal. Although BSSA can reduce noise signal efficiently, ICA consumes huge amount of computational costs. Thus it is difficult to run BSSA in real-time. In this paper, we newly propose a real-time architecture of BSSA and construct a hands-free robot spoken dialogue system based on the real-time BSSA. In the hands-free robot spoken dialogue system with the real-time BSSA, 6% improvement of the speech recognition result can be seen compared with the conventional speech enhancement methods.

In this paper, we investigate a new blind source separation (BSS) structure from a permutation-robustness viewpoint, to mitigate the permutation problem which commonly arises in frequency-domain independent component analysis (ICA). Permutation robustness means that how much the BSS method is not affected under a certain probability of arising permutation, unlike the conventional permutation-solving approaches. We address to analyze our previously proposed BSS architecture, so called blind spatial subtraction array (BSSA). In BSSA, source extraction is achieved by subtracting the power spectrum of the estimated noise via ICA from the power spectrum of partly speech-enhanced signal via delay-and-sum (DS) procedure. Indeed BSSA partially involves permutation problem in the ICA-based noise estimator part. However, BSSA can efficiently reduce the negative affection of the permutation owing to the over-subtraction in the spectral subtraction and defocusing properties in DS. Experiments using artificial and real-recording-based simulations reveal that the proposed method outperforms the conventional ICA.