Convolutional neural networks (CNNs) are among the most widely used machine learning models for computer vision tasks, such as image classification. To improve the efficiency of CNNs, many compression approaches have been developed. Low-rank methods approximate the original convolutional kernel with a sequence of smaller convolutional kernels, leading to reduced storage and time complexities. In this study, we propose a novel low-rank CNN compression method that is based on reduced storage direct tensor ring decomposition (RSDTR). The proposed method offers a higher circular mode permutation flexibility, and it is characterized by large parameter and FLOPS compression rates, while preserving a good classification accuracy of the compressed network. The experiments, performed on the CIFAR-10 and ImageNet datasets, clearly demonstrate the efficiency of RSDTR in comparison to other state-of-the-art CNN compression approaches.

Deep equilibrium (DEQ) models are widely recognized as a memory efficient alternative to standard neural networks, achieving state-of-the-art performance in language modeling and computer vision tasks. These models solve a fixed point equation instead of explicitly computing the output, which sets them apart from standard neural networks. However, existing DEQ models often lack formal guarantees of the existence and uniqueness of the fixed point, and the convergence of the numerical scheme used for computing the fixed point is not formally established. As a result, DEQ models are potentially unstable in practice. To address these drawbacks, we introduce a novel class of DEQ models called positive concave deep equilibrium (pcDEQ) models. Our approach, which is based on nonlinear Perron-Frobenius theory, enforces nonnegative weights and activation functions that are concave on the positive orthant. By imposing these constraints, we can easily ensure the existence and uniqueness of the fixed point without relying on additional complex assumptions commonly found in the DEQ literature, such as those based on monotone operator theory in convex analysis. Furthermore, the fixed point can be computed with the standard fixed point algorithm, and we provide theoretical guarantees of its geometric convergence, which, in particular, simplifies the training process. Experiments demonstrate the competitiveness of our pcDEQ models against other implicit models.

We use fixed point theory to analyze nonnegative neural networks, which we define as neural networks that map nonnegative vectors to nonnegative vectors. We first show that nonnegative neural networks with nonnegative weights and biases can be recognized as monotonic and (weakly) scalable mappings within the framework of nonlinear Perron-Frobenius theory. This fact enables us to provide conditions for the existence of fixed points of nonnegative neural networks having inputs and outputs of the same dimension, and these conditions are weaker than those recently obtained using arguments in convex analysis. Furthermore, we prove that the shape of the fixed point set of nonnegative neural networks with nonnegative weights and biases is an interval, which under mild conditions degenerates to a point. These results are then used to obtain the existence of fixed points of more general nonnegative neural networks. From a practical perspective, our results contribute to the understanding of the behavior of autoencoders, and we also offer valuable mathematical machinery for future developments in deep equilibrium models.

Local damage of bearings can be detected as a weak cyclic and impulsive component in a highly noisy measured signal. A key problem is how to extract the signal of interest (SOI) from the raw signal, i.e., how to identify and design an optimal filter. To tackle this problem, in this article, we propose to use stochastic sampled orthogonal nonnegative matrix factorization for extracting frequency-based features from a spectrogram of the measured signal. The proposed algorithm finds a selective filter that is tailored to the frequency band of the SOI. We show that our approach outperforms the other state-of-the-art selectors that were previously used in condition monitoring. The efficiency of the proposed method is illustrated using both a simulation study and the following real signals: vibration signal from a test rig in the laboratory and acoustic signal from a belt conveyor.

Time-frequency representation (TFR) is often used for non-stationary signal analysis. The most intuitive and interpretable TFR is the spectrogram. Recently, a concept of non-negative matrix factorization (NMF) has been successfully applied to local damage detection in rolling elements of bearings via spectrogram factorization. NMF applied to the spectrogram allows one to find an informative frequency band, which could be further used as a filter characteristic. However, the obtained filter characteristics mostly detect the informative frequency band, which also encompasses a lot of noise. In the case where noise is more problematic, as is the case for acoustic signals from industrial machines, the NMF hardly detects the damage. To solve this problem and obtain more selective filters, which are more robust to noise, we propose the non-negative matrix under-approximation (NMU) as an informative frequency band selector. Due to the more sparse parts-based representation of the NMU compared to NMF, NMU provides more selective filter characteristics, which neglect the non-informative frequency bands related to the noise. In practice, it means that NMU gives a better signal-to-noise ratio for the filtered signal. The efficiency of the proposed approach has been validated on the vibration signal from the test rig and the acoustic signal from an idler.

The topic of convolutional neural networks (CNN) compression has attracted increasing attention as new generations of neural networks become larger and require more and more computing performance. This computational problem can be solved by representing the weights of a neural network with low-rank factors using matrix/tensor decomposition methods. This study presents a novel concept for compressing neural networks using nested low-rank decomposition methods. In this approach, we alternately perform decomposition of the neural network weights with fine-tuning of the network. The numerical experiments are performed on various CNN architectures, ranging from small-scale LeNet-5 trained on the MNIST dataset, through medium-scale ResNet-20, ResNet-56, and up to large-scale VGG-16, VGG-19 trained on the CIFAR-10 dataset. The obtained results show that using the nested compression, we can achieve much higher parameter and FLOPS compression with a minor drop in classification accuracy.