Although the shapes of the parameters are not crucial for designing first-order optimization methods in large scale empirical risk minimization problems, they have important impact on the size of the matrix to be inverted when developing second-order type methods. In this article, we propose an efficient and novel second-order method based on the parameters in the real matrix space [Formula: see text] and a matrix-product approximate Fisher matrix (MatFisher) by using the products of gradients. The size of the matrix to be inverted is much smaller than that of the Fisher information matrix in the real vector space [Formula: see text]. Moreover, by utilizing the matrix delayed update and the block diagonal approximation techniques, the computational cost can be controlled and is comparable with first-order methods. A global convergence and a superlinear local convergence analysis are established under mild conditions. Numerical results on image classification with ResNet50, quantum chemistry modeling with SchNet, and data-driven partial differential equations solution with PINN illustrate that our method is quite competitive to the state-of-the-art methods.

With the rapid progress of the industrial Internet of Things (IIoT), reducing data uncertainty has become a critical issue in predicting the development trends of systems and formulating future maintenance strategies. This article proposes an end-to-end, deep hybrid network-based, short-term, multivariate time-series prediction framework for industrial processes. First, the maximal information coefficient is adopted to extract the nonlinear variate correlation features. Second, a convolutional neural network with a residual elimination module is designed to eliminate data uncertainty. Third, a bidirectional gated recurrent unit network is connected in a time-distributed form to achieve step-ahead prediction. Last, an optimized Bayesian optimization method is adopted to optimize the model's learning rate. A comparison with other state-of-the-art, deep learning-based, time-series prediction methods in the case study illustrates the superiority of the proposed framework in noisy IIoT environments.

In this paper, we consider stochastic second-order methods for minimizing a finite summation of nonconvex functions. One important key is to find an ingenious but cheap scheme to incorporate local curvature information. Since the true Hessian matrix is often a combination of a cheap part and an expensive part, we propose a structured stochastic quasi-Newton method by using partial Hessian information as much as possible. By further exploiting either the low-rank structure or the kronecker-product properties of the quasi-Newton approximations, the computation of the quasi-Newton direction is affordable. Global convergence to stationary point and local superlinear convergence rate are established under some mild assumptions. Numerical results on logistic regression, deep autoencoder networks and deep convolutional neural networks show that our proposed method is quite competitive to the state-of-the-art methods.