In the literature on deep neural networks, there is considerable interest in developing activation functions that can enhance neural network performance. In recent years, there has been renewed scientific interest in proposing activation functions that can be trained throughout the learning process, as they appear to improve network performance, especially by reducing overfitting. In this paper, we propose a trainable activation function whose parameters need to be estimated. A fully Bayesian model is developed to automatically estimate from the learning data both the model weights and activation function parameters. An MCMC-based optimization scheme is developed to build the inference. The proposed method aims to solve the aforementioned problems and improve convergence time by using an efficient sampling scheme that guarantees convergence to the global maximum. The proposed scheme has been tested across a diverse datasets, encompassing both classification and regression tasks, and implemented in various CNN architectures to demonstrate its versatility and effectiveness. Promising results demonstrate the usefulness of our proposed approach in improving models accuracy due to the proposed activation function and Bayesian estimation of the parameters.

There is a significant focus in deep neural networks on finding activation functions that can enhance network performance. Scientists are also interested in developing flexible activation functions that can adapt during training to avoid overfitting and provide reliable parameter learning.This paper presents a Bayesian algorithm that employs Markov Chain Monte Carlo (MCMC) to estimate activation function parameters and model weights. This algorithm accelerates convergence through effective sampling and evaluates its effectiveness on two datasets, achieving high accuracy with low complexity.