This article proposes a new activation function (AF)$\mathrm {sinh}(\beta x) + \mathrm {sinh}^{-1}(\beta x)$called combined hyperbolic Sine (comb-H-sine) to replace existing AFS like$\sigma $and$\tanh $in long short-term memory (LSTM) neural networks. The comb-H-sine function is implemented using purely combinational architectures with a 16-bit data width with fixed-point representation, resulting in improved accuracy. Both software and hardware modeling are used to investigate the proposed architecture. Compared to prior works on$\sigma $and$\tanh $functions, the hardware for the comb-H-sine function shows significant improvements in power consumption, number of cells, cell area, and delay. The proposed LSTM architecture using comb-H-sine outperforms existing AFs in terms of power delay product and accuracy on various datasets.