Deep learning (DL) has been adopted in a broad range of Internet-of-Things (IoT) applications such as auto-driving, intelligent healthcare and smart grids, but limitations such as those relating to data and user privacy can complicate its broader implementation. Seeking to jointly address both privacy and utility, in this paper we connect the layer-wise relevance propagation with gradient descent for injecting proper noise into gradients. We also improve the conventional gradient clipping method by dividing the gradients into several groups; thus, minimizing the gradient distortion. Since the noisy gradient causes the undetermined descent direction and might adversely affect the loss minimization, we use the NoisyMin algorithm to select the best step size for each gradient perturbation. Finally, we integrate the adaptive optimizer into the gradient descent. In addition to improving the model utility, we also leverage the leading Sinh-Normal noise addition mechanism to achieve truncated concentrated differential privacy (tCDP) – as demonstrated by our rigorous analysis. Our experimental evaluations also validate the effectiveness of the proposed algorithm.