Deep learning has profoundly impacted society, yet the inherent nature of deep neural networks hinders further application to high-reliability industries. To demystify these closed-boxes, numerous works attempt to improve the explainability by observing or impacting internal variables of the models. However, existing methods rely on heuristics without rigorous theoretical foundations, often requiring intricate model modifications or redesigns. This work first formalizes two fundamental properties of explainability: alignment and invertibility, serving as theoretical pillars for rigorous interpretability analysis. Building on these, we introduce Bort, a plug-and-play optimizer that enforces Boundedness and orthogonality constraints on model parameters to improve explainability. These constraints are theoretically derived from the alignment and invertibility principles. Considering conventional optimizers can not leverage data features for precise attribution, we present a data-aware extension, termed DBort, which integrates an auxiliary loss term. Intriguingly, in the linear case, DBort converges to Principal Component Analysis (PCA). Our in-depth analysis of penalty term design reveals that $l_{1}$l1-based penalties provide a more stringent adherence to the imposed constraints compared to their $l_{2}$l2 counterparts. Our experiments involve reconstructing and backtracking through the optimized model representations, which reveal a marked enhancement in explainability. Furthermore, leveraging Bort, we successfully synthesize explainable adversarial examples without additional training. Notably, Bort consistently improves the classification accuracy across diverse architectures, including ResNet and DeiT, on benchmark datasets such as MNIST, CIFAR-10, and ImageNet.