MOTIVATION: Accurate model performance on training data does not ensure alignment between the model's feature weighting patterns and human knowledge, which can limit the model's relevance and applicability. We propose Semi-Supervised Data-Integrated Feature Importance (DIFI), a method that numerically integrates a priori knowledge, represented as a sparse knowledge map, into the model's feature weighting. By incorporating the similarity between the knowledge map and the feature map into a loss function, DIFI causes the model's feature weighting to correlate with the knowledge. RESULTS: We show that DIFI can improve the performance of neural networks using two biological tasks. In the first task, cancer type prediction from gene expression profiles was guided by identities of cancer type-specific biomarkers. In the second task, enzyme/non-enzyme classification from protein sequences was guided by the locations of the catalytic residues. In both tasks, DIFI leads to improved performance and feature weighting that is interpretable. DIFI is a novel method for injecting knowledge to achieve model alignment and interpretability. AVAILABILITY AND IMPLEMENTATION: Code and models for our experiments are available at https://github.com/junwkim1/DIFI.