Knowledge graphs are commonly used as sources of information in commonsense question answering, and can also be used to express explanations for the model's answer choice.A common way of incorporating facts from the graph is to encode them separately from the question, and then combine the two representations to select an answer.In this paper, we argue that highly faithful graph-based explanations cannot be extracted from existing models of this type.Such explanations will not include reasoning done by the transformer encoding the question, so will be incomplete.We confirm this theory with a novel proxy measure for faithfulness and propose two architecture changes to address the problem.Our findings suggest a path forward for developing architectures for faithful graph-based explanations.