Knowledge Graphs provide a machine-readable representation of knowledge conforming to graph-based data models. Link prediction methods predict missing facts in incomplete knowledge graphs, often using scalable embedding based solutions that, however, lack comprehensibility which is crucial in many domains. Filling this gap, explanation methods identify supporting knowledge. For evaluating them, user studies are the obvious choice as users are the main recipients of explanations. However, finding domain experts is often challenging. In contrast, an automated approach is to measure the influence of explanations on the very same link prediction task, thus disregarding the perspective of users. Additionally, current evaluation methods vary across different explanation approaches. We propose LP-DIXIT, the first protocol to evaluate the utility of explanations of link predictions. LP-DIXIT is user-aware, algorithmic and unique for different explanation methods. It builds on a typical setting of user studies, but adopts Large Language Models (LLMs) to mimic users. Specifically, it measures how explanations improve the user (LLM) ability to perform predictions, which is key to trust. We experimentally proved an overall agreement between LP-DIXIT and user evaluations. Moreover, we adopted LP-DIXIT to conduct a comparative study of state-of-the-art explanation methods. The outcomes suggest that less is more: the most effective explanations are those consisting of a single fact.