As industrial control systems become increasingly interconnected with information networks, attackers could exploit vulnerabilities across different system layers to create complex exploit chains to compromise field control elements. As such, security decision-making is of essential importance to maintain the operational security of critical industrial infrastructures. In this paper, we consider the problem of designing cost-effective defense strategies to minimize the risk of successful attack paths. To this end, we propose a non-Markovian security game framework on labeled attack graphs to simulate the attack-defense process in industrial control systems. Compared with existing methods, where the cost of exploiting a vulnerability is considered constant, we consider a more dynamic and realistic case where the exploitation cost is discounted with the number of exploitations. Moreover, a state-decomposition based multi-agent reinforcement learning algorithm is developed to obtain the Nash equilibrium of the proposed non-Markovian security game. A case study on a simulated industrial control system is presented to illustrate the feasibility of the proposed approach. The results demonstrate that the discounting exploitation cost could greatly alter the attack and subsequently the defense strategies. In comparison to traditional static intrusion response approaches, our non-Markovian approach offers a more realistic and adaptive framework to anticipate evolving attack paths and allocate defense resources.