Heating, ventilation, and air-conditioning (HVAC) system’s supervisory control is crucial for energy-efficient thermal comfort in buildings. The control logic is usually specified as ‘if-then-that-else’ rules that capture the domain expertise of HVAC operators, but they often have conflicts that may lead to sub-optimal HVAC performance. We propose EImprove, a reinforcement-learning (RL) based framework that exploits these conflicts to learn a resolution policy. We evaluate EImprove through a co-simulation strategy involving EnergyPlus simulations of a real-world office setting and a formal requirement specifier. Our experiments show that EImprove learns 75% faster than a pure RL framework.