We believe that analyzing attention is crucial for understanding bias in large language models (LLMs); in ambiguous comparative prompting frameworks, it provides insight into how the LLM distributes its focus across different entities, and how this contributes to biased decisions.To this end, we first introduce a metric to quantify the "entity preference" of an LLM.We then propose ATLAS, a technique to localize bias to specific layers of the LLM by analyzing attention scores and then reduce bias by scaling attention in these biased layers.To evaluate our method, we conduct extensive experiments across 3 datasets, 4 models, and 4 baseline approaches.Our experiments demonstrate that bias is concentrated in the later layers, typically around the last third.We also show how ATLAS effectively mitigates bias through targeted interventions without compromising downstream performance and an average increase of only 0.34% in perplexity when the intervention is applied.We see an average improvement of 0.28 points in the bias score across all the datasets.