Much research on older people with memory concerns is focused on tracking and informed by the priorities of others. In this paper, we seek to understand the potential that people with memory concerns see in tracking. We conducted interviews with 29 participants with concerns about their memory and engaged in an affective writing approach. We find a range of potentials that can be traced to how participants are already self-tracking. Emotions associated with these potentials vary: from acceptance to resistance, and positive anticipation to aversion. Participants are emotionally motivated to foreclose possibilities in some instances and keep them open in others. While individual and unique, potential is structured by forces that include individual routines, relationships with others, and macro-level institutions and cultural contexts. We reflect on these findings in the context of research on self-tracking with older adults, designing with ambiguity, and forces that structure the experience of living with memory concerns.