Recent advances in vision technology lead to the establishment of non-verbal communication channels from human behaviors toward understanding their intention. An adaptive vision-based attentive tracker (AVAT) is proposed for isolating such human intended actions from the ordinary walking behavior. The algorithm which drives the attentive tracker is divided into two sub-processes: one is for modeling the movement of human body parts as the environment using hidden Markov models, and the other is for learning the model of the tracker's action using a temporal difference algorithm. In the paper, we describe in detail the integration of the two sub-processes and finally show an experimental result of isolating the human sign action during his natural walking motion for demonstrating the feasibility of our proposal. Identification of the sign context determines the action of the tracker with the simulated rewards supplied.