This paper describes the problems and issues involved in developing artificial visual agents that watch moving objects and their interactions in real world situations. One objective of this work is to form conceptual descriptions that capture the behaviours of objects. To do this we use the dynamic scene context but here we go further and incorporate task context and simple learning of behavioural cues. The computational approach uses results from the VIEWS project. The issues concerned with extending computational vision to learn behavioural models and use attention are described for a surveillance system with "task-level control". This means that the visual processing is guided by both the current scene knowledge and the current surveillance task control policy.