We describe the design of a task planning and world modelling mechanism as part of a supervisory or teleassistance control scheme for robots operating in unstructured environments. The mechanism addresses the key difficulties of an unstructured environment which lead to failures in task execution. Reactivity is achieved by mixing planning and execution. Complexity of task and error recovery is managed by hierarchical design of action sequences. The world model and task planner are intimately linked to allow parameterization of actions, and for the operator to focus on involved objects in the world. They work as part of a functional architecture with motion planning, dynamic control and sensor fusion processes, and have been implemented to specify the activity of a pair of underwater manipulators, sensed using force/torque loadcell, joint potentiometers, high frequency sonar and laser striping.