Autonomous Underwater Vehicles (AUVs) have been widely used in scientific or industrial operations over the past years. AUV missions require the vehicle to perform a set of actions autonomously and return to the operators. These missions are currently mostly planned offline by the human operators. Existing solutions in commercial planning software usually only provide an estimate of the mission execution time. The uncertain and dynamic underwater environment can have an effect on the mission performance. More time and energy may be required, disallowing successful mission execution. This work proposes the usage of the correlated orienteering problem (COP) that maximises the utility of a sensing mission while respecting energy and time constraints. We propose a heuristic-based on genetic algorithms (GA) for the solution of the COP. This heuristic is compared against optimal mixed-integer quadratic programming (MIQP) solutions. Results show that the quality of the heuristic solution is in the worst tested case 5.5% less than the 1% optimal solutions. The heuristic proves to be at least 3 times more time efficient than the optimal MIQP solutions in the worst case. The heuristic is finally tested on an embedded platform showing its ability to be used on real robotic platforms.

Autonomous Underwater Vehicles (AUVs) are required to carry out a mission with minimum supervision. Often, the AUV's hardware integrity is compromised amidst operation; thus, jeopardising the mission's success. Thruster failures, for example, may affect AUVs locomotion. Following a thruster failure, the plan may require changes to compensate, if possible, for the loss of mobility. In this paper, we present an algorithm that identifies thruster failures in run-time. Moreover, the algorithm corrects the vehicle's dynamical model to incorporate the defective thruster. The algorithm uses a Mixture of Gaussians representation for the vehicle's state. Variational Bayes Approximation has been utilised to yield the filtering equations. As indicated by experimental evaluation, the algorithm detects thruster-failure events correctly; and, in turn, learns an accurate dynamical model of the vehicle at its current state. Experiments were carried out on a real platform in a wave tank at Heriot-Watt University.

This paper introduces an adaptive tuning method for the controllers of a 4 degrees-of-freedom autonomous underwater vehicle. The proposed scheme consists of two control loops, one for position control and an inner one for velocity control. The gains of the controller are determined on-line, according to the position/velocity errors. Using the proposed adaptive architecture, the uncertainties in the parameters of the system are addressed and the system is able to operate when hydrodynamic disturbances are present. The complexity of the fixed gain tuning procedure is also greatly decreased for underwater vehicles when the algorithm suggested here is used. Experimental results with the Nessie VII AUV show that the adaptive controller is beneficial for underwater vehicles. Finally it is shown that the current approach reduces the energy consumption of the system.

This paper presents a filtering algorithm for non-linear systems in the case of sensor degradation. The algorithm adapts the relative importance of the sensor measurements, compared to the model predictions, in real time; yielding a filter that is robust to noisy observations and sensor blackouts. The filter is constructed using a Variational Bayes Approximation of the conditional probability distribution of the system's state; i.e., the probability distribution of the state, given the measurements from the sensors. The algorithm is evaluated both in simulation and experimentally on a robotic platform. In the experiments, the sensor measurements from an Autonomous Underwater Vehicle (AUV) are altered artificially. The sensor output is either corrupted with outliers or manually stuck to a constant value; simulating in this fashion a sensor defect. In both cases, the filter reconstructs the robot's state accurately, thus enabling the vehicle to resume with mission execution.

This paper presents the novel theory for performing multi-agent activity recognition without requiring large training corpora. The reduced need for data means that robust probabilistic recognition can be performed within domains where annotated datasets are traditionally unavailable. Complex human activities are composed from sequences of underlying primitive activities. We do not assume that the exact temporal ordering of primitives is necessary, so can represent complex activity using an unordered bag. Our three-tier architecture comprises low-level video tracking, event analysis and high-level inference. High-level inference is performed using a new, cascading extension of the Rao–Blackwellised Particle Filter. Simulated annealing is used to identify pairs of agents involved in multi-agent activity. We validate our framework using the benchmarked PETS 2006 video surveillance dataset and our own sequences, and achieve a mean recognition F-Score of 0.82. Our approach achieves a mean improvement of 17% over a Hidden Markov Model baseline.

Navigation is instrumental in the successful deployment of Autonomous Underwater Vehicles (AUVs). Sensor hardware is installed on AUVs to support navigational accuracy. Sensors, however, may fail during deployment, thereby jeopardizing the mission. This work proposes a solution, based on an adaptive dynamic model, to accurately predict the navigation of the AUV. A hydrodynamic model, derived from simple laws of physics, is integrated with a powerful non-parametric regression method. The incremental regression method, namely the Locally Weighted Projection Regression (LWPR), is used to compensate for un-modeled dynamics, as well as for possible changes in the operating conditions of the vehicle. The augmented hydrodynamic model is used within an Extended Kalman Filter, to provide optimal estimations of the AUV's position and orientation. Experimental results demonstrate an overall improvement in the prediction of the vehicle's acceleration and velocity.

This paper proposes a semantic world model framework for hierarchical distributed representation of knowledge in autonomous underwater systems. This framework aims to provide a more capable and holistic system, involving semantic interoperability among all involved information sources. This will enhance interoperability, independence of operation, and situation awareness of the embedded service-oriented agents for autonomous platforms. The results obtained specifically affect the mission flexibility, robustness, and autonomy. The presented framework makes use of the idea that heterogeneous real-world data of very different type must be processed by (and run through) several different layers, to be finally available in a suited format and at the right place to be accessible by high-level decision-making agents. In this sense, the presented approach shows how to abstract away from the raw real-world data step by step by means of semantic technologies. The paper concludes by demonstrating the benefits of the framework in a real scenario. A hardware fault is simulated in a REMUS 100 AUV while performing a mission. This triggers a knowledge exchange between the status monitoring agent and the adaptive mission planner embedded agent. By using the proposed framework, both services can interchange information while remaining domain independent during their interaction with the platform. The results of this paper are readily applicable to land and air robotics.

In this paper we present a new method for obtaining situation awareness via the automatic recognition of agent behaviours. In contrast to many other approaches, the presented method models different behaviour durations without using a fixed classification window, and does not require a distribution of behaviour durations. We introduce the Variable Window Layered Hidden Markov Model (VW-LHMM) as an extension of the LHMM to specifically address behaviours with irregular duration. We validate our approach by simulating three high-level behaviours within the harbour and coastline security domain. We compare performance against the LHMM and show that our approach provides a 10% improvement in classification accuracy, in addition to earlier classification.

This paper proposes a novel approach for autonomous mission plan recovery for maintaining operability of unmanned underwater vehicles. It combines the benefits of knowledge-based ontology representation, autonomous partial ordering plan repair and robust mission execution. The approach uses the potential of ontology reasoning in order to orient the planning algorithms adapting the mission plan of the vehicle. It can handle uncertainty and action scheduling in order to maximize mission efficiency and minimise mission failures due to external unexpected factors. Its performance is presented in a set of simulated scenarios for different concepts of operations for the underwater domain. The paper concludes by showing the results of a trial demonstration carried out on a real underwater platform. The results of this paper are readily applicable to land and air robotics.

In this paper, a new method for the estimation of seabed elevation maps from side-scan sonar images is presented. The side-scan image formation process is represented by a Lambertian diffuse model, which is then inverted by a multiresolution optimization procedure inspired by expectation-maximization to account for the characteristics of the imaged seafloor region. On convergence of the model, approximations for seabed reflectivity, side-scan beam pattern, and seabed altitude are obtained. The performance of the system is evaluated against a real structure of known dimensions. Reconstruction results for images acquired by different sonar sensors are presented. Applications to augmented reality for the simulation of targets in sonar imagery are also discussed.

Efficient path-planning algorithms are a crucial issue for modern autonomous underwater vehicles. Classical path-planning algorithms in artificial intelligence are not designed to deal with wide continuous environments prone to currents. We present a novel Fast Marching (FM)-based approach to address the following issues. First, we develop an algorithm we call FM* to efficiently extract a 2-D continuous path from a discrete representation of the environment. Second, we take underwater currents into account thanks to an anisotropic extension of the original FM algorithm. Third, the vehicle turning radius is introduced as a constraint on the optimal path curvature for both isotropic and anisotropic media. Finally, a multiresolution method is introduced to speed up the overall path-planning process.

Graphical interfaces allow users to issue commands using pull-down menus, icon toolbars, and keyboard shortcuts. Menus and icon toolbars are easier to learn, whereas keyboard shortcuts are more efficient. It would seem natural for users to migrate from the use of easy-to-learn menu and icon methods to the more efficient method of keyboard shortcuts as they gain experience. To investigate the extent to which this transition takes place, 251 experienced users of Microsoft Word were given a questionnaire assessing their choice of methods for the most frequently occurring commands. Contrary to our expectations, most experienced users rarely used the efficient keyboard shortcuts, favoring the use of icon toolbars instead. A second study was done to verify that keyboard shortcuts are, indeed, the most efficient method. Six participants performed common commands using menu selection, icon toolbars, and keyboard shortcuts. The keyboard shortcuts were, as expected, the most efficient. These studies show that even experienced users are inefficient in their use of graphical interfaces. One possible way to improve user efficiency is for training programs to provide a roadmap for users to make the transition from using pull-down menus and clicking icon toolbars to issuing keyboard shortcuts.

This article presents a system for predicting the icon a user will select from an icon toolbar, based on command use frequency and mouse trajectory. The system differs from previous systems in 2 important ways: First, the prediction system does not initiate any action. Instead, it predicts where the mouse is moving and subtly "suggests" a command for the user to verify. Second, the system takes into account the relative likelihood of commands being used when making its predictions. Initial testing suggested that a system that only predicted the most frequently used icon choices was better than one that predicted all choices. A study with 12 test users using a mouse and 10 using a trackpad found substantial benefits of this "limited prediction system." The system resulted in a mean reduction in time to issue a command of 41% for trackpad users and 25% for mouse users. Trackpad users, but not mouse users, preferred the prediction system to the traditional way of pointing. These results suggest that a prediction system such as the one described here has the potential to reduce the time and effort required to issue commands. The utility of the system appears to be especially great in laptops and other devices that use trackpads as their primary pointing devices.

Current generation unmanned underwater vehicles, equipped with robotic manipulators, are teleoperated and consequently place a large workload burden on the human operator. A greater degree of automation could improve the efficiency and accuracy with which underwater tasks are carried out. These tasks can involve manipulator motion that is both unconstrained and/or constrained. For unconstrained motion, where a trajectory requires following, a prerequisite is good joint angle control. An adaptive self-tuning pole-placement controller is used for joint angle control. Practical results show the benefits compared to the conventional fixed-gain control. For constrained motion, simultaneous controls of position and force are often required. An adaptive hybrid position/force controller is proposed and compared to a fixed-gain version. Simulation and practical results illustrate the merits and drawbacks of each scheme.

The need for embedding fault diagnosis into goal-orientated autonomous robotic vehicles for increased mission robustness is described. The RECOVERY system, a method for increasing the diagnostic capability by integrating commonly available heterogeneous knowledge is presented. Initial real-water results using the Ocean Systems Laboratory's RAUVER vehicle are given.

Development and integration of subsystems on advanced robots such as unmanned underwater vehicles, can benefit from the availability of a hardware-in-the-loop (HIL) simulation facility. Although complete interoperability of simulated and real subsystems appears desirable, substantial additional complexity of data flows and hardware can be introduced. Where nonreal time simulations are involved, methods of synchronising subsystems running at different speeds must be employed. These should take account of the realities of starting and stopping real subsystem. The paper reviews some of these issues and presents the CORESIM distributed HIL system based around HLA. The system has been used to evaluate a timeslicing synchronisation approach, and to assist in development of docking, visual servoing and concurrent mapping and localisation systems for underwater vehicles.

This paper introduces a 2D visual servoing technique for the station keeping of an unmanned underwater vehicle (UUV) with respect to planar targets on the sea bed. The underwater vehicle is subject to sea current disturbances which make it drift from its desired position. Feature points from unmarked objects are extracted and tracked with a sparse feature tracker developed in-house. The scene depth is estimated from a planar homography. To validate our approach, we emulate the dynamics of the surge and the sway degrees-of-freedom of an UUV model with a planar Cartesian robot in our water test tank. Successful station keeping experiments obtained with a range of sea current disturbances are presented.

A parallel bellows continuum actuator has been developed for use as a finger in a subsea dextrous hand mounted on a commercial underwater hydraulic manipulator. To improve grasping and manipulation performance, a 3D force control system has been designed for each finger, using in-house proprietary force sensor designs. The paper describes the control system design, focusing on the derivation of the continuum actuator Jacobian matrix and kinematic equations ("continuumatics") necessary to implement the multi-dimensional control.

This paper describes a concurrent mapping and localisation (CML) algorithm suitable for localising an autonomous underwater vehicle (AUV). The proposed CML algorithm uses a standard off-the-shelf sonar for sensing the environment. The returns from the sonar are used to detect targets in the vehicle's vicinity. These targets are used in conjunction with a vehicle model by the CML algorithm to concurrently build an absolute map of the environment and localise the vehicle in absolute coordinates. In order for the algorithm to work, the stored targets must be associated to the sonar returns at each iteration. Given the nature of sonar data, false returns complicate this process. The choice of targets and a suitable data association strategy is, therefore, vital. The chosen targets consist of returns of a significant strength. The segmentation detects these targets and calculates (a) the relative position of their center of mass with respect to the vehicle, (b) the targets' surface size, and (c) the targets' first invariant moment. This information is used by the system to perform the data association. We have chosen to adapt the well known multiple hypothesis tracking filter (MHTF) to the CML structure. This is a measurement oriented approach that finds the probability that an established target gave rise to a certain return. The paper presents results with real sonar data.

To investigate the undulating median fin propulsion and its potential for implementation in man-made underwater vehicles, a "fin actuator" has been developed. The device consists of eight parallel bellows actuators or PBAs (fin rays) arranged in a series and interconnected via a flexible material (fin membrane). The PBAs are pneumatically driven and allow for bending movements in any orientation plane. Position data are provided by flexible electrogoniometers. The forces generated from the fin actuator undulations are measured by a pair of tri-axial force sensors. Initial testing has concentrated on 2D fin ray motions, whereby each PBA moves laterally, with the deflection angle performing a sinusoidal variation. Preliminary measurements, for a range of the propulsive waveform parameters, demonstrate reversible thrust generation and good agreement with the theoretical predictions of a momentum-based model.

The logic relationship among the equality and inequality constraints in a standard constrained optimization problem (SCOP) is the logical AND. Various efficient, convergent and robust algorithms have been developed for such a SCOP. Motivated by a practical application, a more general constrained optimization problem (GCOP) with not only logic AND but also OR relationships is proposed in this paper. In order to solve such a generalized problem, a mathematical transformation which can transfer a set of inequalities with logic OR into one inequality is developed. This transformation provides a necessary and sufficient condition which enables us to use the algorithms developed for SCOPs to solve the generalized optimization problems. The research is motivated by the requirements of developing an efficient, robust, and reliable navigation algorithm for a mobile robot such as an autonomous underwater vehicle (AUV). The original contributions of the paper include threefold: first, from the viewpoint of optimization theory, this paper, to the authors' best knowledge, is the first one to propose such a GCOP. Second, a method is developed to solve such a GCOP. Third, from the viewpoint of robot path planning, this paper presents a new way of using classical optimization approach to solve robot path planning.

Presents the design of the fingertip sensing subsystem constructed for use in a dextrous underwater gripper. Two distinct types of sensor were developed specifically for this application, a force sensor and a slip sensor. The force sensor is used to determine the force exerted by the gripper on the target object, and hence allow the force loop closure required for carrying out dextrous manipulation and the handling of delicate marine samples. The function of the slip sensor is to detect the occurrence of slippage between the fingertip and the target. The information can be used in a variety of ways, for example providing a warning system to the force control loop or enhancing the gripper's exploratory capabilities in areas of poor visibility. Two quite different methods of extracting the force data from the sensor readings are presented and evaluated. The performance of the slip sensor is assessed, as is its potential usefulness in acting in tandem with the force sensor.

AMADEUS is a dexterous subsea robot hand incorporating force and slip contact sensing, using fluid-filled tentacles for fingers. Hydraulic pressure variations in each of three flexible tubes (bellows) in each finger create a bending moment, and consequent motion or increase in contact force during grasping. Such fingers have inherent passive compliance, no moving parts, and are naturally depth pressure-compensated, making them ideal for reliable use in the deep ocean. In addition to the mechanical design, development of the hand has also been considered for closed loop finger position and force control, coordinated finger motion for grasping, force and slip sensor development/signal processing, and reactive world modelling/planning for supervisory "blind grasping". Initially, the application focus is for marine science tasks, but broader roles in offshore oil and gas, salvage, and military use are foreseen. Phase I of the project has been completed, with the construction of a first prototype. This paper summarizes the developments in Phase I. Further details for each area of investigation are referenced.

Reports analysis and simulation studies for the co-ordinated control of multiple manipulators on a free swimming underwater robot. The control objective is to perform an open loop control of the end effector position of one of the arms. A formulation is derived for both the inverse dynamics and inverse kinematics of an underwater robot with multiple manipulators, having a redundancy in degrees of freedom (DOF) of motion, in the presence of external generalised forces. The method is applied to a model of an underwater robot comprising a 6 DOF mission arm and a further pair of 2 DOF paddling arms used for stabilisation in a free swimming condition. The numerical simulation results show the effectiveness of the presented co-ordinated control method with the assistance of the stabilising arms.

We describe the design of a task planning and world modelling mechanism as part of a supervisory or teleassistance control scheme for robots operating in unstructured environments. The mechanism addresses the key difficulties of an unstructured environment which lead to failures in task execution. Reactivity is achieved by mixing planning and execution. Complexity of task and error recovery is managed by hierarchical design of action sequences. The world model and task planner are intimately linked to allow parameterization of actions, and for the operator to focus on involved objects in the world. They work as part of a functional architecture with motion planning, dynamic control and sensor fusion processes, and have been implemented to specify the activity of a pair of underwater manipulators, sensed using force/torque loadcell, joint potentiometers, high frequency sonar and laser striping.

This article addresses a key question in the application of Card, Moran, and Newell's (1983) keystroke-level model to software in which users specify a command by working through a system of hierarchical menus. For example, to insert a row in Lotus 1-2-3®, the user makes three menu choices: W for worksheet, I for insert, and R for row. In the keystroke-level model, it is assumed that a time-consuming mental operation precedes each command. The question in the application of the keystroke-level model to hierarchical menu systems is whether the keystrokes WIR in the previous example constitute the execution of three commands and thus require three mental operations or whether WIR acts as a single command and requires only one mental operation. Data were collected from four highly experienced Lotus 1-2-3 users as they went about their day-to-day work. Strong evidence that only one mental operation is involved in choosing from a hierarchical menu system was obtained. We hypothesize that the discrepancy of our results from the data of others is due to the fact that our subjects were more experienced. The implications of our findings for the design of menus is discussed.

The ability to predict users' commands in Lotus 1-2-3 was investigated. Keystroke data was collected from four subjects in their normal work environment, with over 2700 commands being analysed. The data was used to construct transitional probability matrices that were then used to predict commands based on their immediate predecessors. Approximately half of the users' commands were correctly predicted in this manner. Also of interest was how often the users' commands would be either the most likely or the second most likely command given the previous commands. When the previous two commands were used to derive two predictions, the probability that the users' next command would be one of the two predicted commands was 0.81. The average probability of command repetition was 0.33. It was concluded that users appear to be extremely predictable in the Lotus 1-2-3 environment, and this predictability could serve as the foundation for an interface that aids the user by providing easy access to the next most likely command to be issued and by adapting the probabilities of command use over time.

A knowledge-based architecture currently being implemented is reported, that addresses the problems of responsive and time-constrained reasoning within an autonomous vehicle. The architecture is implemented on a multiprocessor system and uses task scheduling to achieve minimum response times to external stimuli. Time-constrained reasoning is achieved by reasoning at different abstraction levels within the multiprocessor architecture, thereby trading off solution quality against solution time.>

Abstract This paper describes a CAD facility for the design of automatic guidance and control strategies for an unmanned autonomous submersible. The facility utilises a three dimensional colour graphics display of the scene and vehicle trajectory, providing a man/ machine interface for the operator in addition to vehicle motion simulation with six degrees of freedom. A primary application envisaged for the CAD facility is in the design of scene analysis algorithms and the intelligent control system structure, for an autonomous submersible employing scanning sonar as its target sensor.