In this paper, a novel approach for commanding mobile robots using a probabilistic multistroke sketch interface is presented. Drawing from prior work in handwriting recognition, sketches are modeled as a variable duration hidden Markov model, where the distributions on the states and transitions are learned from training data. A forward search algorithm is used to find the most likely sketch given the observations on the strokes, interstrokes, and gestures. A heuristic is implemented to discourage breadth-first search behavior, and is shown to greatly reduce computation time while sacrificing little accuracy. To avoid recognition errors, the recognized sketch is displayed to the user for confirmation; a rejection prompts the algorithm to search for and display the next most likely sketch. Upon confirmation of the recognized sketch, the robot executes the appropriate behaviors. A set of experiments was conducted in which operators controlled a single mobile robot in an indoor search-and-identify mission. Operators performed two missions using the proposed sketch interface and two missions using a more conventional point-and-click interface. On average, missions conducted using sketch control were performed as well as those using the point-and-click interface, and results from user surveys indicate that more operators preferred using sketch control.

A fully probabilistic command interface for controlling robots using multi-stroke sketch commands is presented. Drawing from prior work in handwriting recognition, sketches are modeled as a variable duration hidden Markov model, where the distributions on the states and transitions are learned from training data. A forward search algorithm on the gesture, stroke, and stroke transition observations is used to find the most likely sketch, which is displayed to the user for confirmation. In cases where the most likely sketch is incorrect, the user can reject it, prompting the next most likely sketch to be displayed. Upon confirmation from the user, the robot executes the desired behaviors. A prototype sketch interface was implemented using a pen tablet; two sets of search-and-identify experiments were conducted using a single robot in an indoor environment to test the usability of the proposed framework. Even novice users were able to successfully complete the missions, including those on whom the algorithm was not trained. User surveys indicate that operators generally found the interface to be natural and easy to use.