Objective assessment of surgical skill is gaining traction in a number of specialty fields. In robot-assisted surgery in particular, the availability of data from the operating console and patient-side robot offers the potential to derive objective metrics of performance based on tool movement kinematics. While these techniques are becoming established in the laparoscopic domain, current assessment techniques for robotic endovascular surgery are based primarily on observation, checklists, and grading scales. This work presents an objective and quantitative means of measuring technical competence based on analysis of the kinematics of endovascular tool tip motions controlled with a robotic interface. We designed an experiment that recorded catheter tip movement from 21 subjects performing fundamental endovascular robotic navigation tasks on a physical model. Motion-based measures of smoothness (spectral arc length and number of submovements) were computed and tested for correlation with subjective scores from a global rating scale assessment tool that has been validated for use when performing manual catheterization. Results show that the smoothness metrics that produced significant correlations with the global rating scale for manual catheterization show similar correlations for robotic catheterization. This finding is notable, since with the robotic interface, tool tip motion is commanded discretely via a control button interface, while in manual procedures the tools are controlled through continuous movements of the surgeon's hands. Logistic regression analysis using a single motion metric was capable of classifying subjects by expertise with better than 90% accuracy. These objective and quantitative metrics that capture movement quality could be incorporated into future training protocols to provide detailed feedback on trainee performance.

Current performance assessment techniques in endovascular surgery are subjective or limited to grading scales based solely on an expert's observation of a novice's task execution. Since most endovascular procedures involve performing fine motor control tasks that require complex dexterous movements, this paper evaluates objective and quantitative metrics of performance that capture movement quality through the computation of tool tip movement smoothness. An experiment was designed that involved recording the catheter tip movement from 20 subjects performing four fundamental endovascular tasks in each of three sessions using manual catheterization on a physical model and in a simulation environment. Several motion-based performance measures that have been shown to reliably assess skill in other domains were computed and tested for correlation with subjective data that were simultaneously obtained from the global rating scale assessment tool. Metrics that captured movement smoothness produced statistically significant correlations with the observation-based assessment metrics and were able to differentiate skill among participants. In particular, submovement analysis led to metrics that captured statistically significant differences across ability group, session, experimental platform, and task. Objective and quantitative metrics that capture movement smoothness could be incorporated into future training protocols to provide detailed feedback on trainee performance.

Objective assessment of skill is important in understanding the human performance of precision motor control tasks; however, outcome-based performance measures are most commonly used to provide feedback after the task is completed. In this paper, we propose the use of motion-based performance metrics in order to determine successful task completion strategies for a complex and unconstrained dynamic task. We developed a novel motor control task in a virtual environment and identified two motion-based metrics (mean absolute jerk and average frequency) that correlate with the successful performance of the task measured as a function of completion time and acquired targets. Our methodologies provide insight into the movement strategies used during successful trials of the motor control task, and could be used to provide specific feedback and coaching aimed at altering the task completion strategy and boosting the participants’ performance.

One of the options available to a sizable minority of U.S. voters is the ability to, with a single action, cast votes in multiple races; this is termed straight-party voting (SPV). SPV is implemented inconsistently across the U.S. and this may result in voter confusion, threatening the integrity of elections. We presented survey respondents with multiple SPV scenarios, testing both their understanding of SPV-marked ballots and their likelihood of using SPV to make multiple selections on a ballot. Participants were also asked their opinions on how SPV ought to work. Voters had significant difficulty in interpreting SPV ballots and were reluctant to generate them, though this was improved when ballots had more clear and detailed instructions. Participants also tended to believe that SPV should not work the way they believed it had worked on ballots they had previously seen. Overall, our results suggest that SPV is a likely cause of voter confusion, suggesting further research on the usability of straight-party voting systems.

In the 2006 U.S. election, it was estimated that over 66 million people would be voting on direct recording electronic (DRE) systems in 34% of the nation's counties [8]. Although these computer-based voting systems have been widely adopted, they have not been empirically proven to be more usable than their predecessors. The series of studies reported here compares usability data from a DRE with those from more traditional voting technologies (paper ballots, punch cards, and lever machines). Results indicate that there were little differences between the DRE and these older methods in efficiency or effectiveness. However, in terms of user satisfaction, the DRE was significantly better than the older methods. Paper ballots also perform well, but participants were much more satisfied with their experiences voting on the DRE. The disconnect between subjective and objective usability has potential policy ramifications.

In the United States, computer-based voting machines are rapidly replacing other older technologies. While there is potential for this to be a usability improvement, particularly in terms of accessibility, the only way it is possible to know if usability has improved is to have baseline data on the usability of traditional technologies. We report an experiment assessing the usability of punch cards, lever machines, and two forms of paper ballot. There were no differences in ballot completion time between the four methods, but there were substantial effects on error rate, with the paper ballots superior to the other methods as well as an interaction with age of voters. Subjective usability was assessed with the System Usability Scale and showed a slight advantage for bubble-style paper ballots. Overall, paper ballots were found to be particularly usable, which raises important technological and policy issues.

Because of the visual nature of computer use, researchers and designers of computer systems would like to gain some insight into the visual search strategies of computer users. Icons, a common component of graphical user interfaces, serve as the focus for a set of studies aimed at (1) developing a detailed understanding of how people search for an icon in a typically crowded screen of other icons that vary in similarity to the target, and (2) building a cognitively plausible model that simulates the processes inferred in the human search process. An eye-tracking study of the task showed that participants rarely refixated icons that they had previously examined, and that participants used an efficient search strategy of examining distractor icons to their current point of gaze. These findings were integrated into an ACT-R model of the task using EMMA and a nearest strategy. The model fit the response time data of participants as well as a previous model of the task, but was a much better fit to the eye movement data.

Users of modern GUIs routinely engage in visual searches for various control items, such as buttons and icons. Because this is so ubiquitous, it is important that the visual properties of user interfaces support such searches. The current research is aimed at deepening our understanding of how the visual spacing between icons affects visual search times. We constructed an experiment based on previous icon sets [8] where spacing between icons was systematically manipulated, and for which we had a computational cognitive model that predicted performance. In particular, the model predicted that larger spacing would lead to slower search times. While this prediction was borne out, there was an unanticipated finding: users in this new experiment were substantially slower than in previous similar experiments with smaller spacing. In fact, results from this new experiment were better fit with a model that employed a fundamentally different, and less efficient, search strategy. A second experiment was conducted to explicitly test the surprising result that this varied and larger icon spacing would lead to increased search times. Results were consistent with this hypothesis. These results imply that while small differences in visual layout may not intrinsically produce large differences in user performance, they may cause users to adopt suboptimal strategies that do produce such differences.

Users faced with Web sites containing many possibly relevant pages often have a decision to make about navigation: use the menu of links or use the provided site search function. Two studies were conducted to examine what users do when faced with this decision on e-commerce Web sites, and how users go about deciding which method to attempt. An exploratory study revealed a wide distribution of searching and browsing behavior across sites and users. Counter to some predictions, use of the site search functions did not yield faster or more accurate performance in locating products. Questionnaire data suggested that factors relevant to the menu structure, interface element prominence, information scent and user dispositions all influenced the decision of whether to browse or search a site for a product. A second experiment utilizing novel e-commerce sites and allowing for more control of factors found to be important in the first study found that browsing behavior was influenced by both the breadth and information scent of the menus. These results suggest that providing site search should not be used to compensate for poor menu design, and provide further evidence regarding the design of effective menu structures.

We compare three tools for creating GOMS models, QGOMS [2), CATHCI (17) and GLEAN3 [12], along several dimensions. We examine the representation and available constructs in each tool, the qualitative and quantitative design information provided, the support for building cognitively plausible models, and pragmatics about using each tool (e.g., how easy it is to modify a model). While each tool has its strengths, they all leave something to be desired as a practical UI design tool.

Click-down (or pull-down) menus have long been a key component of graphical user interfaces, yet we know surprisingly little about how users actually interact with such menus. Nilsens [8] study on menu selection has led to the development of a number of models of how users perform the task [6, 21. However, the validity of these models has not been empirically assessed with respect to eye movements (though [l] presents some interesting data that bear on these models). The present study is an attempt to provide data that can help refine our understanding of how users interact with such menus.

A prerequisite to the effective design of user interfaces is an understanding of the tasks for which that interface will actually be used. Surprisingly little task analysis has appeared for one of the most discussed and fastest-growing computer applications, browsing the World-Wide Web (WWW). Based on naturally-collected verbal protocol data, we present a taxonomy of tasks undertaken on the WWW. The data reveal that several previous claims about browsing behavior are questionable, and suggests that that widget-centered approaches to interface design and evaluation may be incomplete with respect to good user interfaces for the Web.

A central goal of many user interface development tools has been to make the construction of high quality interfaces easy enough that iterative design approaches could be a practical reality. In the last 15 years significant advances in this regard have been achieved. However, the evaluation portion of the iterative design process has received relatively little support from tools. Even though advances have also been made in usability evaluation methods, nearly all evaluation is still done “by hand,” making it more expensive and difficult than it might be. This paper considers a partial implementation of the CRITIQUE usability evaluation tool that is being developed to help remedy this situation by automating a number of evaluation tasks. This paper will consider techniques used by the system to produce predictive models (keystroke level models and simplified GOMS models) from demonstrations of sample tasks in a fraction of the time needed by conventional handcrafting methods. A preliminary comparison of automatically generated models with models created by an expert modeler show them to produce very similar predictions (within 2%). Further, because they are automated, these models promise to be less subject to human error and less affected by the skill of the modeler.

One method for user interface analysis that has proven successful is formal interface analysis, such as GOMSbased analysis. Such methods are often criticized for being difficult to learn, or at the very least an additional burden for the system designer. However, if the process of constructing and using formal models could be automated as part of the interface design environment, such models could be of even greater value. This paper describes an early version of such a system, called USAGE (the UIDE System for semi-Automated GOMS Evaluation). Given the application model necessary to drive the UIDE system, USAGE generates an NGOMSL model of the interface which can be “run ” on a typical set of user tasks and provide execution and learning time estimates.