In this paper, we present a framework for the integration of force feedback information in a NASA NextGen Volumetric Cockpit Situation Display (CSD). With the current CSD, the user retrieves operational information solely through visual displays and interacts with the CSD tools through using a mouse. The advanced capabilities of the CSD may require complex manipulation of information which may be difficult to perform with input devices found in today's cockpits. Performance with the CSD could benefit from a new user input device and enhanced user feedback modalities that can be operated safely, effectively, and intuitively in a cockpit environment. In this work, we investigate the addition of force feedback in two key CSD tasks: object selection and route manipulation. Different force feedback models were applied to communicate guidance commands, such as collision avoidance and target contact. We also discuss the development of a GUI-based software interface to allow the integration of a haptic device for the CSD. A preliminary user study was conducted on a testbed system using the Novint Falcon force-feedback device. A full experiment, assessing the effectiveness and usability of the feedback model in the CSD, will be performed in the next phase of our research.

This work focuses on the implementation of a vision-based motion guidance method, called virtual fixtures, on admittance-controlled human-machine cooperative robots with compliance. The robot compliance here refers to the structural elastic deformation of the device. The high mechanical stiffness and non-backdrivability of a typical admittance-controlled robot allow for slow and precise motions, making it highly suitable for tasks that require accuracy near human physical limits, such as microsurgery. However, previous experiments have shown that even small robot compliance degraded virtual fixture performance, especially at the micro scale. In this work, control methods to minimize the effect of robot compliance on virtual fixture performance were developed for admittance-controlled cooperative systems. Based on a linear model of the robot dynamics, we applied a Kalman filter to integrate the measurements obtained from the camera and encoders to estimate the robot end-effector position. A partitioned control law was used to achieve end-effector trajectory following on the desired velocity commanded by the admittance and virtual fixture control laws. The effectiveness of the Kalman filter and the controller was validated on a one degree-of-freedom admittance-controlled cooperative testbed.

In human-machine collaborative systems, robot joint compliance and human-input dynamics lead to involuntary tool motion into undesired regions. To correct this, a set of methods, called dynamically-defined virtual fixtures, was previously proposed to create a movable virtual fixture that stops the user at a safe distance outside the forbidden region. In this work, a new method, called the force-based method, was added. A vision system was introduced for real-time tool tracking. Additionally, we implemented a closed-loop controller with the virtual fixtures that allows the user to reach, but not enter, the forbidden region. Two user experiments were conducted on a 1-DOF testbed to evaluate the virtual fixture methods. The first experiment showed the effectiveness of the virtual fixtures in preventing the penetration. However, the absence of haptic feedback in the closed-loop implementation resulted in boundary penetration. In the second experiment, visual feedback was used to compensate for the lack of haptic feedback. User cognitive load was added as an inhibiting factor in a human-machine cooperative setting. The experiment showed a significant reduction in penetration with visual feedback, while the addition of cognitive load did not significantly increase the penetration.

Virtual fixtures are virtual constraints that enhance human performance in motion tasks. They can either confine and/or guide a user's motion. In this paper, we use a commercially available motion platform to explore the portability and applicability of virtual fixtures and document how people interact with them. Two micromanipulation tasks are analyzed and the effects of similarly designed virtual fixtures are discussed. One task simulates a medical task, retinal vein cannulation, and the other simulates a manufacturing task, fine leads soldering. Preliminary experimental results show that the virtual fixtures increase the accuracy of both medical and manufacturing tasks, lending support to its portability and applicability across unrelated tasks

We present the design and implementation of a vision-based system for cooperative manipulation at millimeter to micrometer scales. The system is based on an admittance control algorithm that implements a broad class of guidance modes called virtual fixtures. A virtual fixture, like a real fixture, limits the motion of a tool to a prescribed class or range of motions. We describe how both hard (unyielding) and soft (yielding) virtual fixtures can be implemented in this control framework. We then detail the construction of virtual fixtures for point positioning and curve following as well as extensions of these to tubes, cones, and sequences thereof. We also describe an implemented system using the JHU Steady Hand Robot. The system uses computer vision as a sensor for providing a reference trajectory, and the virtual fixture control algorithm then provides haptic feedback to implemented direct, shared manipulation. We provide extensive experimental results detailing both system performance and the effects of virtual fixtures on human speed and accuracy.

In this article, we describe and demonstrate control algorithms for general motion constraints. These constraints are designed to enhance the accuracy and speed of a user manipulating in an environment with the assistance of a cooperative or telerobotic system. Our method uses a basis of preferred directions, created off-line or in real-time using sensor data, to generate virtual fixtures that may constrain the user to a curve, surface, orientation, etc. in space. Open loop virtual fixtures seek only to maintain user motion along preferred directions, whereas closed loop fixtures additionally guide the user toward a point, line, or surface. This article demonstrates and compares the effects of open and closed loop fixtures in both autonomous and human-machine cases.

In this paper, we present experiments to determine the effect of different virtual fixture compliance levels in a human-machine cooperative manipulation system. Subjects used the JHU Steady Hand Robot with vision-based virtual fixtures to perform three common tasks: curve following, off-path targeting, and object avoidance. The virtual fixtures provided different levels of guidance to the operator, ranging from no guidance to complete guidance. User performance was evaluated based on the error and time for task execution. We developed an algorithm to determine the appropriate compliance level based on the nature of the task. Task parameters considered were time vs. accuracy and user constraint vs. freedom.