A simulation algorithm to predict expected upper limb movements of prosthesis users performing activities of daily living (ADL) was developed. It is quite challenging to determine the right type and fit of a prosthesis and provide appropriate training to properly utilize it. The amputee care team typically uses prior experiences to provide prescription and training customized for each individual. It is also very difficult to anticipate expected and undesired compensatory motions due to reduced degrees of freedom of a prosthesis user. We have developed a tool to predict and visualize the expected upper limb movements resulting from using a prescribed prosthesis and its suitability to the needs of the amputee. It is expected to help clinicians make decisions such as the type of the prosthesis, and whether to include a wrist joint, based on the impact it will have on the rest of the joints. The main focus of this work is to use robotics-based methods to simulate human use of prostheses and identify the expected posture of the limited joints on the upper limbs. Unlike other works, this paper does not discuss the control of the prosthesis but the posture of the body. A weighted least-norm inverse kinematics algorithm was used to develop a robotics-based model of the upper limbs and torso. Motion capture data from the subjects were used to determine the weighting matrix the algorithm required. Results show that this approach provides human-like simulation of joint motions and matches the motion capture data. The algorithm uses the individual's anthropometrics and level of amputation to create a personalized kinematic model of the upper body and the joint motions during ADLs. A graphic user interface (GUI) was created to allow the clinician to input the relevant data resulting in arm movements of the prospective prosthesis user. A custom-made visualization software was developed to display an animation performing the simulated motion. It should be stressed that this work does not adjust and replay motion capture recordings, but solves the inverse kinematics of the human body.

Robotic manipulators that are attached to mobile platforms are often used in workspaces that require the end-effector to mobilize beyond the manipulator's limited reach, such as in warehouse shelf stacking and similar applications. However, such assistive robots fall short of completing tasks that require the end-effector to be situated in a specific configuration at a critical time during the task. Traditionally, users control the mobile base to situate the arm such that the task can be completed through continuous operation. This requires an experienced operator who can predict the needed end-effector workspace, and can operate the base accordingly to maximize the likelihood of a successful task while avoiding any floor obstacles. In this work, we propose a straightforward control method that provides sufficient freedom to the end-effector to complete a task that is bound by time-dependent constraints. This is achieved by relaxing the time constraints on the mobile base trajectory in a floor space obstructed by obstacles. The trajectory of the platform is determined by sensor-assisted obstacle avoidance algorithm such that a single degree of freedom mobility can be represented through a safe obstacle-free time-independent path. The proposed control method is implemented in simulation and on physical hardware built in our labs. The simulation included a 5-DoF redundant Planar Mobile Manipulator (PMM). The hardware implementation and testing utilized a 9-DoF redundant mobile manipulator. The implementation results demonstrate the effectiveness of the control method in adjusting the mobile platform motion along its allowed obstacle-free path to enable the end-effector to follow its trajectory for task completion that would otherwise fail to complete when conventional control methods are used.

In this work, we present a methodology for enabling a robot to identify an object and grasp configuration of interest and assist the human teleoperating the robot, to grasp the object. The identification is carried out in real-time by detecting the motion intention of the human as they are teleoperating the remote robotic arm towards the object and the grasp configuration. Simultaneously, depending on the detected object and grasp configuration, the human user is assisted to translate and orient the remote arm gripper in order to preshape and grasp the object. The complete process occurs with the human teleoperating the arm, and without them having to interact with another interface. Motion intention recognition is carried out by using Hidden Markov Models (HMMs), trained offline by preshape trials performed by a skilled teleoperator. The environment is unstructured and comprises of a number of objects, each with multiple grasp configurations. Experimental tests on healthy human subjects have validated our intention recognition based assistance method. They show that the method allows objects to be grasped and placed 48% faster, and with much ease compared to unassisted teleoperation. Moreover, we have proved that the model for intention recognition, trained by a skilled teleoperator, can be used by novice users to efficiently execute a grasping task in teleoperation.

Rapidly acquiring the shape and pose information of unknown objects is an essential characteristic of modern robotic systems in order to perform efficient manipulation tasks. In this work, we present a framework for 3D geometric shape recovery and pose estimation from unorganized point cloud data. We propose a low latency multi-scale voxelization strategy that rapidly fits superquadrics to single view 3D point clouds. As a result, we are able to quickly and accurately estimate the shape and pose parameters of relevant objects in a scene. We evaluate our approach on two datasets of common household objects collected using Microsoft's Kinect sensor. We also compare our work to the state of the art and achieve comparable results in less computational time. Our experimental results demonstrate the efficacy of our approach.

Accurately predicting the movements of the human upper body is an obstacle in simulating human movement. This paper describes the optimization and comparison of three inverse kinematic algorithms designed to predict the pose of a 25 degree of freedom robotic human upper body model (RHBM). Motion analysis data of 10 subjects performing 5 activities of daily living were used to evaluate the performance of each method. The first algorithm used a numerically optimized weighted-least-norm (WLN) solution. The second algorithm maximized the joint angle probability density, using the gradient projection method (GP). The third algorithm used a single layer artificial neural network (NN), trained by Levenberg-Marquart backpropagation using the motion analysis data. Error was evaluated using the root mean square of the difference between calculated and recorded joint angles. The robustness was then tested by progressively excluding subject data from the training set, re-training the algorithms, and evaluating the error for all subjects. The numerically optimized WLN solution showed the highest robustness, and the GP and NN solutions had greater accuracy for the data included in training and lower accuracy for the data excluded from training. The gradient projection method showed greater robustness than the artificial neural network, and has potential to be refined and combined with the weighted least norm solution to increase accuracy and robustness. Future work will investigate combined methods and the ability to predict motion of persons using prostheses.

In this work, we analyze the pick and place task for a human-in-the-loop robotic system to determine where human input can be most beneficial to a collaborative task. This is accomplished by implementing a pick and place task on a commercial robotic arm system and determining which segments of the task, when replaced by human guidance, provide the most improvement to overall task performance and require the least cognitive effort. The pick and place task can be segmented into two main areas: coarse approach towards goal object and fine pick motion. For the fine picking phase, we look at the importance of user guidance in terms of position and orientation of the end effector. Results from our experiment show that the most successful strategy for our human-in-the-loop system is the one in which the human specifies a general region for grasping, and the robotic system completes the remaining elements of the task. Our experimental setup and procedures could be generalized and used to guide similar analysis of human impact in other human-in-the-loop systems performing other tasks.

The wheelchair-mounted robotic arm (WMRA) is a mobile manipulator that consists of a 7-DoF robotic arm and a 2-DoF power wheelchair platform. Previous works combined mobility and manipulation control using weighted optimization for dual-trajectory tracking [7]. In this work, we present an image-based visual servoing (IBVS) approach with scale-invariant feature transform (SIFT) using an eye-in-hand monocular camera for combined control of mobility and manipulation for the 9-DoF WMRA system to execute activities of daily living (ADL) autonomously. We also present results of the physical implementation with a simple “Go to and Pick Up” task and the “Go to and Open the Door” task previously published in simulation, using IBVS to aid the task performance.

Commercially available wheelchair-mounted robotic arms (WMRAs) are becoming more prevalent internationally but have yet to be largely developed and approved by the Food and Drug Administration in the United States. The purpose of this study was to perform kinematic and experimental analysis of commercially available WMRAs. The goal was to quantitatively compare each device through a standardized testing protocol. The study and its procedures may serve as a source of quantitative evaluation data for commercially available and developmental WMRAs.

Wheelchair-bound persons with upper limb motion limitations can utilize a wheelchair-mounted robotic arm (WMRA) to perform activities of daily living (ADL) tasks. In this paper, an optimized control of our 9-DoF system, consisting of a 7-DoF robotic arm and a 2-DoF power wheelchair, is achieved. For effective ADL task execution, positioning the end-effector with proper wheelchair orientation was optimized as part of the control algorithm. Separate wheelchair and end-effector trajectories were simultaneously followed to execute a “Go To and Open the Door” task. The control methodology, implementation and test results in simulation are presented in this paper.

When planning robotic grasping and manipulation maneuvers, knowledge of the shape and pose of the object of interest is critical information. In order for an autonomous or semi-autonomous system to operate intelligently in an unstructured environment and interact with novel objects, it must have the ability to recover this information at run time, even when no a priori information of the object is available. In this paper, we describe the development and testing of an algorithm that can reconstruct the full 3D geometry of a novel object from just three images. A variant of shape from silhouettes, the algorithm first generates a rough surface approximation in the form of a point cloud. This approximation is then refined by fitting an eleven parameter geometric surface to the points in such a manner that the surface ignores noise and perspective projection shadows. We test the algorithm in both simulation and on several real world objects. We show that the algorithm provides accurate reconstructions that can be directly used to plan grasping maneuvers. Compared to other attempts in the literature, the proposed algorithm is faster, requires fewer images, is more accurate, and degrades gracefully in the presence of bad data. A real world test case is included that shows that the algorithm still yields usable results when the form of the object is amorphous or otherwise non-geometric.

In this paper, we demonstrate the use of range information from a laser sensor mounted on the end-effector of a remote robot manipulator to assist persons with limited upper body strength carry out Activities of Daily Living in unstructured environments. Laser range data is used to determine goals, identify targets, obstacles, and via points to enable autonomous execution of trajectories. The human operator is primarily involved in higher level decision making; the human user performs minimal teleoperation to identify critical points in the workspace with the laser pointer. Tests on ten healthy human subjects in executing a pick-and-place task showed that laser-based assistance not only increased the speed of task execution by an average of 26.9%while decreasing the physical effort by an average of 85.4%, but also made the task cognitively easier for the user to execute.

A wheelchair-mounted robotic arm (WMRA) system was designed and built to meet the needs of mobility-impaired persons with limitations of upper extremities, and to exceed the capabilities of current devices of this type. The control of this 9-DoF system combines the 7-DoF robotic arm control with the 2-DoF power wheelchair control. The 3-degrees of redundancy can be optimized to effectively perform activities of daily living (ADLs) and overcome some workspace limitations. The control system is designed for teleoperated or autonomous coordinated Cartesian control, and it offers expandability for future research, such as voice or sip and puff control operations and sensor assist functions

A wheelchair-mounted robotic arm (WMRA) was designed and built to meet the needs of mobility-impaired persons with limitations of upper extremities, and to exceed the capabilities of current devices of this type. The mechanical design incorporates DC servo drive, with actuator hardware at each individual joint, allowing reconfigurable link lengths. It has seven degrees of freedom and uses a side mount on a power wheelchair. The control system allows coordinated Cartesian control, and offers expandability for future research, such as coordinated motion with the wheelchair itself. This paper discusses the current state of the art in WMRAs; describes the design goals and user requirements for this device; explains the component selection process; discusses details of the mechanical design, electrical system and low-level controller; covers manufacturing concerns; and describes the testing of the completed arm. Further improvements are also suggested

A wheelchair-mounted robotic arm (WMRA) was designed and built to meet the needs of mobility-impaired persons with limitations of upper extremities, and to exceed the capabilities of current devices of this type. The mechanical design incorporates DC servo drive, with actuator hardware at each individual joint, allowing reconfigurable link lengths. It has seven degrees of freedom and uses a side mount on a power wheelchair. The control system is built and designed for user-driven or autonomous coordinated Cartesian control, and it offers expandability for future research, such as coordinated motion with the wheelchair itself. This paper discusses the current state of the art in WMRAs; describes the design goals and user requirements for this device; explains the component selection and the mechanical/electrical design; describes the controller design of the arm and optimized redundancy resolution for the Cartesian control. Further improvements are also suggested

In telemanipulation systems, assistance through variable position/velocity mapping or virtual fixture can improve manipulation capability and dexterity [3, 5, 6, 7, 8]. Conventionally, such assistance is based on the sensory data of the environment and without knowing user’s motion intention. In this paper, user’s motion intention is combined with real-time environment information for applying appropriate assistance. If the current task is following a path, a virtual fixture is applied. If the task is aligning the end-effector with a target, an attractive force field is produced. Similarly, if the task is avoiding obstacles that block the path, a repulsive force field is generated. In order to successfully recognize user’s motion intention, a Hidden Markov Model (HMM)-based algorithm is developed to classify human actions, such as following a path, aligning target and avoiding obstacles.