Advanced robotic assistance in microsurgery, such as automation, requires an accurate estimation of the state of the robotic forceps. In this paper, we propose a robust and accurate forceps tracking method to estimate the full state of the forceps (i.e., the position, posture, and grip parameters) using visual information obtained from stereo microscopic images and kinematic information obtained from the robotic sensory information, forward kinematics, and hand-eye coordination. An online method for updating the hand-eye coordination was also developed using an extended Kalman filter to cancel the hand-eye coordination errors caused by the repositioning of the microscope. The experimental results showed that the proposed method could accurately and robustly estimate the state of the robotic forceps even after the repositioning of the microscope.

Forceps tracking is an important element of high-level surgical assistance such as visual servoing and surgical motion analysis. In many computer vision algorithms, artificial markers are used to enable robust tracking; however, markerless tracking methods are more appropriate in surgical applications due to their sterilizability. This paper describes a robust, efficient tracking algorithm capable of estimating the full state parameters of a robotic surgical instrument on the basis of projective contour modeling using a 3-D CAD model of the forceps. Thus, the proposed method does not require any artificial markers. The likelihood of the contour model was measured using edge distance transformation to evaluate the similarity of the projected CAD model to the microscopic image, followed by particle filtering to estimate the full state of the forceps. Experimental results in simulated surgical environments indicate that the proposed method is robust and time-efficient, and fulfills real-time processing requirements.

The master and slave surgical assistant systems have been studied actively. However, regarding the master system, the manipulator should be designed for each surgical field because the target workspace and precision are different. Furthermore, operability is important for safety reasons. Therefore, the authors analyzed the motion of surgeon first and then developed a master manipulator suitable for the operation of micro neuro surgery. Some experiments were conducted to evaluate the control method and show the effectiveness of the proposed manipulator control method.

Joint attention, a process to attend to the object that the other attends to is supposed to be important for human-robot communication as well as for human-human communication. We propose an architecture for acquiring joint attention within a certain time period for realizing natural human-robot interaction. The architecture has two featured modules: a self-organizing map that makes the leaning time shorter and an automatic visual attention selector that let the agent communicate with a human synchronously. We implemented the proposed architecture in a real robot agent and found that 30 minutes was enough for acquiring joint attention with two objects. We can conclude from preliminary experiments that even if the gaze preference of the robot is different from that of the human caregiver, it can acquire joint attention.

Joint attention is supposed to be a basis of the competence of communication with others. The authors have been attacking the issue how to learn joint attention only through interactions with caregivers, in other words, without external task evaluation from a viewpoint of a constructivist approach towards both establishing a design principle of communicative robots and understanding the developmental process of human communication. This paper presents a method for quick learning of joint attention with unfamiliar persons based on a hybrid architecture that consists of (1) a quick but person-dependent learning module and (2) a slow but person-independent learning module. Experimental results show the effectiveness of the proposed method.

This paper presents a constructive model by which a robot acquires the ability of joint attention with a human caregiver based on its embedded mechanisms of visual attention and learning with self-evaluation. The former is to look at a salient object in the robot's view, and the latter is to learn sensorimotor co-ordination when visual attention has succeeded. Since the success of visual attention does not always correspond to the success of joint attention, the robot has incorrect learning data for joint attention as well as correct data. However, the robot is expected statistically to lose incorrect data as outliers since such data do not have any correlation in the sensorimotor co-ordination while correct data have a correlation. The robot consequently acquires the ability of joint attention by finding the correlation in the sensorimotor co-ordination even if multiple objects are placed at random positions in an environment and a human caregiver does not provide any task evaluation to the robot. The experimental results show that the proposed model makes the robot reproduce the developmental process of infants' joint attention. Therefore, the proposed model could be one of the models to explain how infants develop the ability of joint attention.