The accuracy of skeleton-based action recognition models can be significantly improved using data processing techniques, particularly in complicated environments such as retail stores where people’s activities are dynamic and prone to noise and occlusions. This paper investigates how various preprocessing techniques, including smoothing, data compensation, and noise augmentation, influence the accuracy of two action recognition models, STGCN and STGCN++. We evaluated these methods using different data sources: key points of YOLO, HRNet, and motion capture (MoCap). Our findings demonstrate that higher quality data, such as projected 2D MoCap, lead to significantly better model performance, with STGCN++ achieving a mean accuracy of $91.26 \%$, compared to $81.06 \%$ with YOLO key points. We also show that applying preprocessing methods to noisier data, such as YOLO, can reduce performance degradation, improving their robustness for real-world applications. These results highlight the importance of preprocessing in implementing robust action recognition systems that can operate in dynamic, real-world settings and open the way for further improvements in the field.

In the domain of artificial neural networks, the comprehensive understanding and optimization of neuron dynamics are crucial. The Dendritic Neuron Model (DNM), noted for its distinct architecture and data processing capabilities, exemplifies this. However, the sophistication of the DNM leads to complexities in hyperparameter tuning. Notably, this complexity manifests in the way parameter changes can alter the dimensions of the solution space, a prime example of the Metameric Variable-length Problem. In this study, we have innovatively integrated chaotic maps into the gene expression mechanisms of Evolutionary Algorithms, enabling the incorporation of all DNM hyperparameters into a singular algorithmic framework. This integration allows for iterative adjustments within a variable-dimensional solution space, representing an evolving neuron model that streamlines the tuning process. Our approach, tested against benchmark datasets from the UCI Machine Learning Repository, demonstrates significant improvements in the DNM's performance, highlighting the effectiveness of incorporating biological chaos phenomena into neural network optimization.

Recent progress of high-speed projectors using DMD (Digital Micromirror Device) has enabled low-latency motion adaptability of displayed images, which is a key challenge in achieving projection-based dynamic interaction systems. This paper presents evaluation of different approaches in achieving fast motion adaptability with DMD projectors through a subjective image evaluation experiment and a discrimination experiment. The results suggest that the approach proposed by the authors, which updates the image position for every binary frame instead of for every video frame, applied to 60-fps video input offers perceptual image quality comparable with the quality offered by 500-fps projection.

This paper presents a fast projection mapping method for moving image content projected onto a markerless planar surface using a low-latency Digital Micromirror Device (DMD) projector. By adopting a closed-loop alignment approach, in which not only the surface texture but also the projected image is tracked by a camera, the proposed method is free from a calibration or position adjustment between the camera and projector. We designed fiducial patterns to be inserted into a fast flapping sequence of binary frames of the DMD projector, which allows the simultaneous tracking of the surface texture and a fiducial geometry separate from a single image captured by the camera. The proposed method implemented on a CPU runs at 400 fps and enables arbitrary video contents to be "stuck" onto a variety of textured surfaces.

We study the problem of mesh-based object generation. We propose a framework that generates mesh-based objects from point clouds in an end-to-end manner by using a combination of variational autoencoder and generative adversarial network. Instead of converting point cloud to other representations like voxels before input into the network, our network directly consumes the point cloud and generates the corresponding 3D object. Given point clouds of objects, our network encodes local and global geometry structures of point clouds into latent representations. These latent vectors are then leveraged to generate the implicit surface representations of objects corresponding to those point clouds. Here, the implicit surface representation is Signed Distance Function (SDF) which preserves the inside-outside information of objects. Then we can easily reconstruct polygon mesh surfaces of objects. This could be very helpful in a situation where there is a need of 3D shapes and only point clouds of objects are available. Experiments demonstrate that our network which makes use of both local and global geometry structure can generate high-quality mesh-based objects from corresponding point clouds. We also show that using PointNet-like structure as an encoder can help to achieve better results.

Bin picking refers to picking up the objects randomly piled in the container (bin) and robotic bin picking is always used to improve the industrial production efficiency. A pose estimation algorithm is necessary to tell the poses of the objects to the robot. This paper proposes a pose estimation algorithm for bin picking using 3D point cloud data. Point Pair Feature algorithm is performed in a fast way to propose possible poses and the poses are verified by a voxel-based verification method. Iterative Closest Point is used to refine the result poses. Our algorithm is proved to be more accurate and faster than Curve Set Feature algorithm and Point Pair Feature algorithm, robust to occlusion and able to detect multiple poses in one scene.

3D measurement of target objects characterized by specular reflection or subsurface scatterings cannot be measured by traditional 3D measurement methods because these targets have multiple light paths that make it difficult to determine the unique surface. We define these objects as complex light reflection objects. In this case, 3D measurement methods based on Light Transport (LT) Matrix estimation may be a solution to measure these complex light reflection objects, because LT Matrix captures every light path, and we can identify all 3D points on the target shape by using LT Matrix. However, these methods either provide low resolution results, or they are too slow for use in robot vision in practice. In this paper, we suppress the computational cost of LT Matrix estimation by dividing LT Matrix estimation into multi-scale. The proposed method reduces the number of candidate combinations between camera pixels and projector pixels greatly by using the information given by low resolution observations. The proposed algorithm allows high resolution measurement of the LT Matrix very efficiently. Furthermore, careful implementation of our method by using a sparse matrix representation achieves memory efficiency. We evaluated our method by measuring 3D points for a 256 × 256 resolution projector and camera system, which is an LT matrix 4096 times larger than that developed in our previous study [1] and 100 times faster than our naïve implementation of [2].

In this paper, we demonstrate an end-to-end spatiotemporal gesture learning approach for 3D point cloud data using a new gestures dataset of point clouds acquired from a 3D sensor. Nine classes of gestures were learned from gestures sample data. We mapped point cloud data into dense occupancy grids, then time steps of the occupancy grids are used as inputs into a 3D convolutional neural network which learns the spatiotemporal features in the data without explicit modeling of gesture dynamics. We also introduced a 3D region of interest jittering approach for point cloud data augmentation. This resulted in an increased classification accuracy of up to 10% when the augmented data is added to the original training data. The developed model is able to classify gestures from the dataset with 84.44% accuracy. We propose that point cloud data will be a more viable data type for scene understanding and motion recognition, as 3D sensors become ubiquitous in years to come.

In this paper, we address an issue on non-existence of a CAD model of an object in a model-based virtual visual servoing system. The approach for reconstructing a 3D model with fewer faces from point cloud data is presented. Because of a number of faces on the model surface, this reconstructed model required less rendering time than a CAD model, consequently, high speed and high accuracy model-based visual servoing can be achieved. We present the experimental results to demonstrate the effectiveness in terms of rendering time, convergence speed, and tracking speed measurements. The comparison of performances of a model-based visual servoing using different models, i.e.- CAD, Coarse mesh, Fine mesh, and Propose models, is also presented.

This paper describes a mobile pedestrian navigation system that provides users with clues that help understanding spatial relationship between mobile camera views and a 2D map. The proposed method draws on the map upright billboards that correspond to the basal planes of past and current viewing frustums of the camera. The user can take photographs of arbitrary landmarks on the way to build billboards with photographs corresponding to them on the map. Subjective evaluation by eight participants showed that the proposed method offers improved experiences over navigation using a standard 2D map.

This paper presents high speed and high accuracy visual servoing system. The algorithm has three major improvements, which can be implemented in practical applications; Firstly high-accuracy pose estimation by using stereo cameras, secondly real time implementation issues with non-real-time image processing platform and thirdly a consideration for industrial position controller. To resolve the issues, position-based visual servoing (PBVS) is adopted and appearance model based virtual visual servoing (VVS) is applied for pose estimation. VVS approach does not compute the stereo matching but directly compares the OpenGL rendered image and camera image for each camera; estimate the position/orientation using VVS independently for each camera; and provides a theoretically optimal compromise among those estimates. To enhance estimation accuracy, a hybrid method of stereo trigonometry for position estimation and weighted least squares for orientation estimation is proposed to combine the information from the stereo cameras. Operation speed is increased by using graphic processing unit (GPU) acceleration and an on-line trajectory generator which can accommodate the variable cycle of the image processing and the fixed cycle of a common robot controller. Finally, some experimental results illustrate the effectiveness of the proposed framework.

This paper presents a system framework to realize the automatic visual inspection with random initial object poses. A series of Computer Graphics (CG) images rendered in OpenGL are compared with a real scene image from a fixed environment camera. With an iterative heuristic search approach for the maximum similarity in these comparisons, robust 3D object pose estimation has been obtained to place an inspection camera to its new initial pose. Then a visual servo technique is used to accurately place this camera so that a reliable visual inspection can be carried out. Contrast with those feature-based or texture-based approaches where textures are required, only the object's geometry information is needed in this framework, therefore this system can deal with the untextured objects (for instance metal parts) well. This paper describes the system configuration and operational procedure in detail. Experiments results are presented to evaluate the system performance.

In biological and medical sciences, optical microscopes are widely used. By using the microscope, we can observe cellular dynamics including intracellular ions and molecules tagged with fluorescent dyes at a high magnification. However, a freely motile cell easily escapes from a field of view and a focal plane of the typical microscope. Therefore, we propose a novel auto-focusing (AF) algorithm and develop an AF and tracking microscope. XYZ positions of a microscopic stage are visual feedback controlled to focus and track the cell automatically. A bright-field image is used to estimate a cellular position. XY centroids are used to estimate XY positions of the tracked cell. To estimate a Z position, we use a diffraction pattern around the cell membrane. This estimation method is so-called Depth from Diffraction (DFDi). However, this method is not robust for individual differences between cells because the diffraction pattern depends on each cellular shape, reflective index and transparency. Therefore, in this study, we propose a real-time correction of DFDi by using 2D Laplacian of a center region of the cell as a goodness of the focus. To evaluate the performance of our developed algorithm and microscope, we auto-focus and track a freely moving paramecium. In this experimental result, the paramecium is auto-focused and kept inside the scope of the microscope during 45s. The evaluated focal error is within 5[µm], while a length and a thickness of the paramecium are about 200[µm] and 50[µm], respectively.

In this paper, to realize robust tracking, we propose a particle filter (PF) model to track a single paramecium. The proposed PF model consists of a system dynamical model and an observation model. The information about our tracking object is described by a state vector and the system state is assumed to evolve according to the system dynamical model. The parallel region-based level set method with displacement correction (PR-LSM-DC) proposed in our previous work now works as the measurements for the PF model. The tracking is achieved by estimating the state of a moving object from the observations. Experiments show that with motion prediction using the PF model, we increase the robustness of tracking and extend the duration of single paramecium tracking. The 2 [ms] computational time indicates that we developed an algorithm and a computer aided system which achieves nonrigid single micro-organisms tracking in real-time as they deform, move and collide with others under optical microscope.

We propose a new approach for augmented reality, in which real-world scene images are augmented with video fragments manipulated in the time domain. The proposed system aims to display slow-motion video sequences of moving objects instantly without accumulated time lag so that a user can recognize and observe high-speed motion on the spot. Images from a high-speed camera are analyzed to detect regions with important visual features, which are overlaid on a normal-speed video sequence.

The problem considered here involves the design and application of a recursive algorithm to extract and predict the position of an object in a 3D environment from one feature correspondence from a monocular image sequence. Translational model involves an object moving in a parabolic path using projectile physics. A state-space model is constructed incorporating kinematic states, and recursive techniques are used to estimate the state vector as a function of time. The measured data are the noisy image plane coordinates of object match taken from image in the sequence. Image plane noise levels are allowed and investigated. The problem is formulated as a tracking problem, which can use an arbitrary large number of images in a sequence. The recursive estimation is done using Recursive Least Squares (RLS). Results on both synthetic and real imagery illustrate the performance of the estimator.

This paper proposes a visual tracking method which is robust to occlusion. This paper also integrates the visual tracking method and visual servo control into a vision-based control method with occlusion handling. The proposed method chooses a set of correctly extracted image features, and it then obtains an estimate of all the image features from the correctly extracted image features. The estimation procedure makes it possible to track image features even when occlusion occurs. The method has low computational complexity, since the image Jacobian is used for the image feature selection and estimation. In addition, even when the controller fails to track a moving image feature, it can find the failed image feature without a global search over the entire image plane. As a result, it can track the failed image feature again quickly.

This paper proposes a method for identifying an object which contains an accelerometer out of many moving objects in the view of a stationary camera using motion data obtained by the camera and the accelerometer. The camera and the accelerometer are assumed to be connected with a network but not synchronized. In order to evaluate the similarity of the motion data despite the unknown time lag between the accelerometer and the camera, NCC (normalized cross-correlation) of the signals is computed and its peak is tracked. Since the coordinate system of the accelerometer is unknown, NCC is computed for the norms of the acceleration vectors, which were compensated for the gravitational acceleration component, obtained by the camera and the accelerometer. The experimental results show that the proposed method successfully identified the person wearing the accelerometer out of three walking people. It is also shown that the hand holding the accelerometer was successfully identified out of three moving hands even though the directions of the accelerometer coordinate axes varied temporally due to the free motion of the hand.

This paper proposes an assistant and training system for controlling an unmanned helicopter. The unmanned helicopter does not have any sensors which measure its position or posture. Stationary cameras are placed on the flight field. The helicopter is controlled by using a visual servo technique as follows. An operator steers the helicopter using sticks on a hand-held input device. The sticks make reference signals. The assistant system designs control signals such that the helicopter tracks the reference signals. The proposed system has the following four functions: automatic takeoff and landing, control channel selection, flight in a desired area, and automatic motion generation. They enables beginners to control an unmanned helicopter. The system provides real actions of unmanned helicopters. This is the main difference from flight simulators.

A 1,000-jps camera-projector system in which projected patterns are adoptively controlled according to image processing results is described. Adaptive structured light projection enables fast and efficient 3-D information acquisition. The prototype system is applied to the tracking of the nearest point of an object, and experimental results show that the system successfully tracked an apex of a fast-moving large! object.

This paper describes a new reconfigurable processor architecture specialized for high frame rate visual processing. This architecture employs a 2-D mesh processing element (PE) array in which the PEs can be configured to operate as SIMD arrays or operation-pipeline trees depending on image processing algorithms so that maximum on-chip memory consumption is reduced. To achieve high on-chip memory utilization, the architecture features that the instruction register in each PE is mapped in its local memory space and that the ALU network and the local memory network can be configured independently. Simulation results show that the proposed architecture effectively utilizes both of the SIMD and operation pipeline modes.

This paper proposes a high frame rate 3-D shape measurement system based on structured light projection using temporally-coded binary patterns. This system consists of a DMD (digital micromirror device) projector and a high-speed camera, both of which operate at over 6,000 fps. Introducing an image processing algorithm of detecting and tracking the projected light patterns enables frame-by-frame 3-D image measurement. Experimental results show that the proposed algorithms worked effectively, and the system achieved 3-D imaging at 3,350 fps.

This paper proposes a switched visual feedback control method for a micro helicopter under occlusions. Two stationary cameras are placed on the ground. They track four black balls attached to rods connected to the bottom of the helicopter. The control input is computed by using the errors between the positions of the tracked objects and pre-specifled reference values for them. The multi-camera configuration is redundant for helicopter control, but it enables us to design a switched controller which is robust against occlusions. An occlusion occurs when an object moves across in front of a camera or when the background color happens to be similar to the color of a tracked object. Multi-camera systems are suitable for designing a robust controller under occlusions, since even when a tracked object is not visible in a camera view, other cameras may track it. The authors have proposed a camera selection approach: If an occlusion is detected in a camera view then the other camera is used to control the helicopter. This paper presents another switched visual feedback control method for a micro helicopter under occlusions. This is called the image feature selection approach. This paper assumes that at most one tracked object is occluded at each time to simplify notations, although we can draw general conclusions. The errors between the positions of the tracked objects and pre-specifled references are used to compute the control input, when all the tracked objects are visible. If one of the tracked objects is invisible, then the controller uses the errors given by the other three tracked objects. The position of the occluded object is also estimated by using the other three tracked objects.

This paper proposes a new autofocusing method for microbiological specimens, such as cells. The proposed focusing method used a quick focus estimation named "depth from diffraction", based on a diffraction pattern in a defocused image of a biological specimen. Since this method can estimate the focal position of the specimen from only a single defocused image, it can easily realize high-speed autofocusing. To demonstrate the proposed method, continuous focusing was applied to focus tracking of a swimming paramecium, in combination with two-dimensional tracking. This allowed three-dimensional tracking of the paramecium for 75 s to be successfully demonstrated

We propose a novel method to control a lot of microorganisms by using visual feedback for micro-robotic application. Our goal is to control a cluster of microorganisms as microscale smart robots for various applications, since microorganisms have efficient actuators and accurate sensors. Compared with single-cell level actuation methods proposed before, this method has the following advantages: (1) a cluster of microorganisms can cancel stochastic perturbations of behavior by taking the average; (2) a cluster of microorganisms can be more powerful than a single microorganism, and thus it can perform a variety of tasks. A visual feedback system is constructed to control cells using galvanotaxis (response to the electric field). Experimental results show the feasibility of visual feedback control of a paramecium cell cluster. In addition, micro-manipulation of small objects by the cell cluster is also demonstrated

In this paper, we present a robot positioning task with respect to a static target using visual servoing and optimization techniques. The vision system is uncalibrated. The displacements of the robot are generated in real time in order to minimize an objective function using a simplex method and a Newton-like method. Our method allows for the inclusion of constraints in the image as well as in joint space. On-line image Jacobian estimation runs at the same time, and is used to accelerate convergence near the target. We successfully validate this method with simulations under the graphic library OpenGL, and practical experiment with industrial manipulators.

We propose a dynamics model of galvanotaxis (locomotor response to electrical stimulus) of the protozoan Paramecium. Our purpose is to utilize microorganisms as micro-robots by using galvanotaxis. For precise and advanced actuation, it is necessary to describe the dynamics of galvanotaxis in a mathematical and quantitative manner in the framework of robotics. However, until now the explanation of Paramecium galvanotaxis in previous works has remained only qualitative. In this paper, we construct a novel model of galvanotaxis as a minimal step to utilizing Paramecium cells as micro-robots. Numerical experiments for our model demonstrate realistic behaviors, such as U-turn motions, like those of real cells.

We propose a visual control system for motile cells. Our goal is to control microorganisms as microscale smart robots for various applications. As a first step, we have developed a visual feedback control system for Paramecium caudatum cells. In order to ensure both detailed measurements and a large working space, "lock-on" tracking of a free-swimming cell with a high frame rate is essential. In our system, high-speed (1-kHz frame rate) tracking hardware and software are used for the continuous observation of moving cells with high magnification. Cells swim in a chamber, and their positions and other properties are measured and computed in real time. The chamber position is visually controlled automatically to track a specific cell. The cell motion is controlled electrically by utilizing the galvanotaxis (intrinsic reaction to electrical stimulus) of microorganisms. Experimental results for open-loop control (periodic zigzag motion) and closed-loop control (trapping within a small region that is 1 mm wide) demonstrate the possibility of using microorganisms as micromachines.

The problem of estimating the camera pose in eye-in-hand visual servoing is considered for the case in which the available point correspondences are less than eight. Two strategies based on the properties of the essential matrix are hence described, which mainly consist of the computation of the roots of a one-variable polynomial in the seven points case and the minimization of a two-variables polynomial through a gradient algorithm in the six points case.

A new robotic hand system with high-speed finger motion of 180 [deg] per 0.1 [s] has been developed. With high-speed visual feedback at a rate of 1 [kHz], various dynamic manipulations are achieved. As examples of dynamic manipulation we describe the dynamic active catching tasks for a falling ball and a falling cylinder using a high-speed multifingered hand. We achieved these difficult tasks by utilizing high-speed motion and impact forces actively. Experimental results using active catching strategy based on visual feedback are shown.

We have proposed a GAG (Gaining Angle of Gaze) strategy for ball catching. The strategy allows single camera configuration and it is robust against camera motion errors. It is superior than previously proposed strategies in the field of experimental psychology in the catchable region and robustness. In this paper, we apply the GAG strategy for tracking and catching a 3D flying target. The GAG strategy is extended for hand-eye robot manipulators and ball catching performance is evaluated by simulations and experiments.

We propose a control system of motile cells. Our goal is to construct a large-scale Organized Bio-Modules (OBM) in which microorganisms are controlled as micro-size smart robots in an organized way. For the first step, we have developed a visual feedback control system of Paramecium caudatum cells. Tracking method is used for observation of moving cells with high magnification. Cells swim in a chamber and their positions are measured by high-speed vision. The chamber position is visually controlled to track a specific cell. The cell motion is controlled electrically by utilizing the galvanotaxis (intrinsic reaction to electrical stimulus) of microorganisms. Experimental results of region-trapping demonstrate the possibility of the OBM system.

This paper presents an examination of ball lifting task using a mobile robot. We have proposed a trajectory control method for tracking and catching a ball flying in 3D space based on GAG (gaining angle of gaze) strategy. The strategy allows single camera configuration and it is robust against camera motion errors. This simple method was implemented to a mobile robot and we demonstrated the quick motion to catch balls. We consider the extension of this method for the ball lifting task and demonstrate the robot can track and rebound a ball repeatedly as a basic examination of the task.

A simple technique for estimating the camera displacement from point correspondences in eye-in-hand visual servoing is presented. The idea for providing more accurate results than existing methods consists of taking into account that the point correspondences used during the camera motion correspond to stationary spatial points, hence exploiting additional information. This is done by first estimating the object Euclidean structure and then estimating the camera displacement from this estimate.

A visual servoing strategy for keeping features in the field of view is proposed which consists of a switching among position-based control strategies and backward motion. In the absence of uncertainty on the extrinsic parameters, all features are kept in the field of view. Moreover, if the intrinsic parameters are also known, the trajectory length is minimized in the rotational space and, for some cases, also minimized in the translational space. Simulation results also show a certain degree of robustness against uncertainty on the intrinsic parameters.

The problem of estimating the camera displacement in eye-in-hand visual servoing is considered, and a simple strategy based on the idea that the estimates accuracy can be improved if the fact that the point correspondences used throughout the visual servoing are relative to the same 3D points is taken into account is presented. In particular, an accurate scaled euclidean reconstruction of the object is built in the first steps of the visual servoing by suitably using existing linear methods, and from this reconstruction the camera displacement is suitably estimated. Extensive proves performed in random conditions have shown that the proposed approach provides significantly better results with respect to the existing linear methods actually used in visual servoing.

In this paper, a visual servoing strategy for dealing with the problem of keeping the observed points in the camera field of view is proposed. The approach consists of a switching control law based on camera displacement estimation and regulated from the position of the points in the image. In absence of uncertainties on the intrinsic parameters and optical axis direction, global stability is achieved and all points are kept in the field of view. Moreover, the trajectory length is minimized in the rotational space and, for some cases, also minimized in the translational one. Robustness against uncertainties is also guaranteed.

The paper proposes an algorithm for object motion estimation. It is assumed that the object motion is composed of primitive motions and the total motion is generated by simply switching these primitive motions. The proposed estimator for visual servoing has two stages: Kalman filter and validation gate. Models of primitive motions are connected and an augmented system is designed. Each primitive motion corresponds to an invariant subset in the state space of the augmented system. The motion switching is generated by a state transition among the state subsets. A Kalman filter is used to predict the object motion on the basis of the augmented system. Switching of actual object motion is detected by the validation gate. The validation gate is implemented by nonlinear least square fitting of the current state onto the nearest state subsets. Real-time experiments of object tracking demonstrate the effectiveness of the proposed estimator.

We propose a visual inspection method of defect detection on textile surfaces; selectively removing the textural information from textile surfaces. For this purpose we use wavelet shrinkage, which was originally proposed by Donoho and Johnstone as a method to remove Gaussian white noise (1994, 1995). We also propose a modification of wavelet shrinkage to selectively remove textural information as noise. Once the textural information, is removed from textile surfaces, the remaining inspection processes can be conducted in the image domain. In this way, we can utilize the image processing methods and techniques developed so far for the visual inspection of other industrial products.

Stability of feature-based visual servoing controllers proposed so far is local. The initial features far from the reference may converge to features different from the reference or, even worse, they may not converge. In this paper, the stability of feature-based visual servoing is considered by using potential. The stable region is visualized and artificial potential switching is proposed to extend the stable region.

This paper proposes a potential switching scheme that enlarges the stable region of feature-based visual servoing. Relay images that interpolate initial and reference image features are generated by using affine transformation. Artificial potentials defined by the relay images are patched around the reference point of the original potential to enlarge the stable region. Simulations with simplified configuration and experiments on a 6 DOF robot show the validity of the proposed control scheme.

The paper proposes a visual servoing scheme based on object motion estimation. A long term prediction algorithm with multiple motion models is developed. The object position and velocity of a few hundreds of samples ahead is predicted. Thus object tracking without visual information can be achieved during this period. As an example, pick up task of moving object is conducted. The task requires object motion prediction because the hand mounted camera loses the object when the hand approaches the object. Simulations show convergence of prediction. Real-time experiments of picking up a moving object demonstrate the effectiveness of the proposed long term predictor.

One of the most important problems in feature-based visual servoing is the slow sampling rate and the delay of the camera that can make the closed loop system oscillative or unstable easily. In the paper, a linearized observer that estimates the object velocity and updates the visual information with the joint sampling rate is proposed. Stability of the observer-based control system and effectiveness of the observer are verified by experiments on a PUMA 560 robot.

Describes the relationship between the control performance and the number/configuration of the image features in feature-based visual servoing. The performance is evaluated by two ways: accuracy and speed. A quantitive definition of the sensitivity is given and the relationship among the sensitivity, the speed of convergence and the accuracy are discussed by using the image Jacobian. It is proved that these performance indices are increased effectively by using a point with different height. Experiments on Puma 560 are given to show the validity of these performance measures.

This paper presents a visual feedback control scheme for a nonholonomic cart without capabilities for dead reckoning. A camera is mounted on the cart and it observes cues attached on the environment. The dynamics of the cart are transformed into a coordinate system in the image plane. An image-based controller which linearizes the dynamics is proposed. Since the positions of the cues in the image plane are controlled directly, possibility of missing cues is reduced considerably. Simulations are carried out to evaluate the validity of the proposed scheme. Experiments on a radio controlled car with a CCD camera are also given.

Yo-yo is a toy made of two thick circular pieces of wood, plastic, etc., connected with a short axle that can be made to run up and down, by moving a string tied to it. Humans can play with a yo-yo without difficulty. However, developing a robot system that can play with a yo-yo presents a significant challenge to controller design, because the dynamics of the yo-yo are difficult to model precisely. Moreover, since the dynamics are not continuous and there are integral-type constraints on the hand trajectory, conventional continuous time feedback control theory does not work well. This paper present a model and a control scheme for robotic yo-yo with visual feedback. Experiments on a PUMA 560 are carried out to evaluate the validity of the discrete dime formulation and the controller design.

This paper proposes a control theoretic formulation and a controller design method for the feature-based visual servoing with redundant features. The linear time-invariant (LTI) formulation copes with the redundant features and provides a simple framework for controller design. The proposed linear quadratic (LQ) method can deal with the redundant features, which is important because the previous LQ methods are not applicable to redundant systems. Moreover, this LQ method gives flexibility for performance improvement instead of the very limited design parameters provided by the generalized inverse and task function controllers. Validity of the LTI model and effectiveness and flexibility of the LQ optimal controller are evaluated by real-time experiments on a PUMA 560 manipulator.

Visual servo system is a robot control system which incorporates the vision sensor in the feedback loop. Since the robot controller is also in the visual servo loop, compensation of the robot dynamics is important for high speed tasks. Moreover estimation of the object motion is necessary for real time tracking because the visual information includes considerable delay. This paper proposes a nonlinear model-based controller and a nonlinear observer for visual servoing. The observer estimates the object motion and the nonlinear controller makes the closed loop system asymptotically stable based on the estimated object motion. The effectiveness of the observer-based controller is verified by simulations and experiments on a two link planar direct drive robot.

This paper proposes a visual feedback control scheme for a manipulator with a camera on the hand. Incorporating the dynamics of the visual information into the manipulator control, we formulate a state space representation of the visual servo mechanism. An optimal control approach to visual servoing is developed based on the state space formulation. The proposed algorithm gives a simple time-invariant state feedback controller, which allows a high sample rate control system. Real-time experiments of object tracking with a PUMA 560 manipulator are conducted to show the validity of the optimal control scheme. Experimental results show the superiority of the proposed approach compared to the conventional visual feedback control algorithms.

A visual feedback control scheme, called image-based visual servo, is proposed for manipulators with cameras on their hands. To accomplish specific tasks in unstructured environments, it is essential to have capabilities of recognizing the object position with respect to the manipulator hand. Using the Jacobian matrix relating the camera motion to the object position change in the acquired image, a state space formulation of a visual feedback system is established. On the basis of the task defined in the image plane, the desired motion of the hand is achieved by time-variant state feedback. The image-based scheme is applied to a task of tracking a moving object by the camera (and the hand). Simulations show that the control scheme is stable even under noisy conditions, while the conventional position-based servo tends to be unstable. Experiments on a PUMA 560 show the validity and the effectiveness of the image-based visual servo.>

Mechanical manipulators are modeled by a set of coupled nonlinear second-order differential equations. When a complete model of the manipulator dynamics is obtained, model-based controllers which decouple and linearize the manipulator dynamics can track the desired trajectories uniformly in all configurations of the manipulator. However, there is a degree of uncertainty in the parameters of the manipulator, especially if it is carrying unknown loads. Therefore, model-based control schemes that are robust against modelling errors and disturbances are required. This paper proposes a robust control scheme based on H/sup infinity / control theory. Given a nominal model of the manipulator dynamics, the controller consists of a model-based controller followed by a H/sup infinity / controller. The model-based controller approximately linearizes the manipulator dynamics. The H/sup infinity / controller compensates for disturbances and unmodeled dynamics. Simulations and experiments on a two-link direct drive manipulator exhibits the effectiveness of the proposed H/sup infinity / model-based robust control scheme.>