Unmanned Aerial Vehicles (UAVs) play a crucial role in various scenarios ranging from disaster response to traffic surveillance. However, aerial video footage often suffers from severe motion blur due to rapid flight maneuvers, vibrations, and camera panning, which can significantly degrade downstream tasks such as target detection. Our goal is to explore a computationally-efficient and effective video deblurring approach to enhance UAV target detection performance. To reduce computational cost, we first propose an Adaptive Latent Scale Selector that dynamically adjusts the latent space resolution according to the intensity of UAV motion, thus balancing detail preservation with inference efficiency. To ensure temporal consistency, we introduce a Multi-Frame Alignment and Learnable Gating module to warp and gate the preceding frames, allowing the model to fuse only relevant temporal information and suppress misaligned or uninformative features. Our method can effectively recover sharp details from the UAV video stream. Extensive experiments on real UAV benchmarks demonstrate that our method not only yields superior deblurring performance but also significantly boosts target detection accuracy, making it highly applicable to robust aerial vision tasks. Code will be publicly available here.

Underwater images suffer from color distortion, haze, and poor visibility due to light refraction and absorption in water. These challenges significantly impact the utilization of Autonomous Underwater Vehicles (AUVs) or marine robots. Typically, color and brightness distortions manifest at lower frequencies, while edge and texture distortions are prevalent at higher frequencies. Traditional methods struggle to concurrently rectify these mixed distortions as they primarily concentrate on the spatial domain. To address these issues, we introduce the Dynamic SpectraFormer, which enhances under-water images through a frequency domain transformer. The Dynamic SpectraFormer introduces an ultra-high-resolution sparse spectrum attention module, which could capture the long-term dependency without losing the universal approximating power. Additionally, we have developed a dynamic spectrum weight generation layer that serves as an adaptive spectrum band selector, accentuating critical frequency bands and suppressing less relevant ones. Consequently, this method significantly improves underwater image quality by addressing both high-and low-frequency distortions. Our extensive ablation studies and comparative evaluations consolidate the Dynamic SpectraFormer’s efficacy across multiple underwater image enhancement benchmarks. The source code is available at https://github.com/arifence2024/DynamicSpectraFormer.git.

Detecting the micromovements of eyes that reflect a person’s inner state can be an essential step in many applications, but most eye trackers need to fix the subject’s head using a chin rest to obtain sufficient data quality. We propose a stand-alone eye tracker that utilizes two 500 fps cameras, a pair of rotating mirrors for gaze control, a liquid lens for focus control and an intensity-controllable light source, and describe how the proposed system works in real-time. The experimental results show that our system covers more than twice as wide a measurement range in the depth direction as the conventional eye tracker while achieving sufficient data quality to analyze microsaccades with an amplitude of down to 0.2 deg. We also examine the practical use of the proposed system for microsaccade detection.

Projection mapping (PM) exhibits suboptimal performance in well-lit environments because of the interference caused by ambient light. This interference degrades the contrast of the projected images. Consequently, conventional methodologies restrict the application of PM to dimly lit settings, leading to an unnatural visual experience, as only the PM target is prominently illuminated. To overcome these limitations, we introduce an innovative approach that leverages a mixed light field, blending traditional PM with ray-controllable ambient lighting. This methodological combination, despite its simplicity, is effective because it ensures that the projector exclusively illuminates the PM target, preserving the optimal contrast. Precise control of ambient light rays is essential to prevent them from illuminating the PM target while adequately illuminating the surrounding environment. Furthermore, we propose the integration of a kaleidoscopic array with integral photography to generate dense light fields for ray-controllable ambient lighting. Additionally, we present an efficient binary-search-based calibration method tailored to this intricate optical system. Our optical simulations and the developed system collectively validate the effectiveness of our approach. Our results show that PM targets and ordinary objects coexist naturally in environments that are brightly lit as a result of our method, enhancing the overall visual experience.

The authors propose a spatiotemporal noise reduction system called Foveated Noise Reduction. This system reduces ambient motion noise synchronized with eye movements and spatiotemporal blurring in peripheral vision. Preliminary experiments were conducted with the visual search task in ambient motion noise. The authors evaluated task performance when varying spatiotemporal blurring parameters. The results suggest that the spatial reduction affected the task performance, and the blur size would be suitable under the visual search task with ambient noise.

In daily life, we realize various complex tasks with our two upper limbs based on the so-called bimanual coordination phenomenon. A fundamental feature of bimanual coordination is the natural tendency to synchronize the motion of two upper limbs, resulting in some preferred patterns of interlimb coordination. In this study, based on the coarse-to-fine human-robot collaboration framework, we investigate the possibility of realizing human-robot interaction and collaboration for accurate manipulation utilizing voluntary bimanual coordination. The practical motivation of utilizing the voluntary bimanual coordination, which can be perceived as an indirect way of implementing interlimb transmission of force feedback information, is to avoid the bad effect on force feedback presentation due to the counterforce if the human-robot collaboration was realized in an unimanual manner. For experimental studies, we firstly evaluated the synchronous protocols in terms of in-phase and anti-phase between two arms. Based on the studied protocol, moving target tracking was then demonstrated as an application scenario of the proposed human-robot collaboration utilizing voluntary bimanual coordination.

In this study, we discuss the imaging of depth and velocity using heterodyne-mode time-of-flight (ToF) cameras. In particular, Doppler ToF (D-ToF) imaging utilizes heterodyne modulation to measure the velocity from the Doppler frequency shift, which uniquely facilitates the instantaneous radial velocity estimation. However, theoretical discussion on D-ToF is limited to orthogonal frequency and sinusoidal waveform modulation. This study extends the formulation of the D-ToF imaging, and proposes an arbitrary-frequency, arbitrary-waveform framework considering a phase-compensated, symmetrical two-dimensional correlation map. With the proposed framework, the optimal heterodyne frequency for frequency decoding is found. A differential frequency sampling and decoding method is then proposed, which computes the frequency and phase from as few as four simultaneously captured images. With an experiment platform we built, it is confirmed that the minimum velocity sensing error is half that of the orthogonal frequency method, and the sensible phase range is approximately 2.5 times larger. The conclusions in this study allow the ToF velocity imaging to be applied at the optimal sample frequencies for a wide range of ToF sensors. This pushes one step further to the practical use of ToF velocity imaging.

High-speed 3D shape sensing is an essential technology for three-dimensional recognition and manipulation in dynamic scenes. However, conventional high-speed sensing methods mainly focus on the image capturing speed and the actual processing time to obtain a point cloud is not optimized in the measurement scheme and sensing pattern configuration. On the other hand, measurement latency is critical to respond in real-time and physically handle the dynamic scenes. This paper introduces a physically stereo-rectified projector-camera system for high-speed and low-latency 3D sensing with fast sequential memory access in the decoding process. Moreover, we configure a structured light pattern named “Parallel-bus pattern” with a De Bruijn torus and clock lines to maximize information density and robustly decode the pattern as data transfer in a parallel bus interface. We measured dynamically moving and deforming objects with the proposed system and evaluated the measurement performance. As a result, the developed 3D sensing system with the parallel-bus pattern achieved 27K-points measurement at higher than 1000 fps with 0.336 ms latency and 0.838 mm accuracy on average.

In spatial augmented reality, a widely dynamic projection mapping system has been developed as a novel approach to graphics presentation for widely moving objects in dynamic situations. However, this method necessitates a novel tracking marker design that is resistant to random and complex occlusion and out-of-focus blurring, which conventional markers have not achieved. This article presents a uniform circumferential marker that becomes an ellipse in perspective projection and expresses geometric information. It can track the relative posture of a dynamically moving sphere with high speed, high accuracy, and robustness owing to continuous contour lines, thereby supporting both wide-range movement in the depth direction and human interaction. Moreover, a biased circumferential marker is proposed to embed unique coding, where the absolute posture is decoded with a novel recognition algorithm. Moreover, rough initialization using the geometry of multiple ellipses is proposed for both markers to start the automatic and precise tracking. Real-time rotation visualization onto the surface of a moving sphere is made possible with the high-speed, widely dynamic projection mapping system. The tracking performance is demonstrated to exhibit sufficient basic tracking performance as well as robustness against blurring and occlusion compared to conventional dot-based markers.

We propose an accurate and robust inter-vehicle distance estimation method using highspeed stereo vision. The framework involves two phases: a tracking phase, wherein a preceding vehicle is accurately and stably tracked by a tracking algorithm optimized for stereo high-speed vision, and a distance estimation phase, wherein the inter-vehicle distance is estimated via a highly accurate scale estimation and aggregation method for multiple scale-based distance estimations to ensure that it is more accurate and robust without introducing a delay. Further, we propose patch multiplexing to realize accurate and efficient aggregation even in situations where the scale changes rapidly (e.g., emergency braking). Through comparative analysis using three real-world scenarios, we verify that the accuracy of inter-vehicle distance estimation using our approach is comparable to that of laser rangefinders. We also demonstrate that differential quantities, such as velocity and acceleration, could be accurately estimated using an adaptive Kalman filter. Our results will help develop safe and accurate truck platooning and adaptive cruise control systems.

High-speed finger tracking is necessary for augmented reality and operation in human-machine cooperation without latency discomfort, but conventional markerless finger tracking methods are not fast enough and the marker-based methods have low wearability. In this paper, we propose an ellipses ring marker (ERM), a finger-ring marker consisting of multiple ellipses and its high-speed image recognition algorithm. The finger-ring shape has highly wearing continuity, and the surface shape is suitable for various viewing angle observation. The invariance of the ellipse in the perspective projection enables accurate and low-latency posture estimation. We have experimentally investigated the advantage in normal distribution, validated the sufficient accuracy and computational cost in the marker tracking, and showed a demonstration of dynamic projection mapping on a palm.

Image projection on spheres and their surroundings have been researched for all-around display and motion visualization. However, wide-range projection onto a dynamic sphere can suffer from tracking errors and projection latency. We propose a high-speed projection mapping system for dynamic spheres, and circumferential markers for sphere posture estimation. Marker detection based on ellipse appearance can easily start tracking, and is robust against interactive occlusion; therefore, the system enables projection mapping and rotational visualization for a dynamically moving sphere. We experimentally confirmed sufficient pose estimation accuracy of the circumferential markers compared with initialized dot markers, and sufficient tracking ability against initialization, occlusion, and depth-direction movement. Demonstrations showed accurate, rotation-visualized projection onto a dynamic sphere, which is useful for sports practice applications.

High-speed camera imaging (e.g., 1,000 fps) is effective to detect and recognize objects moving at high speeds because temporally dense images obtained by a high-speed camera can usually capture the best moment for object detection and recognition. However, the latest recognition algorithms, with their high complexity, are difficult to utilize in real-time applications involving high-speed cameras because a vast number of images need to be processed with no latency. To tackle this problem, we propose a novel framework for real-time object recognition with high-speed camera imaging. The proposed framework has the key processes of population data cleansing and data ensemble. Population data cleansing improves the recognition accuracy by quantifying the recognizability and by excluding part of the images prior to the recognition process, while data ensemble improves the robustness of object recognition by merging the class probabilities with multiple images from the object tracking sequence. Experimental results with a real dataset show that our framework is more effective than existing methods.

In this study, a visual shock absorber capable of adapting to free-fall objects with various weights and speeds is designed and realized. The key element is a magnetic gear to passively absorb shock in the moment of contact, which is difficult for traditional feedback control to deal with. The magnetic gear allows the seamless transfer of control from the non-contact state to the contact state. 1000 Hz high-speed visual object tracking is used for preparation with position and velocity control in the object non-contact state. In the moment of object contact, the high backdrivability of the magnetic gear response by hardware provides high responsiveness to external force. After the impact, the plastic deformation control of a parallel-expressed Maxwell model handles the contact state.

In this study, Magripper, a highly backdrivable gripper, is developed to achieve high-speed hitting grasping executed seamlessly from reaching. The gripper is designed to achieve both high speed and environmental adaptability. The key element is backdrivability in terms of both hardware and control. In Magripper, a magnetic gear is introduced to passively absorb shock in the moment of contact as a means of hardware backdrivability, and backdrive control is implemented based on the Zener model. After developing a hitting grasping framework, high-speed hitting grasping tasks with a wood block, a wood cylinder, and a plastic coin are conducted using only servo control without sensors, such as cameras and tactile sensors. In particular, coin grasping with high-speed movement is very difficult because collisions with environmental objects such as the floor and desk, are likely, which may break a robot.

We propose a projector-based method that provides an illusion of geometry change, similar to that caused by a change in physical properties such as elasticity, in response to inertia caused by physical motion. The proposed method is named “ElaMorph projection.” Although several projection mapping methods capable of deforming targets have been previously proposed, these methods require the preparation of animations in advance. Moreover, these methods are unable to deform the shape in real-time according to actual movements of the object. To address these issues, we perform real-time geometry deformation and rendering based on the spatial motion of the object. To render the projection image, we extend the conventional method of deformation for 2D pictures or static 3D objects to include dynamic 3D objects. This study involves projection onto a dynamic 3D object; however, the projection quality decreases if a part of the rendered image extends beyond the projection target. To address this issue, the proposed algorithm ensures that the vertices after deformation always remain within the projection target. In addition, we develop a robust algorithm to generate projection images under dynamic illuminative conditions, through real-time estimation of the environmental lighting required for rendering. Moreover, using an elasticity map that can be easily constructed using a UV map, our method enables users to specify the vertices to be deformed, using an elasticity map. We present projections under several different sets of elasticity maps, environmental lighting, and elasticities. Finally, we evaluate the latency and throughput of our system.

In this study, human-robot interaction with multimodal perception interface combining human visual and haptic perception is introduced. In the proposed human-robot collaboration method, cognitive capabilities of human and accurate motion control capabilities of robot are integrated based on a coarse-to-fine strategy. Human operator is designated for coarse global motion under feedback interfaces utilizing human visual as well as haptic modalities. Simultaneously, fine local motion in an active manner without involving human intention-aware is realized by a robotic module with high-speed actuators and high-speed sensory feedback. Experiments demonstrated the effectiveness of the proposed method.

This study proposes a dynamic observable contact (DOC) hand as a new multifingered hand to ensure high- speed insertion in an assembly process with a small clearance between objects. To achieve insertion with a small clearance at high speed, a robot hand must realize both impact reduction and position-error compensation when the two objects contact each other. The DOC hand, with its features of 6-degrees-of- freedom dynamic passivity and object-pose observability, can realize both impact reduction and position-error compensation. To evaluate the effectiveness of the DOC hand, we construct a robot system using the DOC hand. We evaluate the performance of the system in the task of ring insertion with a small clearance (0-36um). The results indicate that the robot system performs with a higher speed than a human. In fact, the average cycle time is 2.42 s for the robot, whereas it is 2.58 s for a human. The DOC hand has opened up the possibility for achieving high-speed precision assembly using robots.

Color-forming display can present digital vision by repeatedly changing its color; this display has been studied in recent research on augmented reality (AR) displays. Previous methods exhibit large-latency and low-speed problems, leading to the lack of interaction in immersive AR experiences. This study proposes a highspeed and low-latency drawing method on dynamically moving objects, using photochromic material colored by ultraviolet light laser, high-speed mirror control, and visual feedback. The experiment evaluated the method's drawing accuracy and we confirmed that this method has an adequate tracking ability against random rotation to produce accurate drawings.

We propose a novel displaying method that provides completely different visual perception to multiple observers simultaneously using afterimage effects for specific motion. Initially, the displayed patterns do not reveal any information; however, when seen by a user moving his or her gaze in a certain direction and speed, they are spatially integrated and appear as 2D afterimages. Our method only requires a high-speed display system to produce the user-oriented perception, which expands the range of applications in various situations such as in road signs.

Effects of latency in a visual feedback system during path-steering tasks were examined. Input movements were recorded in high speed and projected with controlled latency, while a user was required to drag an object through a tunnel path. We tested the system with 10 people and analyzed the duration based on the steering law and revealed that the latency of visual feedback beyond 64.3 ms was associated with reductions in user performance, which provides suggestions in developing user-oriented interface designs.

Based on the concept of spatial augmented reality (SAR), a number of applications have been developed by extending the type of target objects to include dynamically moving objects; however, the systems themselves still remain fixed. In this paper, we propose a new paradigm for dynamic SAR implemented with a mobile device and a practical architecture using a 3D-stacked vision chip and a high-speed optical gaze controller. We evaluated the prototype system that we developed, called “Portable Lumipen”, and confirmed that image capturing at 1000 fps, rapid processing and a 3 ms response time for projection were achieved on a mobile device. We also propose potential applications using this portable dynamic SAR system and demonstrated miniaturization of a high-speed visual feedback system.

Human-robot collaboration taking both human and robot's advantages becomes very promising in recent years. In this study, we propose a new active assistant robotic system for human-robot collaboration with the aim of optimally combining the cognitive capabilities of human and accurate motion control capabilities of robot. Under our method, human operator is provided with pneumatic haptic feedback to guide the system for coarse global-motion. Fine local-motion in an active manner is realized by a dynamic compensation robotic module with high-speed visual feedback. Since the active assist behavior here is confined to local motion and is accordingly safe to human operator. Experiments of preliminary investigation for set-point positioning and contour tracing under human-robot collaboration were implemented. Experimental results verified the effectiveness of the proposed method.

In this paper, we propose a robotic assistance system for carrying out a highly accurate peg-in-hole task by using a high-speed vision system and high-accuracy actuators. In this system, the vision system measures the position and the posture of the workpieces, and the actuators are moved according to this information. By using the high-speed vision system and operating the system at 500 Hz, a hole formed in one workpiece can be made to follow the complicated motion of a peg operated by a human and can be accurately aligned with the peg. In addition, since the torque required to move the hole is calculated on the basis of the position and direction data of the workpieces in the coordinate plane of the image captured by the vision system, the system is not affected by camera installation errors. Moreover, by using two cameras and combining the information from them, we made the tracking field wider. In order to evaluate the performance of the system, we executed a peg-in-hole task using a 50μm peg and a 70μm hole, and showed that the system completed the task with a success rate of 90%.

The regrasping function of a robotic hand and arm has been investigated by many studies. Dynamic regrasping is performed by accelerating objects and it has the advantage of being able to perform the regrasp function at high speed. However, the difficulty of increasing the success rate is a persistent problem. In this study, we aimed to realize this continuous high-speed operation by increasing the success rate of the regrasping function. The handling of the Rubik's cube was used as the specific task to be performed. The action that was required to handle the Rubik's cube consisted of two types of regrasping motion and one type of one-face turn motion. In this study, a Rubik's cube was placed in a plane and manipulated by combining these three types of motion. Continuous operation was realized with a robotic hand and high-speed vision by utilizing environmental constraints in order to minimize the error. As a result, we succeeded 3 times in turning and regrasping in 1 s. Additionally, we were able to succeed 30 times in turning and regrasping in 10 s, with a success rate of 70%.

This video introduces our recent studies on human-robot collaboration based on dynamic compensation framework with the aim of optimally combining the cognitive capabilities of human and accurate motion capabilities of robot. Under the dynamic compensation approach, human operator is for cognitive global-motion without caring much about accuracy. Fine local-motion in an active manner is realized by a dynamic compensation robotic module based on high-speed visual feedback. Application scenarios from micro-manipulation to macro-manipulation are implemented.

This study aims to investigate the impact on the user's performance when there is latency between the user's physical input to the system and the visual feedback. We developed a video latency control system to film the user's hand movements and control the latency when displaying the video (The standard deviation is 0.38 ms). The minimum latency of the system is 4.3 ms, hence this enables us to investigate the performance for unknown low latency ranges. Using this system, we conducted experiments wherein 20 subjects performed a pointing task based on Fitts' law to clarify the effect of video latency, particularly for low latency. Experimental results showed that when the latency is over 24.3 ms, the user performance begins to decrease. This result will be applied to determine a standard limit for video latency in interactive video devices.

Projectors, as information display devices, have improved substantially and to achieve both the wide range and high resolution is desired for the dynamic human gaze. However, a fixed projector has a trade-off between the angle of projection and a resolution with limited pixels. Conventional methods with dynamic optical axis control lack the potential speed of the devices. We propose a tracking projection mosaicing with a high-speed projector and a high-speed optical axis controller for a randomly moving position, such as the gaze. We also propose a synchronization strategy by queuing and alternating operations to reduce motion-based artifacts, which realize a high-quality static image projection during the dynamic optical axis control. We have experimentally validated the geometric and temporal consistency of the proposed synchronization method and have attempted a demonstration of the tracking projection mosaicing for the dynamically moving bright spot of a laser pointer.

The visual appearance of an object can be disguised by projecting virtual shading as if overwriting the material. However, conventional projection-mapping methods depend on markers on a target or a model of the target shape, which limits the types of targets and the visual quality. In this paper, we focus on the fact that the shading of a virtual material in a virtual scene is mainly characterized by surface normals of the target, and we attempt to realize markerless and modelless projection mapping for material representation. In order to deal with various targets, including static, dynamic, rigid, soft, and fluid objects, without any interference with visible light, we measure surface normals in the infrared region in real time and project material shading with a novel high-speed texturing algorithm in screen space. Our system achieved 500-fps high-speed projection mapping of a uniform material and a tileable-textured material with millisecond-order latency, and it realized dynamic and flexible material representation for unknown objects. We also demonstrated advanced applications and showed the expressive shading performance of our technique.

In this paper, we propose a pixel-wise deblurring imaging (PDI) system based on active vision for compensation of the blur caused by high-speed one-dimensional motion between a camera and a target. The optical axis is controlled by back-and-forth motion of a galvanometer mirror to compensate the motion. High-spatial-resolution image captured by our system in high-speed motion is useful for efficient and precise visual inspection, such as visually judging abnormal parts of a tunnel surface to prevent accidents; hence, we applied the PDI system for structural health monitoring. By mounting the system onto a vehicle in a tunnel, we confirmed significant improvement in image quality for submillimeter black-and-white stripes and real tunnel-surface cracks at a speed of 100 km/h.

Humans interact with deformable objects on a daily basis but this still represents a challenge for robots. To enable manipulation of and interaction with deformable objects, robots need to be able to extract and learn the deformability of objects both prior to and during the interaction. Physics-based models are commonly used to predict the physical properties of deformable objects and simulate their deformation accurately. The most popular simulation techniques are force-based models that need force measurements. In this paper, we explore the applicability of a geometry-based simulation method called meshless shape matching (MSM) for estimating the deformability of objects. The main advantages of MSM are its controllability and computational efficiency that make it popular in computer graphics to simulate complex interactions of multiple objects at the same time. Additionally, a useful feature of the MSM that differentiates it from other physics-based simulation is to be independent of force measurements that may not be available to a robotic framework lacking force/torque sensors. In this work, we design a method to estimate deformability based on certain properties, such as volume conservation. Using the finite element method (FEM) we create the ground truth deformability for various settings to evaluate our method. The experimental evaluation shows that our approach is able to accurately identify the deformability of test objects, supporting the value of MSM for robotic applications.

The technique of Projection Mapping, which is useful for merging real-world geometry with an augmented appearance, is a promising core technology for augmented reality (AR). In recent years, dynamically changing environments, mainly a consequence of the growing demand for interactive user experiences, have contributed to a new style of AR applications. However, performance levels of current systems for realizing 3D effects, in terms of the tracking speed and projection ability, are insufficient to meet these demands. In this paper, we present a high-speed, occlusion-robust marker-based 3D tracking method achieved by only using a monocular monochrome image. The objective of our research is to develop an automatic marker design method for any 3D shape and an effective framework for stabilizing tracking at high throughput by extending the latest promising work based on a deformable dot cluster marker [46]. Furthermore, this tracking method was used in combination with a high-speed projector, both of which can achieve high throughput and low latency, on the order of milliseconds. This enabled us to realize a high-quality computational display capable of representing the material appearance of a dynamically moving target. The demonstration showed that the effect of a dynamically changing appearance with nearly imperceptible latency drastically enriches the sense of immersion in the recognition of augmented materials with the naked eye.

Dynamic projection mapping for moving objects has attracted much attention in recent years. However, conventional approaches have faced some issues, such as the target objects being limited to rigid objects, and the limited moving speed of the targets. In this paper, we focus on dynamic projection mapping onto rapidly deforming non-rigid surfaces with a speed sufficiently high that a human does not perceive any misalignment between the target object and the projected images. In order to achieve such projection mapping, we need a high-speed technique for tracking non-rigid surfaces, which is still a challenging problem in the field of computer vision. We propose the Deformable Dot Cluster Marker (DDCM), a novel fiducial marker for high-speed tracking of non-rigid surfaces using a high-frame-rate camera. The DDCM has three performance advantages. First, it can be detected even when it is strongly deformed. Second, it realizes robust tracking even in the presence of external and self occlusions. Third, it allows millisecond-order computational speed. Using DDCM and a high-speed projector, we realized dynamic projection mapping onto a deformed sheet of paper and a T-shirt with a speed sufficiently high that the projected images appeared to be printed on the objects.

Conventional active 3D sensing systems do not work well when other objects get between the measurement target and the measurement equipment, occluding the line of sight. In this paper, we propose an active 3D sensing method that solves this occlusion problem by using a light field created using aerial imaging. In this light field, aerial luminous spots can be formed by focusing rays of light from multiple directions. Towards the occlusion problem, this configuration is effective, because even if some of the rays are occluded, the rays of other directions keep the spots. Our results showed that this method was able to measure the position and inclination of a target by using an aerial image of a single point light source and was robust against occlusions. In addition, we confirmed that multiple point light sources also worked well.

This paper focuses on high-performance robotic contour tracking under the uncertainties that commonly exist in actual robotic applications. These uncertainties can be attributed to the robot itself (such as modeling errors or mechanical defects like backlash) or to environmental issues (such as calibration errors or misalignment of the workpiece). We propose a non-model-based dynamic compensation approach based on the coarse-to-fine philosophy, which enables contour tracking with both high speed and good accuracy. This is achieved by adopting a methodology in which a main robot performs fast but coarse motion, while an add-on module conducts accurate compensation for the overall uncertainties using a high-speed camera and high-speed compensation actuator. An algorithm called pre-compensated proportionalderivative sliding mode control (pre-compensated PD-SMC) is proposed to control the compensation actuator. The effectiveness of the proposed contour tracking approach and control algorithm are experimentally verified using two typical planar-contour shapes: a random smooth-curvature and rectangle.

This work employs an adaptive learning mechanism to perform tracking of an unknown object through RGBD cameras. We extend our previous framework to robustly track a wider range of arbitrarily shaped objects by adapting the model to the measured object size. The size is estimated as the object undergoes motion, which is done by fitting an inscribed cuboid to the measurements. The region spanned by this cuboid is used during tracking, to determine whether or not new measurements should be added to the object model. In our experiments we test our tracker with a set of objects of arbitrary shape and we show the benefit of the proposed model due to its ability to adapt to the object shape which leads to more robust tracking results.

The purpose of this work is to achieve high-speed dynamic manipulation. As one example, we deal with throwing a breaking ball. In this article we deal with two types of throwing with different grasping forms. We modeled a manipulator and a ball with a sphere rolling on the tip link of a three-degrees-of-freedom manipulator in the case where a ball is thrown with spin, and with a sphere held between the tip links in the case where the ball is thrown with less spin. We simulated the motion of the ball by using these models in each case. In addition, experimental results obtained with balls thrown by a high-speed manipulator showed that the spin rates with the two types of throwing differed by a factor of 4.3 at most.

In this study, a deformation control with spatial decoupling properties is designed and implemented. This control strategy treats the shift in position and posture attributable to an external force as the deformation of the robot. The deformation dynamics are constructed from the Maxwell and Voigt models, which describe plastic and elastic deformation, respectively. Next, a control method is proposed to passively achieve different deformation characteristics in each direction. Two physical simulations with a robotic arm are executed to validate the proposed control law.

Recently, information about reaction times in response to perceptive stimulation has been used to develop several applications focused on movement instruction. However, knowledge is still lacking about reaction time related to haptics, especially electrical stimulation. Therefore, in this study we measured and analyzed reaction time in response to electrical and visual stimulation in human participants using a 250 fps high-speed camera. In our experimental process, participants were placed in a separate room and they wore soundproof headphones to shield them from external noise. Subsequently, they conducted reaching tasks involving visual and electrical stimuli. Using a high-speed camera, we recorded participants' hand motions from a top view (thereby focusing on the backs of their hands) and calculated their reaction time via image processing. The results show that reaction time in response to electrical stimulation is approximately 200 ms, which is more than 35% faster than reaction time in response to visual stimulation. Consequently, our results could augment basic knowledge about electrical stimulation in real-time feedback systems. The data obtained may help in electrical application in high-speed use.

A computer display that is sufficiently realistic such that the difference between a presented image and a real object cannot be discerned is in high demand in a wide range of fields, such as entertainment, digital signage, and design industry. To achieve such a level of reality, it is essential to reproduce the three-dimensional (3D) shape and material appearances simultaneously; however, to date, developing a display that can satisfy both conditions has been difficult. To address this problem, we propose a system that places physical elements at desired locations to create a visual image that is perceivable by the naked eye. This configuration can be realized by exploiting characteristics of human visual perception. Humans perceive light modulation as perfectly steady light if the modulation rate is sufficiently high. Therefore, if high-speed spatially varying illumination is projected to the actuated physical elements possessing various appearances at the desired timing, a realistic visual image that can be transformed dynamically by simply modifying the lighting pattern can be obtained. We call the proposed display technology Phyxel. This paper describes the proposed configuration and required performance for Phyxel. We also demonstrate three applications: dynamic stop motion, a layered 3D display, and shape mixture.

Reality is the most realistic representation. We introduce a material display called ZoeMatrope that can reproduce a variety of materials with high resolution, dynamic range and light field reproducibility by using compositing and animation principles used in a zoetrope and a thaumatrope. With ZoeMatrope, the quality of the material is equivalent to that of real objects and the range of expressible materials is diversified by overlaying a set of base materials in a linear combination. ZoeMatrope is also able to express spatially-varying materials, and even augmented materials such as materials with an alpha channel. In this paper, we propose a method for selecting the optimal material set and determining the weights of the linear combination to reproduce a wide range of target materials properly. We also demonstrate the effectiveness of this approach with the developed system and show the results for various materials.

Visual tracking of unknown objects is an essential task in robotic perception, of importance to a wide range of applications. In the general scenario, the robot has no full 3D model of the object beforehand, just the partial view of the object visible in the first video frame. A tracker with this information only will inevitably lose track of the object after occlusions or large out-of-plane rotations. The way to overcome this is to incrementally learn the appearances of new views of the object. However, this bootstrapping approach is sensitive to drifting due to occasional inclusion of the background into the model.

Realizing accurate robotic manipulation of flexible objects is very challenging. It asks for dexterous manipulation skill, and demands good adaptation to dynamic uncertainties due to deformation. Unlike traditional methods that basically depend on complex modeling, we follow the opposite approach by exploring possible simplifications based on high-speed motion strategy as well as high-speed visual sensing. As an application task concerned with robotic needle threading, this work demonstrates our philosophy by proposing a novel and straightforward high-speed strategy to speed up the manipulation while solves the issue of deformation uncertainty at the same time. The methodology is based on a very simple physical fact, namely that by rotating a thread with sufficient velocity, the thread can be approximated as a rigid object caused by the constantly applied centrifugal force during the rotation. Therefore, the needle threading problem is converted into a simpler peg-insertion problem, and the complexity of interaction between robot and deformable thread is significantly simplified. Based on the proposed strategy, simple technical methods then become possible for adoption to accomplish the challenging task. Experiments show that the proposed approach allows the needle threading task to be realized very rapidly.

In this paper we focus on master-slave robot hand systems that can realize non-contact sensing and intuitive mapping between human hand motion and robot hand motion. Such a master-slave robot hand system can be effective from a viewpoint of usability. However, conventional systems are not able to adapt to dynamically changing environments because they have high latency from input to output. Therefore, we developed a fast-response master-slave robot hand system using a high-speed vision system and a high-speed robot hand. The latency of the proposed system is so small that humans cannot recognize it. The motion of a human hand is obtained with high-speed non-contact 3D sensing, and this motion is mapped to a high-speed robot hand, while taking account of structural differences between the human hand and the robot hand. We confirmed the effectiveness of our proposed system through experiments.

We developed an optical high-speed image rotation controller and realized a high-speed roll camera that is able to cancel the rotational motion blur of a rotating target. This system is composed of a hollow motor, a Dove prism, and a high-speed camera and controls optical image rotation according to the target rotation by using high-speed image processing. This so-called optical lever formed of the Dove prism worked effectively also for a high-speed rotating target, and our prototype system shows that rotational motion blur was suppressed to 0.125 [°] at 1420 [r/min].

A visually guided shock absorber for high-speed catching without rebound is designed and realized. This visual shock absorber is based on the Maxwell model, which uses plastic deformation to suppress rebounding. The shock absorber is constructed from a spring-based passive elastic body and a servo control-based software damper connected in series. The visual shock absorber is realized in a robot with two degrees of freedom and high-speed vision. The robot receives the impact of a rolling object without repelling it.

A precise calibration method for optical high-speed gaze controllers using rotational mirrors is proposed. The mirror-based pan/tilt camera can change gaze direction more quickly than a conventional pan-tilt-zoom camera, which actuates the camera itself. The high-speed gaze controller can track moving objects at high magnification and can utilize new applications of three-dimensional measurement. However, the controller is difficult to calibrate owing to shallow depth of field (DOF) at close focus positions and mechanical differences. We propose to control optics and illumination to detect calibration pattern control points precisely against shallow DOF; we deepen the DOF using a small aperture, coaxial episcopic illumination, and a retroreflective checkerboard pattern. We introduce mirror thickness for an exact gaze control model and bundle adjustment for large center-of-projection translation. We confirm that wide gaze range calibration is achieved more accurately than when mirror thickness is not considered.

High-speed vision technology in which not just image capturing and recording but also image processing are executed simultaneously at high frame rates, exceeding video rates (30 Hz), has recently been considered an important technology for various applications, such as robotics, and man-machine interfaces. However, the image processing performed in conventional high-speed vision systems is mainly based on two-dimensional pattern recognition. In order to extend the possibilities of this technology, here we focus on real-time three-dimensional sensing at the speeds achievable by high-speed vision systems. Although a related approach for high-speed 3D sensing can achieve a frame rate of over 200 fps, there are disadvantages, including the need for multiple captured frames during sensing, a limited measurement range and low resolution. Our proposed real-time 3D sensing system consists of a projector and two cameras. By projecting a well-designed segmented pattern and using three-viewpoint epipolar constraints, the proposed system can obtain 3D points at high speed. The developed system robustly obtained a 3D shape at 450-500 fps in real time.

Dynamic Projection Mapping, projection-based AR for a moving object without misalignment by a high-speed optical axis controller by rotational mirrors, has a trade-off between stability of highspeed tracking and high visibility for a variety of projection content. In this paper, a system that will provide robust high-speed tracking without any markers on objects against illumination changes, including projected images, is realized by introducing a retroreflective background with the optical axis controller for Dynamic Projection Mapping. Low-intensity episcopic light is projected with Projection Mapping content, and the light reflected from the background is sufficient for high-speed cameras but is nearly invisible to observers. In addition, we introduce adaptive windows and peripheral weighted erosion to maintain accurate high-speed tracking. Under low light conditions, we examined the visual performance of diffuse reflection and retroreflection from both camera and observer viewpoints. We evaluated stability relative to illumination and disturbance caused by non-target objects. Dynamic Projection Mapping with partially well-lit content in a low-intensity light environment is realized by our proposed system.

In recent years, projection mapping has attracted much attention in a variety of fields. Generally, however, the objects in projection mapping are limited to rigid and static or quasistatic objects. Dynamic projection mapping onto a deformable object could remarkably expand the possibilities. In order to achieve such a projection mapping, it is necessary to recognize the deformation of the object even when it is occluded. However, it is still a challenging problem to achieve this task in real-time with low latency. In this paper, we propose an efficient, high-speed tracking method utilizing high-frame-rate imaging. Our method is able to track an array of dot markers arranged on a deformable object even when there is external occlusion caused by the user interaction and self-occlusion caused by the deformation of the object itself. Additionally, our method can be applied to a stretchable object. Dynamic projection mapping with our method showed robust and consistent display onto a sheet of paper and cloth with a tracking performance of about 0.2 ms per frame, with the result that the projected pattern appeared to be printed on the deformable object.

We propose a novel three-dimensional motion sensing method using lasers. Recently, object motion information is being used in various applications, and the types of targets that can be sensed continue to diversify. Nevertheless, conventional motion sensing systems have low universality because they require some devices to be mounted on the target, such as accelerometers and gyro sensors, or because they are based on cameras, which limits the types of targets that can be detected. Our method solves this problem and enables noncontact, high-speed, deterministic measurement of the velocity of a moving target without any prior knowledge about the target shape and texture, and can be applied to any unconstrained, unspecified target. These distinctive features are achieved by using a system consisting of a laser range finder, a laser Doppler velocimeter, and a beam controller, in addition to a robust 3D motion calculation method. The motion of the target is recovered from fragmentary physical information, such as the distance and speed of the target at the laser irradiation points. From the acquired laser information, our method can provide a numerically stable solution based on the generalized weighted Tikhonov regularization. Using this technique and a prototype system that we developed, we also demonstrated a number of applications, including motion capture, video game control, and 3D shape integration with everyday objects.

In this paper, we propose an algebraic solution for rapid SVBRDF measurement. The algebraic approach requires only a few reflectance samples to obtain the parameters described by the physically based Cook - Torrance model. This solution, however, also involves constraints concerning light and the normal direction in the acquisition process. To meet these constraints, we developed a system that changes the illumination according to the target 3D shape at high speed. As a result, the proposed method provides BRDF parameters at each texel without optimization and over-sampling. We demonstrated rapid measurement with real objects that do not have uniform reflectance and confirmed the validity of this approach by comparison with conventional methods.

We propose a driving safety support system (DSSS) that employs a high-speed vision system installed in the environment surrounding, for instance, highways, urban roads, and intersections. The aim of the system is to recognize potentially dangerous traffic situations, including those that are undetectable from a moving vehicle, and to use this information for supporting safe driving. The system consists of a vision network of synchronized high-speed cameras that are capable of acquiring images at one-millisecond intervals, and vehicles that are capable of communicating with this network through communication hubs. We conducted collision avoidance experiments and demonstrated that, by introducing high-speed vision, the proposed system can resolve the issue of slow reaction time, which is common to environmental vision systems.

This paper presents an approach for document rectification using a quasi-isometric mapping derived from a novel model that represents a deformed document surface. The model is composed of rectangle fragments of the developed document plane. This was realized by introducing gap relaxation which permits unconnected fragments. Our experiments show that our rectification approach can be applied to different types of document deformation.

In this paper, a two-dimensional analysis of biped robot sliding dynamics is considered. First, the dynamics of a biped robot based on feet slip are derived using the Coulomb friction model. The state transition can be formulated in the centroid acceleration space whose diagram is defined as a “triangle of sliding friction.” The triangle of sliding friction's characteristics are explained by focusing on comparison with the cone of friction, which has a similar state decision diagram. Moreover, the behavioral simulation of a concrete model 2-DOF biped robot is used to analyze the sliding features in terms of the asymmetry of the dynamics of both legs.

In this paper, we propose a new book digitization system that can obtain high-resolution document images while flipping the pages automatically. The distinctive feature of our system is the adaptive capturing that has a crucial role in achieving high speed and high resolution. This adaptive capturing requires observing the state of the flipped pages at high speed and with high accuracy. In order to meet this requirement, we newly propose a method of obtaining the 3D shape of the book, tracking each page, and evaluating the state. In addition, we explain the details of the proposed high-speed book digitization system. We also report some experiments conducted to verify the performance of the developed system.

In this paper, we propose a method for improving the resolution of a 3D shape reconstructed from multiple range images acquired from a moving target. In our approach, the alignment and surface estimation problems are solved in a simultaneous estimation framework based on multi-level implicit surface. We present results of experiments for evaluating the reconstruction accuracy with different point cloud densities and noise levels which shows that our method achieved good performance even when the input range images were low-resolution and sparse.

Augmented reality (AR) allows us to experience real-world objects onto which some extra visual information has been added by illumination from a projector. Most of the projected objects in AR research remain stationary, such as a white figure mounted on a desk, a wall, a screen, and so on. We have proposed, and successfully demonstrated, an AR system for dynamic objects, such as a flying ball, a moving object held in a person's hand, and so on. In the work described in this paper, we introduced a high-speed optical axis controller (a mirror-based device that we have developed) and an original target tracking algorithm for dynamic objects, which is based on HSV color detection.

In this paper, we propose an entirely new manipulation strategy for dynamic manipulation of a ribbon with a high-speed manipulator. The manipulation strategy involves manipulating the object at a constant, high speed. Then, we can assume that the dynamic behavior of the ribbon can be obtained by performing algebraic calculations of the robot motion using the proposed strategy. Based on this assumption, we derive a model of the ribbon and suggest a motion planning method using the proposed model. Finally, we show experimental results of shape control of a ribbon based on the proposed method.

In this paper, we focus on unknown parameters such as the position and attitude of the object, and describe a short-range, high-speed and noncontact sensing method for obtaining the position and attitude of the object using IR Net-Structure Proximity Sensor (“IR-NSPS”) which complements the dead region of sensory information between visual and tactile sensing. To be more precise, we propose two effective control methods which are pre-shaping and object positioning using IR-NSPS for robust grasping by adjusting the gripper configuration in response to attitude error of up to ±45 deg and the position error of up to ±80 mm of the unknown object. The methods therefore can significantly increase the speed and effectiveness of grasping objects without requiring a specific approach that depends on a vision sensor. Furthermore, to demonstrate the advantages of pre-shaping and object positioning, object grasping experiments were performed using these two operations to grasp objects placed randomly on a tabletop.

We propose a method for tracking the boundary of an object at frame rates beyond 1 kHz, based on a novel contour propagation mechanism operating in polar image space. The work draws inspiration from well established methodologies in object tracking and segmentation. The main contribution is how the polar representation is exploited for tracking, including enabling parallelization for sub millisecond performance. The presented results show the feasibility of the method in a wide range of settings.

This paper presents a visual compliance strategy to deal with the problem of fast peg-and-hole alignment with large position and attitude uncertainty. With the use of visual compliance and adoption of a light-weight 3-DOF active peg, decoupled alignment for position and attitude is realized. The active peg is capable of high-speed motion and with less dynamic defects than a traditional robot arm. Two high-speed cameras, one configured as eye-in-hand and the other as eye-to-hand are adopted to provide with the task-space feedback. Visual constraints for effecting the visual compliant motion are analyzed. Alignment experiments show that peg-and-hole alignment with the proposed approach could be successfully realized with robust convergence, and on average, the alignment could be realized within 0.7 s in our experimental setting.

In recent years, there has been an increasing demand to digitize a huge number of books. A promising new approach for meeting this demand, called Book Flipping Scanning, has been proposed. This is a new style of scanning in which all pages of a book are captured while a user continuously flips through the pages without stopping at each page. Although this new technology has had a tremendous impact in the field of book digitization, page turning is still done manually, which acts as a bottleneck in the development of high-speed book digitization. Against this background, this paper proposes a newly designed high-speed, high-precision book page turner machine. Our machine turns the pages in a contactless manner by utilizing the elastic force of the paper and an air blast. This design enables high-speed performance that is ten times faster than conventional approaches and, in addition, causes no obstruction in the digitization process. This paper reports the evaluation of the proposed machine using various types of paper with different qualities. Our machine achieved almost 100% success rate when turning pages at around 300 pages/min, showing that it is a promising technology for turning pages at high-speed and with high precision.

Recently, mixed reality (MR) involving the use of a projector, which is often called projection-based MR, has attracted much attention and is being widely studied. However, most MR research is mainly focused on static objects or situations. In contrast, we propose "Lumipen," which is a projection-based MR system for high-speed or high-frequency objects, using a high-speed vision sensor and a projector with a high-speed optical gaze controller. The high-speed vision sensor detects the dynamics of the object, and the high-speed optical gaze controller controls the projection direction in a few milliseconds. Lumipen is expected to enable addition of sophisticated and diverse visual information even in dynamic live content (e.g., a play, a sports game, a music concert, etc.), like computer graphics added to video content.

In this paper, we propose an entirely new manipulation strategy for dynamic manipulation of a flexible rope with a high-speed robot arm. The manipulation strategy involves manipulating the object at a constant, high speed. Then, we can assume that the dynamic behavior of the flexible rope can be obtained by performing algebraic calculations of the robot motion using the proposed strategy. Based on this assumption, we derive a model of the flexible rope and suggest a motion planning method using the proposed model. Finally, we show experimental results of rope deformation control based on the proposed method.

This video introduces an ultra high-speed robot as a milestone in the history of intelligent manipulation systems. To develop the ultra high-speed robot under the concept of dynamics matching, we began with the development of a 1 kHz vision system. Next, we developed a sensory-motor fusion system by introducing the 1 kHz vision system. In addition, we have developed a new high-torque mini actuator and a high-speed multi-fingered hand with these incorporated. Integration of these components brings real-time dexterous manipulations unlike commonly-used control based on prediction or learning.

In this paper, dynamic horizontal movement is considered with the goal of achieving high-speed dynamic leg motion. We propose a new movement principle using frictional asymmetry for legged robots. This motion strategy consists of a sliding motion based on kinematic constraints and a jumping motion which makes use of lightweight high-torque motors. In addition, the motion characteristics based on the dynamics are analyzed. Experimental results are shown in which a 2-DOF bipedal robot takes short rapid repetitive steps with compensation for the landing time controlled by high-speed visual feedback.

In this research, we successfully demonstrated dexterous manipulation of sheet-like elastic objects, namely, playing cards, using a high-speed robot system. In particular, our goal was to achieve card flicking by vibrating a fingertip of a robot hand at high speed and card catching by using high-speed visual feedback based on a high-speed vision system. We discuss card grasping in the initial state based on the geometry conditions of the card and the kinematics of the robot hand, and we propose a strategy for card flicking. We also suggest a card catching method based on information obtained by the high-speed vision system. In addition, we obtain the card flicking conditions by analyzing the slip between the card and the fingertip of the robot hand. Finally, we show experimental results of card flicking and card catching.

We propose a new method of indoor-scene stereo vision that uses probabilistic prior knowledge of indoor scenes in order to exploit the global structure of artificial objects. In our method, we assume three properties of the global structure - planarity, connectivity, and parallelism/orthogonality - and we formulate them in the framework of maximum a posteriori (MAP) estimation. To enable robust estimation, we employ a probability distribution that has both high peaks and wide flat tails. In experiments, we demonstrated that our approach can estimate shapes whose surfaces are not constrained by three orthogonal planes. Furthermore, comparing our results with those of a conventional method that assumes a locally smooth disparity map suggested that the proposed method can estimate more globally consistent shapes.

We developed an optical high-speed gaze controller, called the “Saccade Mirror”, and used it to realize a high-speed pan/tilt camera with a high-speed image processor. Generally, in a pan/tilt camera, the gaze is controlled mechanically by rotational actuators. However, because pan/tilt cameras will be expected to use high-speed image processing (≥ 1000 fps), sufficiently high-speed performance of the gaze controller, comparable to the high frame rate, cannot be obtained with the usual method. In our system, the camera itself was fixed, and an external Saccade Mirror subsystem was used for optical gaze control. An arbitrary gaze direction within ±30 deg could be achieved in less than 3.5 ms.

Tactile sensors for slip detection are essential for implementing human-like gripping in a robot hand. In previous studies, we proposed flexible, thin and lightweight slip detection sensors utilizing the characteristics of pressure-sensitive conductive rubber. This was achieved by using the high-frequency vibration component generated in the process of slipping of the gripped object in order to distinguish between slipping of the object and changes in the normal force. In this paper, we design a slip detection sensor for a multi-fingered robot hand and examine the influence of noise caused by the operation of such a hand. Finally, we describe an experiment focusing on the adjustment of the gripping force of a multi-fingered robot hand equipped with the developed sensors.

The purpose of the work described in this paper is to achieve dynamic manipulation of a sheet-like flexible object. As one example, we examine dynamic folding of a cloth with two high-speed multifingered hands mounted on two sliders. First, dynamic folding by a human subject is analyzed in order to extract the necessary motions for realizing this task. Second, a model of a sheet-like flexible object is proposed by extending a linear flexible object model (algebraic equation) that takes advantage of high-speed robot motion. Third, motion planning of the robot system is performed by using the proposed model, and the simulation results are shown. Finally, an experiment was conducted with the robot motion obtained by the simulation.

This paper presents a proximity sensor for recognizing an object in 360 degrees all around with response time only 1[ms]. The sensor applies an unique design of analog net-structure circuit, which enables all-face mounting on various peripheries and fast-response/simplification of information processing. Thus, the sensor is especially available for avoiding sudden contact with rapidly approaching objects for mobile robots, robot arms, robot hands, and other robotic systems. In this paper, the structure and the detecting principle of the sensor are firstly described, then, its detection property is tested by experiments. The result shows that the sensor recognizes the angle of the nearby object with accuracy of 3[deg]. Furthermore, as practical use of the sensor, a large size model is produced for a mobile robot to achieve omni-directional sensing. By using the application model, two detection methods are discussed on situations where the number or the positions of the nearby objects differs.

In this paper, we propose a novel concept called VolVision that encompasses using a camera to reconstruct 6DoF unconstrained motion. "VolVision" is designed to handle imagery falling, tossed or thrown cameras. And VolVision also allows users to reconstruct dynamic images and generate a 3D-mapped scene from image sequences. It could be used to model severe environments like valley and mountains that are normally not easily viewed by humans. We produced a prototype that embodies the concept above, and were able to reconstruct the camera's path, perform image mosaicing, and track 3D information of feature points in images.

We focus on the growing need for a technology that can achieve motion capture in outdoor environments. The conventional approaches have relied mainly on fixed installed cameras. With this approach, however, it is difficult to capture motion in everyday surroundings. This paper describes a new method for motion estimation using a single wearable camera. We focused on walking motion. The key point is how the system can estimate the original walking state using limited information from a wearable sensor. This paper describes three aspects: the configuration of the sensing system, gait representation, and the gait estimation method.

In this paper, a jumping motion is considered with the goal of achieving high-speed dynamic motion for legged robots. We propose a two-step jumping pattern which makes use of the characteristic of a lightweight and fast actuator. In addition, jumping strategies based on the dynamics of the robots are presented, and the effect of structural parameter on jumping height is analyzed. Experimental results are shown in which a 1-DOF two-legged robot jumps with three patterns.

In this paper, we propose a wide-range image sensing method using a camera thrown up into the air. By using camera thrown up in this way, we can get images that are otherwise difficult to obtain, such as those taken from overhead. As an example of wide-range image sensing, we integrated video images captured by a thrown-up camera using an image mosaicing technique. When rotation about the optical axis of the camera can be ignored, we can integrate images by mosaicing using a translational approximation, which preferentially pastes pixels around the image center. To obtain the information about the camera direction, a rotational approximation using the angles of incident light rays is required. We also propose use of a high-frame-rate camera (HFR camera) in order to acquire a large amount of information. A seamless large image was obtained by synthesizing the images captured by a thrown-up HFR camera. We found that high frame rates of around 1000 fps were necessary.

There is a strong demand for a method of acquiring a non-rigid shape under deformation with high accuracy and high resolution. However, this is difficult to achieve because of performance limitations in measurement hardware. In this paper, we propose a model based method for estimating non-rigid deformation of a developable surface. The model is based on geometric characteristics of the surface, which are important in various applications. This method improves the accuracy of surface estimation and planar development from a low-resolution point cloud. Experiments using curved documents showed the effectiveness of the proposed method.

To achieve the skillful task like the human, many researchers have been working on robot hand. An interaction with vision and tactile information are indispensable for realization of skillful tasks. In the existing research, the method using a camera to get the vision information is often found. But, in the boundary area of a non-contact phase and a contact phase, there are problem that lack of sensor information because the influence of occlusion comes up to surface. We devise to introduce the proximity sensor in this area. And we call the robot hand which is equipped with proximity, tactile and slip sensor “intelligent robot hand”. In this research, we show the constitution example of the intelligent robot hand and propose the method to realize Pick&Place as concrete task.

We report a high-speed liquid lens, called a Dynamorph Lens (DML), and its applications in the field of computer vision. The developed lens uses a liquid-liquid interface as a variable refractive surface. The interface shape is controlled by the volume of a liquid chamber using a high-speed piezostack actuator. A developed prototype demonstrated both millisecond-order response speed and practical optical performance. A high-speed focusing vision system that can adaptively control its focal length in based on visual feedback by using the DML prototype coupled with a high-speed imager is also proposed. High-speed autofocusing within 15.8 ms and focus tracking of a moving object were demonstrated using the developed system. The DML prototype was also applied to high-speed focus scanning of a camera lens at 500 Hz. An image sequence was captured at 8000 fps through the lens so that every eight images were captured with a different focus, and a 1000 fps video with extended depth of field was successfully synthesized.

Slip-detecting tactile sensors are essential for achieving human-like gripping motion with a robot hand. In previous research, we developed a flexible, thin and lightweight slip sensor that exploits the characteristics of pressure conductive rubber. However, using this sensor, it was difficult to distinguish between object slip and a change in the normal force. Therefore, in the present research, we investigated a method for identifying object slip by analyzing the frequency components of the output signal from the sensor. As a result, we found that high-frequency components of 1 kHz or more are included in the complex voltage signal generated by object slip. Therefore, by using this high-frequency component, we developed a simple sensor that distinguished between both contact and initial slip with high sensitivity.

In this paper, we propose an entirely new strategy for dexterous manipulation of a linear flexible object with a high-speed robot arm. The strategy involves manipulating the object at high speed. By moving the robot at high speed, we can assume that the dynamic behavior of the linear flexible object can be obtained by performing algebraic calculations of the robot motion. Based on this assumption, we derive a model of the linear flexible object and confirm the validity of the proposed model. Finally, we perform simulation of dynamic knotting based on the proposed model. Results of an experiment demonstrating dynamic knotting with a high-speed robot arm are shown.

In this paper, we proposed an architecture of embedded systems for high-frame-rate real-time vision on the order of 1000 f/s, which achieved both hardware reconfigurability and easy algorithm implementation while fulfilling performance demands. The proposed system consisted of an embedded microprocessor and field programmable gate arrays (FPGAs). A coprocessor consisting of memory units, direct memory access controller units, and image processing units were implemented in each FPGA. While the number of units and functions are reconfigurable by reprogramming the FPGAs, users can implement algorithms without hardware knowledge. A descriptor method in which the central processing unit gave instructions to each coprocessor through a register array enabled task-level parallel processing as well as pixel-level parallel processing in the processing units. The specifications of an evaluation system developed based on the proposed architecture, the results of performance evaluation, and application examples using the system were shown.

Recognition of dynamic scenes based on shape information could be useful for various applications. In this study, we aimed at improving the resolution of three-dimensional (3D) data obtained from moving targets. We present a simple clean and robust method that jointly estimates motion parameters and a high-resolution 3D shape. Experimental results are provided to illustrate the performance of the proposed algorithm.

This video introduces the demonstration of skillful manipulation using a high-speed robot system. The system consists of visual and tactile sensors at a rate of 1 kHz and a high-speed hand-arm manipulator. The high-speed sensory-motor fusion improves not just the speed of existing robot manipulations, but robotic skills by introducing the features peculiar to high-speed motion. Based on such a concept, new variations of skillful manipulation were achieved.

Even with the eyes closed, humans are able to grip an object with minimal force without such information as the coefficient of friction or the weight. Tactile sensors capable of detecting slippage are necessary for this gripping action to be realized in a robot hand. Heretofore, many slip sensors were developed and produced, but there was not a slip sensor of simple structure practical for installation on fingertips of a robot hand. Therefore, we propose a low-profile/lightweight slip sensor of simple structure. The special properties of pressure conductive rubber are utilized as a detection device in this sensor. In this paper, we discuss the results of trial manufacture and of slip detection property testing of this sensor. Moreover, we will report the results of slip prevention experiments by this prototype slip sensor, and indicate that the pressure conductive rubber is promising as a material of slip sensor.

We present here a first prototype of the Virtual Haptic Radar (VHR), a wearable device helping actors become aware of the presence of invisible virtual objects in their path when evolving in a virtual studio (such as a "bluescreen" filming stage [Figure 1]). The VHR is a natural extension of the Haptic Radar (HR) and its principle [Cassinelli et al. 2006] in the realm of virtual reality: while each module of the HR had a small vibrator and a rangefinder to measure distance to real obstacles, the VHR module lacks the rangefinder but accommodates instead a (cheap) ultrasound-based indoor positioning system that gives it the ability to know exactly where it is situated relatively to an external frame of reference.

This paper presents a design and an implementation of document image retrieval to support reading mokkans. A mokkan is a wooden tablet with text written by a brush in India ink. Despite the archaeological and historical value of the mokkans excavated from ancient ruins, many of the mokkans have not been decoded yet due to the lost or too much damaged character patterns on them. Character recognition for damaged patterns is useful to decode such mokkans. Furthermore, if the recognition results show not only the character codes but also the images of the character patterns and the whole mokkans, the recognition becomes useful document retrieval to complement the lost or unreadable part of the mokkans. In the implementation, we built a public database of historical mokkans with their photographs and a character recognition module working on our support system to search the database. The evaluation by archaeologists is in progress.

In our project, we are developing a system that helps learners in vocabulary acquirement. In the system, learners can create their own materials for mobile learning, and they can also exchange their materials. This paper focuses on a sub system, SIGMA, to share and exchange materials among learners. In addition, SIGMA enables learners to give evaluation scores and comments to all learning materials stored in the system. In this paper, we will discuss the management policy of the system based on the reliability of the evaluation scores given by the learners to select appropriate materials. From the result of our experiment, we concluded that the evaluation scores have some reliability but an instructor should guide the learners.

In this paper we propose a high-S/N imaging method involving combining many images captured with small blur using a video camera capable of high-frame-rate image capturing at 1000 frames/s. Use of a high-frame-rate camera makes the image change between frames small, enabling easy motion estimation, and makes it possible to use more light information, even when the exposure time is reduced to avoid blurring. To obtain a clear picture without misalignment due to motion parallax, it is necessary to determine both the motion and a depth map of the subject from noisy input images. We show results when applying the proposed algorithm to an image sequence captured by a high-frame-rate camera.

The recognition of dynamic scenes using 3D shapes could provide useful approaches for various applications. However, the conventional 3D-shape sensing systems dedicated for such scenes have had problems in spatial resolution, though they have achieved high sampling rate in temporal domain. In order to solve this limits, we present a method that integrates time-sequential partial range images capturing moving targets to reconstruct a high-resolution range image. In the proposed method, multiple range images are set in the same coordinate system based on multi-frame simultaneous alignment. This paper also demonstrates the performance of the proposed method using some example rigid bodies.

To achieve a human like grasping with a multi- fingered robot hand, the grasping force should be controlled without using information from the grasped object such as its weight and friction coefficient. In this study, we propose a method for detecting the slip of a grasped object using the force output of Center of Pressure (CoP) tactile sensors. CoP sensors can measure the center position of a distributed load and the total load applied on the surface of the sensor, within 1 ms. These sensors are arranged on the fingers of the robot hand, and their effectiveness as slip detecting sensors is confirmed in tests of slip detection during grasping. Finally, we propose a method for controlling grasping force to resist tangential force applied to the grasped object using a feedback control system with the CoP sensor force output.

We propose a new framework and novel visual control system for motile cells in three-dimensional (3-D) space. Our goal is to utilize microorganisms as micro-robots in various applications by exploiting galvanotaxis (locomotor response to electrical stimulus) to actuate them. This requires automated motion control of swimming cells in 3-D space; in contrast, our previous work has been limited to 1-D or 2-D motion control on the focal plane. The system is capable of 3-D tracking and control of swimming cells by utilizing a high-speed vision system. A combination of "lock-on" tracking within the focal plane and automated focusing using a Depth-From-Diffraction method executed at 1-kHz frame rate ensures both detailed measurement and a large working space. Experimental results for closed-loop 3-D motion control of Paramecium cells trapped within a small 3-D region demonstrate the possibility of using microorganisms as micromachines.

In this paper, we propose the concept of a "mobile microscope". This is a hand-held microscope that can be used for observation in various places. Because microscopes which have a small observable area and are vulnerable to vibrations, images from hand-held microscopes are usually blurred. We propose a method of stabilizing images from a microscope which are affected by shaking, and we use the method to realize a mobile microscope. The method is based on high-speed visual feedback with an inclined image sensor for measuring the 3D movements of the microscope. We developed a prototype microscope system, without a built-in actuator, by employing this method. Experimental results showed that images from the hand-held microscope could be stabilized by our method.

In this paper, a new serial algorithm for highspeed autofocusing of cells is proposed. The proposed algorithm extracts depth information of target cells from their diffraction patterns. High-speed focusing of yeast cells with 20 ms response time was demonstrated. Continuous autofocusing of yeast cells with a scanning microscope was also demonstrated. The successful continuous autofocusing suggests that image-based high- throughput measurement of cells could be realized using the proposed algorithm.

In order to achieve as skillful handling as a human hand, it is necessary that a hand has the capability to manipulate tools. One important problem in tool handling with a multifingered hand is that the relative positions between the robot hand, the tool and the handled object change during the handling. For this reason, it is necessary to measure these relative positions in realtime, and to control the hand so as to cancel the changes. We have resolved this problem by using a high speed visual servoing. In this paper, as an example, a tweezers is handled by a multifingered hand. Further, a new method to control tools along with experiment result are shown.

In this paper the robotic throwing task is considered with the goal of achieving high-speed dynamic manipulation. We propose a kinetic chain approach for swing motion focused on torque transmission. In addition the release method using a robotic hand is analyzed for ball control. Experimental results are shown in which a high-speed manipulator throws a ball toward a target.

Slip detecting tactile sensors is essential to achieving a human-like gripping motion with a robot hand. Up until now, we have developed flexible, thin and lightweight center of pressure (CoP) sensor. The sensor, constructed of pressure conductive rubber sandwiched between two sheets of conductive film, is able to detect the center position of the load distribution and the total load. Recently, detection of initial slip has been shown to be possible. However the detection principles are unclear. Therefore, we carried out verification experiments of the slip detection properties of the CoP sensor and the detection principle. In the results, we found a change in electrical conductivity produced with a shear deformation of the pressure conductive rubber. In this paper, we will discuss the slip detection properties of the CoP sensor and detection principle.

In this paper, we examine the relationship between a knotting process and the individual skills of which a robot hand is capable. To determine the necessary hand skills required for knotting, we first analyzed the knotting action performed by a human subject. We identified loop production, rope permutation, and rope pulling skills. To take account of handling of the two ends of the rope, we added a rope moving skill. We determined the characteristics of these skills using an intersection-based description. The knotting process was examined based on the analysis of knots and the characteristics of the robot hand skills. Finally, we show experimental results of an overhand knot and a half hitch performed using a high-speed multifingered hand system.

We describe a high-speed vision system for real-time applications, which is capable of processing visual information at a frame rate of 1 kfps, including both imaging and processing. Our system performs moment-based analysis of numerous objects. Moments are useful values providing information about geometric features and invariant features with respect to image-plane transformations. In addition, the simultaneous observation of numerous objects allows recognition of various complex phenomena. The proposed system achieves high-speed image processing by providing a dedicated massively parallel co-processor for moment extraction. The co-processor has a high-performance core based on a pixel-parallel and object-parallel calculation method. We constructed a prototype system and evaluated its performance. We present results obtained in actual operation.

In this paper we propose a method of realizing continuous tracking of a three-dimensional object by calculating moments of a translating and rotating object whose shape is known, either analytically or by using a table, and matching them with those of the input image. In simulation, the position and orientation is accurately recognized. Using a noise model and particle filter, we show that the position and orientation can be recognized even with noisy images.

This paper describes a new realtime collision avoidance algorithm for a robot manipulator based on a new visual servo control. In the proposed algorithm, it is not necessary to compute the 3D position of the obstacle. Further, the manipulator can avoid the fast moving obstacle. The developed system uses several small light-weight high-speed vision chips placed on the surface of the robot manipulator. Experimental results of avoiding high-speed motion are shown.

This paper describes real-time shape measurement using a newly developed high-speed vision system. Our proposed measurement system can observe a moving/deforming object at high frame rate and can acquire data in real-time. This is realized by using two-dimensional pattern projection and a high-speed vision system with a massively parallel co-processor for numerous-point analysis. We detail our proposed shape measurement system and present some results of evaluation experiments. The experimental results show the advantages of our system compared with conventional approaches.

A tactile sensor is developed with the aim of covering a robot’s entire structure, while reducing wiring requirement and ensuring high-speed response. The sensor detects the center point of load distribution on 2-dimensional surfaces as well as the overall load. There are only 4 signal wires from the sensor. The sensor response time is nearly constant (within 1ms) regardless of the number of detection elements, their placements or sensor areas. In this paper, the principles behind the operation of this sensor and the results of experiments using the sensor are described. Further, a developed tactile element composed of pressure-sensitive materials and chip-resistors is described.

This paper proposes a new strategy for making knots with a high-speed multifingered robot hand having tactile sensors. The strategy is divided into three skills: loop production, rope permutation, and rope pulling. Through these three skills, a knot can be made with a single multifingered robot hand. The dynamics of the rope permutation are analyzed in order to improve the success rate, and an effective tactile feedback control method is proposed based on the analysis. Finally, experimental results are shown.

Increasing demands for robust image recognition systems require vision processors not only with enormous computational capacities but also with sufficient flexibility to handle highly complicated recognition tasks. We describe a multi-SIMD architecture and the design of a vision processor based on it for carrying out such difficult image recognition tasks. The proposed architecture consists of two SIMD parallel processing modules and a shared memory, allowing highly parallelized and flexible computation of complicated recognition tasks, which were difficult to process on a conventional massively parallel SIMD architecture. We designed a prototype vision processor for evaluation purposes and confirmed that the processor could be implemented in FPGA.

In most previous studies, it has been difficult for a robot hand to regrasp a target quickly because its motion was static or quasi-static, in a constant contact state. In order to achieve high-speed regrasping, we propose a new strategy which we call dynamic regrasping. In this strategy, the regrasping task is achieved by throwing a target up and by catching it. In this paper, a regrasping strategy based on visual feedback is presented and experimental results using a high-speed multifingered robot hand and a high-speed vision system are shown. A cylinder is chosen as a specific example of target for dynamic regrasping, with which successful dynamic regrasping tasks are experimentally achieved

This paper proposes a new autofocusing method for microbiological specimens, such as cells. The proposed focusing method used a quick focus estimation named "depth from diffraction", based on a diffraction pattern in a defocused image of a biological specimen. Since this method can estimate the focal position of the specimen from only a single defocused image, it can easily realize high-speed autofocusing. To demonstrate the proposed method, continuous focusing was applied to focus tracking of a swimming paramecium, in combination with two-dimensional tracking. This allowed three-dimensional tracking of the paramecium for 75 s to be successfully demonstrated

Speeding up robot motion provides not only improvement in operating efficiency but also improves dexterous manipulation by taking advantage of an unstable state or noncontact state. In this paper we describe a hybrid trajectory generator that produces high-speed manipulation. This algorithm produces both mechanical high-speed motion and sensor-based reactive motion. As an example of high-speed manipulation, a robotic ball control in a batting task has been achieved. Performance evaluation is also analyzed

A sensor attached to the sole of the foot of a biped robot to detect a zero moment point (ZMP) is proposed and basic experimental results are presented. The sensor simultaneously detects the center position of a two-dimensional distributed load on the surface of the sensor and the total load of the distribution. The sensor is sheet-like in form, lightweight (0.2 g/cm2), offers a high-speed response (within 1 ms), and needs a little wiring (four wires). In the present paper, the principle of the above-described sensor and its characteristics are described, and the ZMP locus is shown for a walking biped robot with the attached sensors

This paper newly proposes the four-fingered robot hand with dual turning mechanism where two and two other fingers can independently rotate inner and outer circles with the common center, respectively. Due to this mechanical configuration, it has the particular rotating axis where the manipulation around the axis can be completely decomposed into the velocity control around the axis and the internal force control in the contact plane. We achieved a manipulation task around the axis with the time of 0.8[sec] for one rotation, while it is relatively slow for another axis.

A new multifingered hand system with quick finger motion of 180deg per 0.1s has been developed. With high-speed visual feedback at a rate of 1kHz, high-speed catching of a falling ball are achieved. In this catching task, a target was dropped from about 1m in height, and the speed of the falling ball became more than about 4m/s just before it hit the ground. The hand could precisely catch it using two fingertips, and the percentage of the success was almost 100%. This successful catching depends on not only high-speed precise motion of the hand, but also softness of the target surface. In this paper, the dynamical analysis of the high-speed catching is proposed, and the softness of the target surface makes very important roles in this tasks. As a result it is shown that the balanced impedance of the target surface and the feedback control of the hand is needed in order to achieve stable catching.

We propose a dynamics model of galvanotaxis (locomotor response to electrical stimulus) of the protozoan Paramecium. Our purpose is to utilize microorganisms as micro-robots by using galvanotaxis. For precise and advanced actuation, it is necessary to describe the dynamics of galvanotaxis in a mathematical and quantitative manner in the framework of robotics. However, until now the explanation of Paramecium galvanotaxis in previous works has remained only qualitative. In this paper, we construct a novel model of galvanotaxis as a minimal step to utilizing Paramecium cells as micro-robots. Numerical experiments for our model demonstrate realistic behaviors, such as U-turn motions, like those of real cells.

The coexistence of humans and industrial robots in a common workspace provides the advantage of increased flexibility in production or longer system up-time during maintenance. However, it is fundamentally necessary to guarantee the safety of the human. This paper presents an approach that uses a specialized tracking-vision-chip to realize a high-speed emergency-stop for safe human-robot-coexistence. The presented approach uses the ability of the vision-chip to perform pixel-parallel masking and fast summation-operations on binary images to detect whether the robot and human are too close. After initial evaluation in a (semi)-simulation, the approach was realized in an experimental system. Even with only a small 8-bit microcontroller controlling the vision-chip and the communication, a cycle time of more than 500Hz was achieved.

In order to achieve faster and more dexterous manipulations, we propose a strategy called "dynamic holding". In the dynamic holding condition, the object is held in a stable condition while moving at high-speed. In this paper, we first explain the concept of dynamic holding, then we describe an experiment of dribbling by a robot as an example of dynamic holding using a high-speed multifingered hand and a high-speed vision system.

We propose a visual control system for motile cells. Our goal is to control microorganisms as microscale smart robots for various applications. As a first step, we have developed a visual feedback control system for Paramecium caudatum cells. In order to ensure both detailed measurements and a large working space, "lock-on" tracking of a free-swimming cell with a high frame rate is essential. In our system, high-speed (1-kHz frame rate) tracking hardware and software are used for the continuous observation of moving cells with high magnification. Cells swim in a chamber, and their positions and other properties are measured and computed in real time. The chamber position is visually controlled automatically to track a specific cell. The cell motion is controlled electrically by utilizing the galvanotaxis (intrinsic reaction to electrical stimulus) of microorganisms. Experimental results for open-loop control (periodic zigzag motion) and closed-loop control (trapping within a small region that is 1 mm wide) demonstrate the possibility of using microorganisms as micromachines.

A new robotic hand system with high-speed finger motion of 180 [deg] per 0.1 [s] has been developed. With high-speed visual feedback at a rate of 1 [kHz], various dynamic manipulations are achieved. As examples of dynamic manipulation we describe the dynamic active catching tasks for a falling ball and a falling cylinder using a high-speed multifingered hand. We achieved these difficult tasks by utilizing high-speed motion and impact forces actively. Experimental results using active catching strategy based on visual feedback are shown.

A sensor selection method for distributed real-time sensing is proposed where the effect of communication delays is taken into account. First, a multi-sensor fusion algorithm that allows communication delays is proposed. This method takes advantage of the information form of Kalman filtering in order to maintain scalability with the number of sensors. Based on this algorithm, an algorithm is developed that calculates mutual information between an observation and the state to be estimated, which provides a criterion for sensor selection. Simulated results of a target-tracking task show its effectiveness.

We propose a control system of motile cells. Our goal is to construct a large-scale Organized Bio-Modules (OBM) in which microorganisms are controlled as micro-size smart robots in an organized way. For the first step, we have developed a visual feedback control system of Paramecium caudatum cells. Tracking method is used for observation of moving cells with high magnification. Cells swim in a chamber and their positions are measured by high-speed vision. The chamber position is visually controlled to track a specific cell. The cell motion is controlled electrically by utilizing the galvanotaxis (intrinsic reaction to electrical stimulus) of microorganisms. Experimental results of region-trapping demonstrate the possibility of the OBM system.

In this paper a robotic batting algorithm using a high-speed arm and high-speed stereo vision is proposed. With this strategy, the desired trajectory of the manipulator is generated so that both high-speed swing motion and tracking motion combine to meet the ball squarely with the bat. As a result the manipulator can follow the ball while swinging the bat at high speed even if it is difficult to predict the trajectory of a ball. Experimental results are shown in which a high-speed manipulator hits a ball thrown by a human.

Dynamic preshaping is important for high speed capturing robot to catch a moving object successfully. This paper discusses dynamic preshaping for a capturing robot driven by a single wire. To control the finger posture while approaching an object, there are three essential factors, wiring, spring distribution among joints, and mass distribution in each finger link. Giving the final link posture of finger, mass distribution, and tension with respect to time, we explore hoe to determine both the wiring and the spring distribution. We first show how to find them under a quasi-static motion, where all dynamic effects are neglected. We further explore an approach to find them under a dynamic motion where the tension increases fast enough.

In this paper a robotic catching algorithm based on a nonlinear mapping of visual information to the desired trajectory is proposed. The nonlinear mapping is optimized by learning based on constraints of dynamics and kinematics. As a result a reactive and flexible motion is obtained due to the real-time high-speed visual information. Experimental results on catching a moving object using a high-speed vision system and a manipulation are presented.

Research attempting to merge human and mechanical systems through a nervous system is one of the most dynamic fields in interface technology. Recently, artificial cochlear and retinal nerves were connected via a nervous system. However, there has been no report concerning the replacement of human tactile sensation by artificial sensors connected to the tactile sensory nerve system. The present study reports the development of a system which enables the touch strength applied to a robot hand to be sensed and applied to the hand of a human operator via the microstimulation method. The operator controls the grasping action of a robot hand equipped with tactile sensors at a remote location via a network. The contact information between an object and the robot hand is transmitted via the network, and the tactile sensory nerve of the operator is stimulated by a microelectrode. The force applied to the robot hand can be sensed as an applied force to the hand of the human operator. Applications of the proposed system include the presentation of tactile perception in remote manipulation. In addition, the proposed system is an effective method for transmitting sensation from artificial limbs. The present paper describes the experimental system and results.

In this paper we introduce a newly developed high-speed multi-fingered robotic hand. The hand has 8-joints and 3-fingers. A newly developed small harmonic drive gear and a high-power mini actuator are fitted in each finger link, and a strain gauge sensor is in each joint. The weight of the hand module is only 0.8 kg, but high-speed motion and high-power grasping are possible. The hand can close its joints at 180 deg per 0.1 s, and the fingertips have an output force of about 28 N. The hand system is controlled by a massively parallel vision system. Experimental results are shown in which a falling object was caught by the high-speed hand.

A real-time visual processing system using a general-purpose vision chip, an image sensor in which photo detectors and processing elements are integrated, is described. In order to control the vision chip and process its output at high speed, a novel architecture called SPARSIS, in which the control process of the vision chip is pipelined and integrated with a RISC type integer pipeline, was developed. This architecture can guarantee real-tune operation with high temporal resolution, and even makes possible software-controlled A/D conversion. Sample algorithms demonstrating its fine-grained real timeliness, and experimental results with the implemented system, are also described.

This paper discusses the design of a capturing system with an extremely high response. The key for achieving a quick response is the arm/gripper coupling mechanism, where the potential energy initially accumulated in the arm is transferred to the kinetic energy of the arm and, continuously, to the kinetic energy for closing the gripper at the capturing point without any time lag. The experimental results show the maximum acceleration of 91G corresponding to almost two times higher than the conventional world record, and the total capturing time of 0.03 sec. Experiments on capturing a ball are also shown.

When considering real-world applications of robot control with visual servoing, both 3D information and a high feedback rate are required We have developed a 3D target tracking system with a 1ms feedback rate using two high-speed vision systems called column parallel vision (CPV) systems. To obtain 3D information, such as position, orientation, and shape parameters of the target object, a feature-based algorithm has been introduced using moment feature values extracted from vision systems for a spheroidal object model. Also, we propose a new 3D self windowing method to extract the target in 3D space, which is an extension of the conventional self windowing method in 2D images.

We have developed a 1 ms vision system, to provide a much faster frame rate than that of the conventional systems. Our 1 ms vision system has a 128/spl times/128 PD array and an all parallel processor array connected to each other in a column parallel architecture, so that the bottleneck of an image transfer is solved. 1 ms visual feedback has been realized in this system, in which the image feature value is extracted in 1 ms cycle-time for visual servoing. We have also developed a high speed active vision system (AVS)-II. Finally, we provide a detail discussion on our 1 ms vision system and its performance through some experiments.

We describe our research on optically interconnected optoelectronic parallel computing systems. Our architecture is based on a multilayer pipeline of two-dimensional optoelectronic device arrays in which each pixel is composed of an optical input channel, a general-purpose programmable processor, local memory, and a surface-emitting laser diode as an optical output channel. Free-space optics provides parallel, global communication between layers in the pipeline via optical paths that are dynamically reconfigurable. Design and initial realization of a system are described.

In high speed real-time vision systems, such as vision chips, whose frame rate is much higher than video rates (NTSC 30 Hz/PAL 25 Hz), many conventional image processing techniques can be realized by more simplified algorithms by considering that in such a high speed vision system the change in images between frames is small. We propose segmentation/corresponding algorithms, "self windowing", as simplified algorithms using such a property. In fact we show that the self-windowing algorithms can work on our vision chip, and we show experimental results for several image sequences at high frame rates.

In most conventional manipulation systems, changes in the environment cannot be observed in real time because the vision sensor is too slow. As a result the system is powerless under dynamic changes or sudden accidents. To solve this problem we have developed a grasping system using high-speed visual and force feedback. This is a multi-fingered hand-arm with a hierarchical parallel processing system and a high-speed vision system called SPE-256. The most important feature of the system is the ability to process sensory feedback at high speed, that is, in about 1 ms. By using an algorithm with parallel sensory feedback in this system, grasping with high responsiveness and adaptivity to dynamic changes in the environment is realized.

We introduce a visual impedance scheme for vision based control which realizes task-level dynamical robot control. This method is simply described as applying image features to the impedance equation so that integration of a visual servo and a conventional servo system can be naturally accomplished. With visual impedance, an adaptive motion is obtained for real robot tasks in dynamically changing or unknown environments based on the framework of impedance control. In such cases, very high rate visual feedback is necessary to control the robot dynamics but most conventional vision systems using CCD cameras can never satisfy this condition because their sampling rate is limited by the video signal. To solve this problem, we developed a general-purposed vision chip SPE-256 and the 1 ms visual feedback system which can achieve an adequate servo rate to control dynamics. The 1 ms visual feedback system for a robot control system is described. Experimental results with some real robot tasks are shown.

Most conventional visual feedback systems using CCD camera are restricted by video rates and therefore cannot be adapted to the changing environment sufficiently quickly. To solve this problem we developed a 1 ms visual feedback system using a general purpose massively parallel processing vision system in which photo-detectors and processing elements are directly connected. For high speed visual feedback fast image processing algorithms are also required. In particular, the difference of images between frames is very small in our system because of its high speed frame rate. Using this feature, we can realize several image processing techniques by simpler algorithms. In this paper we propose a simple algorithm for target tracking using the feature of high speed vision, and realize target tracking on the 1 ms visual feedback system.

A new method for sensor fusion is proposed and a sensor fusion system using the method with parallel processing architecture is shown. The method utilizes mappings between sensor data and can be applied not only to static coordination transforms between sensor data, but also to the dynamics of actuators. Capabilities of the method are confirmed by edge tracking experiments using the experimental sensor fusion system with a PSD (position sensitive detector), a CCD camera and an XY-recorder. The system is based on parallel processing architecture using transputers in a network. It is concluded that parallel processing is an intrinsic architecture for sensor fusion.

The authors describe a new sensor data integration architecture that utilizes a physical network paradigm. The authors have realized a prototype system on a parallel processor network using Inmos's transputers. Simple demonstrations of 2-D shape recognition and target tracking are shown and the authors have demonstrated its ability to respond appropriately to varying environmental constraints on the structure of the signal flow. It is shown that one can gain higher reliability in the architecture using the flexibility of the signal flow topology.

Abstruct-A visual processing system for early vision using massively parallel processing is described. The system named SPE-4k (Sensory Processing Element - 4k) has 64x64~4096 processing elements (PE’s) which are directly connected with photo detectors in parallel. Eight PE’s are integrated into an originally developed LSI. An mesh type networks is used for interconnection among PE’s. The system carries out input and processing of visual information at extremely high speed by using massively parallel processing. Edge detection using four neighborhoods, for example, can be done in 3.3~s. Since the PE is designed to be so compact that the system can be integrated, the system is regarded as a scale up model of integrated parallel processing vision system in future. In this paper, architectures of the parallel processing LSI and the SPE-4k system are shown. Processing performance of the system is evaluated by carrying out some applications on the system.

A high-resolution and sheetlike form imaging tactile sensor with video signal output has been developed. The sensor has a 64*64 array of sensing elements on a flexible PC board with 1-mm spatial resolution. Since the sensor outputs pressure distribution as a video signal, a real-time tactile image can be observed using a TV monitor. As the sensor is made up of a sheet of pressure-sensitive conductive rubber and stripe electrodes, there are undesirable currents passing between sensing elements. Thus these undesirable current passes must be cut within a short scanning period (500 ns for each element). The authors used a ground potential method and proved it useful under such a high-speed scanning condition. The properties of pressure-sensitive conductive rubber, including hysteresis and creep effects, are presented. A spatial filtering effect of an elastic cover for a tactile sensor is analyzed.>