In this paper, we develop a multiple-object zooming system which can capture clear images of different objects at an ultrafast speed. The system consists of a panoramic HFR stereo camera and a galvanometer-based reflective pan-tilt-zoom (PTZ) camera. In order to alleviate the impact of brightness, noise, and viewing angle in the image, we use the high speed motion information of the object for stereo correspondence. According to the spatial positions of all objects obtained from HFR stereo correspondence, we can obtain the control voltage of the pan and tilt mirror of the galvanometer-based reflective PTZ camera through the mapping relationship. Then, PTZ camera captures clear images of multiple objects in a time-division multiplexed manner at an extremely fast speed. Experimental results show that we can distinguish multiple fast-moving people indoors in the HFR stereo camera and capture their high-definition facial images simultaneously.

This paper presents an innovative continuous autofocus (C-AF) approach based on high-speed vision. It consistently provides focused images with stable and sufficiently high frame rates, aiming to improve the ability to track small, fast-moving objects in a highly magnified scene. To achieve this, we propose the concept of a dynamic-range focal sweep enabled by a high-speed camera and a focus-tunable liquid lens with high adjustment capability. The focal sweep consistently covers a small range around the object’s focus position, guided by previous depth results obtained through the depth-from-focus (DFF) technique. We conducted verification experiments to thoroughly analyze the capability of the proposed C-AF approach. By integrating a 2-axis Galvano mirror, we built a high-speed C-AF active vision system with rapid focus and pan-tilt adjustments. The comprehensive experiment results highlight the advanced capabilities of our seamless C-AF in tracking magnified objects.

Analysis of the behavior between the wheel and the road surface is important for understanding a vehicle's safe traveling performance. In contrast, sensors attached to the wheels have many limitations, and it is difficult to capture slipping phenomena such as spinning and sliding with rotation sensors alone. In addition, it is difficult for a fixed-point camera to capture phenomena between the wheel and the road surface that cannot be predicted when and where they will occur because the angle of view is limited. Here, high-speed zoom tracking was performed remotely using ultra-high-speed active vision. Image analysis of the rotating wheel, which is always magnified at the center of the high frame rate image, enabled non-contact measurement of wheel behavior while the vehicle moved. Experiments were conducted using actual N-gauge model vehicles and Segways to clarify the slipping phenomena of running vehicles and analysis of wheel deformation behavior. By analyzing tracking zoomed images taken at 500 fps, we quantitatively verified the measurement of slip during sudden stop and sudden acceleration and wheel deformation behavior during running over hump.

Traditional gimbal-based bionic eye systems usually use a multi-degree-of-freedom mechanical platform to move the camera freely, which makes the structure complex and bulky. The galvanometer-based reflective bionic eye system uses a galvanometer to replace the traditional mechanical rotation structure, which separates the camera from the gimbal system, greatly simplifying the structure. However, there are currently few methods for calibrating such systems, mostly for object detection and tracking. In this paper, a flexible method for high-precision calibration of a galvanometer-based reflective bionic eye system is proposed. In this method, a planar target is used for the calibration of the bionic eye system. The effectiveness and accuracy of the method are evaluated by the reprojection error of the control voltage and the spatial localization of the binocular system. Experiments show that the error of the control voltage after calibration is less than 0.2%. At an indoor distance of about 7 m, the RMSE of spatial visual localization is less than 0.3 cm.

This study proposes a dual-camera system for indoor high magnification surveillance which is capable of achieving always-in-focus and non-delay gaze control based on high-speed vision. The users are enabled to move the mouse freely on the wide-view screen while observing its in-focal zoom-in monitoring video in real-time. The proposed system consists of a wide-angle camera for wide-view and a Galvano mirror-enabled ultra-fast pan-tilt-zoom (PTZ) camera for zoom-in view. To achieve always-in-focus, a high-speed focus scanning system is proposed that is comprised of a high-speed camera, a parfocal zoom lens, and a gear mechanism. Through continuously reciprocating rotational motion of the focusing ring driven by the servo motor, the high-speed camera captures sets of images with varying focal distances. Moreover, we proposed a most-in-focus (MIF) frame extraction algorithm to select the sharpest images as output. The experimental results are obtained to confirm the effectiveness of our system.

We developed a dual-camera-based multi-face zoom-tracking system that captures zoomed targets simultaneously by combining a mirror-drive pan-tilt camera with ultrafast gaze control for zoomed views and an RGB-D camera for 3D wide view. In our system, the RGB-D camera detects multiple persons in the wide view and outputs the 3-D positions of their keypoints with CNN-based pose estimation. The mirror-drive pan-tilt camera simultaneously switches its viewpoints to the directions of the nose keypoints to zoom on their face images to perform person identification. We demonstrate our system performance using experimental results of a 3-D people-flow analysis with person identification, where the mirror-drive pantilt camera with 120-viewpoints/s-switching functions effectively as five virtual 24-fps pan-tilt cameras to track and zoom on the faces of five walking persons in an indoor scene.

In recent years, vision-based analysis systems of micro-objects in a microchannel have been actively developed. However, it is difficult to focus on high-speed micro-objects in a microchannel because the general height of a microchannel is approximately 10-100 μm, whereas the depth of focus of the objective lens is approximately 1-4 μm. Therefore, we propose a high-speed well-focused image-capturing microscope, which is a system with an objective lens attached to a vibration machine that moves the focus position rapidly by oscillating it up and down to capture well-focused images using a histogram-based algorithm. The proposed microscope system is verified experimentally to capture well-focused images of moving micro- objects.

This paper proposes an intelligent and fast tracking method for robust trackability against appearance changes. The method hybridizes a correlation-based tracking algorithm operating at hundreds of frames per second (fps) with a deep learning-based recognition algorithm operating at dozens of fps. A prototype intelligent mechanical tracking system was developed by implementing our hybridized tracking algorithm on a 500-fps vision platform. A complex-shaped target can be robustly tracked at the center of the camera view in real time by controlling a pan-tilt active vision system with 500 Hz visual feedback. The tracking performance of our proposed algorithm was verified by showing several experimental results for pre-learned objects, which were quickly manipulated against complex backgrounds.

Recent progress in LOC technology have made it possible to analyze cells in microfluidic devices in a microscopic environment. Vision-based cell analysis systems have a limitation in their analysis range owing to the small view area of the microscopic objective lens; thus, an expansion of the view area of microscopic observation is required. In this paper, we propose a view expansion system for photomicroscopy using the observing point movement of a Galvano mirror. We develop a prototype of the proposed view expansion system that consists of a microscope, Galvano mirror, and vision system. Then, the visual field of the developed microscopic view expansion system is derived experimentally. Finally, our system is demonstrated through panoramic image developing experiments. The effectiveness of the proposed system is verified by comparing it with a normal microscopic video shooting area.

In this paper, we propose a design method for the mechanism of a robotic arm suitable for a multicopter. The motion of an arm attached to a multicopter can possibly disturb the multicopter attitude. Our aim is to suppress this attitude disturbance by mechanical methods. In the proposed design method, the arm has the following features for suppressing the disturbance of the multicopter attitude. 1. The robotic arm can adjust its center of gravity (COG). 2. In the simplified model, the sum of the angular momentum of the rotating parts constituting the robot arm is equal to zero. These features can compensate for both the displacement of the COG of the arm and the counter torque generated from rotating parts, which cause the disturbance to the multicopter attitude. Since the disturbance of the multicopter attitude can be suppressed mechanically, the robotic arm does not require a special flight control law. In other words, this robotic arm has the advantage of being attached to a multicopter using a common, off-the-shelf, multipurpose flight controller. Furthermore, we discuss the mass distribution, power transmission method, and the moment of inertia of rotating parts, such as the speed reducer, motor, and arm. Additionally, we fabricate a prototype robotic arm with four joints and three degrees of freedom (DOFs), based on the proposed design method. Finally, the experiments showed that the robotic arm suppressed the disturbance of the multicopter attitude caused by the arm motion.

The research field of nongrasping manipulation is a maturing area in robotic motion control. However, the common principles of motion planning for nongrasping manipulation systems have not yet been established. This paper proposes the concept of virtual connecting manipulation as a generalized motion planning framework for nongrasping manipulation systems. As a preliminary result, we had previously realized a flower-stick juggling task called “propeller motion” using an actual experimental system. In this paper, we apply the virtual connecting manipulation concept to a flower-stick juggling task and analyze the generated motion from the view point of analytical methodology. We conduct a stability analysis of the generated cyclic motion of the flower stick by using a Poincaré map, and the analytical results show that the generated cyclic motion is asymptotically stable.

We propose a novel scheme for manipulating a passive object using an active plate. The objective of this study is to control an object's orientation with respect to the gravitational force direction by using an active plate for realizing hitherto unrealized object motion. In this context, motions of the object and active plate are designed to be cyclic. A state vector composed of the object's angle and angular velocity is defined, and the cyclic motion is expressed as a nonlinear discrete system. Fixed points of the state vector are searched for in the designed cyclic motion. A stability analysis around the fixed points is conducted using a Poincaré map. As a result, the fixed points are shown to be asymptotically stable. Finally, experimental results are used to verify that the object's angle can be manipulated with the designed cyclic motion using the plate.

Flower stick juggling is a dexterous task done by skillful jugglers. We aim to realize dexterous tasks done by humans using robotic systems. This work focuses on flower stick juggling and proposes a feedback control strategy for a flower stick juggling task called “propeller” as one of the robotic dexterous manipulations. The propeller motion is modeled by considering combined flower stick and a robotic manipulator. We developed a control strategy that allows stable cyclic rotation of the flower stick in the air. The control parametars in the control strategy are searched through numerical simulation. Finally, the flower stick propeller motion is realized using an actual robotic system.

This paper proposes a novel concept for simultaneous cell shape and motion analysis in fast microchannel flows by implementing a multiobject feature extraction algorithm on a frame-straddling high-speed vision platform. The system can synchronize two camera inputs with the same view with only a tiny time delay on the sub-microsecond timescale. Real-time video processing is performed in hardware logic by extracting the moment features of multiple cells in 512 × 256 images at 4000 fps for the two camera inputs and their frame-straddling time can be adjusted from 0 to 0.25 ms in 9.9 ns steps. By setting the frame-straddling time in a certain range to avoid large image displacements between the two camera inputs, our frame-straddling high-speed vision platform can perform simultaneous shape and motion analysis of cells in fast microchannel flows of 1 m/s or greater. The results of real-time experiments conducted to analyze the deformabilities and velocities of sea urchin egg cells fast-flowing in microchannels verify the efficacy of our vision-based cell analysis system.

We present a high-speed vision-based morphological analysis system for fast-flowing cells in a microchannel. Real-time video processing is performed in hardware logic by extracting the moment features and bounding boxes of multiple cells in 512 × 256-pixel images at 2000 fps. The extracted cell regions are pushed into a first-in-first-out (FIFO) buffer for real-time image-based morphological analysis after being shrunk proportionally to a certain size. By extracting the bounding boxes of the cell regions using hardware logic and shrinking the cell region to a certain size to reduce processing time, our high-speed vision system can perform fast morphological analysis of cells at 500 cells/s in fast microchannel flows. Moreover, snapshots of all passed cells under the microscope are stored for offline verification. The results of real-time experiments conducted to analyze the size, eccentricity, and transparency of sea urchin embryos flowing fast in microchannels verify the efficacy of our vision-based cell analysis system.

In this paper, we report on the development of a projection mapping system that can project RGB light patterns that are enhanced for three-dimensional (3-D) scenes using a GPU-based high-frame-rate (HFR) vision system synchronized with HFR projectors. Our system can acquire 512×512 depth images in real time at 500 fps. The depth image processing is accelerated by installing a GPU board for parallel processing of a gray-code structured light method using infrared (IR) light patterns projected from an IR projector. Using the computed depth images, suitable RGB light patterns to be projected are generated in real time for enhanced application tasks. They are projected from an RGB projector as augmented information onto a 3-D scene with pixel-wise correspondence even when the 3-D scene is time-varied. Experimental results obtained from enhanced application tasks for time-varying 3-D scenes such as (1) depth-based color mapping and (2) augmented reality (AR) spirit level, confirm the efficacy of our system.

This paper proposes a novel method for simultaneous cell shape and motion analysis in rapid microchannel flows based on a multi-object feature extraction algorithm with a frame-straddling high-speed vision platform. This system can synchronize two camera inputs that share the same view with only a very small sub-microsecond time delay. Real-time video processing is performed using the hardware logic by extracting the moment features of multiple cells at 2000 fps or more, which are obtained from the two camera inputs, and their frame-straddling time can be adjusted from 0 to 0.5 ms in 9.9 ns steps. After setting the frame-straddling time within a certain range to avoid large image displacements between the two camera inputs, the frame-straddling high-speed vision platform can perform simultaneous shape and motion analysis of cells in fast microchannel flows of 1 m/s or greater. The results of real-time experiments conducted to analyze the deformabilities, velocities, and shapes of fast-flowing sea urchin egg cells in straight and L-type microchannels verified the efficacy of our vision-based cell analysis system.

A manipulator which enables noncontact control of the position and attitude of an object on a plane by controlling a fluid field is proposed. The manipulator comprises 2 boards with grid pattern arrangements of holes and a parallel link robot which is capable of translational motion with 2 degrees of freedom and rotational motion with 1 degree of freedom in a plane. Air jets are discharged from the grid of holes. The direction of the air jets from the holes can be controlled by changing the positional relationship of the 2 boards by means of the parallel link robot. This makes it possible to form an air flow with a unidirectional fluid field or a vortex-like fluid field. When an object is placed in the fluid field, translational motion and rotational motion of the object are possible. If the position and attitude of the object are photographed and calculated using high speed camera and feedback control is performed, the position and attitude of the object can be controlled. The structure of the manipulator and the principle of control of the fluid field are described, and the possibility of controlling the translational motion and rotational motion of an object is demonstrated experimentally.

In this paper, a high-speed vision-based morphological analysis system for fast-flowing cells in a microchannel implementing a multi-object feature extraction algorithm on a high-speed vision platform is proposed. Real-time video processing is performed in hardware logic by extracting the moment features and bounding boxes of multiple cells in 512×256-pixel images at 2000 fps. The extracted cell regions are pushed into a first-in-first-out (FIFO) buffer for real-time image-based morphological analysis after being shrunk proportionally to a certain size. By extracting the bounding boxes of the cell regions using hardware logic and shrinking the cell region to a certain size to reduce processing time, our high-speed vision system can perform fast morphological analysis of cells at 2 ms/cell in fast microchannel flows. The results of real-time experiments conducted to analyze the size, eccentricity, and transparency of fertilized sea urchin eggs fast flowing in microchannels verify the efficacy of our vision-based cell analysis system.

Improved robot mobility is necessary in order to realize the expected uses of robots in the human life environment, which includes obstacles such as stairs and narrow passages. Because objects which humans require are often located on desks or tables, which are at some height above ground level, a long vertical dimension is also necessary in the robot design. To meet these requirements, this study focused on a type of mobile robot using an inverted pendulum, which enables a robot with a long vertical dimension to negotiate narrow passages. Since it is difficult for inverted pendulum robots to climb steps, this study proposes a planetary wheel mechanism which makes it possible for an inverted pendulum robot to ascend differences in level. This mechanism has an extremely simple structure, comprising a belt, pulley, and arm, and has the advantage of not requiring addition of a new actuator. This paper describes the composition of the proposed mechanism and its principle of operation, and also describes the control method. A prototype of the robot was produced, and it was shown experimentally that the robot was capable of climbing stairs with a height of 120 mm-130 mm at a rate of approximately 2.4 s per one step.

We propose a novel dot-pattern-projection three-dimensional (3-D) shape measurement method that can measure 3-D displacements of blink dots projected onto a measured object accurately even when it moves rapidly or is observed from a camera as moving rapidly. In our method, blinking dot patterns, in which each dot changes its size at different timings corresponding to its identification (ID) number, are projected from a projector at a high frame rate. 3-D shapes can be obtained without any miscorrespondence of the projected dots between frames by simultaneous tracking and identification of multiple dots projected onto a measured 3-D object in a camera view. Our method is implemented on a field-programmable gate array (FPGA)-based high-frame-rate (HFR) vision platform that can track and recognize as much as 15×15 blink-dot pattern in a 512×512 image in real time at 1000 fps, synchronized with an HFR projector. We demonstrate the performance of our system by showing real-time 3-D measurement results when our system is mounted on a parallel link manipulator as a sensing head.

We conducted high-frame-rate (HFR) video mosaicing for real-time synthesis of a panoramic image by implementing an improved feature-based video mosaicing algorithm on a field-programmable gate array (FPGA)-based high-speed vision platform. In the implementation of the mosaicing algorithm, feature point extraction was accelerated by implementing a parallel processing circuit module for Harris corner detection in the FPGA on the high-speed vision platform. Feature point correspondence matching can be executed for hundreds of selected feature points in the current frame by searching those in the previous frame in their neighbor ranges, assuming that frame-to-frame image displacement becomes considerably smaller in HFR vision. The system we developed can mosaic 512×512 images at 500 fps as a single synthesized image in real time by stitching the images based on their estimated frame-to-frame changes in displacement and orientation. The results of an experiment conducted, in which an outdoor scene was captured using a hand-held camera-head that was quickly moved by hand, verify the performance of our system.

We have developed a fast target tracking system that utilizes four cameras with lenses of different focal lengths to track an object without blurring images, even when the object moves in the depth direction away from the cameras in three-dimensional (3-D) space. This system can maintain a well-focused camera view by switching among the four input images, instead of using lens motor control. The multi-camera system was mounted on a two-axis mechanical active vision platform. The active vision control and camera-view switching are executed by processing color 512 × 512 images from the four camera inputs at 500 fps in real time on a high-speed vision platform. The performance of our system was verified by its tracking results for objects moving rapidly in 3-D space.

This paper describes a high-frame-rate (HFR) vision system that can extract locations and features of multiple objects in an image at 2000 f/s for 512 × 512 images by implementing a cell-based multiobject feature extraction algorithm as hardware logic on a field-programmable gate array-based high-speed vision platform. In the hardware implementation of the algorithm, 25 higher-order local autocorrelation features of 1024 objects in an image can be simultaneously extracted for multiobject recognition by dividing the image into 8 × 8 cells concurrently with calculation of the zeroth and first-order moments to obtain the sizes and locations of multiple objects. Our developed HFR multiobject extraction system was verified by performing several experiments: tracking for multiple objects rotating at 16 r/s, recognition for multiple patterns projected at 1000 f/s, and recognition for human gestures with quick finger motion.

In this study, we introduce a novel concept of real-time microscopic particle image velocimetry (micro-PIV) for high-speed microchannel flows in a lab-on-a-chip using a frame-straddling high-speed vision system with two camera inputs; it can synchronize two camera inputs with the same view field with a time delay on a submicrosecond time scale. To improve the measurable range of velocity in microchannel flow observation, we propose a variable-frame-straddling optical flow (VFS-OF) algorithm that can simultaneously estimate the microchannel flow distribution as gradient-based optical flows using frame-straddled images from the two camera inputs; their frame-straddling time is determined by the amplitude of the estimated optical flow to avoid large image displacements between frames that often generate serious errors in optical flow estimation. We built a real-time micro-PIV system by software-implementing the VFS-OF algorithm in a high-speed vision system with two frame-straddled cameras; it can execute real-time video processing and recording of 512×512-pixel images at 2000 frames per second for the two cameras and control their frame-straddling time from 0 to 0.5 ms with 9.9-ns steps. Our micro-PIV system can estimate the velocity distribution of high-speed microchannel flows at 1 m/s or more in real time by controlling the frame-straddling time. Experimental results were performed for microfluidic flows on microchannels with widths of hundreds of micrometers to verify the performance of our micro-PIV system based on the VFS-OF algorithm.

In this paper, we develop a high-frame-rate (HFR) vision system that can estimate the optical flow in real time at 1000 f/s for 1024×1024 pixel images via the hardware implementation of an improved optical flow detection algorithm on a high-speed vision platform. Based on the Lucas-Kanade method, we adopt an improved gradient-based algorithm that can adaptively select a pseudo-variable frame rate according to the amplitude of the estimated optical flow to accurately detect the optical flow for objects moving at high and low speeds in the same image. The performance of our developed HFR optical flow system was verified through experimental results for high-speed movements such as a top's spinnning motion and a human's pitching motion.

Real-time face tracking at 500 fps was realized for an 8-bit 512×512 color image on a GPU-based high-speed vision platform by implementing a boosting-based face detection algorithm. This algorithm was developed by enhancing the Viola-Jones face detector. In the implementation, the size and position of a face pattern in an image can be efficiently extracted at high speed by reducing the number of window searches for Haar-like features and by combining skin color extraction with Haar classifiers, while assuming a property of high-frame-rate (HFR) vision, i.e., a small interframe image displacement. Several experiments were performed by using rapidly moving face patterns in order to verify the effectiveness of the proposed face tracking system.

In this paper, we propose a concept for the real time target tracking of vibrating objects to extract vibration image regions using digital filters at all the pixels in an image, assuming that periodic changes in image intensities exist at pixels around vibrating objects. By combining vibration-based processing with color extraction as a masking process, our algorithm can prevent the latency effect on a digital filter, which may degrade its tracking performance. Digital filters at a pixel level are implemented on a 1000-fps vision system that can measure vibration distribution over a wide range of frequencies. On our target tracking system, we can track a vibrating object at the center of the camera view by controlling a 2-DOF (Degree Of Freedom) active vision system with vibration-based image features. We have verified its effectiveness by presenting several experimental results using objects vibrating at high frequencies, which cannot be observed with the naked eye.

Abstract-In this paper, we propose an automatic behavior recognition algorithm for laboratory mice to determine their typical model behaviors and hence detect the degrees of diseases via animal testings for behavioral pharmacology. The algorithm can quantify several model behaviors of mice by detecting the repetitive motions of fore or hind limbs at several hertz; these motions, too rapid for the naked eye and the NTSC cameras, are captured on a high-frame-rate video. Even when a mouse changes its position and orientation, the algorithm can always detect these repetitive motions as periodic frame differential image features on four segmented regions-head, left side, right side, and tail-because the silhouette image of a mouse is converted in the polar coordinate system as its shift and rotation-invariant shape by detecting the positions of its head and tail. The effectiveness of the algorithm is evaluated for six typical model behaviors, including scratching and grooming, by analyzing the experimental results for several laboratory mice, obtained by capturing long-term videos at 240 fps. We have provided the detection/correction ratios and compared the time durations of the model behaviors of these mice.

In this study, we propose the concept of dynamics-based visual inspection for the verification of structural dynamic properties of a vibrating object; the inspection method involves the analysis of the vibration distribution of the object by using a high-frame-rate video. Under unknown ambient excitation, modal parameters of an excited object are simultaneously estimated to determine its input-invariant dynamic properties by using a fast output-only modal analysis algorithm, SSI-CPAST. The algorithm was implemented on a 2000-fps vision platform, and it facilitates non-destructive monitoring of the structure of beam-shaped objects vibrating at dozens of hertz; the algorithm detects small changes in the dynamic properties of the objects caused by internal defects such as fatigue cracks. The modal parameters resonant frequency and mode shape were actually estimated for beam-shaped objects excited by human finger tapping to verify the performance of 2000-fps real-time dynamics-based visual inspection.

Abnormal behavior detection in periodic operations and high-frame-rate (HFR) video logging were realized by introducing a visual machinery surveillance algorithm, which includes a phase-encoding process for periodic operations to match input images with pre-stored reference images efficientl. Assuming that the periodic machinery operations to be monitored are perfectly regular, the surveillance algorithm can encode the phase of a periodic operation just by inspecting temporal changes in the brightnesses at several significan pixels in an input image; abnormal behavior in the periodic operation can be detected by comparing the input image with a reference image synchronized with its encoded phase without the heavy computation needed to search all the reference images. This algorithm was software-implemented on a high-speed vision platform, IDP express, which can record input images of 512 × 512 pixels and process the results at 1000 fps. An experiment was performed using a sewing machine with periodic operation at a frequency of 12 Hz, and a vidio of the abnormal behavior was automatically recorded at 1000 fps to verify the effectiveness of HFR-video-based machinery surveillance.

Real-time multi-object extraction at 2000 fps was realized by designing a cell-based labeling algorithm. The algorithm can label the divided cells in an image by scanning the image only once to obtain their moment features, and the computational complexity required for labeling can be remarkably reduced. The cell-based labeling algorithm for 8 × 8 pixel cells was implemented on a high-speed vision platform, and multiple objects in an image of 512 × 512 pixels could be extracted at 2000 fps. An experiment was performed using a quickly rotating object to verify the performance of our multi-object extraction system.

This paper introduces an intelligent high-frame-rate video logging system that can automatically detect high-speed unpredictable behavior and record video comprising images with dimensions of 512 × 512 pixels at 1000 fps. To capture high-frame-rate video of crucial moments of abnormal behavior in high-speed periodic motion, a real-time abnormal behavior detection algorithm is implemented on a highspeed vision platform, IDP Express, which can record both input images and processed results at 1000 fps for video logging. Several experiments were performed using a machine with repetitive operation at a frequency of 15 Hz and video of the abnormal behavior was automatically recorded to verify the effectiveness of our system.

This paper introduces a high-speed vision system called IDP Express, which can execute real-time image processing and high frame rate video recording simultaneously. In IDP Express, a dedicated FPGA (Field Programmable Gate Array) board processes 512 × 512 pixel images from two camera heads by implementing image processing algorithms as hardware logic; the input images and processed results are transferred to standard PC memory at a rate of 2000 fps or more. Owing to the simultaneous high-frame-rate video processing and recording, IDP Express can be used as an intelligent video logger for long-term high-speed phenomenon analysis even when the measured objects move quickly in a wide area. We applied IDP Express to a mechanical target tracking system to record a high-frame-rate video at high resolution for a crucial moment, which is magnified by tracking the measured objects with visual feedback control. Several experiments on moving objects that undergo sudden shape deformation were performed. The results of the experiments involving the explosion of a rotating balloon and the crash of falling custard pudding have been provided to verify the effectiveness of IDP Express.

This paper presents a force visualization mechanism for endoscopic surgical instruments using a Moirée fringe. This mechanism can display fringes or characters that correspond to the magnitude of a force between the surgical instruments and internal organs without the use of electronic elements, such as amplifiers and strain gauges. As this mechanism is simply attached to the surgical instruments, there is no need for additional devices in the operating room or wires to connect these devices. The structure is simple, and its fabrication is inexpensive. An example is shown with the mechanism mounted on a 10-mm forceps. We experimentally verified in vivo, using a pig, that it can display characters corresponding to the magnitude of the force, thus visually displaying the force even in endoscopic image.

A big advantage of minimally invasive surgery is the quick recovery afforded by the minimal physical injury sustained by patients. However, the operative field provided by the endoscope camera is rather narrow. This paper discusses a new broad-view camera system, which is capable of providing a wider view of the internal organs during minimally invasive surgery. This system consists of a camera unit for capturing the image and a monitor for displaying the captured image. With the use of this system, the invasiveness of the procedure does not increase, and the other specifications of current minimally invasive surgery are maintained. Furthermore, it can function as a second camera (in addition to the one attached to the endoscope) and can help assess the overall state of the target area. In this paper, we present this a newly developed camera system, which is intended for use along with the proposed system. The camera system is composed of a miniature color CMOS camera, an indwelling needle, and an extra-thin connector. The specific design of the camera unit and the method for positioning it are shown here. The performance of this camera system has been confirmed through basic and laparoscopic surgery on animals.

In the recent years, intensive studies have been carried out on the measurement of distance by using stereo camera systems. However, image systems with multiple image sensors are usually very complex and are inconvenient to use. In this paper, we present a new method that is based on the fact that the viewpoint depends on the position of aperture. We show images from different viewpoints can be obtained by using a monocular system. In the proposed method, stereo image analysis is carried out by using a viewpoint switching iris. A prototype of the viewpoint switching iris has be developed, which can be used along with a monocular camera for distance measurement. Furthermore, we have verified that this system can be used to a moving object by using a compensation method.

In this paper, we introduce a high-speed vision platform, H3(Hiroshima hyper human) vision, which can simultaneously process a 1024 × 1024 pixel image at 1000 fps and a 256 × 256 pixel image at 10000 fps by implementing image processing algorithms as hardware logic on a dedicated FPGA board. Various types of algorithms are actually implemented to show that H3Vision can work a high-speed image processing engine. Experimental results are shown for multi-target color tracking, feature point tracking, optical flow detection, and pattern recognition by using high-order local auto-correlation (HLAC) feature at a frame rate of 1000 fps or more.

Quantifying the rapid action of a mouse scratching its head with its hind leg can provide an objective behavioral indicator for atopic dermatitis, and the development of new drugs for this disease can be significantly expedited by automating such quantification. In this paper, we propose methods for extracting the scratching patterns of a mouse by focusing on its rapid and periodic behavioral patterns. We extracted the scratching patterns from the high-speed video images of actual laboratory mice, and our experimental results show that scratching can be automatically quantified without misinterpretations. Note to Practitioners - The proposed pattern detection method is completely free from the detection of false behaviors caused by markers painted on their skins. Therefore, practitioners in laboratory experiments can also easily apply it to other periodic behavior analyses, such as gait pattern analysis and tremble detection for automated quantitative evaluations of behaviors related to brain, heart, or other diseases, even if the experimented animals are sensitive to painting.

A 1,000-jps camera-projector system in which projected patterns are adoptively controlled according to image processing results is described. Adaptive structured light projection enables fast and efficient 3-D information acquisition. The prototype system is applied to the tracking of the nearest point of an object, and experimental results show that the system successfully tracked an apex of a fast-moving large! object.

This paper proposes dynamic capturing strategies where a 2D stick-shaped object with both translational and rotational velocities is completely stopped by two robotic fingers. We first show the fingertip position and the object orientation for generating a desired velocity of the object under the friction independent collision. Once the object results in a pure translational motion whose direction is perpendicular to the longitudinal axis of object, it is guaranteed that two fingers can always capture the object irrespective of friction coefficient. By using this nature, we show both 2-step and 3-step capturing strategies for a 2D stick-shaped object whose width is negligibly small. The 3-step capturing strategy can guide the object in an arbitrary direction, while the 2-step one can do it only in a particular direction. The 3-step capturing strategy is demonstrated by experiment for verifying our idea.

In various application fields, high-speed cameras are used to analyze high-speed phenomena. Coded structured light projection methods have been proposed for acquiring three-dimensional images. Most of them are not suitable for measuring high-speed phenomena because the errors are caused when the measured objects move due to light projection of multiple patterns. In this paper, we propose a new coded structured light projection method that can select not only a temporal encoding but also a spatial encoding adaptively for obtaining three-dimensional images at a high-speed frame rate. We also develop an evaluation system that uses a DLP projector and an off-line high-speed video camera, and verify the effectiveness of the proposed method by showing the obtained three-dimensional shapes for moving objects.

This paper proposes dynamic capturing strategies where a 2D stick-shaped object with both translational and rotational velocity is completely stopped by two robotic fingers. We first show the fingertip position and the object orientation for generating a desired velocity of the object under the friction independent collision. Once the object results in a pure translational motion whose direction is perpendicular to the longitudinal axis of object, it is guaranteed that two fingers can always capture the object irrespective of friction coefficient. By using this nature, we show both 2-step and 3-step capturing strategies for a 2D stick-shaped object whose width is negligibly small. The 3-step capturing strategy can guide the object in an arbitrary direction, while the 2-step one can do it only in a particular direction. The proposed strategies are demonstrated by experiments for verifying our idea.

This paper discusses jumping pattern generation for a serial link robot in order to maximize its jump height under torque limitation. By applying a genetic algorithm (GA) for determining torque assignment, we obtain various jumping patterns with respect to the torque limitation for a fixed mass of the robot. With the increase of the torque limitation, double-leg based jump, single-leg based jump, and spring-type jump are generated for achieving the largest jump height. Under an additional joint angle limitation, we also obtain an interesting solution where one end of the link is first lifted up and the other end finally kicks the ground strongly

In this paper, we propose a finger-tapping interface called FINGERTAP, which can estimate the contact states and forces on human fingertips in real time by using only the fingertip positions measured by high-speed vision cameras. In particular, we focus on the high-frequency component that develops at the moment a fingertip contacts something. We introduce a real-time estimation algorithm for the fingertip contact state and force by using frequency filtering to detect the high-frequency movement component of the fingertip positions. We also introduce a clickable pointing interface and a virtual musical instrument as a demonstration of an application using the FINGERTAP system.

We discuss the optimum design issue for tracing type legged robots in the sense that it can jump as high as possible. By applying dimensional analysis techniques, we introduce four non-dimensional parameters that control the jump ratio(= h/l) for a jumping robot. An interesting observation is that there exists the optimum design point where the jump ratio becomes maximum. Through experiments, we found that the robot with the optimum design specification can achieve the jump ratio of 3.6, while the jump ratio decreases for other design points.

This paper newly proposes the four-fingered robot hand with dual turning mechanism where two and two other fingers can independently rotate inner and outer circles with the common center, respectively. Due to this mechanical configuration, it has the particular rotating axis where the manipulation around the axis can be completely decomposed into the velocity control around the axis and the internal force control in the contact plane. We achieved a manipulation task around the axis with the time of 0.8[sec] for one rotation, while it is relatively slow for another axis.

This paper discusses 2D dynamic preshaping issue for a stick-shaped object changing both position and orientation with respect to time. We propose the two-step grasping strategy where it first stops the rotational motion by choosing an appropriate finger position, and then stops the translational motion step by step for completely making the object stationary. We confirm the basic idea by simulation.

In this paper, we propose a new vision chip architecture specialized for target tracking and recognition, and describe a prototype vision chip. Both higher order correlation features and moment features extraction algorithms are implemented in the proposed architecture in order to achieve highspeed image processing and enhanced pixel integration.

A real-time visual processing system using a general-purpose vision chip, an image sensor in which photo detectors and processing elements are integrated, is described. In order to control the vision chip and process its output at high speed, a novel architecture called SPARSIS, in which the control process of the vision chip is pipelined and integrated with a RISC type integer pipeline, was developed. This architecture can guarantee real-tune operation with high temporal resolution, and even makes possible software-controlled A/D conversion. Sample algorithms demonstrating its fine-grained real timeliness, and experimental results with the implemented system, are also described.

When considering real-world applications of robot control with visual servoing, both 3D information and a high feedback rate are required We have developed a 3D target tracking system with a 1ms feedback rate using two high-speed vision systems called column parallel vision (CPV) systems. To obtain 3D information, such as position, orientation, and shape parameters of the target object, a feature-based algorithm has been introduced using moment feature values extracted from vision systems for a spheroidal object model. Also, we propose a new 3D self windowing method to extract the target in 3D space, which is an extension of the conventional self windowing method in 2D images.

In high speed real-time vision systems, such as vision chips, whose frame rate is much higher than video rates (NTSC 30 Hz/PAL 25 Hz), many conventional image processing techniques can be realized by more simplified algorithms by considering that in such a high speed vision system the change in images between frames is small. We propose segmentation/corresponding algorithms, "self windowing", as simplified algorithms using such a property. In fact we show that the self-windowing algorithms can work on our vision chip, and we show experimental results for several image sequences at high frame rates.

In most conventional manipulation systems, changes in the environment cannot be observed in real time because the vision sensor is too slow. As a result the system is powerless under dynamic changes or sudden accidents. To solve this problem we have developed a grasping system using high-speed visual and force feedback. This is a multi-fingered hand-arm with a hierarchical parallel processing system and a high-speed vision system called SPE-256. The most important feature of the system is the ability to process sensory feedback at high speed, that is, in about 1 ms. By using an algorithm with parallel sensory feedback in this system, grasping with high responsiveness and adaptivity to dynamic changes in the environment is realized.

Most conventional visual feedback systems using CCD camera are restricted by video rates and therefore cannot be adapted to the changing environment sufficiently quickly. To solve this problem we developed a 1 ms visual feedback system using a general purpose massively parallel processing vision system in which photo-detectors and processing elements are directly connected. For high speed visual feedback fast image processing algorithms are also required. In particular, the difference of images between frames is very small in our system because of its high speed frame rate. Using this feature, we can realize several image processing techniques by simpler algorithms. In this paper we propose a simple algorithm for target tracking using the feature of high speed vision, and realize target tracking on the 1 ms visual feedback system.