In this study, we propose a method for an aerial display. The method uses a high-speed gaze control system and a laser display to perform projection mapping on a screen at a distance, which is suspended from a flying drone. A prototype system was developed and successfully demonstrated dynamic projection mapping on a screen attached to a flying drone at a distance of about 36 m, which indicated the effectiveness of the proposed method.

In this paper, we propose an edible lens made from foodstuffs. The optical lens is used for forming an optical system. It is expected that it will be possible to make optical systems edible by preparing an edible lens that has the same function as the conventional optical lens. In order to realize this, prototypes of edible lens made of agar were developed, because the agar had been reported as a material to form edible retroreflector materials. Furthermore, we investigated its optical performance.

In this research, we propose an edible retroreflector made of candy. Previously proposed edible retroreflector was made of agar, granulated sugar, and water, so it was weak against drying and lost its function in a short period of time. However, solid foodstuffs like candy are stable against drying. Thus, if it was possible to create a retroreflector from candy, longer lifetime is expected to be achieved. Therefore, edible retroreflectors made of candy was developed, and the performances as an optical device were evaluated.

A retroreflector is a kind of optical device that can reflect incident light back to the light source. A retroreflector is often used as an optical marker for machine vision since it becomes very bright in the scene and is easy detect by image processing.

This paper reports the prototype of a new optical high-speed gaze controller, named Saccade Mirror 3, that overcomes the limited gaze control range (~60 deg) of our previously proposed high-speed gaze controller, called Saccade Mirror. Saccade Mirror 3 is based on three automated rotational mirrors, and this gaze control mechanism can achieve an ultrawide gaze control range of 360 deg in theory. A prototype was developed based on this mechanism. A gaze control range of over 260 degrees in the pan direction and a high-speed response of about 10 ms were confirmed. Furthermore, the prototype was applied to high-speed visual tracking coupled with a 1000 fps high-speed vision system. A visual tracking algorithm was developed for this mechanism, and high-speed tracking of a table tennis ball in play and a flying drone was successfully demonstrated.

A precise calibration method for optical high-speed gaze controllers using rotational mirrors is proposed. The mirror-based pan/tilt camera can change gaze direction more quickly than a conventional pan-tilt-zoom camera, which actuates the camera itself. The high-speed gaze controller can track moving objects at high magnification and can utilize new applications of three-dimensional measurement. However, the controller is difficult to calibrate owing to shallow depth of field (DOF) at close focus positions and mechanical differences. We propose to control optics and illumination to detect calibration pattern control points precisely against shallow DOF; we deepen the DOF using a small aperture, coaxial episcopic illumination, and a retroreflective checkerboard pattern. We introduce mirror thickness for an exact gaze control model and bundle adjustment for large center-of-projection translation. We confirm that wide gaze range calibration is achieved more accurately than when mirror thickness is not considered.

Dynamic Projection Mapping, projection-based AR for a moving object without misalignment by a high-speed optical axis controller by rotational mirrors, has a trade-off between stability of highspeed tracking and high visibility for a variety of projection content. In this paper, a system that will provide robust high-speed tracking without any markers on objects against illumination changes, including projected images, is realized by introducing a retroreflective background with the optical axis controller for Dynamic Projection Mapping. Low-intensity episcopic light is projected with Projection Mapping content, and the light reflected from the background is sufficient for high-speed cameras but is nearly invisible to observers. In addition, we introduce adaptive windows and peripheral weighted erosion to maintain accurate high-speed tracking. Under low light conditions, we examined the visual performance of diffuse reflection and retroreflection from both camera and observer viewpoints. We evaluated stability relative to illumination and disturbance caused by non-target objects. Dynamic Projection Mapping with partially well-lit content in a low-intensity light environment is realized by our proposed system.

Augmented reality (AR) allows us to experience real-world objects onto which some extra visual information has been added by illumination from a projector. Most of the projected objects in AR research remain stationary, such as a white figure mounted on a desk, a wall, a screen, and so on. We have proposed, and successfully demonstrated, an AR system for dynamic objects, such as a flying ball, a moving object held in a person's hand, and so on. In the work described in this paper, we introduced a high-speed optical axis controller (a mirror-based device that we have developed) and an original target tracking algorithm for dynamic objects, which is based on HSV color detection.

Recently, mixed reality (MR) involving the use of a projector, which is often called projection-based MR, has attracted much attention and is being widely studied. However, most MR research is mainly focused on static objects or situations. In contrast, we propose "Lumipen," which is a projection-based MR system for high-speed or high-frequency objects, using a high-speed vision sensor and a projector with a high-speed optical gaze controller. The high-speed vision sensor detects the dynamics of the object, and the high-speed optical gaze controller controls the projection direction in a few milliseconds. Lumipen is expected to enable addition of sophisticated and diverse visual information even in dynamic live content (e.g., a play, a sports game, a music concert, etc.), like computer graphics added to video content.

We developed an optical high-speed gaze controller, called the “Saccade Mirror”, and used it to realize a high-speed pan/tilt camera with a high-speed image processor. Generally, in a pan/tilt camera, the gaze is controlled mechanically by rotational actuators. However, because pan/tilt cameras will be expected to use high-speed image processing (≥ 1000 fps), sufficiently high-speed performance of the gaze controller, comparable to the high frame rate, cannot be obtained with the usual method. In our system, the camera itself was fixed, and an external Saccade Mirror subsystem was used for optical gaze control. An arbitrary gaze direction within ±30 deg could be achieved in less than 3.5 ms.

We report a high-speed liquid lens, called a Dynamorph Lens (DML), and its applications in the field of computer vision. The developed lens uses a liquid-liquid interface as a variable refractive surface. The interface shape is controlled by the volume of a liquid chamber using a high-speed piezostack actuator. A developed prototype demonstrated both millisecond-order response speed and practical optical performance. A high-speed focusing vision system that can adaptively control its focal length in based on visual feedback by using the DML prototype coupled with a high-speed imager is also proposed. High-speed autofocusing within 15.8 ms and focus tracking of a moving object were demonstrated using the developed system. The DML prototype was also applied to high-speed focus scanning of a camera lens at 500 Hz. An image sequence was captured at 8000 fps through the lens so that every eight images were captured with a different focus, and a 1000 fps video with extended depth of field was successfully synthesized.

We propose a new framework and novel visual control system for motile cells in three-dimensional (3-D) space. Our goal is to utilize microorganisms as micro-robots in various applications by exploiting galvanotaxis (locomotor response to electrical stimulus) to actuate them. This requires automated motion control of swimming cells in 3-D space; in contrast, our previous work has been limited to 1-D or 2-D motion control on the focal plane. The system is capable of 3-D tracking and control of swimming cells by utilizing a high-speed vision system. A combination of "lock-on" tracking within the focal plane and automated focusing using a Depth-From-Diffraction method executed at 1-kHz frame rate ensures both detailed measurement and a large working space. Experimental results for closed-loop 3-D motion control of Paramecium cells trapped within a small 3-D region demonstrate the possibility of using microorganisms as micromachines.

In this paper, we propose the concept of a "mobile microscope". This is a hand-held microscope that can be used for observation in various places. Because microscopes which have a small observable area and are vulnerable to vibrations, images from hand-held microscopes are usually blurred. We propose a method of stabilizing images from a microscope which are affected by shaking, and we use the method to realize a mobile microscope. The method is based on high-speed visual feedback with an inclined image sensor for measuring the 3D movements of the microscope. We developed a prototype microscope system, without a built-in actuator, by employing this method. Experimental results showed that images from the hand-held microscope could be stabilized by our method.

In this paper, a new serial algorithm for highspeed autofocusing of cells is proposed. The proposed algorithm extracts depth information of target cells from their diffraction patterns. High-speed focusing of yeast cells with 20 ms response time was demonstrated. Continuous autofocusing of yeast cells with a scanning microscope was also demonstrated. The successful continuous autofocusing suggests that image-based high- throughput measurement of cells could be realized using the proposed algorithm.

This paper proposes a new autofocusing method for microbiological specimens, such as cells. The proposed focusing method used a quick focus estimation named "depth from diffraction", based on a diffraction pattern in a defocused image of a biological specimen. Since this method can estimate the focal position of the specimen from only a single defocused image, it can easily realize high-speed autofocusing. To demonstrate the proposed method, continuous focusing was applied to focus tracking of a swimming paramecium, in combination with two-dimensional tracking. This allowed three-dimensional tracking of the paramecium for 75 s to be successfully demonstrated

We propose a novel method to control a lot of microorganisms by using visual feedback for micro-robotic application. Our goal is to control a cluster of microorganisms as microscale smart robots for various applications, since microorganisms have efficient actuators and accurate sensors. Compared with single-cell level actuation methods proposed before, this method has the following advantages: (1) a cluster of microorganisms can cancel stochastic perturbations of behavior by taking the average; (2) a cluster of microorganisms can be more powerful than a single microorganism, and thus it can perform a variety of tasks. A visual feedback system is constructed to control cells using galvanotaxis (response to the electric field). Experimental results show the feasibility of visual feedback control of a paramecium cell cluster. In addition, micro-manipulation of small objects by the cell cluster is also demonstrated

We propose a dynamics model of galvanotaxis (locomotor response to electrical stimulus) of the protozoan Paramecium. Our purpose is to utilize microorganisms as micro-robots by using galvanotaxis. For precise and advanced actuation, it is necessary to describe the dynamics of galvanotaxis in a mathematical and quantitative manner in the framework of robotics. However, until now the explanation of Paramecium galvanotaxis in previous works has remained only qualitative. In this paper, we construct a novel model of galvanotaxis as a minimal step to utilizing Paramecium cells as micro-robots. Numerical experiments for our model demonstrate realistic behaviors, such as U-turn motions, like those of real cells.

We propose a visual control system for motile cells. Our goal is to control microorganisms as microscale smart robots for various applications. As a first step, we have developed a visual feedback control system for Paramecium caudatum cells. In order to ensure both detailed measurements and a large working space, "lock-on" tracking of a free-swimming cell with a high frame rate is essential. In our system, high-speed (1-kHz frame rate) tracking hardware and software are used for the continuous observation of moving cells with high magnification. Cells swim in a chamber, and their positions and other properties are measured and computed in real time. The chamber position is visually controlled automatically to track a specific cell. The cell motion is controlled electrically by utilizing the galvanotaxis (intrinsic reaction to electrical stimulus) of microorganisms. Experimental results for open-loop control (periodic zigzag motion) and closed-loop control (trapping within a small region that is 1 mm wide) demonstrate the possibility of using microorganisms as micromachines.

We propose a control system of motile cells. Our goal is to construct a large-scale Organized Bio-Modules (OBM) in which microorganisms are controlled as micro-size smart robots in an organized way. For the first step, we have developed a visual feedback control system of Paramecium caudatum cells. Tracking method is used for observation of moving cells with high magnification. Cells swim in a chamber and their positions are measured by high-speed vision. The chamber position is visually controlled to track a specific cell. The cell motion is controlled electrically by utilizing the galvanotaxis (intrinsic reaction to electrical stimulus) of microorganisms. Experimental results of region-trapping demonstrate the possibility of the OBM system.