Microexpressions are expressions that people inadvertently express, and therefore often represent a person’s true emotion. However, because it has a low intensity and a short duration, it is hard to be recognized correctly. In this paper, we propose a deep learning magnification method to generate macroexpressions from a single microexpression image. In the first stage, we extract the expression information from a single microexpression image. Then, We combine the idea of cyclegan and optical flow consistency to model the extracted expression features as the optical flow field between the neutral face and microexpressions. To extract a reliable optical flow field from the expression information, we design an optical flow refiner. In the second stage, we adopt an encoder-decoder network and let it learn to magnify the optical flow. Finally, the magnified optical flow guided the microexpression images to generate macroexpression images. We compare our single input based network with current two-frames-input based networks. The results show that our method performs better, even in wild images. We fed our magnified images directly into a simple ResNet18 network for recognition, achieving a competitive score under the MEGC2019 standard, compared with recent complex recognition networks.

The inclination of a spherical camera results in nonupright panoramic images. To carry out upright adjustment, traditional methods estimate camera inclination angles firstly, and then resample the image in terms of the estimated rotation to generate upright image. Since sampling an image is a time-consuming processing, a lookup table is usually used to achieve a high processing speed; however, the content of a lookup table depends on the rotational angles and needs extra memory to store also. In this paper we propose a new approach for panorama upright adjustment, which directly generates an upright panoramic image from an input nonupright one without rotation estimation and lookup tables as an intermediate processing. The proposed approach formulates panorama upright adjustment as a pixelwise image-to-image mapping problem, and the mapping is directly generated from an input nonupright panoramic image via an end-to-end neural network. As shown in the experiment of this paper, the proposed method results in a lightweight network, as less as 163MB, with high processing speed, as great as 9ms, for a 256×512 pixel panoramic image.

In this article, we propose a dataset named Wheelchair-OmniGaze and a method of estimating the gaze point of a wheelchair user that can cope with large head motion as the first step for understanding the intention of a wheelchair user. In the proposed method, the wheelchair user's face is observed by two remote cameras, called face cameras, mounted on the two front sides of the wheelchair, while a spherical camera with a 360° field of view, called a spherical scene camera, is mounted to the top rear to observe the surrounding scenes. A user's gaze point at the environment is determined by combining the facial information acquired by the face cameras with the spherical image acquired by the spherical scene camera. Concretely, a convolutional neural network–long short-term memory network is used to learn gaze points of the spherical image from the two face image sequences. Using two face cameras enables a user's face to be observed even for a large head motion. Using a spherical scene camera mounted on a wheelchair, the representation of a user's gaze points at a wheelchair-centered spherical camera coordinate can be obtained, which is independent of a user's head motion. The experimental results show that our method achieved competitive performance in comparison with the state-of-the-art method. Since our problem definition is different from the existing research, the experimental results of this article can be seen as a baseline for our built dataset.

How to detect the subtle intensity change and find the inherent relation of feature maps in a face image is the two key issues for micro-expression recognition. In this study, in order to cope with the subtle intensity change, we design an innovative Cascade Micro-Expression Magnification Network (Cascade-MEMN) to magnify the low amplitude of the intensity change for learning the intensity progressively, which helps suppress overlapping artifacts and produce more realistic magnification. In order to find the inherent relation of facial feature maps, we design a Self-attention-based Convolution Layer (SCL), which introduces self-attention to every pixel when calculating the weights in the sliding window, considering that the variation of micro-expressions is very subtle and local on face. Concretely, the SCLs are used to replace 3 Resblocks from the last stage of ResNet50 with 3 SCL blocks. Finally, a new micro-expression recognition system is realized by combining the progressive-learning-based intensity magnification network with the modified self-attention-convolution classification network. Consequently, the proposed method achieves competitive results for the composite database evaluation (CDE) protocol from MEGC 2019, as shown in the experimental result.

From a spherical image, a gaze point, instead of gaze vectors, can be estimated directly because a remote spherical camera can observe a user's face and a gaze target simultaneously. This paper investigates the problem of estimating a gaze point in a spherical image from facial landmarks. In contrast with the existing methods which usually assume gaze points move on a narrow plane, the proposed method can cope with the situation where gaze points vary in depth for a relatively wide field of view. As shown in the results of comparative experiments, we find the orthogonal coordinates of facial landmarks on a unit sphere is a reasonable representation in comparison with spherical polar coordinates; the cues of head pose is helpful to improve the accuracy of gaze points. Consequently, the proposed method achieves a performance on the accuracy of gaze points estimation which is comparable to the state of the art methods.

Facial micro-expressions (MEs) recognition has attracted much attention recently. However, because MEs are spontaneous, subtle and transient, recognizing MEs is a challenge task. In this paper, first, we use transfer learning to apply learning-based video motion magnification to magnify MEs and extract the shape information, aiming to solve the problem of the low muscle movement intensity of MEs. Then, we design a novel graph-temporal convolutional network (Graph-TCN) to extract the features of the local muscle movements of MEs. First, we define a graph structure based on the facial landmarks. Second, the Graph-TCN deals with the graph structure in dual channels with a TCN block. One channel is for node feature extraction, and the other one is for edge feature extraction. Last, the edges and nodes are fused for classification. The Graph-TCN can automatically train the graph representation to distinguish MEs while not using a hand-crafted graph representation. To the best of our knowledge, we are the first to use the learning-based video motion magnification method to extract the features of shape representations from the intermediate layer while magnifying MEs. Furthermore, we are also the first to use deep learning to automatically train the graph representation for MEs.

Facial expression is controlled by facial muscle and can be considered as appearance and geometric variation of key parts. One key challenging issue of static-based facial expression recognition is to capture effective information from a single facial image. In this paper, we propose a graph representation with Bidirectional RNN (BRNN) for static-based facial expression recognition. Each node on the graph represents appearance information around the facial landmarks. Edges represent the geometric information encoded by the distance between two nodes. A bidirectional recurrent neural network utilized to process the graph extracts the appearance and geometric representation. The final representation from BRNN is fed into a fully connected layer and a Softmax layer to infer expressions. Experimental results show that this method achieves significant improvements over the state-of-art methods on three widely used facial databases (Oulu-CASIA, CK+, and MMI), and our method reduces the error rates of the previous best methods by 42.2%, 35.9% and 18.7%, respectively.

The fitting of an ellipse to the iris contour is usually performed using five unknown parameters. In this study, we divide the continuous gaze estimation of a head-mounted eye camera into two phases. One phase, known as the calibration phase, is used to estimate the eyeball center position in relation to the coordinate system of the head-mounted eye camera. The other phase is used to fit the iris contour in 2D images employing only two parameters for gaze estimation. As seen from the experimental results, the proposed method demonstrates both credible eyeball center estimation and an accurate iris contour estimation in comparison with the conventional five unknown parameter approach.

This paper proposes a novel method of gaze estimation based on an eye model with known head pose. The most crucial factors in the eye-model-based approach to gaze estimation are the 3-D positions of the eyeball and iris centers. In the proposed method, an RGB-D camera, Kinect sensor, is used to obtain the head pose as well as the eye region of the color image. The 3-D position of the eyeball center is determined in the calibration phase by gazing at the center of the color image camera. Then, to estimate the 3-D position of the iris center, the 3-D contour of the iris is projected onto the color image with the known head pose obtained from color and depth cues of an RGB-D camera. Thus, the ellipse of the iris in the image can be described using only two parameters: the yaw and pitch angles of the eyeball in the iris coordinate system, rather than the conventional five parameters of an ellipse. The proposed method can fit an iris that is not complete due to eyelid occlusion. The average errors of vertical and horizontal angles of the gaze estimation for seven subjects are 5.9° and 4.4°, respectively. The processing speed is as high as 330 ms per frame. However, for lower resolution and poor illumination images, as tested on the public database EYEDIAP, the performance of the proposed eye-model-based method is inferior to that of the-state-of-the-art appearance-based method.

In this paper we propose a novel method of estimating indoor scenes from a single full-view image. On the one hand, the conventional methods cope with limited field of view images, such as perspective images and hemispherical omnidirectional images, which result in visually open boundary condition, called open geometry. On the other hand, a full-view image results in a visually close boundary condition, called close geometry. In this paper we employ the characteristics of close geometry to explore indoor scene understanding from a single full-view image. The proposed close geometry is tested in comparison with the conventional open geometry. The comparative experiment is also carried out between the proposed method and the state-of-the-art method [13]. The experimental results show that the proposed approach can achieve a better performance, and imply that close geometry plays an important role on the interpretation of indoor scenes.

This paper introduces a new approach to calibrating inward- and outward-facing in-vehicle cameras using a laser pointer. Since inward cameras, which observe the driver's behavior, and outward cameras, which observe the driver's field of view, do not have overlapping views, the conventional calibration methods for stereo cameras cannot be applied directly. A method is presented in which the two cameras can be connected by a laser emitted from a laser pointer. The laser pointer is mounted on a calibration board so that the laser ray's pose within the calibration board's coordinate system can be calculated. Two algorithms are presented based on this idea: the coplanar method and the collinear method. We compare the performance of the two proposed algorithms with the conventional mirror-based method through simulations and experiments. Finally, the laser-based collinear method is applied to the calibration of an in-vehicle camera system. In contrast to other methods, the proposed method is simple, practical, and particularly well suited to the calibration of nonoverlapping in-vehicle cameras in a factory or a garage.

In this paper, we propose a novel method of calibrating non-overlapping RGB-D cameras using one chessboard fixed with a laser pointer. A laser pointer is fixed at one calibration board so that its pose at the coordinate system of the calibration board can be obtained easily. While one of the RGB-D cameras observes the calibration board fixed with the laser pointer, the laser pointer project a spot to the scene which is observed by the other. Thus, two 3D points, respectively located in the field of views of the two RGB-D cameras, are connected by a laser ray. The relative pose of two RGB-D cameras can be estimated through this collinear constraint. The experiment results show the effectiveness of the proposed method.

The use of brain waves as biometrics has been proposed. The brain wave is spontaneously occurring; therefore, it enables to transparently present biometric data, and makes on-demand authentication possible. User management is one of supposable and plausible applications in which the on-demand authentication is required, While using a system, users are authenticated on demand from the system. Assuming driver authentication, we have evaluated the verification performance of brain waves measured in simulated driving environments. In this paper, we divide α-β band to several sub-bands and evaluate verification performance in each sub-band. As a result, it is confirmed that each verification performance is quite different from other ones. This result may lead to some kind of knowledge about feasible frequency bands in the brain wave for person authentication.

Mounting three fisheye cameras, on the sides and rear of a vehicle, can help a driver maneuver his/her vehicle in restricted environments by providing a top view that is generated from these three fisheye cameras. To generate the top view, the pose of each fisheye camera must first be calibrated. In this paper, we propose an easy method of calibrating such a fisheye camera system by observing the scene of a parking space. First, each camera pose relative to the ground is estimated from the typical line pattern of a parking space. Then, the relative pose among the three cameras is refined using the overlapping region of the ground between the neighboring cameras. Finally, if necessary, any small deviation of the pose of the camera system can manually be adjusted by an interactive interface. Since the calibration of the fisheye camera system can be performed without preparing a specific calibration pattern for this particular purpose beforehand, the proposed method can reduce the workload on the user. Experimental results reveal the effectiveness of the proposed method.

This paper proposes a full-view spherical image format which is based on the geodesic division of a sphere. In comparison with the conventional 3D array representation which consists of five parallelograms, the proposed spherical image format is a simple 2D array representation. The algorithms of finding the neighboring pixels given a pixel of a spherical image and mapping between spherical coordinate and spherical image pixel are given also.

This paper proposes a method of determining two pairs of vanishing points from an H-pattern, which consists of two parallel lines and another line perpendicular to the two, in a spherical image. In contrast to the conventional methods which determine a vanishing point (a pair of vanishing points at a spherical image) from a group of parallel lines, the proposed method first determines a pair of vanishing points by the two parallel lines and then compute the other pair of vanishing points from the remaining line in terms of the inherent relation of the H-pattern. The experimental result of estimating the pose of a camera mounted at the rear of a vehicle is given to show the effectiveness of the proposed method.

In this paper we propose a method of detecting lane markings by a side fisheye camera which is mounted at the side of a vehicle and observes the side scene in the FFCVS (Four Fisheye Camera Vision System). Since the side fisheye camera has a wider FOV (Field Of View) than a hemispherical one the whole lane marking at the side may be observed by the fisheye camera and the two vanishing points of straight lane markings may appear in the fisheye image. When a vehicle moves along straight lanes the translation vector of the camera is the same as the direction vector of the straight lanes. In this paper, we first compute the epipoles of a pair of fisheye images by spherical stereo method. Then the epipoles are used as the initial position of the vanishing points of lanes. Finally, the straight lane markings are detected by the estimated position of the vanishing points and the gradient values of edge points of lane markings.

A fish-eye camera has a wide field of view (FOV), and the realization of a binocular fish-eye stereo for sensing the surrounding 3-D information of the environment around a vehicle is useful for safe driving. However, since a fish-eye camera may have a wider-than-hemispherical FOV, the conventional stereo approach of obtaining a perspective image based on the pinhole camera model cannot directly be applied. However, using a spherical camera model and defining the disparity of a spherical stereo, the conventional binocular stereo problem is reformulated as abinocularsphericalstereoproblem. A binocular spherical stereo is a generalized paradigm that can cope with cameras having any FOV, including conventional cameras and fish-eye cameras. Moreover, by transforming the rectified spherical images to latitude-longitude representation, the feature point matching of the spherical stereo images can be sped up by using the processing used for perspective stereo images. The effectiveness of this approach is demonstrated by realizing a binocular spherical stereo using a pair of fish-eye cameras. Finally, the application of the proposed approach to vehicles in the future is considered.

In contrast with the conventional trinocular planar image stereo in which the third camera is usually used to reduce the ambiguity on matching, every spherical camera of trinocular spherical stereo plays a role which cannot be replaced by the other two since binocular spherical stereo cannot recover the distance information of scene points whose projection is on the baseline in spherical images. Using three spherical cameras which are not aligned with a straight line the distance cue of points near the baseline of one pair of spherical images can be recovered from the other; thus, we can acquire the whole surrounding structure of environment by trinocular spherical stereo approach. In this paper we give the definition, algorithm and experimental results of trinocular spherical stereo

This paper describes a prototype of a full-view spherical image sensor, gives a method for sensor calibration, and discusses display modalities of the captured full-view image for monitoring around a vehicle. To monitor the whole surrounding situation of a dynamic environment by a single camera, a spherical field of view (FOV) is divided into two hemispherical views. Each hemispherical FOV is imaged by a fisheye lens. Both of the hemispherical views are fused by a mirror to acquire them on a single image plane. To calibrate the full-view spherical image sensor, a three-dimensional (3-D) calibration pattern is used to compute the internal parameters of each fisheye lens and their relative orientation based upon a spherical camera model. Finally, several display modalities are discussed to show drivers the relevant spherical image information on planar displays for monitoring around a vehicle

The sense of balance is the basic skill of biological system. It is combined and unified with other motion skills of animals. By sensing the direction of the gravity from accelerometers, we can acquire a universal vertical reference which makes it much easier to estimation head orientation from visual sensors. This paper proposes a new method of estimating head orientation of a humanoid-like mobile robot by mounting a fisheye camera on its head. The camera rotation is estimated based upon a sky-ground representation, which is a spherical image combined with a vertical reference determined by sensing the direction of gravity. The spherical image can then be divided into two parts, sky part and ground part, along the horizon. Since the direction of the gravity is acquired from accelerometers, what we require to estimate about camera rotation is only the yaw component. Since the yaw rotation does not give rise to any content change for the sky part (the upper part of a spherical image), but a shift along the azimuth angle in a polar spherical coordinate system, the yaw component can be estimated by a phase correlation method which is referred to as the Fourier shift property. By combining the gravity with visual cues, we can estimate camera rotation without accumulated errors for a local scene. In our method no complicated computation is required and robust results are acquired for a dynamic and cluttered outdoor environment even if there is some illumination change.

By representing projection of a point at space on a spherical image, called spherical projection, the conventional stereo algorithm can be reformulated. This means that we can acquire the whole surrounding 3D structure of environments for the construction of immersive VR (virtual reality) environment by full view spherical images, called spherical stereo, instead of the conventional approach that uses a normal camera with a limited view to take multiple plane images, reconstructs parts of the environment structure from multiple image pairs, respectively, and finally fuses them as the whole surrounding 3D structure of environments. In this paper we give the concrete approach to realize this spherical stereo method and show the effectiveness of this method.

In the walk-through via the Internet, we move a mouse to imitate a walk by our foot (Chen, 1995). Here, we consider the case which maps a real motion to an immersive virtual reality environment. For example, we wander in a computer room as if we do in a museum for watching some arts. The user's head motion at a relative large scale environment is needed to be known for realizing this immersive virtual reality. This research proposes a method of estimating the head motion based upon vision approaches for mapping a head motion to an immersive virtual reality environment. We put fiducial markers around the room to be changed virtually and use an omnidirectional image sensor to observe these markers. By mounting the visual sensor on a helmet, the head motion of a user who wears the helmet can be estimated by processing images captured from the omnidirectional image sensor. Since it has a 360 degree view, it can cope with a large head rotation motion compared with a normal camera. Since a head motion at every time is directly estimated from the observed markers, there is no accumulated errors in our method compared with an inertial sensor.

Being increasingly popular, the Internet greatly changes our live. We can conveniently receive and send information via the Internet. With the information explosion in Web, it is becoming crucial to develop means to automatically extract important sentences from the Web articles. In this paper, we propose a method which uses both statistical and structural information in sentence extraction. In addition, following the analysis of human's extractions, several heuristic rules are added to filter out non-important sentences and to prevent similar sentences from being extracted. Our experimental results proved the effectiveness of these means. In particular, once the heuristic rules being added, a significant improvement has been observed.

This paper proposes a method of making a local map of indoor environments by swiveling a camera and a sonar. By swiveling a camera, an omni-directional panoramic image of high azimuth angle solution can be obtained. Meanwhile, regions of constant depth (RCD) corresponding to boundary edges and walls can be obtained by swiveling a sonar. Since many vertical lines exist in indoor environments, a local map of indoor environments can be acquired based upon position of a vertical line in space. From a 2D image, we cannot know its depth in space. At the same time, we cannot also distinguish the RCDs of boundary vertical lines from those of walls and others. However, since a boundary vertical line in space not only appears in images but also has a corresponding RCD, a local map can be acquired by fusing the metrical information from a sonar and the texture information from a camera. This paper proposes the method of fusing and gives the experimental results in a real indoor environment.

This paper proposes a method of robot navigation in outdoor environments by using GPS (Global Positioning System) information and panoramic views. Since an outdoor environment is very large scaled, it is very difficult to determine the global position based on a visual sensor. Although a global position can be obtained from a GPS navigator, it is difficult to locate a robot accurately since it has a distance error about 30/spl sim/100 meter. Our system is equipped with a GPS navigator and a camera. A coarse-to-fine method is used to generate an outdoor environment map and locate a mobile robot. First, a robot finds its approximate position based on the GPS information. Then, it identifies its location from the image information. Experimental results in outdoor environments show the effectiveness of this method.

This paper proposes an idea of representing qualitatively a route in an outdoor environment in terms of landmarks. A robot continuously scans the side-view and yields an intermediate representation called panoramic representation. The objects in the scene are segmented based upon range and color information in the panoramic representation, and they are then arranged into a structure map which provides qualitative information on the environment structure. The objects which have distinctive features along the route are selected as the landmarks. Since their relative spatial management is invariant with the camera speed, the memorized landmark can be matched with the objects in the new structure map by a dynamic programming-based method. The correctness of matching is further verified by using the disparity and color cues. Results of experiments in, a real outdoor world show the effectiveness and and robustness of this method.

A navigation system based upon panoramic representation is explored in this paper. The panoramic representation is an intermediate 2(1/2)D representation which is transformed from an image sequence taken continuously by a camera on a mobile robot. While iconic information is contained in a panoramic view, estimating the disparity of feature points by image processing costs a lot of computation. Here the authors' system obtains disparity values quickly by an active method. The system is constructed by a mobile robot mounted with cameras and slit light projectors. A camera and a projector are directed toward the ground in front of the robot to detect obstacles while other cameras and projectors are directed to two sides for acquiring the panoramic representation of two side scenes. The robot can move freely in an indoor environment and the preliminary experimental results on acquiring environment maps along a closed path are given.

A human being often looks around to locate him/herself from the arrangement of surrounding objects. Here, the authors propose an idea to determine camera motion by line correspondence in omni-directional images. An omni-directional image has a 360 degree view and contains much more information than those taken by a conventional imaging method. Since the camera motion is estimated from line displacements in images and the line displacements change due to line arrangement in space and camera motion, an omni-directional view makes it possible to select a better line combination for estimating more accurate motion. Experimental results show that camera motion can be estimated more accurately using lines surrounding the camera in omni-directional images than using lines in a narrow view. First, the authors use lines surrounding the camera to determine the camera rotational component and find the true solution from multiple ones by voting from multiple line combinations. Then, the authors select an optimal line combination to estimate the translation component by examining their linear independence. Simulation and real-world experimental results are given.>

A mobile robot memorizes scenes along a route by landmarks selected in a trial navigation and locates itself in the route based upon the memorized landmarks when the robot pursues the same route by itself. Since the stream of images observed continuously along the route is highly redundant, it is transformed into an intermediate 2(1/2)D representation, called panoramic representation with a much less amount of data. A structure map representing the object arrangement in space is then yielded in terms of 3D objects along the route found in the panoramic representation by fusing their range estimates and color attributes. By examining the spatial relationships, shapes and color attributes of the objects in the structure map, distinctive objects are selected as the landmarks. The route described by selected landmarks are recognized successfully, although the speed of camera motion and the illumination condition are changed considerably.>

The authors have developed a method for applying the active vision paradigm for a vision-guided mobile robot to acquire 3D structure of a man-made environment. The robot plans a circular path by itself, moves along the planned circular path by the visual feedback, computes the camera rotation from the vanishing points of horizontal lines in the scene, and estimates the 3D structure of the environment. Experimental results showed that this method was robust to the disturbances in real environments.>

Autonomous finding of landmarks for guiding long-distance navigation by a mobile is explored. In a trial, the robot continuously views and memorizes scenes along the route. When the same route is subsequently pursued again, the robot locates and orients itself based on the memorized scene. Since the stream of images is highly redundant, it is transformed into an intermediate 2/sup 1///sub 2/D representation, called the panoramic representation (PR), with a much smaller amount of data. Although the PR can be used for guidance of the autonomous navigation, it still contains a huge amount of data for a very long route. A human memorizes only very impressive objects along the route and uses them as landmarks. The robot also finds distinctive objects along the route and memorizes their features and spatial relationships. 3D objects are segmented in the PR by fusing range estimates and color attributes, and then a structure map representing their arrangement in space is obtained. In order to find distinctive objects for use as landmarks, the spatial relationships, shapes, and color attributes of the objects are examined.>

A novel method that uses vanishing points of horizontal lines, but not parallel to each other, for estimating the change of camera azimuth is proposed. Although indoor scenes contain many such lines, there also exist non-horizontal lines which should be discarded from the group of lines for estimating the azimuth change. Distinguishing non-horizontal lines from horizontal ones is difficult for a static image, but possible for the dynamic imagery because non-horizontal lines show different motion patterns as the camera moves.>