Plastic bags are challenging objects for robot manipulation due to transparency and deformability. This paper proposes a learning approach for a robot to insert its hand into a container- or bag-shaped object based on visual and tactile sensing. The basic idea is to utilize probing action that allows to acquire rich information about the object even with a simple tactile sensor. The structure of the object is estimated by unsupervised learning with contact and reachability information. The result is transferred to visual recognition as self-supervised learning. Based on the unsupervised learning result, the robot can verify whether the hand truly reached the interior of the bag by additional probing actions. The proposed method was evaluated experimentally by a robot hand with a simple tactile sensor.

We propose an image reconstruction method from event data, assuming the target images belong to a prespecified class like QR codes. Instead of solving the reconstruction problem in the image space, we introduce a code space that covers all the noiseless target class images and solves the reconstruction problem on it. This restriction enormously reduces the number of optimizing parameters and makes the reconstruction problem well posed and robust to noise. We demonstrate fast and robust QR-code scanning in difficult, high-speed scenes with industrial high-speed cameras and other reconstruction methods.

Event cameras are bio-inspired vision sensors that mimic retinas to asynchronously report per-pixel intensity changes rather than outputting an actual intensity image at regular intervals. This new paradigm of image sensor offers significant potential advantages; namely, sparse and non-redundant data representation. Unfortunately, however, most of the existing artificial neural network architectures, such as a CNN, require dense synchronous input data, and therefore, cannot make use of the sparseness of the data. We propose EventNet, a neural network designed for real-time processing of asynchronous event streams in a recursive and event-wise manner. EventNet models dependence of the output on tens of thousands of causal events recursively using a novel temporal coding scheme. As a result, at inference time, our network operates in an event-wise manner that is realized with very few sum-of-the-product operations---look-up table and temporal feature aggregation---which enables processing of 1 mega or more events per second on standard CPU. In experiments using real data, we demonstrated the real-time performance and robustness of our framework.

We propose a novel three-dimensional (3D)-convolution method, cv3dconv, for detecting spatiotemporal features from videos. It reduces the number of sum-of-products of 3D convolution by thousands of times by assuming the constant moving velocity of the camera. We observed that a specific class of video sequences, such as those captured by an in-vehicle camera, can be well approximated with piece-wise linear movements of 2D features in the temporal dimension. Our principal finding is that the 3D kernel, represented by the constant-velocity, can be decomposed into a convolution of a 2D kernel representing the shapes and a 3D kernel representing the velocity. We derived the efficient recursive algorithm for this class of 3D convolution which is exceptionally suited for sparse data, and this parameterized decomposed representation imposes a structured regularization along the temporal direction. We experimentally verified the validity of our approximation using a controlled dataset, and we also showed the effectiveness of cv3dconv for the visual odometry estimation task using real event camera data captured in urban road scene.

The mechanism behind collisions between vehicles and pedestrians must be thoroughly studied in order to prevent future traffic accidents. In particular, preventing collisions where pedestrian steps out onto the road from behind an obstruction such as buildings, walls or vehicles is a challenging problem. To tackle this problem, we propose situation dependent topic model (SDTM), a regression model that predicts dangerous vehicle-pedestrian encounter in response to different driving situations, which also provides a framework to analyze and understand the underlying factors that lead to dangerous situations. Complex nature of situations where collisions with pedestrians happen can be expressed well by defining how dangerous situations arise differently for each driving situation pattern retrieved using statistical topic modeling. In experiments, we compare the performance of SDTM with orthodox logistic regression models using vehicle-pedestrian encounters in near-miss incidents. We also show the result of acquired knowledge that can form the basis of many other researches concerning pedestrian safety.

We propose a method of ego-motion estimation for a self-driving vehicle using multiple cameras. By finding corresponding points between the multi-camera images, we aim to enhance the accuracy of the ego-motion estimation. However since the viewing directions are very different from one camera to the other, a conventional algorithm such as SURF cannot detect a sufficient number of correspondences. We propose a novel matching algorithm by warping feature patches detected in different cameras based on urban 3D structure. We assume that detected features exist on the surface of buildings or roads and the patch around the feature is planar. Based on this assumption, we can warp the patches so that the feature descriptors are similar for the corresponding feature points. We apply Bundle Adjustment to the found correspondences to optimizes the odometry. The result shows higher estimation accuracy when compared to other matching method.