Autonomous driving cars currently under development use many sensors, among which CCD/CMOS cameras are the most important. To improve real-time image processing technology and FPGA utilization for autonomous driving, FPGA design contests have been held on the theme of creating an automated small robotic car using FPGA. This paper describes our FPGA robot car under development for participation in the FPGA design contest to be held at FPT2022. Our design incorporates a Xilinx SoC FPGA and uses both hardware and software for control. We use Xilinx DPU IP for object detection with Yolo v3 tiny. Furthermore, we are working on improving the stability and reliability of the FPGA robot car by modifying our previous FPGA robot car.

We are developing an FPGA-based robot car for the FPT'21 FPGA design competition. The FPGA we are using is Xilinx UltraScale+ MPSoC, which is an SoC type FPGA and mainly consists of a processor part and a programmable logic part. Among the functions of the autonomous driving system, lane-keeping, localization, driving planning, and obstacle avoidance are implemented as software in the processor part. For object detection, Yolo, a machine learning algorithm, is implemented as hardware in the programmable logic part using the DPU IP provided by Xilinx. The PWM circuits for controlling the DC motor and servo motor are also implemented as the original hardware.

Autonomous technology has been actively developed. The theme of FPT 2018 FPGA design competition is autonomous vehicle driving using FPGA. In this paper, we describe our autonomous driving robot car for the design competition. Our robot car is developed using Xilinx FPGA Zynq 7020, which has PL (Programmable Logic) part and PS (Processor System) part. Image recognition to detect dolls used as humans is implemented on PL as hardware logic. Image processing of lane keeping, navigation and motor control are implemented on PS as software program. Our system is using PYNQ environment which can control hardware by Python language.

This paper proposes a novel algorithm that calibrates multiple cameras scattered across a broad area. The key idea of the proposed method is “using the position of an omnidirectional camera as a reference point.” The common approach to calibrating multiple cameras assumes that the cameras capture at least some common points. This means calibration becomes quite difficult if there are no shared points in each camera's field of view (FOV). The proposed method uses the position of an omnidirectional camera to determine point correspondence. The position of an omnidirectional camera relative to the calibrated camera is estimated by the theory of epipolar geometry, even if the omnidirectional camera is placed outside the camera's FOV. This property makes our method applicable to multiple cameras scattered across a broad area. Qualitative and quantitative evaluations using synthesized and real data, e.g., a sports field, demonstrate the advantages of the proposed method.

This paper proposes a novel framework for image and video completion that removes and restores unwanted regions inside them. Most existing works fail to carry out the completion processing when similar regions do not exist in undamaged regions. To overcome this, our approach creates similar regions by projecting a low dimensional space from the original space. The approach comprises three stages. First, input images/videos are converted to a lower dimensional feature space. Second, a damaged region is restored in the converted feature space. Finally, inverse conversion is performed from the lower dimensional space to the original space. This generates two advantages: (1) it enhances the possibility of applying patches dissimilar to those in the original color space and (2) it enables the use of many existing restoration methods, each having various advantages, because the feature space for retrieving the similar patches is the only extension. The framework’s effectiveness was verified in experiments using various methods, the feature space for restoration in the second stage, and inverse conversion methods.

In this paper, we describe our Trax player implemented on FPGA for FPT2016 design competition, which is designed by using a high-level synthesis tool, Xilinx Vivado HLS. The previous version of our design had recursive functions in the AI part, which could not be synthesized by Vivado HLS. They were processed by software using a hard IP processor. We rewrite them to non-recursive functions to synthesize using the HLS tool. The most part of our code becomes synthesizable in the current version.

This paper proposes a method that blindly predicts preference order between inpainted images, aiming at selecting the best one from a plurality of results. Image inpainting, which removes unwanted regions and restores them, has attracted recent attention. However, it is known that the inpainting result varies largely with the method used for inpainting and the parameters set. Thus, in a typical use case, users need to manually select the inpainting method and the parameter that yields the best one. This manual selection takes a great deal of time and thus there is a great need for a way to automatically estimate the best result. Although some methods, such as estimating perceptual preference score from image features, have been proposed in recent years, none of them are considered very promising approaches. Our method focuses on the following two points: (1) what we essentially need is a preference order relation rather than an absolute score, and (2) we consider that image features for order estimation can be effectively designed by using actually measured human visual attention. Comparison with other image quality assessment methods shows that our method estimates the preference order with high accuracy.

We present a new unsupervised technique to segment 3D Lidar points in outdoor environments. The main idea of this work is to identify artificial objects according to the existence of extruded shapes. Many artificial objects are composed of extruded shapes such as cylinders, planes, cubes, and lines. Therefore, we detect these arbitrarily extruded shapes on the basis of an indicator for repetitive crosssection shapes, and connect the components according to the strength between the overlapping areas in the extruded surfaces. Conventional segmentation methods that use local geometry information may sometimes produce erroneous results in scenes where there are many objects that are very near to and partially in contact with each other. In contrast, our method is more robust against these complex scenes using large scale surface overlapping strength. Experiments show it provides good results in urban environments and expressway scenes.

In this paper, we describe our implementation of Trax player on Xilinx Zynq programmable SoC for ICFPT2015 design competition. Our design uses Alpha-Beta pruning algorithm to find the best next move from the game tree. As thinking time for one turn is limited to one second by the competition rule, we also use iterative deepening algorithm and timer hardware. Evaluation function which is used for the leaf nodes of the game tree detects simple attacks, simple corners, and victory line candidates and calculates the value of the existing tiles. Hotspot software code detected by profiling is translated into HDL code using a high level synthesis tool.

A novel framework for image/video content completion comprising three stages is proposed. First, input images/videos are converted to a lower dimensional feature space, which is done to achieve effective restoration even in cases where a damaged region includes complex structures and changes in color. Second, a damaged region is restored in the converted feature space. Finally, an inverse conversion from the lower dimensional feature space to the original feature space is performed to generate the completed image in the original feature space. This three-step solution generates two advantages. First, it enhances the possibility of applying patches dissimilar to those in the original color space. Second, it enables the use of many existing restoration methods, each having various advantages, because the feature space for retrieving the similar patches is the only extension. Experiments verify the effectiveness of the proposed framework.

A method for blindly predicting inpainted image quality is proposed for enhancing the robustness of diminished reality (DR), which uses inpainting to remove unwanted objects by replacing them with background textures in real time. The method maps from inpainted image features to subjective image quality scores without the need for reference images. It enables more complex background textures to be applied to DR.

A system providing visual feedback of a trainee's motions for effectively enhancing motor learning is presented. It provides feedback in synchronization with a reference motion from multiple view angles automatically with only a few seconds delay. Because the feedback is provided automatically, a trainee can obtain it without performing any operations while the memory of the motion is still clear. By employing features with low computational cost, the system achieves synchronized video feedback with four cameras connected to a consumer tablet PC.

We propose a system that can provide the physical presence of remote events through a head mount display (HMD) and a headphone. It can stream omnidirectional video within a limited network bandwidth at a high bitrate without sending regions that users are not viewing. It can also reproduce binaural sounds by convoluting head related transfer functions and angular region-wise separated signals. Technical demos of the system using an Oculus Rift HMD with a headphone will be performed to enable users to experience the visual and acoustic immersion it provides.

NTT Media Intelligence Laboratories and DWANGO Co., Ltd. have jointly developed a virtual reality system that enables users to have an immersive experience visiting a remote site. This system makes it possible for users to watch video content wherever they want to watch it by using interactive streaming technology that selectively streams the user's watching section at a high bitrate in a limited network bandwidth. Applying this technology to omnidirectional video allows users to experience feelings of presence through the use of an intuitive head mount display. The system has also been released on a commercial platform and successfully streamed a real-time event. A demonstration is planned in which the details of the system and the streaming service results obtained with it will be presented.

Blokus Duo is an abstract strategy game for two players. In this paper, we describe our FPGA implementation of Blokus Duo player for ICFPT2014 design contest, which is the revised version of the previous design for ICPFT2013 design contest. Our design consists of hardware logic part and software part using soft IP processor. The hardware logic part calculates evaluation value of the board status which is a heavy task for the software part. Our implementation uses recursive Alpha-Beta pruning and iteration deepening algorithm by the software part, which are complex to implement as the hardware logic circuit. The current version of our implementation on Xilinx Artix7 can run at 142MHz. The hardware logic part evaluates about 90,000 nodes in one second at the beginning of the game.

We propose a video streaming system that lets users view their favorite sections through an HMD (head mount display) from a video recorded by a spherical (360-degree) camera. Although spherical video streaming tends to consume a lot of bitrate for its huge image area, our system consumes a reasonable amount of bitrate by assigning higher bitrate only for the user's viewing area, not for the area outside of it. Technical Demos of the system with an Oculus Rift HMD will be performed to demonstrate it enables them to view images at a bitrate of about 2.5 Mbps.

View synthesis prediction is critical for efficient compression of 3D video, which consists of multiview video and depth maps. However, its performance is limited in practical situations since it is necessary to use erroneous depth information and to perform block-based compensation instead of pixel-based warping. This paper proposes a novel view synthesis prediction scheme where motion field is synthesized by utilizing coarse disparity filed derived from erroneous depth information. As part of the proposed depth-based motion field synthesis, occlusion-aware backward mapping and 3D motion field warping are performed. In order to improve prediction performance with block-based compensation, an adaptive prediction sample generation that utilizes both temporal and inter-view correlations is also proposed. Experiments show that the proposed scheme achieves average bitrate reductions of 1.38% and 1.19% for coded views and synthesized views. The maximum gain is 11.57% for dependent view.

Inter-view residual prediction and advanced residual prediction (ARP) are efficient tools for coding dependent views of 3D video. It can predict the residue of motion compensated prediction (MCP) using the additional derived disparity vector. However, its coding performance depends on the accuracy of disparity vector derivation. In this paper, we propose the improved disparity vector derivation scheme for ARP. In the proposed scheme, the disparity vector is derived from the corresponding block in the reference picture. And that corresponding block is pointed by the same as MCP. Moreover, the disparity vector can be derived not only from the current reference block, but also from the blocks on the all other reference pictures included in the current reference picture lists. In addition, the disparity vector can be derived from both of the blocks predicted by disparity compensated prediction (DCP) and the blocks predicted by MCP and ARP because the derived disparity vector for ARP is stored in additional disparity vector field. Experimental results show that 0.2% bitrate reduction of synthesized views, and up to 0.5% for each dependent views in the reference software of 3D-HEVC.

Blokus Duo is a board game for two players, which uses 21 different shapes of tiles and 14×14 board. We implement Blokus Duo player design on FPGA by using hardware-software co-design method. In this paper, we describe the structure and the algorithm which are used in our Blokus Duo player design. It uses alpha-beta pruning and iterative deepening depth-first search by software and evaluation function by hardware in the current version. The hardware-software co-design implementation is about seven times faster than the original software code.

View synthesis prediction has been studied as an efficient inter-view prediction scheme. Existing view synthesis prediction schemes fall into two types according to the pixel warping direction. While backward warping based view synthesis prediction enables block-based processing, forward warping based view synthesis prediction can handle occlusions properly. This paper proposes a two-step warping based view synthesis prediction; a virtual depth map is first generated by forward warping, and then prediction signals are generated by block-based backward warping using the virtual depth map. The technique of backward-warping-aware depth inpainting is also proposed. Experiments show that the proposed VSP scheme can achieve the decoder runtime reductions of about 37% on average with slight bitrate reductions relative to the conventional forward warping based VSP. Compared to the conventional backward warping based VSP, the proposed method reduces the bitrate for the synthesized views by up to 2.9% and about 2.2% on average.

Video with depth is attracting attention as a promising approach to future 3D video. Unlike conventional video coding technologies, video with depth should take the correlations between video and depth into account. This paper proposes a novel bi-prediction scheme for video coding that exploits the structure similarity between video and depth. In the proposed scheme, a prediction image is generated by the pixel-wise weighted average of motion compensated reference images, and weights are decided by considering pixel similarities in the depth domain. The proposed pixel-wise weighted bi-prediction offers implicit depth-based block partitioning with consideration of the pixel correlations between predicted and reference blocks. Compared to using explicit, arbitrary shaped block partitioning, the proposed method offers better prediction for non-rigid objects, and requires no increase in motion information. Experiments show that the proposed method can achieve up to 17% bitrate reduction for bi-predictive frames and about 8% on average for 6 sequences relative to the existing biprediction scheme.

Pixel domain methods for content-aware image retargeting have a performance limitation in that they distort important objects when an image is shrunken too much. To address this problem, we have designed and developed a novel 3D space domain image retargeting which rearranges objects with occlusions and deforms objects. With this method image formation is modeled as a quasi-3D space. The method also reduces image width by shrinking a part of the image space. Instead of directly optimizing the position and shape of objects, it closes objects by shrinking the gap space between them and deforms objects by shrinking spaces that include parts of them. It is capable of being extended to higher dimension problems such as multi-view image retargeting because it needs to optimize only one quasi-3D space whether the input is ordinary images or more complex ones such as video and stereo images. We have also designed a simple algorithm that makes use of dynamic programming. An experiment we conducted demonstrated the algorithm's effectiveness.

We propose a new interactive panorama video delivery system that enables users not only to select favorite parts of an event site, but also to watch repeatedly with stable and the user-selected camera trajectory. In this paper, we present a system that makes a favorable and stable camera trajectory as a scenario from user's manipulations of a tablet device in a format that can be easily shared with others. We also evaluate the validity of the camera trajectory results obtained our stabilization technique.

We propose a scenario-driven interactive panorama video delivery system that allows users to repeatedly watch the enjoyable parts of an event and share them with others. It provides functions for interactively watching panorama video parts, as well as for scenario making (with user-selected camera trajectory), and delivery that allows users to share their panorama watching experiences. In this technical demos, we demonstrate the system that can make a favorable and stable camera trajectories as a scenario from intuitive user manipulations of a tablet device in a format that can be easily shared with others.

In recent years, exploring the rich web image resources has been offering promising solutions to the problem of how to perform low-manual-cost concept learning. However, concept classifiers trained using web images perform poorly when they are directly applied to video concept detection. We propose a novel scheme to address video concept learning using web images, one that includes the selection of web training data and the transfer of subspace learning within a unified framework. Starting with a small set of video keyframes related to a video concept, we select web training data of good quality from the web by referring to the content of video keyframes. Then, by exploiting both the selected dataset and video keyframes, we train a robust concept classifier by means of a transfer subspace learning method. Experiment results demonstrate the robustness and effectiveness of our method.

Open Course Ware (OCW) was first adopted by universities to release text-based lecture notes, and lecture videos are a recent addition. We have designed Web site which can be shared among OCW member universities without their own facility for sharing lecture video and lecture-scene-related-knowledge. We initially propose the ‘Social Curator Collaboration Model’ which creates an environment that general public user other than course lecturer can participate as learning assistant to reply to other learner’s questions or opinions. We also propose Web-API function mash-up architectural design to realize OCW common platform. We have evaluated the functionality of our proposed model by prototyping the infrastructure and challenged it with real OCW lecture video.

This paper presents a content-based movie trailer generation method, named Vid2Trailer (V2T). Since trailers are intended to advertise movies, they must show specific symbols such as the title logo and the main theme music. Moreover, it is expected to attract viewers by its visual and audio content. V2T satisfies these two requirements when creating a trailer from the original movie content. First, the title logo and the main theme music are extracted. Second, impressive speech and video segments are extracted by using an affective content analysis technique. Third, all of the extracted components are concatenated into the form of a trailer; to realize this, we propose a method that estimates the affective impact of shot sequences, and introduce an algorithm that arranges a set of shots so as to maximize the affective impact of the sequence. Experiments show that our V2T is more appropriate to trailer generation than conventional techniques.

This paper presents a novel method for movie affective scene classification that outputs the emotion (in the form of labels) that the scene is likely to arouse in viewers. Since the affective preferences of users play an important role in movie selection, affective scene classification has the potential to develop more attractive user-centric movie search and browsing applications. Two main issues in designing movie affective scene classification are considered. One is “how to extract features that are strongly related to the viewer's emotions”, and the other is “how to map the extracted features to the emotion categories”. For the former, we propose a method to extract emotion-category-specific audio-visual features named affective audio-visual words (AAVWs). For the latter issue, we propose a classification model named latent topic driving model (LTDM). Assuming that viewers' emotions are dynamically changed by the movie scene sequences, LTDM models emotions as Markovian dynamic systems driven by the sequential stimuli of the movie content. Experiments on 206 movie scenes extracted from 24 movie titles and the corresponding labels of eight emotion categories given by 16 subjects show that our method outperforms conventional approaches in terms of the subject agreement rate.

This paper proposes a latent topic driving model (LTDM) as a novel approach to movie affective scene classification. LTDM is a discriminative model of emotions driven by movie affective contents. Unlike existing methods, our approach is based on movie topic extraction via the latent Dirichlet allocation (LDA) and emotion dynamics modeling with reference to Plutchik's emotion theory. The classification procedure starts by segmenting movie scenes into movie shots, each of which is represented by a histogram of quantized affect-related audio-visual features. LDA is applied to detect topics of each movie shot. Emotions for the current movie shot are estimated based on both the topics of the shot and emotion transition weights determined by Plutchik's emotion theory. We conduct experiments using 206 movie scenes extracted from 24 movie titles (total 6 hours 20 min. 12 sec.) and the labels of eight emotion categories given by 16 subjects are collected. The results show that LTDM outperforms conventional modeling approaches in terms of the subject agreement rate.

A novel region-based approach is proposed to model semantic concepts using web images. Web images are mined to obtain multiple visual patterns automatically that then are used to model a semantic concept. First, the salient region groups corresponding to the representative visual patterns of a concept are mined and selected as positive samples. Next, a representative visual pattern is built in each salient region group by using a BDA classifier. Finally all the visual patterns are aggregated to describe the concept by using a BDA ensemble approach. Because the proposed method models a semantic concept utilizing multiple visual patterns, it enhances the visual variability of a visual model when learning from diverse web images and improves the robustness of the visual model in handling segmentation-related uncertainties. Experiment results demonstrate our method performs well on generic images including not only "object" concepts, but also complex "scene" concepts.