In this paper, we present a framework that jointly retrieves and spatiotemporally highlights actions in videos by enhancing current deep cross-modal retrieval methods. Our work takes on the novel task of action highlighting, which visualizes where and when actions occur in an untrimmed video setting. Action highlighting is a fine-grained task, compared to conventional action recognition tasks which focus on classification or window-based localization. Leveraging weak supervision from annotated captions, our framework acquires spatiotemporal relevance maps and generates local embeddings which relate to the nouns and verbs in captions. Through experiments, we show that our model generates various maps conditioned on different actions, in which conventional visual reasoning methods only go as far as to show a single deterministic saliency map. Also, our model improves retrieval recall over our baseline without alignment by 2-3% on the MSR-VTT dataset.

The latest generative adversarial networks (GANs) can generate realistic high resolution images. However, to the best of our knowledge, there are no GANs for generating 360-degree images. Therefore, this paper proposes the novel problem setting that by using a known area from the 360-degree image as an input, the remainder of the image can be completed with the GANs. To do so, we propose the approach of two-stage generation using network architecture with series-parallel dilated convolution layers. Moreover, we present how to rearrange images for data augmentation, simplify the problem, and make inputs for training the 2ndstage generator. Our experiments show that these methods generate the distortion seen in 360-degree images in the outlines of buildings and roads, and their boundaries are clearer than those of baseline methods. Furthermore, we discuss and clarify the difficulty of our proposed problem. Our work is the first step towards GANs predicting an unseen area within a 360-degree space.

We have developed a Virtual Museum with realtime 3DCG capable of exhibiting arbitrary planar artifacts such as paintings which have been specified by a user. The pictures are collected from the Internet sites such as Wikimedia Commons by giving bookmarks by the user. The artifact images are displayed in its life size automatically aligned on the wall of the museum with picture frames and generated captions. This process is done based on metadata extracted using a technique called Web scraping to extract necessary information from the target Web sites. The museum space is realistically modeled with high resolution and sophisticated illumination where the user can walk through in the space. The system enables the users to create their own personalized museums with their favorite pictures exhibited in the realistic way.

Driver education is quite important even if ITS(Intelligent Transport System)s spread. Therefore, we have developed an Assistant System for Safe Driving by Informative Supervision and Training (ASSIST), a system created to prevent accidents based on our safe-driving theory. In this paper, we report development of the ASSIST based on IoT architecture.

We have been studying and developing the system which enables to generate Computer Graphics Animation (CGA) automatically by processing HTML data of Web site. In this paper, we propose an open framework to facilitate this. The framework is functioning all at a server side, obtaining the HTML, converting it to a script describing the CGA story and updating the script. And at a client side, a user accesses the script on the server to visualize it by using real-time CG character with synthesized voice, camera work, superimposing, sound file playback etc. We have constructed the framework on the server and deployed the substantial engines to convert Web sites to CGAs. This paper describes the detail of the framework and also shows some example projects providing automatically generated News show, Talk show and personal Blog visualization.

Virtual Museum service has been carried out in many places owing to the advanced video and network technology in recent years. In the virtual museum, people primarily experience the prepared content actively with a mouse, a touch panel and specially designed tangible devices. On the other hand, in a real museum space, people appreciate the artifacts passively walking around the space freely without stress. It can be said that the virtual museum is designed to urge people to deal with it rather actively when compared to the real museum. We have been studying and developing a new type of virtual museum enabling people to participate the space with both active and passive way, by implementing various new functions. In this time, we developed the new virtual museum equipped with a video content generator using the virtual exhibition space modeled with 3D computer graphics (CG). This video content is created in real-time by using the 3DCG-modeled museum space as it is, adding appropriate visual and audio effects such as camerawork, superimposing text, synthesized voice narration, back ground music etc. Since this system is working in the 3DCG space, a user can easily go back and forth between the two modes of watching the video content passively and doing walkthrough in the space actively by a wheel mouse. In this paper, we first introduce primary virtual museums in the world. Then, we describe our method: 1) specially designed walkthrough algorithm, 2) the video content generator using the 3DCG museum space and 3) seamless integration of the 1) and the 2). We then describe our functioning prototype followed by the conclusion and the future plans.

This poster gives an introductory summery of one of the approaches in a multi-nodal framework. The approach presented here is the Study of physiological responses of in-game player experience for engagement. Other components of a multi-nodal approach that are not included in this poster are, literature studies, the study of player accounts of the play experience and the study of in-game metric data in computer and video games. The approach described here is the study of real-time data acquisition from physiological responses generated during play sessions. The data acquisition is generated through, Eye-tracking, using a Tobii X2-30 Eye-tracker, temperature and galvanism in fingers, EEG, with the Emotive EEG, Neuroheadset, EPOC Neuroheadset and 3D Brain Activity Map and by measuring heart rate variability. The real-time physiological data acquisition from the play session is conducted and displayed together with the game on a “4K," extreme high-resolution 3840 × 2160 pixel display system. This allows users to record the play-session and the acquisition data all on one display. This provides optimized access to synchronized data when recorded sessions are later under review for analysis. The display is 4 times larger than conventional HD displays and configuration by using a generic PC "Quad-CORE" equipped computer with an AMD "V7900" graphics card running on the Microsoft Windows 7 OS. The four Display-Port outputs from the card are converted to four DVIs, which are connected to the "DM-3410-A", 4K monitor manufactured by Astrodesign Inc. The physiological data acquisition is operated by the PC and displayed in the seven fields on the screen as shown in fig 1. This method of studying the physiological responses of in-game player experience provides a large spectrum of data. This data together with the other approaches in a multi-nodal framework can help researchers and designers in gaining further knowledge about correlations between player engagement and computer and video games.

We have been developing T2V (Text-To-Vision) technology which enables to produce CG animation from text. This technology is built on a TVML (TV program Making Language) core engine which produces TV-program-like animation from text input using real-time CG, voice synthesizing technique and so on. Recently, we have ported the TVML engine on a game engine UNITY. The TVML SDK on UNITY is provided to game developers enabling to make various interactive applications using T2V technology. Our aim is to integrate two different media: TV and Game to make new type of media named 'gaming TV' owing to the T2V technology. In this paper, as one of the functioning developments of gaming TV, we will introduce 'Interactive TV' application using the SDK. In this application, a user can interrupt the on-going show at any time to have a dialogue with the actors in the show, then return to the point that had been interrupted and resume the show. We have confirmed the potential of gaming TV through the user experience of our functioning development.

Summary form only given. We have been researching and developing 'Virtual Museum' in an extreme high-definition real-time computer graphics system with a resolution of 4K and 8K (Super Hi-Vision). We had first developed a functioning test system, which exhibited Japanese artifacts 'Ukiyoe' in 4K resolution. In this time, we have enhanced the virtual museum with a capability to exhibit 3D objects such as statues and archaeological items. In order to achieve this, the following technology is needed, 1) acquisition of 3D object with very high-resolution digitization for both geometry and texture data, 2) new representation technique to enable a user to walkthrough in the museum space and appreciate the artifacts with less stress. For 1), we have established the digitization pipeline converting a real object to the virtual object in the virtual space. And for 2), we have made a new walkthrough algorithm with a human gesture sensing device which enables a user to walkthrough by hand gesture. We have developed and demonstrated a prototype of the new virtual museum and have confirmed its functionality and usability.

We are interested in the problem of automatically tracking and identifying players in sports video. While there are many automatic multi-target tracking methods, in sports video, it is difficult to track multiple players due to frequent occlusions, quick motion of players and camera, and camera position. We propose tracking method that associates tracklets of a same player using results of player number recognition. To deal with frequent occlusions, we detect human region by level set method and then estimates if it is occluded group region or unoccluded individual one. Moreover, we associate tracklets using the results of player number recognition at each frame by keypoints-based matching with templates from multiple viewpoints, so that final tracklets include occluded region.

Currently, there are many occasions to communicate with each other in the collaborative virtual environment. However, it is more difficult to keep high-motivation for participating in communication in such virtual environment than in face-to-face environment. It is considered that the reasons are 1) the difficulty of understanding how their own motivations are and 2) the boring expression due to the fixed view point. To address this issue, we display electroencephalogram information to a virtual space by forward-backward model - zoom in/out of whole virtual space. According to the results of experiment, our system successfully provides an environment in which people can keep high-motivations for participating in communication.

A video blogging system has been developed for easily producing your own video programs that can be made available to the public in much the same way that blogs are created. The user merely types a program script on a webpage, the same as creating a blog, selects a direction style, and pastes in some additional material content to create a CG-based video program that can be openly distributed to the general public. The script, direction style, and material content are automatically combined to create a movie file on the server side. The movie file can then be accessed by referring to an RSS feed and viewed on the screens of various devices.

Article Free Access Share on Image compositing system capable of long-range camera movement Authors: Masaki Hayashi NHK Science and Technical Research Laboratories, 1-10-11, Kinuta, Setagaya-ku, Tokyo, 157, Japan NHK Science and Technical Research Laboratories, 1-10-11, Kinuta, Setagaya-ku, Tokyo, 157, JapanView Profile , Kazuo Fukui NHK Science and Technical Research Laboratories, 1-10-11, Kinuta, Setagaya-ku, Tokyo, 157, Japan NHK Science and Technical Research Laboratories, 1-10-11, Kinuta, Setagaya-ku, Tokyo, 157, JapanView Profile , Yasumasa Itoh NHK Science and Technical Research Laboratories, 1-10-11, Kinuta, Setagaya-ku, Tokyo, 157, Japan NHK Science and Technical Research Laboratories, 1-10-11, Kinuta, Setagaya-ku, Tokyo, 157, JapanView Profile Authors Info & Claims MULTIMEDIA '96: Proceedings of the fourth ACM international conference on MultimediaFebruary 1997 Pages 153–161https://doi.org/10.1145/244130.244181Published:01 February 1997Publication History 1citation368DownloadsMetricsTotal Citations1Total Downloads368Last 12 Months19Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

In this paper a motion planning method for mobile robots that coexist and cooperate with human beings to avoid collision is proposed. Human motion, which has large uncertainty, is predicted by a motion predictor using a stochastic process model to generate probability distribution maps of human existence. An evaluating function is defined considering the danger of collision and efficiency of trajectories. A genetic algorithm determines the robot trajectory by optimizing the function. This procedure is repeated for each sampling time. The robot moves maintaining high safety against various possible human motion. Simulation results revealed that robots can reach goals avoiding danger along appropriate trajectories and that this method is effective especially for the case where optimal trajectories dynamically vary according to change of human motion.