Segmenting and determining the 3D bounding boxes of objects of interest in RGB videos is an important task for a variety of applications such as augmented reality, navigation, and robotics. Supervised machine learning techniques are commonly used for this, but they need training datasets: sets of images with associated 3D bounding boxes manually defined by human annotators using a labelling tool. However, precisely placing 3D bounding boxes can be difficult using conventional 3D manipulation tools on a 2D interface. To alleviate that burden, we propose a novel technique with which 3D bounding boxes can be created by simply drawing 2D bounding rectangles on multiple frames of a video sequence showing the object from different angles. The method uses reconstructed dense 3D point clouds from the video and computes tightly fitting 3D bounding boxes of desired objects selected by back-projecting the 2D rectangles. We show concrete application scenarios of our interface, including training dataset creation and editing 3D spaces and videos. An evaluation comparing our technique with a conventional 3D annotation tool shows that our method results in higher accuracy. We also confirm that the bounding boxes created with our interface have a lower variance, likely yielding more consistent labels and datasets.

We propose a method to integrate more interactivity into automatic hyperparameter optimization systems to leverage the user’s prior knowledge on parameter distribution. In our method, the user continuously observes automatic optimization’s progress and dynamically specifies where to search in the parameter space. We present a prototype implementation of an interactive dashboard for an optimizer to show our method’s feasibility. The interactive dashboard’s main feature is “paintable timeline” where the user can not only observe the past parameter values tested as in standard timeline but also specify the range of future parameters to be tested with simple painting operations. We show three examples where user intervention might improve the performance of automatic optimizations. We run a user study with experts and the results show that, with prior knowledge about parameter distribution of the target problem, interactive optimization can reach better results compared to fully automatic optimization.

People with visual impairments often require mobility assistance of sighted guides but they are not always available. Recent technological strides have opened up new directions for sighted guidance services, assigning guides from a network of remote workers to provide real-time assistance via audio/video communication. However, little has been known regarding desirable support characteristics of remote guides or challenges experienced in guide practices without the requisite expertise. To recommend support strategies that contribute to facilitating a successful platform for remote sighted guidance, this paper presents a comparative study of the performance of trained and untrained sighted guides who are recruited for a remote scenario in assisting people with visual impairments in indoor navigation. As an outcome of this research, we provide a deeper understanding of design opportunities for HCI to scaffold requirements of remote guides, such that their collaborative efforts and environmental knowledge influence the user experience. Based on our empirical insights, we suggest to develop the expertise of remote guides through: a) preliminary guidance cooperation awareness b) guidelines for verbal description methods, and c) approaches to compensate for the lack of environmental knowledge.

We propose two online-learning algorithms for modeling the personal preferences of users of interactive systems. The proposed algorithms leverage user feedback to estimate user behavior and provide personalized adaptive recommendation for supporting context-dependent decision-making. We formulate preference modeling as online prediction algorithms over a set of learned policies, i.e., policies generated via supervised learning with interaction and context data collected from previous users. The algorithms then adapt to a target user by learning the policy that best predicts that user’s behavior and preferences. We also generalize the proposed algorithms for a more challenging learning case in which they are restricted to a limited number of trained policies at each timestep, i.e., for mobile settings with limited resources. While the proposed algorithms are kept general for use in a variety of domains, we developed an image-filter-selection application. We used this application to demonstrate how the proposed algorithms can quickly learn to match the current user’s selections. Based on these evaluations, we show that (1) the proposed algorithms exhibit better prediction accuracy compared to traditional supervised learning and bandit algorithms, (2) our algorithms are robust under challenging limited prediction settings in which a smaller number of expert policies is assumed. Finally, we conducted a user study to demonstrate how presenting users with the prediction results of our algorithms significantly improves the efficiency of the overall interaction experience.

We present an assistive suitcase system, BBeep, for supporting blind people when walking through crowded environments. BBeep uses pre-emptive sound notifications to help clear a path by alerting both the user and nearby pedestrians about the potential risk of collision. BBeep triggers notifications by tracking pedestrians, predicting their future position in real-time, and provides sound notifications only when it anticipates a future collision. We investigate how different types and timings of sound affect nearby pedestrian behavior. In our experiments, we found that sound emission timing has a significant impact on nearby pedestrian trajectories when compared to different sound types. Based on these findings, we performed a real-world user study at an international airport, where blind participants navigated with the suitcase in crowded areas. We observed that the proposed system significantly reduces the number of imminent collisions.

This paper presents CoSummary, an adaptive video fast-forwarding technique for browsing surgical videos recorded by wearable cameras. Current wearable technologies allow us to record complex surgical skills, however, an efficient browsing technique for these videos is not well established. In order to assist browsing surgical videos, our study focuses on adaptively changing playback speeds through the learning and detecting collaborative scenes based on surgeon hand placement and gaze information. Our evaluation shows that the proposed method is able to highlight important collaborative scenes and skip less important scenes during surgical procedures. We have also performed a subjective study with surgeons in order to have professional feedback. The results confirmed the effectiveness of the proposed method in comparison to uniform video fast-forwarding.

This paper presents a novel interface to support video coding of social attention in the assessment of children with autism spectrum disorder. Video-based evaluations of social attention during therapeutic activities allow observers to find target behaviors while handling the ambiguity of attention. Despite the recent advances in computer vision-based gaze estimation methods, fully automatic recognition of social attention under diverse environments is still challenging. The goal of this work is to investigate an approach that uses automatic video analysis in a supportive manner for guiding human judgment. The proposed interface displays visualization of gaze estimation results on videos and provides GUI support to allow users to facilitate agreement between observers by defining social attention labels on the video timeline. Through user studies and expert reviews, we show how the interface helps observers perform video coding of social attention and how human judgment compensates for technical limitations of the automatic gaze analysis.

StratoJump is an installation that allows participants to explore the experience of high altitude jumping toward the edge of space. In this peper, we presents an vision-based algorithm for generating hyperlapse of spherical videos captured in high altitude space. Unlike previous work, we consider high altitude videos that cannot perfectly be stabilized by 3D reconstruction-based algorithms. We proposed recursive intelligent sampling to simultaneously stabilize and shorten spherical videos. Our preliminary results show that the proposed method can reduce cumulative errors of stabilization compared to a frame by frame method in reasonable time.

This work presents a novel user interface applying 3D visualization to understand complex group activities from multiple first-person videos. The proposed interface is designed to assist video viewers to easily understand the collaborative relationships of group activity based on where the individual worker is located in a workspace and how multiple workers are positioned to one another during the group activity. More specifically, the interface not only shows all recorded first-person videos but also visualizes the 3D position and orientation of each view point (i.e., the 3D position of each worker wearing a head-mounted camera) with a reconstructed 3D model of the workspace. Our user study confirms that the 3D visualization helps video viewers to understand geometric information of a worker and collaborative relationships of group activity easily and accurately.

Active involvement of users with disabilities is difficult to employ during the iterative stages of the design process due to high costs and effort associated with user studies. This research proposes a user centered design (UCD) strategy to incorporate the use of gaze-contingent tunnel vision simulation with sighted individuals to facilitate rapid prototyping of accessible interfaces. Through three types of validation studies, we examined how our simulation techniques can provide the opportunity for continued evaluation and refinement of the design. Our simulation approach was effective in emulating scanning behaviors caused by tunnel vision along with grasping user feedback to recognize user interface and usability criteria early in the design cycle.

This work presents EgoScanning, a novel video fast-forwarding interface that helps users to find important events from lengthy first-person videos recorded with wearable cameras continuously. This interface is featured by an elastic timeline that adaptively changes playback speeds and emphasizes egocentric cues specific to first-person videos, such as hand manipulations, moving, and conversations with people, based on computer-vision techniques. The interface also allows users to input which of such cues are relevant to events of their interests. Through our user study, we confirm that users can find events of interests quickly from first-person videos thanks to the following benefits of using the EgoScanning interface: 1) adaptive changes of playback speeds allow users to watch fast-forwarded videos more easily; 2) Emphasized parts of videos can act as candidates of events actually significant to users; 3) Users are able to select relevant egocentric cues depending on events of their interests.

In this work, we investigate how remote collaboration between a local worker and a remote collaborator will change if eye fixations of the collaborator are presented to the worker. We track the collaborator's points of gaze on a monitor screen displaying a physical workspace and visualize them onto the space by a projector or through an optical see-through head-mounted display. Through a series of user studies, we have found the followings: 1) Eye fixations can serve as a fast and precise pointer to objects of the collaborator's interest. 2) Eyes and other modalities, such as hand gestures and speech, are used differently for object identification and manipulation. 3) Eyes are used for explicit instructions only when they are combined with speech. 4) The worker can predict some intentions of the collaborator such as his/her current interest and next instruction.

"Shepherd Pass" is a method of tuning sport abilities that focuses on passing a ball. The method can be used to overcome skill gaps between the players for design novel sport games. Sports have been changed by technological innovations. Professional players are improving their skills using updated tools, clothing, and training methods. These innovations can help those who enjoy leisure sports as a way to promote communication or better health. Augmented sports is terms of designing novel sports with augmented fields, tools, or players using information technology. Our research involve the use of information technology to tune the sports abilities of expert and non-expert players to fill gaps in their skills and enable a freer sports design. We developed a self-actuated ball that flies via a ball-shaped quadcopter. The ball can change its speed and trajectory based on a players skill. In this paper, we explain the design concept and implementation of this system, and discuss the user experience and future research directions.

ImmerseBoard is a system for remote collaboration through a digital whiteboard that gives participants a 3D immersive experience, enabled only by an RGBD camera (Microsoft Kinect) mounted on the side of a large touch display. Using 3D processing of the depth images, life-sized rendering, and novel visualizations, ImmerseBoard emulates writing side-by-side on a physical whiteboard, or alternatively on a mirror. User studies involving three tasks show that compared to standard video conferencing with a digital whiteboard, ImmerseBoard provides participants with a quantitatively better ability to estimate their remote partners' eye gaze direction, gesture direction, intention, and level of agreement. Moreover, these quantitative capabilities translate qualitatively into a heightened sense of being together and a more enjoyable experience. ImmerseBoard's form factor is suitable for practical and easy installation in homes and offices.