We present a computational approach to haptic design embedded in everyday tangible interaction with digital fabrication. To generate haptic feedback, the use of permanent magnets as the mechanism potentially contributes to simpleness and robustness; however, it is difficult to manually design how magnets should be embedded in the objects. Our approach enables the inverse design of magnetic force feedback; that is, we computationally solve an inverse problem to obtain an optimal arrangement of permanent magnets that renders the user-specified haptic sensation. To solve the inverse problem in a practical manner, we also present techniques on magnetic simulation and optimization. We demonstrate applications to explore the design possibility of augmenting digital fabrication for everyday use.

This paper presents Songle Sync, a web-based platform on which hundreds of Internet-connected devices - including smartphones, computers, and other physical computing devices - can be controlled to synchronize with music playback. It uses music-understanding technologies to dynamically synthesize music-driven multimedia performances from a musical piece of choice. To simultaneously control hundreds of devices, a conventional architecture keeps always-on connections between them. However, it does not scale and suffers from latency and jitter issues when there are various devices with potentially unstable networks. We address this with a novel autonomous control architecture in which each device is notified of forthcoming musical events (e.g., beats and chorus sections) to automatically drive various changes in multimedia performances. Moreover, we provide a development kit of an event-driven multimedia framework for JavaScript, example programs, and an interactive tutorial. To evaluate the platform, we compared latencies, jitters, and amounts of network traffic between ours and the conventional architecture. To examine use cases in the wild, we deployed the platform to drive over a hundred of a variety of devices. We also developed a web browser-based application for a multimedia performance with music playback. It provided audiences of hundreds with a bring-your-own-device experience of synchronized animations on smartphones. In addition, the development kit was used in a two-day hackathon. We report lessons learned from these studies and discuss the future of the Internet of Musical Things.

We present magneto-haptics, a design approach of haptic sensations powered by the forces present among permanent magnets during active touch. Magnetic force has not been efficiently explored in haptic design because it is not intuitive and there is a lack of methods to associate or visualize magnetic force with haptic sensations, especially for complex magnetic patterns. To represent the haptic sensations of magnetic force intuitively, magneto-haptics formularizes haptic potential from the distribution of magnetic force along the path of motion. It provides a rapid way to compute the relationship between the magnetic phenomena and the haptic mechanism. Thus, we can convert a magnetic force distribution into a haptic sensation model, making the design of magnet-embedded haptic sensations more efficient. We demonstrate three applications of magneto-haptics through interactive interfaces and devices. We further verify our theory by evaluating some magneto-haptic designs through experiments.

This article presents a novel smart eyewear that recognizes the wearer’s facial expressions in daily scenarios. Our device uses embedded photo-reflective sensors and machine learning to recognize the wearer’s facial expressions. Our approach focuses on skin deformations around the eyes that occur when the wearer changes his or her facial expressions. With small photo-reflective sensors, we measure the distances between the skin surface on the face and the 17 sensors embedded in the eyewear frame. A Support Vector Machine (SVM) algorithm is then applied to the information collected by the sensors. The sensors can cover various facial muscle movements. In addition, they are small and light enough to be integrated into daily-use glasses. Our evaluation of the device shows the robustness to the noises from the wearer’s facial direction changes and the slight changes in the glasses’ position, as well as the reliability of the device’s recognition capacity. The main contributions of our work are as follows: (1) We evaluated the recognition accuracy in daily scenes, showing 92.8% accuracy regardless of facial direction and removal/remount. Our device can recognize facial expressions with 78.1% accuracy for repeatability and 87.7% accuracy in case of its positional drift. (2) We designed and implemented the device by taking usability and social acceptability into account. The device looks like a conventional eyewear so that users can wear it anytime, anywhere. (3) Initial field trials in a daily life setting were undertaken to test the usability of the device. Our work is one of the first attempts to recognize and evaluate a variety of facial expressions with an unobtrusive wearable device.

Toothbrushing plays an important role in daily dental plaque removal for preventive dentistry. Prior work has investigated improvements on toothbrushing with sensing technologies. But existing toothbrushing support focuses mostly on estimating brushing coverage. Users thus only have indirect information about how well their toothbrushing removes dental plaque. We present LumiO, a toothbrush that offers users continuous feedback on the amount of plaque on teeth. Lumio uses a well-known method for plaque detection, called Quantitative Light-induced Fluorescence (QLF). QLF exploits a red fluorescence property that bacterium in the plaque demonstrates when a blue-violet ray is cast. Blue-violet light excites this fluorescence property, and a camera with an optical filter can capture plaque in pink. We incorporate this technology into an electric toothbrush to achieve improvements in performance on plaque removal in daily dental care. This paper first discusses related work in sensing for oral activities and interaction as well as dental care with technologies. We then describe the principles of QLF, the hardware design of LumiO, and our vision-based plaque detection method. Our evaluations show that the vision-based plaque detection method with three machine learning techniques can achieve F-measures of 0.68 -- 0.92 under user-dependent training. Qualitative evidence also suggests that study participants were able to have improved awareness of plaque and build confidence on their toothbrushing.

This paper presents a novel smart eyewear that uses embedded photo reflective sensors and machine learning to recognize a wearer's facial expressions in daily life. We leverage the skin deformation when wearers change their facial expressions. With small photo reflective sensors, we measure the proximity between the skin surface on a face and the eyewear frame where 17 sensors are integrated. A Support Vector Machine (SVM) algorithm was applied for the sensor information. The sensors can cover various facial muscle movements and can be integrated into everyday glasses. The main contributions of our work are as follows. (1) The eyewear recognizes eight facial expressions (92.8% accuracy for one time use and 78.1% for use on 3 different days). (2) It is designed and implemented considering social acceptability. The device looks like normal eyewear, so users can wear it anytime, anywhere. (3) Initial field trials in daily life were undertaken. Our work is one of the first attempts to recognize and evaluate a variety of facial expressions in the form of an unobtrusive wearable device.

FluxPaper is a new paper-based medium that enables physical movement and dynamic interaction between a high-power magnetized paper and a programmable magnetic field. FluxPaper has a very thin patterned magnetic layer (0.1 mm) pasted behind the paper. A thin but strong neodymium-based magnet realizes fast, powerful, and precise physical actions while retaining the original characteristics of the paper that is widely used in our daily lives. Owing to an effective magnetic pattern and a computer-controlled magnetic field, FluxPaper can add new interaction modality to ordinary paper. We describe the functions of magnetized paper; challenges through realization; and the interaction scenarios in several applications, such as self-alignment, self-construction, floating on the board, and quickly picking out a target card from a stack.

The purpose of our study is to investigate human teaching behavior and robot learning behavior when a human teaches a robot. Agents for learning support need to build a pedagogical relationship, in which a teacher agent and a student agent change their behaviors as they recognize the other's characteristic behaviors. In order to investigate how a robot that behaves as a student should respond to humans' teaching behaviors in a pedagogical relationship between human and robot, we conducted a case study using a game played on a tablet with a robot. In the case study, we analyzed how humans changed their teaching behaviors when the humanoid robot failed to understand what they taught. From the results of this case study, we observed that some subjects carefully taught the robot in each trial in order to allow the robot to understand the subjects. Moreover, we also observed that subjects' teaching behavior changed when the subject received feedback from the robot about the teaching.

We propose an active display robot for tele-communication system combining 3-axis display arm manually controlled by touch input, and automatically controlled by human tracking. This system is designed to present distance change behavior between user and robot by implementing two types of user interaction during communication; 1) Providing robot's movement to follow the user who is far from or going to pass by the display robot, 2) providing touchable display to activate and manipulate the direction of the display during the user is sited in front of the display system. Both local and remote users use the same system to communicate, make user's attention to distanced user via the system. Manipulating the movable part of the display by the movement of the finger of the user's touch makes the display correspond to user's intention and attention. There are two advantages of the movable display by the operation of the user's touch. One is to enhance remote communication by making the display arm moving with touch operation. Second is to enable the transmission of non-verbal information through the operation of the display. We designed hardware and software associated with user behavior and user input via the display. It is expected that the system will solve that the problem of misunderstanding of remote device operation, then it contribute to intuitive operation. In order to evaluate how this system does contribute to intuitive operation, we conducted an assessment experiment by using Questionnaire of Likert scale and interview. We verified that the display can move in response to operator's touch input contribute to intuitive experience.

The study of the telepresence robot as a tool for telecommunication from a remote location is attracting a considerable amount of attention. However, the problem arises that a telepresence robot system does not allow the volume of the user's utterance to be adjusted precisely, because it does not consider varying conditions in the sound environment, such as noise. In addition, when talking with several people in remote location, the user would like to be able to change the speaker volume freely according to the situation. In a previous study, a telepresence robot was proposed that has a function that automatically regulates the volume of the user's utterance. However, the manner in which the user exploits this function in a practical situation needs to be investigated. We propose a telepresence conversation robot system called “TeleCoBot.” TeleCoBot includes an operator's user interface, through which the volume of the user's utterance can be automatically regulated according to the distance between the robot and the conversation partner and the noise level in the robot's environment. We conducted a case study, in which the participants played a game using TeleCoBot's interface. The results of the study reveal the manner in which the participants used TeleCoBot and the additional factors that the system requires.