Research and design in Human-Computer Interaction (HCI) have increasingly engaged with embodied, situated, and material-mediated forms of interaction. Across many such systems, sensing is not performed solely by embedded, off-the-shelf components, but arises through material properties and transformations. Yet HCI still lacks a coherent framework for articulating how materials themselves participate in sensing beyond serving as passive substrates. This paper introduces Ambient Material-Embodied Sensing (AMES), a generative theoretical framework that reframes sensing by foregrounding materials as the sensing medium in their own right. AMES conceptualizes it as an emergent property of material transduction, material-body coupling, and ambient computational inference. We articulate three core mechanisms and demonstrate how AMES can reinterpret existing systems while informing future design and research on material-mediated sensing in interaction design.

We propose a method to fabricate objects composed of 3D printed flattened pieces with integrated zipper-like structures. The object is manually assembled into a 3D shape by connecting the zipper components. By employing a zipper design that allows for angle-independent connections between patches, our method enables both the surface and zipper components to be printed in the same orientation, resulting in high-quality reconstruction of the input model with a faster 3D printing process that wastes less material. We implement a fully automated pipeline that takes a 3D model as input, converts it into developable patches, generates the zipper structures, and flattens them for subsequent 3D printing. We demonstrate that our approach significantly reduces the fabrication time and support material consumption. We also present application examples that highlight the versatility of our method.

We propose an approach to detect finger hover inputs on a smartphone screen using corneal reflection images captured by the device’s built-in front camera. This method requires no external sensors or hardware, enabling hover input detection in the near-screen space that is not directly visible to the camera. By leveraging a convolutional neural network (CNN), we estimate the two-dimensional position of a hovering finger and classify it into a predefined screen grid. Experimental results show that our model achieves approximately 95% accuracy for coarse grids and maintains over 88% accuracy for finer divisions. Furthermore, our system demonstrates real-time processing capability with an end-to-end latency of approximately 22 ms on a standard smartphone. These findings highlight the practical feasibility of camera-only hover sensing and suggest a wide range of touchless interaction applications, enabling touchless interaction when touch is undesirable, pre-touch UI adaptation, and accessibility support on commodity mobile devices.

Understanding smartphone users’ grasp postures can improve the mobile interaction experience; for instance, the user interface (UI) layout can be dynamically adjusted based on the detected grasp. In our previous work, we introduced ReflecTouch, a grasp posture detection method that leverages corneal reflection images captured by the front camera. Building on this, we conducted a follow-up study using crowdsourced experiments in uncontrolled, in-the-wild environments. Based on the results, we discuss design considerations and practical strategies, including recommendations for integrating ReflecTouch with complementary sensing modalities, such as touch and IMU-based methods, to enhance the model’s accuracy in real-world settings.

Figure 1: FlexEar-Tips overview and application examples.A: FlexEar-Tips, an ear tip system for hearables that allows shape adjustment through pressure control, B: user wearing the device, C: haptics features during gameplay, D: music listening experience enhancement and notification features, E: an image of ear tip status transitions.

Cervical myelopathy (CM) is a progressive spinal disorder affecting hand function. We developed two machine learning–based screening methods: one using tablet drawings and another using smartphone-captured paper drawings. In the tablet method, users trace spiral, square, and triangular figures with a stylus, which a convolutional neural network classifies with 95% sensitivity and 85% specificity, exceeding conventional examinations. The smartphone method captures paper drawings, enabling screening without specialized equipment. Together, these approaches provide accessible and reliable CM screening in clinical and non-clinical settings.

Smart rings are used for contactless payment, smart lock operation, and health monitoring. For applications such as electronic payment and unlocking smart locks, the implementation of a user authentication system in smart rings is essential; however, some challenges remain. Fingerprint authentication is sensitive to fingertip conditions, while face authentication faces difficulties with miniaturization, power efficiency, and privacy. This study proposes ScanRing, a hybrid authentication system using a distance sensor and an IMU sensor in a smart ring. By moving the ring device laterally in front of the face, the distance sensor captures facial structure data, while the IMU sensor records the user’s motion characteristics. These combined datasets enable robust user authentication without relying on cameras, which enhances privacy while supporting a compact and power-efficient design. A user study (N = 30) demonstrated that ScanRing achieved an average authentication accuracy of 98.41 \(\%\) under stable conditions.

Earphone-type wearable devices, also known as “hearables,” will have many functions in the future. Some of those functions will require authentication of the wearer for access to the user's privacy information or settlement of payments. In this study, we propose a new personal authentication system for hearables called EarLock. EarLock authenticates the wearer by acquiring and analyzing ear canal and auricle shape information using sound leakage from the device. The system can be implemented using a speaker and external microphone that are highly compatible with hearables. We implemented three prototype devices and investigated EarLock's authentication performance under various practical scenarios, including walking conditions, noisy environments, and situations with object interference. Experimental results showed that the in-ear, open-ear, and bone-conduction devices achieved balanced accuracy (BAC) scores of 87.2–93.7%, 83.4–94.7%, and 85.9–90.0%.

We introduce EarHover, an innovative system that enables mid-air gesture input for hearables. Mid-air gesture input, which eliminates the need to touch the device and thus helps to keep hands and the device clean. However, existing mid-air gesture input methods for hearables have been limited to adding cameras or infrared sensors. By focusing on the sound leakage phenomenon unique to hearables, we have realized mid-air gesture recognition using a speaker and an external microphone that are highly compatible with hearables. The signal leaked to the outside of the device due to sound leakage can be measured by an external microphone, which detects the differences in reflection characteristics caused by the hand’s speed and shape during mid-air gestures. Among 27 types of gestures, we determined the seven suitable gestures for EarHover in terms of signal discrimination and user acceptability. We then evaluated the gesture detection and classification performance of two prototype devices (in-ear type/open-ear type) for real-world application scenarios.

Hearables are highly functional earphone-type wearables; however, existing input methods using stand-alone hearables are limited in the number of commands, and there is a need to extend device operation through hand gestures. In previous research on hearables for hand input, user understanding and gesture recognition systems have been developed. However, in the realm of user understanding, investigation concerning hand input with hearables remains incomplete, and existing recognition systems have not demonstrated proficiency in discerning user-defined gestures. In this study, we conducted a gesture elicitation study (GES) assuming hand input using hearables under six conditions (three interaction areas x two device shapes). Then, we extracted ear-level gestures that the device's built-in IMU sensor could recognize from the user-defined gestures and investigated the recognition performance. The results of sitting experiments showed that the gesture recognition rate for in-ear devices was 91.0% and that for ear-hook devices was 74.7%.

Guiding users with limb exercise can assist in muscle training or physical recovery. However, traditional vision-based methods often require multiple camera angles to help users understand the motions and require them to be within the range of the screen. Therefore, we propose a non-visual system that can guide users with multiple-directional limb motions utilizing spatial audio, AudioMove , with commercial-off-the-shelf (COTS) devices (i.e., smartphones and earphones). The proposed system addresses the challenge of conveying directional information encompassing multiple planes in real-time. We conduct a mixed-method user study to evaluate the effectiveness of the system with three methods combining motion data with spatial audio perception. Additionally, a user interface is built to collect users' comments. The results conclude that spatial audio guidance could create a natural, pervasive, and non-visual exercise training solution in daily life.

Embedding technology into day-to-day wearables and creating smart devices such as smartwatches and smart-glasses has been a growing area of interest. In this paper, we explore the interaction around face masks, a common accessory worn by many to prevent the spread of infectious diseases. Particularly, we propose a method of using the straps of a face mask as an input medium. We identified a set of plausible gestures on mask straps through an elicitation study (N = 20), in which the participants proposed different gestures for a given referent. We then developed a prototype to identify the gestures performed on the mask straps and present the recognition accuracy from a user study with eight participants. Our results show the system achieves 93.07% classification accuracy for 12 gestures.

Human activities can introduce variations in various environmental cues, such as light and sound, which can serve as inputs for interfaces. However, one often overlooked aspect is the airflow variation caused by these activities, which presents challenges in detection and utilization due to its intangible nature. In this paper, we have unveiled an approach using mist to capture invisible airflow variations, rendering them detectable by Time-of-Flight (ToF) sensors. We investigate the capability of this sensing technique under different types of mist or smoke, as well as the impact of airflow speed. To illustrate the feasibility of this concept, we created a prototype using a humidifier and demonstrated its capability to recognize motions. On this basis, we introduce potential applications, discuss inherent limitations, and provide design lessons grounded in mist-based airflow sensing.

By sensing how a user is holding a smartphone, adaptive user interfaces are possible such as those that automatically switch the displayed content and position of graphical user interface (GUI) components following how the phone is being held. We propose ReflecTouch, a novel method for detecting how a smartphone is being held by capturing images of the smartphone screen reflected on the cornea with a built-in front camera. In these images, the areas where the user places their fingers on the screen appear as shadows, which makes it possible to estimate the grasp posture. Since most smartphones have a front camera, this method can be used regardless of the device model; in addition, no additional sensor or hardware is required. We conducted data collection experiments to verify the classification accuracy of the proposed method for six different grasp postures, and the accuracy was 85%.

While boxing games allow players to learn techniques at home, they seldom check players’ movements. We propose tracking the players’ performance by smartphone sensors and classifying punches with convolutional neural network models. As a result, we achieved accuracies of 79.2% and 84.6% for 10 participants with two experiments, implying the possibilities and facilitation of smartphone sensors in boxing classifications of people with varying experiences.

During recruitment interviews, the facial impressions of the interviewers likely affect the nervousness of the interviewees and make it difficult to conduct fair and consistent interview processes. To minimize the difference in facial impressions among interviewers, we have investigated a method to convert an interviewer’s face into an avatar. In this study, we used a digital full-face mask display capable of replacing the wearer’s face with an avatar in the real world. By reproducing the interviewer’s facial expression with an avatar in real time, we investigated the effect of avatar appearance on interviewee’s nervousness during interviews. Two types of avatars (a dignified face and a gentle face) were applied to three male interviewers in different age groups (10s, 20s and 60s). We compared the level of interviewee’s nervousness between before and after augmenting interviewer’s face with avatar. 172 college students were recruited as interviewees to assess the variation in the level of nervousness. Our experimental results show that avatar appearance can elicit more unique and consistent impressions than the interviewer’s real face and reduce the variation in interviewee’s nervousness level across interviewers.

We propose the use of face-related gestures involving the movement of the face, eyes, and head for augmented reality (AR). This technique allows us to use computer systems via hands-free, discreet interactions. In this paper, we present an elicitation study to explore the proper use of facial gestures for daily tasks in the context of a smart home. We used Amazon Mechanical Turk to conduct this study (N=37). Based on the proposed gestures, we report usage scenarios and complexity, proposed associations between gestures/tasks, a user-defined gesture set, and insights from the participants. We also conducted a technical feasibility study (N=13) with participants using smart eyewear to consider their uses in daily life. The device has 16 optical sensors and an inertial measurement unit (IMU). We can potentially integrate the system into optical see-through displays or other smart glasses. The results demonstrate that the device can detect eight temporal face-related gestures with a mean F1 score of 0.911 using a convolutional neural network (CNN). We also report the results of user-independent training and a one-hour recording of the experimenter testing two of the gestures.

This paper presents a thin digital full-face mask display that can reflect an entire facial expression of a user onto an avatar to support augmented face-to-face communication in real environments. Although camera-based facial expression recognition technology has enabled people to augment their faces with avatars, application was limited to face-to-face communication in virtual environments. To enable digital facial augmentation with an avatar in a real space, we propose a digital face mask display system that integrates a lightweight flexible display with a thin facial expression recognition system. The thin wearable facial expression recognition system was implemented with photo reflective sensor arrays which can measure facial expressions at 40 feature points distributed across an entire face. We investigated a ten-class facial expression identification model based on an SVM training algorithm. The trained model achieved an average accuracy of 79% when identifying the facial expressions of multiple users. User experiments indicated that the proposed thin digital full-face mask display allows the wearer to control the facial expression of the avatar with a fast response rate and create a positive sense of self-agency and self-ownership toward the augmented avatar face.

In this paper, we propose a method to predict impulsive input on a gamepad. We use a force-sensitive resistor to observe pressure on the gamepad button, and prediction is achieved by simple filtering processes. To evaluate our method, we conducted a user study in which users were encouraged to make impulsive inputs. The results showed that the system predicted an ON event of the button 30.82 ms in advance on average and an OFF event 29.30 ms in advance. Prediction accuracy was 97.87% for predicting ON events and 81.74% for predicting OFF events.

In this research, we present a cushion interface for operating smart home applications. We developed a gesture recognition system using convolutional neural networks and embedded the acceleration sensor arrays in the cushion cover. To evaluate the system, we conducted experiments and measured recognition accuracy.

In this paper, we propose a wrist rehabilitation support system using a smartphone app. There are several issues with the conventional wrist rehabilitation systems, primarily that there is no method for quantitatively evaluating whether it has been appropriately carried out or not, that it is difficult for doctors to observe the condition of patients at home, and that the content is often boring. In our system, using a smartphone means that we can easily introduce it to homes and have medical doctors observe the condition of patients by accessing their data on the cloud. We also aim to help patients maintain their motivation for rehabilitation by playing a game using their wrist and a smartphone.

The photo reflective sensor (PRS), a tiny distant-measurement module, is a popular electronic component widely used in wearable user-interfaces. An unavoidable issue of such wearable PRS devices in practical use is the need of user-independent training to have high gesture recognition accuracy. Each new user has to re-train a device by providing new training data (we call the inter-user setup). Even worse, re-training is also necessary ideally every time when the same user re-wears the device (we call the intra-user setup). In this paper, we propose a domain adaptation framework to reduce this training cost of users. Specifically, we adapt a pre-trained convolutional neural network (CNN) for both inter-user and intra-user setups to maintain the recognition accuracy high. We demonstrate, with an actual PRS device, that our framework significantly improves the average classification accuracy of the intra-user and inter-user setups up to 87.43% and 80.06% against the baseline (non-adapted) setups with the accuracy 68.96% and 63.26% respectively.

In this paper, we propose EarTouch, a new sensing technology for ear-based input for controlling applications by slightly pulling the ear and detecting the deformation by an enhanced earphone device. It is envisioned that EarTouch will enable control of applications such as music players, navigation systems, and calendars as an "eyes-free" interface. As for the operation of EarTouch, the shape deformation of the ear is measured by optical sensors. Deformation of the skin caused by touching the ear with the fingers is recognized by attaching optical sensors to the earphone and measuring the distance from the earphone to the skin inside the ear. EarTouch supports recognition of multiple gestures by applying a support vector machine (SVM). EarTouch was validated through a set of user studies.

Airborne Ultrasound Tactile Displays (AUTD) are tactile displays that can generate vibrotactile sensation on human skin. Combining an AUTD with a projector-camera system, it is possible to present synchronous visual and haptic stimuli that is physically aligned in the 3D space. To maintain the synchronous sensation as realistic as possible, an accurate crucial to spatial calibration between the AUTD and the projector-camera system is required. This paper thereby proposes a calibration method for the AUTD system by utilizing fur material to recognize the output focal points of an AUTD in the camera coordinate system. Our method simplifies the calibration procedure with calibration error of 2.63 mm.

We propose a drawing method to create large-scale pictures in public space. We use a property of anisotropic reflection to show images on the grass field. We created a prototype of roller type device which can control the angle of grass. We observed that our system entertains people in public exhibition.

We propose a facial expression mapping technology between virtual avatars and Head-Mounted Display (HMD) users. HMD allow people to enjoy an immersive Virtual Reality (VR) experience. A virtual avatar can be a representative of the user in the virtual environment. However, the synchronization of the the virtual avatar's expressions with those of the HMD user is limited. The major problem of wearing an HMD is that a large portion of the user's face is occluded, making facial recognition difficult in an HMD-based virtual environment. To overcome this problem, we propose a facial expression mapping technology using retro-reflective photoelectric sensors. The sensors attached inside the HMD measures the distance between the sensors and the user's face. The distance values of five basic facial expressions (Neutral, Happy, Angry, Surprised, and Sad) are used for training the neural network to estimate the facial expression of a user. We achieved an overall accuracy of 88% in recognizing the facial expressions. Our system can also reproduce facial expression change in real-time through an existing avatar using regression. Consequently, our system enables estimation and reconstruction of facial expressions that correspond to the user's emotional changes.

In this paper, we propose a novel technology called "CheekInput" with a head-mounted display (HMD) that senses touch gestures by detecting skin deformation. We attached multiple photo-reflective sensors onto the bottom front frame of the HMD. Since these sensors measure the distance between the frame and cheeks, our system is able to detect the deformation of a cheek when the skin surface is touched by fingers. Our system uses a Support Vector Machine to determine the gestures: pushing face up and down, left and right. We combined these 4 directional gestures for each cheek to extend 16 possible gestures. To evaluate the accuracy of the gesture detection, we conducted a user study. The results revealed that CheekInput achieved 80.45 % recognition accuracy when gestures were made by touching both cheeks with both hands, and 74.58 % when by touching both cheeks with one hand.

This article presents a novel smart eyewear that recognizes the wearer’s facial expressions in daily scenarios. Our device uses embedded photo-reflective sensors and machine learning to recognize the wearer’s facial expressions. Our approach focuses on skin deformations around the eyes that occur when the wearer changes his or her facial expressions. With small photo-reflective sensors, we measure the distances between the skin surface on the face and the 17 sensors embedded in the eyewear frame. A Support Vector Machine (SVM) algorithm is then applied to the information collected by the sensors. The sensors can cover various facial muscle movements. In addition, they are small and light enough to be integrated into daily-use glasses. Our evaluation of the device shows the robustness to the noises from the wearer’s facial direction changes and the slight changes in the glasses’ position, as well as the reliability of the device’s recognition capacity. The main contributions of our work are as follows: (1) We evaluated the recognition accuracy in daily scenes, showing 92.8% accuracy regardless of facial direction and removal/remount. Our device can recognize facial expressions with 78.1% accuracy for repeatability and 87.7% accuracy in case of its positional drift. (2) We designed and implemented the device by taking usability and social acceptability into account. The device looks like a conventional eyewear so that users can wear it anytime, anywhere. (3) Initial field trials in a daily life setting were undertaken to test the usability of the device. Our work is one of the first attempts to recognize and evaluate a variety of facial expressions with an unobtrusive wearable device.

In this paper, we present a wearable facial expression recognition system that can analyse and enhance a daily experience. Our aim is to create a mindful experience in daily life by connecting the device with everyday objects and service. To this end, we made two prototypes that supports users to keep right side of emotions: 1) a text chatting system that automatically inserts an emoticon based on his/her facial expressions in the end of a comment a user typed, 2) a plant interface controlled by facial expressions. We also analysed multiple users' facial expressions while they played video games. We confirmed that visualization of sensor data from the device shows the possibility for estimating the transition of different facial expressions.

This paper presents a novel smart eyewear that uses embedded photo reflective sensors and machine learning to recognize a wearer's facial expressions in daily life. We leverage the skin deformation when wearers change their facial expressions. With small photo reflective sensors, we measure the proximity between the skin surface on a face and the eyewear frame where 17 sensors are integrated. A Support Vector Machine (SVM) algorithm was applied for the sensor information. The sensors can cover various facial muscle movements and can be integrated into everyday glasses. The main contributions of our work are as follows. (1) The eyewear recognizes eight facial expressions (92.8% accuracy for one time use and 78.1% for use on 3 different days). (2) It is designed and implemented considering social acceptability. The device looks like normal eyewear, so users can wear it anytime, anywhere. (3) Initial field trials in daily life were undertaken. Our work is one of the first attempts to recognize and evaluate a variety of facial expressions in the form of an unobtrusive wearable device.

We devised a display technology that utilizes the phenomenon whereby the shading properties of fur change as the fibers are raised or flattened. One can erase drawings by first flattening the fibers by sweeping the surface by hand in the fiber's growth direction, and then draw lines by raising the fibers by moving the finger in the opposite direction. These material properties can be found in various items such as carpets in our living environments. We have developed three different devices to draw patterns on a "fur display" utilizing this phenomenon: a roller device, a pen device and pressure projection device. Our technology can turn ordinary objects in our environment into rewritable displays without requiring or creating any non-reversible modifications to them. In addition, it can be used to present large-scale image without glare, and the images it creates require no running costs to maintain.

We present Reality Jockey, a system that confuses the participant's perception of the reality by mixing in a recorded past-reality. The participant will be immersed in a spatialized 3D sound environment that is a mix of sounds from the reality and from the past. The sound environment from the past is augmented with haptic feedback in cross-modality. The haptic feedback is associated with certain sounds such as the vibration in the table when stuff is placed on the table to make the illusion of it happening in live. The seamless transition between live and past creates immersive experience of past events. The blending of live and past allows interactivity. To validate our system, we conducted user studies on 1) does blending live sensations improve such experiences, and 2) how beneficial is it to provide haptic feedbacks in recorded pasts. Potential applications are suggested to illustrate the significance of Reality Jockey.

FlashTouch is a new technology that enables data communication between touchscreen-based mobile devices and digital peripheral devices. Touchscreen can be used as communication media using visible light and capacitive touch. In this paper, we designed a stylus prototype to describe the concept of FlashTouch. With this prototype, users can easily transfer data from one mobile device to another. It eliminates the complexity associated with data sharing among mobile users, which is currently achieved by online data sharing services or wireless connections for data sharing that need a pairing operation to establish connections between devices. Therefore, it can prove to be of particular significance to people who are not adept at current software services and hardware functions. Finally, we demonstrate the valuable applications in online settlements via mobile device, and data communication for mobile robots.

We present a sensing technology and input method that uses skin deformation estimated through a thin band-type device attached to the human body, the appearance of which seems socially acceptable in daily life. An input interface usually requires feedback. SenSkin provides tactile feedback that enables users to know which part of the skin they are touching in order to issue commands. The user, having found an acceptable area before beginning the input operation, can continue to input commands without receiving explicit feedback. We developed an experimental device with two armbands to sense three-dimensional pressure applied to the skin. Sensing tangential force on uncovered skin without haptic obstacles has not previously been achieved. SenSkin is also novel in that quantitative tangential force applied to the skin, such as that of the forearm or fingers, is measured. An infrared (IR) reflective sensor is used since its durability and inexpensiveness make it suitable for everyday human sensing purposes. The multiple sensors located on the two armbands allow the tangential and normal force applied to the skin dimension to be sensed. The input command is learned and recognized using a Support Vector Machine (SVM). Finally, we show an application in which this input method is implemented.

PINOKY is a wireless ring-like device that can be externally attached to any plush toy as an accessory that animates the toy by moving its limbs. A user is thus able to instantly convert any plush toy into a soft robot. The user can control the toy remotely or input the movement desired by moving the plush toy and having the data recorded and played back. Unlike other methods for animating plush toys, PINOKY is non-intrusive, so alterations to the toy are not required. In a user study, 1) the roles of plush toys in the participants' daily lives were examined, 2) how participants played with plush toys without PINOKY was observed, 3) how they played with plush toys with PINOKY was observed, and their reactions to the device were surveyed. On the basis of the results, potential applications were conceptualized to illustrate the utility of PINOKY.

We present the iRing, an intelligent input ring device developed for measuring finger gestures and external input. iRing recognizes rotation, finger bending, and external force via an infrared (IR) reflection sensor that leverages skin characteristics such as reflectance and softness. Furthermore, iRing allows using a push and stroke input method, which is popular in touch displays. The ring design has potential to be used as a wearable controller because its accessory shape is socially acceptable, easy to install, and safe, and iRing does not require extra devices. We present examples of iRing applications and discuss its validity as an inexpensive wearable interface and as a human sensing device.

We have developed a simple skin-like user interface that can be easily attached to curved as well as flat surfaces and used to measure tangential force generated by pinching and dragging interactions. The interface consists of several photoreflectors that consist of an IR LED and a phototransistor and elastic fabric such as stocking and rubber membrane. The sensing method used is based on our observation that photoreflectors can be used to measure the ratio of expansion and contraction of a stocking using the changes in transmissivity of IR light passing through the stocking. Since a stocking is thin, stretchable, and nearly transparent, it can be easily attached to various types of objects such as mobile devices, robots, and different parts of the body as well as to various types of conventional pressure sensors without altering the original shape of the object. It can also present natural haptic feedback in accordance with the amount of force exerted. A system using several such sensors can determine the direction of a two-dimensional force. A variety of example applications illustrated the utility of this sensing system.

We present an actuated handheld puppet system for controlling the posture of a virtual character. Physical puppet devices have been used in the past to intuitively control character posture. In our research, an actuator is added to each joint of such an input device to provide physical feedback to the user. This enhancement offers many benefits. First, the user can upload pre-defined postures to the device to save time. Second, the system is capable of dynamically adjusting joint stiffness to counteract gravity, while allowing control to be maintained with relatively little force. Third, the system supports natural human body behaviors, such as whole-body reaching and joint coupling. This paper describes the user interface and implementation of the proposed technique and reports the results of expert evaluation. We also conducted two user studies to evaluate the effectiveness of our method.

We present the FuwaFuwa sensor module, a round, hand-size, wireless device for measuring the shape deformations of soft objects such as cushions and plush toys. It can be embedded in typical soft objects in the household without complex installation procedures and without spoiling the softness of the object because it requires no physical connection. Six LEDs in the module emit IR light in six orthogonal directions, and six corresponding photosensors measure the reflected light energy. One can easily convert almost any soft object into a touch-input device that can detect both touch position and surface displacement by embedding multiple FuwaFuwa sensor modules in the object. A variety of example applications illustrate the utility of the FuwaFuwa sensor module. An evaluation of the proposed deformation measurement technique confirms its effectiveness.

We propose a cooking system that operates in an open environment. The system cooks a meal by pouring various ingredients into a boiling pot on an induction heating cooker and adjusts the heating strength according to the user's instructions. We then describe how the system incorporates robotic- and human-specific elements in a shared workspace so as to achieve a cooperative rudimentary cooking capability. First, we use small mobile robots instead of built-in arms to save space, improve flexibility and increase safety. Second, we use detachable visual markers to allow the user to easily configure the real-world environment. Third, we provide a graphical user interface to display detailed cooking instructions to the user. We hope insights obtained in this experiment will be useful for the design of other household systems in the future.