Voice Activity Projection (VAP) has been actively studied to enable natural turn-taking in spoken dialogue systems, relying primarily on acoustic features. Visual cues such as head movements are also known to contribute to turn-taking prediction; however, camera-based approaches are affected by placement and lighting conditions and are not always reliably available to dialogue systems. As a camera-independent approach for directly capturing head motion, earable devices offer a promising solution. In this study, we propose Sensor-Augmented VAP, a framework that integrates in-ear inertial measurement unit (IMU) signals with a pre-trained VAP model via a lightweight residual fusion module. To validate our proposed method, we collected a dataset pairing conversational audio with in-ear IMU data, comprising 12 dyadic Japanese dialogues recorded using microphones and earbuds. Experiments in speaker-independent and speaker-dependent settings demonstrate that IMU fusion consistently improves weighted F1 score for shift detection and reduces VAP loss over the audio-only baseline. These results confirm that head-motion cues are effective for enhancing turn-taking prediction.

In Affective computing, recognizing users' emotions accurately is the basis of affective human-computer interaction. Understanding users' interoception contributes to a better understanding of individually different emotional abilities, which is essential for achieving inter-individually accurate emotion estimation. However, existing interoception measurement methods, such as the heart rate discrimination task, have several limitations, including their dependence on a well-controlled laboratory environment and precision apparatus, making monitoring users' interoception challenging. This study aims to determine other forms of data that can explain users' interoceptive or similar states in their real-world lives and propose a novel hypothetical concept "cyberoception," a new sense (1) which has properties similar to interoception in terms of the correlation with other emotion-related abilities, and (2) which can be measured only by the sensors embedded inside commodity smartphone devices in users' daily lives. Results from a 10-day-long in-lab/in-the-wild hybrid experiment reveal a specific cyberoception type "Turn On" (users' subjective sensory perception about the frequency of turning-on behavior on their smartphones), significantly related to participants' emotional valence. We anticipate that cyberoception to serve as a fundamental building block for developing more "emotion-aware", user-friendly applications and services.

In Smart Things and smart city applications, IoT devices generate large amounts of data and deep learning technologies are used to acquire useful information from it. Based on the kind of application and data, there are various non-functional requirements, such as low latency for information presentation by MR and privacy for video processing. To serve these requirements, a computing platform needs to make appropriate use of computing resources, namely Cloud, Fog, and Dew. However, there are some technical challenges in designing such a platform: i) transparently satisfying application QoS; ii) running the application across various hardware and OSes without modification; iii) sharing the application context taking into account the validity of values (temporal locality) and the data privacy (spatial locality). In this paper, we introduce Laqista, a novel Cloud-Fog-Dew computing platform. Laqista serves applications in a serverless manner via the Edgeless API, which schedules requests and abstracts the details of the platform. Applications are separated into Logics and Models, which are converted to lightweight, platform-agnostic formats such as WebAssembly and ONNX, respectively. Additionally, the Context Store synchronizes application context among the nodes, handling the privacy and validity of data. We developed a prototype implementation of Laqista in Rust and evaluated its performance. Experimental results show that the Laqista design has practical performance and is applicable to real-time applications such as video processing and MR.

In this work, we propose a novel model that focuses on object features by combining object detection with CNN and LSTM networks. In recent years, a multitude of deep learning-based methods for Visual Localization, have been extensively researched. However, conventional methods do not adequately account for object-level features. Therefore, it is difficult to use indoors where similar objects appear frequently. Our method applies CNN for feature extraction on detected objects cropped by YOLOv8, an object detection algorithm, and then integrates these features into a single feature vector using LSTM, enabling location estimation that takes into account multiple object features.

In this paper, we propose a MiRRoR(Mixed-Reality of Robust Rendering) that enables user localization in the mobility scenario. Mixed Reality (MR) is an emerging technology for interacting with spatial digital content. It merges virtual content and the actual world through the camera. Nowadays, many kinds of Head-Mount Display (HMD) devices for MR have been released, and they can be more popular in the future. However, MR is not available in mobility situations because of localization problems. This research aims to solve this problem using a sensor fusion approach. We developed an MR bus tour with visual-based localization and GPS, then conducted a preliminary experiment to evaluate localization accuracy and MR experience.

Sustained input of lifelog data is critical for conversational health applications, where algorithms and AI advise users based on recorded lifelog data such as meals, exercise, and sleep. However, an effective method for presenting information to encourage this continuity has not been identified. This study developed three intervention methods for prompting lifelog entries: (a) wording adjustment based on individual characteristics, (b) timing adjustment based on physical activity, and (c) a combination of these adjustments. An empirical experiment with 422 participants was conducted to evaluate the effects.

This study presents SatisFace, a novel approach to measuring customer satisfaction in amusement parks by analyzing facial expressions at entry and exit points. Recognizing satisfaction's subjectivity, SatisFace assesses facial Action Unit (AU) changes, tailoring assessments to individual user characteristics. Data were collected through an experiment with 200 participants at Soleil Hill, using questionnaires and OpenFace for facial analysis. A gradient-boosting machine learning model revealed a significant correlation between changes in facial expressions and satisfaction levels.

Class-incremental semantic segmentation (CISS) is a challenging task for operating image analysis from mobile devices in a changing real-world environment. In CISS, data-replay method is effective which selects and stores a part of past class data as the exemplar memory to learn increasing class objects sequentially without forgetting. We set SSUL-M as our baseline and propose the exemplar memory selection method considering the objects' classes.

This study aims to develop a heat stroke risk estimation system that takes into account not only environmental information but also internal factors of individuals in order to prevent the increasing number of heat stroke deaths in Japan. 26 outdoor workers were subjected to comprehensive measurements of environmental and biological data between August 15 and September 17, 2023, A prediction model was developed using a random forest classifier. While the model is able to predict with high accuracy cases with no risk, there is room for improvement in the detection of risky cases. The system aims to personalize heat stroke prevention measures to achieve more effective risk management.

The preparation of a large and diverse dataset is essential for training a robust deep learning model. However, there are instances where certain data are theoretically possible but challenging to observe in reality (e.g., traffic light violations). We refer to these unique instances as 'Scarce Realities', highlighting their rarity and the difficulties they present in data collection and model training. One effective and emerging approach involves using generative models to generate and augment such data. In this study, we demonstrate the promising potential of combining object detection models with simple image generation models as a way to generate fake videos. We achieve this by partially editing existing videos to artificially create 'Scarce Realities', using the generation of fake dashboard camera footage of traffic light violations as an example.

Many smart city applications utilize deep learning technologies to process the data generated by sensors and smart devices. However, current application hosting platforms are not suitable for deep learning applications, because of their special requirements, including GPU and large trained model data. We propose a smart city application hosting platform named MLess, which serves and scales deep-learning applications across servers. MLess adds an extra abstraction layer between applications and executing hosts, thus it allows developers to write applications in a serverless manner. We developed a Proof-of-Concept implementation of MLess and made preliminary evaluations against it. In future work, we plan to add QoS support like inference accuracy or throughput.

In this paper, we propose a MeowSorter that identifies stray and pet cats using deep learning technology, to address the problem that lost cats with owners are mistakenly identified as stray cats and wrongfully euthanized. We made a dataset of 800 facial images and compared the accuracy of six image recognition algorithms. The optimal algorithm for classifying stray and pet cats is ResNet-50, achieving an accuracy of 85.44%. Our findings confirm that cat eyes and ears are key differentiators, rivaling cat expert judgments.

Deep learning models need a lot of labeled data, which is costly and time-consuming to collect. A method that learns a common knowledge (meta-knowledge) from similar tasks to train new tasks efficiently with fewer data is effective. We propose "Feature-adaptive Re-MAML" (FARe-MAML), a novel method for acquiring the most optimal learning method, taking into account the features of the new training data.

Data binding in front-end user interface web development allows for the UI to update automatically in sync with data, removing complexity from development and simplifying programming. However, data binding often causes website performance to degrade due to its increased complexity. In this paper, we propose "Visible Anchors" to solve the performance degradation caused by data binding. We present a novel web front-end compiler called FaST that builds upon this idea. We also compare the rendering speed of websites built by existing methods and the FaST compiler. The evaluation result revealed that the websites built by the FaST compiler are rendered at a minimum 17.1 times faster than the ones built by the existing methods. Most notably, FaST achieves all of this without any additional cost to the developer.

Crowdedness sensing of buses is playing an important role in the disease control of COVID-19 and bus resource scheduling. This research analyzes the relationship between carbon dioxide concentration, bus environment and the number of passengers by linear regression. Our prototype system collects the data of bus environment and carbon dioxide concentration to estimate the number of passengers in real time. By collecting the sensing data from a shuttle bus of university campus, we experimentally evaluate the feasibility and sensing performance of the crowdedness estimation model.

Emotions are essential for constructing social relationships between humans and interactive systems. Although emotional and empathetic dialogue generation methods have been proposed for dialogue systems, appropriate dialogue involves not only mirroring emotions and always being empathetic but also complex factors such as context. This paper proposes Emotion Regulation Chat (ER-Chat) as an end-to-end dialogue framework for emotion regulation. Emotion regulation is concerned with actions to approach appropriate emotional states. Learning appropriate emotion and intent when responding on the basis of the context of the dialogue enables the generation of more human-like dialogue. We conducted automatic and human evaluations to demonstrate the superiority of ER-Chat over the baseline system. The results show that inclusion of emotion and intent prediction mechanisms enable generation of dialogues with greater fluency, diversity, emotion awareness, and emotion appropriateness, which are greatly preferred by humans.

Many researchers and companies have engaged in estimating users' interests so that an online shopping system can tell what he/she wants now. This paper tackles the next challenge in online shopping, i.e., predicting the times that users go shopping online. To predict the timing of online shopping, we focus on "wandering behavior" in web search activities and propose a "search wandering score" (SWS). Online shopping behavior can be categorized into three states: "wandering shop-ping", "focused shopping", and others. Wandering shopping is a state in which users make purchases in high SWS situations; focused shopping is a state in which users buy things in low SWS situations. Unlike previous studies, our work is based on an analysis of large-scale data containing shopping and search logs produced by approximately 200,000 users of a real web portal site for over a year. The results of an extensive evaluation show that our methodology can predict user's future shopping behavior types with 86% accuracy. This research is the first step towards understanding the relationship between users' mental states and their online shopping behavior.

With an increase in the number of IoT devices, the amount of data transferred between the devices and applications is becoming huge. To ensure that these data are properly used, a new IoT data transfer system is needed to better control the timing and the content of the data to transmit. One of the promising means for such a large-scale city-data transfer is the publish/subscribe messaging model, which can separate data senders and receivers so that they can run independently. However, existing pub/sub systems cannot cope well with the mobility of senders and receivers, thereby limiting its applicability to real-world uses. In concrete, they don't consider Time-to-live (TTL) of data. Users can use the data anytime and within TTL of it. IoT platforms can improve controlling data transmission by used to this characteristic. In this paper, we focus on the Time-to-Live of data (data-TTL). Our system can control data transmission by using data-TTL and combine with the user's movement information. We have constructed a system, that is capable of control data transmission for mobility aware.

Conversational agents are increasingly becoming digital partners of our everyday computing experiences offering a variety of purposeful information and utility services. Although rich on competency, these agents are entirely oblivious to their users' situational and emotional context today and incapable of adjusting their interaction style and tone contextually. To this end, we present a mixed-method study that informs the design of a situation- and emotion-aware conversational agent for kinetic earables. We surveyed 280 users, and qualitatively interviewed 12 users to understand their expectation from a conversational agent in adapting the interaction style. Grounded on our findings, we develop a first-of-its-kind emotion regulator for a conversational agent on kinetic earable that dynamically adjusts its conversation style, tone, volume in response to users emotional, environmental, social and activity context gathered through speech prosody, motion signals and ambient sound. We describe these context models, the end-to-end system including a purpose-built kinetic earable and their real-world assessment. The experimental results demonstrate that our regulation mechanism invariably elicits better and affective user experience in comparison to baseline conditions in different real-world settings.

The limited attentional resource of users is a bottleneck to delivery of push notifications in today's mobile and ubiquitous computing environments. Adaptive mobile notification scheduling, which detects opportune timings based on mobile sensing and machine learning, has been proposed as a way of alleviating this problem. However, it is still not clear if such adaptive notifications are effective in a large-scale product deployment with real-world situations and configurations, such as users' context changes, personalized content in notifications, and sudden external factors that users commonly experience (such as breaking news). In this paper, we construct a new interruptibility estimation and adaptive notification scheduling with redesigned technical components. From the deploy study of the system to the real product stack of Yahoo! JAPAN Android application and evaluation with 382,518 users for 28 days, we confirmed several significant results, including the maximum 60.7% increase in the users' click rate, 10 times more gain compared to the previous system, significantly better gain in the personalized notification content, and unexpectedly better performance in a situation with exceptional breaking news notifications. With these results, the proposed system has officially been deployed and enabled to all the users of Yahoo! JAPAN product environment where more than 10 million Android app users are enjoying its benefit.

Conversational agents are increasingly becoming digital partners of our everyday computing experiences offering a variety of purposeful information and utility services. Although rich on competency, these agents are entirely oblivious to their users' situational and emotional context today and incapable of adjusting their interaction style and tone contextually. To this end, we present a first-of-its-kind situation-aware conversational agent on kinetic earable that dynamically adjusts its conversation style, tone, volume in response to users emotional, environmental, social and activity context gathered through speech prosody, ambient sound and motion signatures.

In correlation with the socio-economic development, changes in people's lifestyle brought about significant impact on dietary patterns. Though public concerns over healthy eating are increasing, many are still uncertain when choosing a well balanced meal amid welter of information. In this paper, we propose "KomaFLens'', a mobile system and application built for Microsoft HoloLens, which aims to motivate long term dietary habit modification through gamification. The primary purpose of this research is to enhance the users' nutritional knowledge and to guide them to make healthier choices in their diet. Our preliminary evaluation revealed interesting points for discussion regarding the procedure for capturing food labels. Streamlining the operational method to boost tractability will improve the accuracy when recording food intakes.

We propose "SmileWave", the first selfie social networking service to reveal the existence of emotional cognation through smiling selfies on the social network. We conducted multiple rounds of in-the-wild user studies with 86 cumulative total users for total duration of 5 weeks. Throughout the entire study, we confirmed the occurrence of smile-based emotional contagion over social network, not only in the momentary duration but in longer term period.

In today's advancing ubiquitous computing age, with its ever-increasing amount of information from various applications and services available for consumption, the management of people's attention has become very important. In particular, the high volume of notifications on mobile devices has become a major cause of interruption of users. There has been much research aimed at detecting the opportune moment to present such information to users with in a way that lowers the cognitive load or frustration. However, evaluation of such systems in the real-world production environment with real users and notifications, and evaluation on user's engagement to the presented notification beyond simple responsiveness have not been adequately studied. To the best of our knowledge, this study is the first to investigate user interruptibility and engagement using a real-world large-scale mobile application and real-world notifications consisting of actual news content. We equipped the Yahoo! JAPAN Android app, one of the most popular applications on the national market, with our mobile-sensing and machine-learning-based interruptibility estimation logic. We conducted a large-scale in-the-wild user study with more than 680,000 users for three weeks. The results show that in most cases delaying the notification delivery until an interruptible moment is detected is beneficial to users and results in significant reduction of user response time (49.7%) compared to delivering the notifications immediately. We also observed a higher number of notifications opened in our system as well as constant improvement in user engagement levels throughout the entire study period.

This paper proposes ''SmileWave", a system for revealing smile-based emotional contagion, propagation effect of the similar emotion through smiley facial expression, on the social network where users interact each other through web-based user interface rather than in-person interaction. SmileWave is a picture-based networking service and detects the change of smile degree when the user looks at posted smile images of others. Our extensive user study with 50 participants for 30 days confirmed the emotional contagion effect on SmileWave. Users' smile degree improved by 27% when the user looked at posted smile images. The result also proved that there is a stronger effect on smile-based emotional contagion when the examinee and the person in the image are in close relationship.

Recent technological trends in mobile/wearable devices and sensors have been enabling an increasing number of people to collect and store their "lifelog" easily in their daily lives. Beyond exercise behavior change of individual users, our research focus is on the behavior change of teams, based on lifelogging technologies and lifelog sharing. In this paper, we propose and evaluate six different types of lifelog sharing models among team members for their exercise promotion, leveraging the concepts of "competition" and "collaboration." According to our experimental mobile web application for exercise promotion and an extensive user study conducted with a total of 64 participants over a period of three weeks, the model with a "competition" technique resulted in the most effective performance for competitive teams, such as sports teams.

In today's ubiquitous computing environment where users carry, manipulate, and interact with an increasing number of networked devices, applications and web services, human attention is the new bottleneck in computing. It is therefore important to minimize a user's mental effort due to notifications, especially in situations where users are mobile and using multiple wearable and mobile devices. To this end, we propose Attelia II, a novel middleware that identifies breakpoints in users' lives while using those devices, and delivers notifications at these moments. Attelia II works in real-time and uses only the mobile and wearable devices that users naturally use and wear, without any modifications to applications, and without any dedicated psycho-physiological sensors. Our in-the-wild evaluation in users' multi-device environment (smart phones and smart watches) with 41 participants for 1 month validated the effectiveness of Attelia. Our new physical activity-based breakpoint detection, in addition to the UI Event-based breakpoint detection, resulted in a 71.8% greater reduction of users' perception of workload, compared with our previous system that used UI events only. Adding this functionality to a smart watch reduced workload perception by 19.4% compared to random timing of notification deliveries. Our multi-device breakpoint detection across smart phones and watches resulted in about 3 times greater reduction in workload perception than our previous system.

We present QueueVadis, a system that addresses the problem of estimating, in real-time, the properties of queues at commonplace urban locations, such as coffee shops, taxi stands and movie theaters. Abjuring the use of any queuing-specific infrastructure sensors, QueueVadis uses participatory mobile sensing to detect both (i) the individual-level queuing episodes for any arbitrarily-shaped queue (by a characteristic locomotive signature of short bursts of "shuffling forward" between periods of "standing") and (ii) the aggregate-level queue properties (such as expected wait or service times) via appropriate statistical aggregation of multi-person data. Moreover, for venues where multiple queues are too close to be separated via location estimates, QueueVadis also uses a novel disambiguation technique to separate users into multiple distinct queues. User studies, performed with 138 cumulative total users observed at 23 different real-world queues across Singapore and Japan, show that QueueVadis is able to (a) identify all individual queuing episodes, (b) predict service and wait times fairly accurately (with median estimation errors in the 10%--20% range), independent of the queue's shape, (c) separate users in multiple proximate queues with close to 80% accuracy and (d) provide reasonable estimates when the participation rate (the fraction of QueueVadis-equipped people in the queue) is modest.

In today's ubiquitous computing environment where the number of devices, applications and web services are ever increasing, human attention is the new bottleneck in computing. To minimize user cognitive load, we propose Attelia, a novel middleware that identifies breakpoints in user interaction and delivers notifications at these moments. Attelia works in realtime and uses only the mobile devices that users naturally use and wear, without any modifications to applications, and without any dedicated psycho-physiological sensors. Our evaluation proved the effectiveness of Attelia. A controlled user study showed that notifications at detected breakpoint timing resulted in 46% lower cognitive load compared to randomly-timed notifications. Furthermore, our “in-the-wild” user study with 30 participants for 16 days further validated Attelia's value, with a 33% decrease in cognitive load compared to randomly-timed notifications.

Recent technological trends on mobile/wearable devices and sensors have been enabling increasing number of people to collect and store their “life-logs” easily in their daily lives. Beyond exercise behavior change of individual user, our research focus is on the behavior change of teams, based on life-logging technologies and information sharing. In this paper, we propose and evaluate six different types of information sharing model among team members for their exercise promotion, leveraging concepts of “competition” and “collaboration”. According to our experimental mobile web application for exercise promotion and extensive user study among 64 total users for three weeks, the model with “external competition” technique resulted the most effective performance for competitive teams such as sport teams.

In this paper, we present the design and implementation of Focus, a system for effectively and efficiently reducing power consumption of OLED displays on smartphones. These displays, while becoming exceedingly common still consume significant power. The key idea of Focus is that we use the notion of saliency to save display power by dimming portions of the applications that are less important to the user. We envision Focus being especially useful during low battery situations when usability is less important than power savings. We tested Focus using 15 applications running on a Samsung Galaxy S III and show that it saves, on average, between 23 to 34% of the OLED display power with little impact on task completion times. Finally, we present the results of a user study, involving 30 participants that shows that Focus, even with its dimming behaviour, is still quite usable.

Remote execution can transform the puniest mobile device into a computing giant able to run resource-intensive applications such as natural language translation, speech recognition, face recognition, and augmented reality. However, easily partitioning these applications for remote execution while retaining application-specific information has proven to be a difficult challenge. In this paper, we show that automated dynamic repartitioning of mobile applications can be reconciled with the need to exploit application-specific knowledge. We show that the useful knowledge about an application relevant to remote execution can be captured in a compact declarative form called tactics. Tactics capture the full range of meaningful partitions of an application and are very small relative to code size. We present the design of a tactics-based remote execution system, Chroma, that performs comparably to a runtime system that makes perfect partitioning decisions. Furthermore, we show that Chroma can automatically use extra resources in an over-provisioned environment to improve application performance.

In this paper, we present a new mobility-support model for applications, Application Module Request Broker (AMRB). We focus on two types of mobility: host mobility and application code mobility. These two types of mobility dynamically change the binding between applications name and location in the network. AMRB conceals these changes of banding to reduce a complexity in development of applications. In AMRB, we deal with mobile application codes that communicate with each other as Application Modules (AMs). AMRB provides AM's service transparent communication for applications by using a specifier which does not need to include any network location information. Furthermore, applications can use AM's service transparently of migration by exploiting location management mechanism. In this paper, we describe the design and implementation of AMRB and some evaluations. Also, we demonstrate a sample application that AMRB is effective for developing mobile sensor type applications.

This paper proposes "MobileSocket" which realizes session layer communication continuity support for Java Applications towards the continuous operations for mobile applications. In the mobile computing environment where mobile hosts move around the network even during communications with the remote, maintenance of the communication continuity between the applications is significant. Not only mobility support but virtual circuit continuity support is required for communication continuity. Existing approaches have not provided the complete communication continuity for applications. "MobileSocket" is a user-level enhanced socket library written in Java, and provides library-based session layer mobility and virtual circuit continuity support for applications. Two mechanisms, dynamic socket switching (DSS) and the application layer window (ALW) were developed for MobileSocket and enable a simple implementation. The MobileSocket applications can be used in Java mobile applications and the agents, as well as for ordinary network applications. In this paper, after we clarify the communication continuity and existing approaches, we present the MobileSocket design, mechanism, and results of evaluation.