Human activity recognition (HAR) in industrial domains is important for workflow optimization, throughput estimation, and bottleneck detection. Skeleton-based models achieve high HAR accuracy by exploiting rich spatial and temporal cues, but they are difficult to deploy in industrial sites due to occlusions, camera placement, and privacy concerns. IMU sensors, especially smartwatches, are practical for deployment but lack spatial awareness, resulting in weaker performance. This work aims to enable robust HAR using only a wrist-worn IMU by distilling knowledge from richer modalities. Knowledge distillation allows transferring information from a skeleton teacher to a single-IMU student, but the large modality gap has limited the success of prior teacher–student approaches. To address this issue, we propose a teacher-assistant-student (TAS) learning framework, in which a multi-IMU assistant model bridges the skeleton-based teacher and the single-IMU student. To support TAS, we develop the following techniques: (i) Dense temporal Contrastive Learning, aligning structural representations of skeleton and IMU segments; (ii) Spatial Relationship Learning, guiding models to capture spatial priors from skeleton data; and (iii) Temporal Attention Transfer, distilling attention patterns for key atomic actions. We further boost the robustness to behavioral variation with motion-guided IMU data diversification using physics-based simulation. Experiments on industrial HAR sensor data show that our framework consistently improves single-IMU recognition across diverse operational scenarios, highlighting its potential for practical deployment.

Foundation models pretrained on large-scale unlabeled data using self-supervised learning (SSL) have demonstrated remarkable capabilities in natural language processing (NLP) and computer vision (CV). Applying this paradigm to sensor-based Human Activity Recognition (HAR) holds promise for robust and generalizable models, but also poses unique challenges due to diverse sensor mounting positions and application domains. Despite this potential, unlike in NLP and CV, there has been a lack of systematic evaluation frameworks in HAR to rigorously assess the generalization and adaptation capabilities of foundation models across these critical dimensions. To address this gap, we introduce HARBench, a comprehensive benchmark designed to evaluate HAR foundation models along key axes, including Domain Robustness, Position Robustness, Few-shot Performance, and Zero-shot Performance. By leveraging a large collection of datasets from multiple domains, totaling over 2.4 million hours of sensor data, HARBench enables evaluation of HAR foundation models from multiple perspectives, providing a thorough and standardized assessment of their capabilities. We conduct large-scale evaluation and analysis of existing HAR approaches, ranging from conventional supervised methods to state-of-the-art SSL models. Our results quantitatively map the capabilities and limitations of current models, thereby providing guidance for future research and the development of next-generation foundation models for HAR. HARBench is publicly released to support the HAR research community, promoting reproducibility and accelerating the development of robust foundation models.

This study proposes a new deep learning method for reconstructing depth images of moving objects within a specific area using Wi-Fi channel state information (CSI). The Wi-Fibased depth imaging technique has novel applications in domains such as security and elder care. However, reconstructing depth images from CSI is challenging because it is difficult to learn the mapping function between CSI and depth images, both of high-dimensionalities. To address the challenge, we propose Wi-Depth. The main idea behind the design of Wi-Depth is that a depth image of a moving object can be decomposed into three core components: the shape, depth, and position of the target. Therefore, in the depth-image reconstruction task, Wi-Depth simultaneously estimates the three core pieces of information as auxiliary tasks in our proposed VAE-based teacher-student architecture, enabling it to output images with the consistency of a correct shape, depth, and position. In addition, the design of Wi-Depth is based on our idea that this decomposition efficiently takes advantage of the fact that shape, depth, and position relate to primitive information inferred from CSI such as angle-of-arrival, time-of-flight, and Doppler frequency shift.

Spatio-temporal forecasting addresses the prediction of phenomena that evolve over space and time. A key application is headcount prediction, which enables socially beneficial functions, such as the intelligent control of HVAC systems, by estimating occupancy within defined areas. Conventional headcount-prediction methods rely on sensor-derived data that are often corrupted by noise and missing values. Therefore, existing studies require extensive preprocessing, including data cleaning, to mitigate these issues. To overcome this limitation, we propose a robust spatio-temporal graph convolutional network (RSTGCN) that integrates a reliability-weighting module into the feature-extraction process, thereby attenuating the influence of unreliable measurements. Experiments show that our proposed model, RSTGCN, achieves superior forecasting performance on unprocessed real-world datasets containing noise and missing values.

Optimizing building energy consumption is essential for advancing global sustainability goals. Data-driven occupancy estimation using non-intrusive sensors plays a vital role in this effort. However, large-scale deployment remains limited due to a fundamental challenge: existing models fail to generalize across diverse environments, performing poorly under domain shift caused by variations in physical dynamics and sensor behavior unique to each space. In this study, we present a novel learning strategy centered on a dynamic peak-aware loss (DPL) to address this issue within a strict zero-shot framework. Our method directly addresses the core physical problem of sensor lag, where variables, such as$C O_{2}$respond slowly to occupancy changes, by asymmetrically penalizing estimation errors. This process trains the model to compensate for delayed sensor readings, particularly for complex occupancy patterns with multiple peaks and valleys. Evaluated on a public real-world dataset, our learning-centric approach demonstrated model-agnostic effectiveness, consistently enhancing the performance of diverse backbone models and, notably, surpassing fully supervised baselines. Therefore, this study presents a robust and scalable solution that bridges the gap between data-driven models and real-world physical constraints, advancing the development of practical smart building energy management systems.

Complex work activity recognition based on wearable sensors is crucial for streamlining work processes in industrial domains. Unlike basic activities such as walking or running, which involve simple repetitive motions, a complex work activity consists of discrete atomic actions such as an action of spreading a shipping label or cutting tape in a packaging task. In addition, the atomic actions sometimes involve characteristic short-term sensor data patterns. In addition, these actions can be performed in different orders by different workers to achieve similar outcomes, resulting in different long-term sensor data trends for different workers. Because multilayer networks for activity recognition may learn short-term features from shallow-level layers and long-term trends from deeper layers, we propose a new transfer learning method called multilevel knowledge transfer (MLKT), which performs level-wise source selection according to trend similarity across workers in different levels. For example, for training shallow layers, highly similar workers are selected for specific short motions (e.g., pasting a shipping label), and to train the deeper layers, workers with similar cadence are selected. This method also enables the adaptive thresholding of source data selection for each layer level during network training using the proposed adaptive level-wise discerning module.

This study proposes a new deep learning method for hand-washing activity recognition using a series of hand skeleton data extracted from an RGB-D camera. Assessment of hand-washing activity based on recognized hand-washing steps is crucial in both industrial and medical domains, as well as in promoting healthy habits. However, recognizing hand-washing activities presents unique challenges compared to typical activity recognition for a single person due to the specific nature of hand-washing tasks. First, the steps of hand-washing can be better explained by the interaction between objects, i.e., the two hands, such as rubbing palms and fingers. Second, occlusion occurs much more frequently during hand-washing due to the frequent interaction between both hands. Therefore, we propose a new neural network tailored for hand-washing recognition called InterHandNet to address these challenges. To capture the interaction, we propose two novel modules in InterHandNet: Interaction Graph and Interaction Attention. These modules enable to exchange information across skeleton graphs of the two hands within a graph neural network framework and to focus on important keypoints in one hand by referencing the other hand through the query-key-value mechanism, respectively. To address the issue of missing data caused by occlusion, we propose Inter-hand Temporal Fusion, which fills in the missing information by referencing data from the other hand and other time steps within a time window. InterHandNet outperforms other state-of-the-art skeleton-based and RGB-based methods in terms of accuracy, and significantly surpasses RGB-based methods in runtime efficiency on edge devices.

Owing to the cost of collecting labeled sensor data, self-supervised learning (SSL) methods for human activity recognition (HAR) that effectively use unlabeled data for pretraining have attracted attention. However, applying prior SSL to COMPLEX activities in real industrial settings poses challenges. Despite the consistency of work procedures, varying circumstances, such as different sizes of packages and contents in a packing process, introduce significant variability within the same activity class. In this study, we focus on sensor data corresponding to characteristic and necessary actions (sensor data motifs) in a specific activity such as a stretching packing tape action in an assembling a box activity, and propose to train a neural network in self-supervised learning so that it identifies occurrences of the characteristic actions, i.e., Motif Identification Learning (MoIL). The feature extractor in the network is subsequently employed in the downstream activity recognition task, enabling accurate recognition of activities containing these characteristic actions, even with limited labeled training data. The MoIL approach was evaluated on real-world industrial activity data, encompassing the state-of-the-art SSL tasks with an improvement of up to 23.85% under limited training labels.

Unlike human daily activities, existing publicly available sensor datasets for work activity recognition in industrial domains are limited by difficulties in collecting realistic data as close collaboration with industrial sites is required. This also limits research on and development of methods for industrial applications. To address these challenges and contribute to research on machine recognition of work activities in industrial domains, in this study, we introduce a new large-scale dataset for packaging work recognition called OpenPack. OpenPack contains 53.8 hours of multimodal sensor data, including acceleration data, keypoints, depth images, and readings from IoT-enabled devices (e.g., handheld barcode scanners), collected from 16 distinct subjects with different levels of packaging work experience. We apply state-of-the-art human activity recognition techniques to the dataset and provide future directions of complex work activity recognition studies in the pervasive computing community based on the results. We believe that OpenPack will contribute to the sensor-based action/activity recognition community by providing challenging tasks. The OpenPack dataset is available at https://open-pack.github.io.

This seminar introduces recent trends in sensor-based activity recognition technology. Technology to recognize human activities using sensors has been a hot topic in the field of mobile and ubiquitous computing for many years. Recent developments in deep learning and sensor technology have expanded the application of activity recognition to various domains such as industrial and natural science fields. However, because activity recognition in the new domains suffers from various real problems such as the lack of sufficient training data and complexity of target activities, new solutions have been proposed for the practical problems in applying activity recognition to real-world applications in the new domains. In this seminar, we introduce recent topics in activity recognition from the viewpoints of (1) recent trends in state-of-the-art machine learning methods for practical activity recognition, (2) recently focused domains for human activity recognition such as industrial and medical domains and their public datasets, and (3) applications of activity recognition to the natural science field, especially in animal behavior understanding.

In this study, we proposed a new method for pedestrian safety named ObsSense that estimates the distance and relative velocity of both static and mobile roadside obstacles using probing signals emitted by an off-the-shelf smartphone carried by a pedestrian. Sound-based estimation of the movement properties of an obstacle has been actively studied in pervasive computing, and generally provides the distance from a smartphone user to an obstacle and the relative velocity of the obstacle. However, most prior studies have focused on estimating the properties of either static or mobile obstacles. Furthermore, they estimated either the distance to obstacles or their relative velocity. Nonetheless, both the distance and relative velocity of the oncoming obstacles are necessary to derive the timing of a possible collision. To address this issue, we proposed a new probing signal composed of sine waves and sine sweeps, designed to acquire information about both the distance and relative velocity of static and mobile obstacles. In addition, we proposed a neural network model that can jointly predict distance and relative velocity from reflected sounds of the probing signal captured by a smartphone. The proposed neural network was designed to consider the mobility status of obstacles, i.e., whether they are mobile or static, in order to select the most appropriate method to estimate their distance and relative velocity. In addition, our method uses the relationship between distance and velocity to increase the precision of its estimations. For example, the distance can be estimated by integrating the velocity-time curve. The performance of ObsSense was evaluated using real-world data, and the experimental results demonstrated the effectiveness of our proposed probing signal and neural network architecture.

Owing to the recent proliferation of smartphones and the SNS, a large number of images taken by smartphones at various places have been uploaded to SNSs. In addition, smartphones are equipped with various sensors such as Wi-Fi modules that enable us to generate an image associated with the sensory information that represents the context in which the image was captured. This study demonstrates the benefits of images associated with Wi-Fi signals in the automated construction of a Wi-Fi-based indoor logical location classifier that predicts a semantic location label of a user’s position for shopping complexes. In this study, a logical location class refers to the store class label in a shopping complex, such as Starbucks and H&M. Given a collection of images associated with Wi-Fi signals taken at a shopping complex and the complex’s floor plan, the proposed method first estimates the store label at which an image was taken by analyzing the image and crawled online images of branch stores. Then, the 2D coordinates of the images taken at branch stores on the floor coordinate system can be estimated using the floor plan. Subsequently, by using the Wi-Fi signals of the branch store images and their estimated 2D coordinates, we construct a transformation function that maps Wi-Fi signals onto the 2D coordinates, and we adopt this function to predict an indoor location class of an observed Wi-Fi scan from a smartphone possessed by an end user. The proposed transformation function comprises an ensemble of sub-functions designed based on CVAEs. Finally, we demonstrate the effectiveness of the proposed method for three actual shopping complexes.

This study presents a new method for recognizing complex human activities within the logistics domain, such as packaging operations, using acceleration data from a body-worn sensor. The recognition of packaging tasks using standard supervised machine learning is complex because the observed data vary considerably depending on the number of items to be packed, the size of the items, and other parameters. In this study, we focused on the characteristics and necessary key actions (motions) that occur during a specific operation. For instance, when the packaging tape is stretched while assembling the shipping boxes. To focus on these characteristic actions when recognizing data, we propose the use of an attention-based neural network. With our method, the attention-based neural network’s training is guided such that its focus is on the motifs. In addition, this method was designed to accurately recognize short operations by leveraging data augmentation techniques. We tested our method on two logistics datasets and achieved a 3.9% improvement over the previous MGA-Net approach.

This study presents a new method for recognizing complex human activities in a logistical domain, such as packaging, using acceleration data from a body-worn sensor. Recognition of packaging tasks using standard supervised machine learning is difficult because the observed data vary considerably depending on the number of items to pack, the size of the items, and other parameters. In this study, we focus on characteristic and necessary actions (motions) that occur in a specific operation such as an action of stretching packing tape when assembling shipping boxes. We propose the use of an attention-based neural network to focus on these characteristic actions when recognizing the data. However, training of a such deep network model is a data-intensive process, and obtaining a huge amount of labeled training data in actual industrial settings is difficult. To address this problem, we employ motif-detection algorithms to detect sensor data motifs (segments corresponding to characteristic actions) that can be useful for recognizing operations in advance. Moreover, we propose that the training of the attention-based network should be guided such that it pays attention to the detected motifs, i.e., motif-guided training.

Recommender systems (RSs) have been extensively studied in academia and industry, while few works focus on flash sale recommendations. In flash sale scenarios, period-specific high discounts are applied on ordinal sales during each flash sale period to attract users. According to periodic sales strategies, available and discounted items change significantly across periods. Users are attracted by the high discounts and show a period-specific preference. Besides, the frequently changed available items provoke a cold-start problem, i.e., an impaired recommendation caused by lacking interactions. However, most existing RSs either cannot handle users’ period-specific preferences or suffer from the cold-start problem. Therefore, this work proposes a novel meta-learning-based RS to mitigate the cold-start problem and simultaneously handle users’ period-specific preferences. Moreover, we introduce a novel hierarchical meta-training algorithm to guide the learning of our recommendation model via period-and user-specific gradients. In this way, the learned model contains user- and period-shared knowledge and can fast adapt to the recommendations for new flash sale periods and users. To evaluate the effectiveness of our system in flash sale recommendations and non-flash sale recommendations, we conduct experiments on a real-world flash sale e-commerce dataset and a widely used recommendation dataset, considering both warm and cold scenarios. The experimental results show that our proposed model is remarkably improved over the current state-of-the-art methods in flash sale recommendations and most of the non-flash sale cold-start recommendations.

Trajectory analysis has been a central problem in applications of location tracking systems. Recently, the (discrete) Fréchet distance becomes a popular approach for measuring the similarity of two trajectories because of its high feature extraction capability. Despite its importance, the Fréchet distance has several limitations: (i) sensitive to noise as a trade-off for its high feature extraction capability; and (ii) it cannot be incorporated into machine learning frameworks due to its non-smooth functions. To address these problems, we propose the Fréchet kernel (FRK), which is associated with a smoothed Fréchet distance using a combination of two approximation techniques. FRK can adaptively acquire appropriate extraction capability from trajectories while retaining robustness to noise. Theoretically, we find that FRK has a positive definite property, hence FRK can be incorporated into the kernel method. We also provide an efficient algorithm to calculate FRK. Experimentally, FRK outperforms other methods, including other kernel methods and neural networks, in various noisy real-data classification tasks.

Although deep learning-based activity recognition using wearable sensors has been actively studied to implement smart applications such as supporting elderly care, healthcare, and home automation, techniques for understanding the inside of activity recognition networks have not yet been investigated thoroughly. In the computer vision research field, activation maximization (AM) was proposed to visualize the internal functions of networks. However, when conventional AM techniques, which are tailored to image-based recognition, are directly applied to acceleration-based activity recognition networks, meaningless and noisy signals are generated because of the difficulties in regularizing AM by directly using the values of the acceleration signals that are generated. This study proposes novel regularization techniques for AM using activity recognition networks that leverage activation values used in AM to indirectly control the acceleration signals that are generated. We evaluated the proposed method quantitatively using publicly available datasets and confirmed the effectiveness of the proposed techniques.

In recommender systems, a cold-start problem occurs when there is no past interaction record associated with the user or item. Typical solutions to the cold-start problem make use of contextual information, such as user demographic attributes or product descriptions. A group of works have shown that social media background can help predicting temporal phenomenons such as product sales and stock price movements. In this work, our goal is to investigate whether social media background can be used as extra contextual information to improve recommendation models. Based on an existing deep neural network model, we proposed a method to represent temporal social media background as embeddings and fuse them as an extra component in the model. We conduct experimental evaluations on a real-world e-commerce dataset and a Twitter dataset. The results show that our method of fusing social media background with the existing model does generally improve recommendation performance. In some cases the recommendation accuracy measured by hit-rate@K doubles after fusing with social media background. Our findings can be beneficial for future recommender system designs that consider complex temporal information representing social interests.

This paper presents a new human activity recognition method that uses a camera mounted on a mobile robot. We assume that the robot's camera captures images of a person and recognizes his/her activities based on skeletal and visual features extracted from the images. A key issue encountered with this method for activity recognition is that it requires the robot to position itself so that it has an adequate field of view of the activities being conducted. For example, if the robot is directly behind a person while observing that person making tea, it will be difficult for the robot to distinguish that activity from other similar activities such as preparing a meal or washing dishes. Our method employs deep reinforcement learning to control the movements of the mobile robot that is observing the activities in order to maximize its recognition accuracy while minimizing its energy consumption related to its movement. We propose effective action- and state-space designs that can achieve early training convergence and highly accurate activity recognition by: (i) incorporating the confidence of the activity recognition output when evaluating the quality of the current state (position), (ii) incorporating the costs of subsequent actions when estimating values for those actions, and (iii) designing an effective action space that accelerates reinforcement learning by restricting the movement space of the robot to the circumference of a circle with a predefined radius centered on the person.

Nowadays, advertising (ad) deliveries are conducted in a targeted manner to improve their effectiveness and efficiency. However, human behavior data in ad-platforms such as Web browsing history is complex and contains a lot of “noise”. On the other hand, information in the advertiser's domain (e.g. e-commerce sites) seems to contain less noise (e.g. product browsing history) with respect to ad-targeting. We introduce a new denoising method for behavioral ad-targeting based on the idea of feature distribution alignment induced by the advertiser's domain. This denoised distribution induction can be achieved by employing domain adversarial training with stabilization techniques. We evaluate our model on real world data originating from an e-commerce site and an ad-platform. The results of an ablation study have demonstrated the advantage of utilizing an advertiser's domain for denoising human behavior data of an ad-platform domain.

In terms of nonlinear machine learning classifier such as Deep Learning, machine-learning model is generally a black box which has issue not to be clear the causality among its output classification and input attributes. In this paper, we propose a causality analysis method with self-organizing map and locally approximation to linear model. In this method, self-organizing map generates the cluster of input data and local linear models for each node on the map provides explanation of the generated model. Applying this method to the member rank prediction model based on Deep Learning, we validated our proposed method.

Inertial sensor data collected from wearable smart devices such as smartwatches are expected to be used in various smart applications such as video game controllers, hand drawing, hand writing, gestural input devices, human activity recognition, and remote communication using sign language. However, since the maximum sampling rate of inertial sensors in commercial smartwatches is restricted, capturing fine-grained body movements using the low-sampled signals is difficult for these sensors. Therefore, this study proposes a new method for generating high sampling rate signals from the low-sampled signals by upsampling the low-sampled signals using interpolation with an artificial neural network. Because it is impossible to obtain "non-existent" data from low-sampled signals according to the information theory, we estimate these data from experience, i.e., using high-sampled signals prepared in advance for training. This is possible because trajectories of a sensor are restricted by the skeletal structure of the body part to which the sensor is attached.

Recognizing activities of daily living (ADLs) in the real world is an important task for understanding everyday human life. However, even though our life events consist of chronological ADLs with the corresponding places and objects (e.g., drinking coffee in the living room after making coffee in the kitchen and walking to the living room), most existing works focus on predicting individual activity labels from sensor data. In this paper, we introduce a novel framework that produces an event timeline of ADLs in a home environment. The proposed method combines semantic concepts such as action, object, and place detected by sensors for generating stereotypical event sequences with the following three real-world properties. First, we use temporal interactions among concepts to remove objects and places unrelated to each action. Second, we use commonsense knowledge mined from a language resource to find a possible combination of concepts in the real world. Third, we use temporal variations of events to filter repetitive events, since our daily life changes over time. We use cross-place validation to evaluate our proposed method on a daily-activities dataset with manually labeled event descriptions. The empirical evaluation demonstrates that our method using real-world properties improves the performance of generating an event timeline over diverse environments.

Behavioral targeting (BT) is an important function for a company to reach a wide range of potential users. Trans-domain BT which targets potential users of one (source) domain (e.g. E-Commerce) who lie in another (target) domain (e.g. Ad Network) is a promising method to expand the range. However, it is difficult for trans-domain BT to keep its targeting quality high in case when ID linkage across domains is limited. To realize high quality trans-domain BT with limited ID linkage, we propose a method to cross-connect private touchpoints to users in each domain, which we call Virtual Touch-Point (VTP). Here, we utilize transfer learning to acquire knowledge to tie two domains. We made a VTP prototype by implementing typical transfer learning algorithms and evaluated its effectiveness using real-world data of two domains: (source) E-Commerce → (target) Ad Network.

This paper presents a robust machine learning-based method for detecting malicious nodes in mobile ad hoc networks (MANETs). Since general machine learning methods rely on training data, trained detectors do not work well in test environments that are different from training environments. This is an inherent problem in malicious detection in dynamic MANETs environments, where network parameters, such as the average node speed and density of nodes, differ in each environment. In this study, we propose a method for environment-adaptive malicious node detection based on ensemble learning. We first prepare weak malicious node detectors trained in diverse environments, and then construct a strong ensemble malicious node detector, which is tailored to a given test environment, by fusing weak detectors whose performances are estimated to be high in the test environment. We investigate the performance of our method and confirm that our method significantly outperforms the state-of-the-art methods in terms of detection accuracy and false detection rate.

This study shows a preliminary investigation of daily activity recognition based on a wearable camera without using training data prepared by a user in her environment. Recently, deep learning frameworks have been publicly available, and we can now easily use deep convolutional neural networks (CNNs) pre-trained on a large image data set. In our method, we first detect objects used in the user's activity from her first-person images using a pre-trained CNN for object recognition. We then estimate an activity of the user using the object detection result because objects used in an activity strongly relate to the activity. To estimate the activity without using training data, we utilize knowledge on the Web because the Web is a repository of knowledge that reflects real-world events and common sense. Specifically, we compute semantic similarity between a list of the detected object names and a name of each activity class based on the Web knowledge. The activity class with the largest similarity value is the estimated activity of the user.

We propose a language-independent datadriven method to exhaustively extract bursty phrases of arbitrary forms (e.g., phrases other than simple noun phrases) from microblogs.The burst (i.e., the rapid increase of the occurrence) of a phrase causes the burst of overlapping Ngrams including incomplete ones.In other words, bursty incomplete N-grams inevitably overlap bursty phrases.Thus, the proposed method performs the extraction of bursty phrases as the set cover problem in which all bursty N-grams are covered by a minimum set of bursty phrases.Experimental results using Japanese Twitter data showed that the proposed method outperformed word-based, noun phrase-based, and segmentation-based methods both in terms of accuracy and coverage.

Indoor positioning is a core technology for indoor daily life applications such as elderly care and home automation. This study presents a device-free and privacy preserving indoor positioning method using infrared (IR) cameras and retroreflectors. The proposed method uses IR cameras equipped with IR LEDs to capture retroreflections from markers attached to walls in the environment, and detects a person who passes between the camera and a marker by observing the occlusion. Because our method employs occlusion of markers, it can track a person without attaching tags to the person. Also, our camera device permits us to filter out visible light and thus the appearance of the person is not recorded. Our evaluation in real environments showed that our method achieved an average positioning error of about 0.3 meters.

Retrieving past egocentric videos about personal daily life is important to support and augment human memory. Most previous retrieval approaches have ignored the crucial feature of human-physical world interactions, which is greatly related to our memory and experience of daily activities. In this paper, we propose a gesture-based egocentric video retrieval framework, which retrieves past visual experience using body gestures as non-verbal queries. We use a probabilistic framework based on a canonical correlation analysis that models physical interactions through a latent space and uses them for egocentric video retrieval and re-ranking search results. By incorporating physical interactions into the retrieval models, we address the problems resulting from the variability of human motions. We evaluate our proposed method on motion and egocentric video datasets about daily activities in household settings and demonstrate that our egocentric video retrieval framework robustly improves retrieval performance when retrieving past videos from personal and even other persons' video archives.

We propose a method for selecting home appliances using a smart glass, which facilitates the control of network-connected appliances in a smart house. Our proposed method is image-based appliance selection and enables smart glass users to easily select a particular appliance by just looking at it. The main feature of our method is that it achieves high precision appliance selection using user contextual information such as position and activity, inferred from various sensor data in addition to camera images captured by the glass because such contextual information is greatly related in the home appliance that a user wants to control in her daily life. We design a state-of-the-art appliance selection method by fusing image features extracted by deep learning techniques and context information estimated by non-parametric Bayesian techniques within a framework of multiple kernel learning. Our experimental results, which use sensor data obtained in an actual house equipped with many network-connected appliances, show the effectiveness of our method.

In a line production system of a factory, a worker repetitively performs predefined operation processes. This paper tries to recognize work by factory workers in an unsupervised manner. Specifically, we propose an unsupervised measurement method for estimating lead time (duration) of each period of an operation process using a wrist-worn accelerometer because the lead time greatly affects productivity of the line production system. Our proposed method automatically finds a frequent sensor data segment as a "motif" that occurs once in each operation period using only prior knowledge about predefined standard lead time of the operation process, and uses the occurrence intervals of the motif to estimate the lead time. We evaluated our method using real factory data and the estimation error was only about 3.5%.

This paper presents a method for estimating a user's indoor location without using training data collected by the user in his/her environment. Specifically, we attempt to predict the user's location semantics, i.e., location classes such as restroom and meeting room. While indoor location information can be used in many real-world services, e.g., context-aware systems, lifelogging, and monitoring the elderly, estimating the location information requires training data collected in an environment of interest. In this study, we combine passive sensing and active sound probing to capture and learn inherent sensor data features for each location class using labeled training data collected in other environments. In addition, this study modifies the random forest algorithm to effectively extract inherent sensor data features for each location class. Our evaluation showed that our method achieved about 85% accuracy without using training data collected in test environments.

This paper presents novel unconstrained person identification for tabletop systems using a ceiling-mounted depth camera that overlooks a table. Recent state-of-the-art ubicomp, computer-vision, and CSCW studies have tried to recognize a user's activities and actions on a table using a ceiling-mounted device that overlooks the table. However, conventional unconstrained person identification methods such as face identification cannot be used for providing personalized services in such settings. In this study, we focus on a user's soft biometrics that can be captured from the ceiling such as the shoulder length, shape of the head, and posture of the back to achieve unconstrained person identification by using a ceiling-mounted depth camera. We achieve robust person identification by combining the soft biometrics within a framework of multiview learning. Multiview learning allows us to deal effectively with data consisting of features from multiple sources with different data distributions, i.e., multiple soft biometrics in our case. To the best of our knowledge, this is the first study that investigates the feasibility of person identification for tabletop users by a ceiling-mounted depth camera.

This paper presents a new indoor positioning system that utilizes home lighting. We design a beacon for use in the system that is inserted between a home light bulb and its socket, and is supplied with electricity from the socket. This means that a user can easily install the system in his/her environment.

This paper presents a new method for evaluating tooth brushing performance using audio collected from a smartphone. To do this, we use hidden Markov models (HMMs) to recognize audio data that include various types of tooth brushing actions, such as brushing the outer surface of the front teeth and brushing the inner surface of the back teeth. We then use the output of the HMMs to build regression models to estimate tooth brushing performance scores, such as stroke quality of brushing for the back inner teeth and duration of brushing for the front teeth. The scores used to train these regression models are obtained from a dentist who specializes in dental care instruction, with the resulting regression models estimating performance scores that closely correspond to the scores assigned by the dentist.

This paper proposes a new method that makes it easy for us to construct a positioning model for device-free passive indoor localization by using model transfer techniques. With device-free passive indoor positioning, a wireless sensor network is used to detect the movement of a person based on the fact that RF signals transmitted between a transmitter and a receiver are affected by human movement. However, because device-free passive indoor positioning relies on machine learning techniques, we must collect labeled training data at many training points in an end user's environment. This paper proposes a method that transfers a signal strength model used for locating a person obtained in another environment (source environment) to the end user environment. With the transferred models, we can construct a positioning model for the end user environment inexpensively. Our evaluation showed that our method achieved almost the same positioning performance as a supervised method that requires labeled training data obtained in an end user's environment.

In this article, we propose a new method for automatically updating a Wi-Fi indoor positioning model on a cloud server by employing uploaded sensor data obtained from the smartphone sensors of a specific user who spends a lot of time in a given environment (e.g., a worker in the environment). In this work, we attempt to track the user with pedestrian dead reckoning techniques, and at the same time we obtain Wi-Fi scan data from a mobile device possessed by the user. With the scan data and the estimated coordinates uploaded to a cloud server, we can automatically create a pair consisting of a scan and its corresponding indoor coordinates during the user's daily life and update an indoor positioning model on the server by using the information. With this approach, we try to cope with the instability of Wi-Fi-based positioning methods caused by changing environmental dynamics, that is, layout changes and moving or removal of Wi-Fi access points. Therefore, ordinary users (e.g., customers) who do not have rich sensors can benefit from the continually updating positioning model.

This paper presents a new method for estimating which outlet an electrical appliance is plugged into by using the electrical wiring installed in the building. By making use of the voltage drop caused by the wire, we can estimate the distance between the sensor and an electrical appliance plugged into an outlet on an electrical circuit. If we have a floor plan of an environment of interest showing a wiring diagram and where a sensor is attached, we can determine which outlet an electrical appliance is plugged into from the distance between the sensor and the appliance. The estimated outlet position of an appliance is very useful for understanding real-world events and developing real-world applications, e.g., providing user location and appliance location aware services, daily activity recognition, and estimating a user's indoor location through electrical appliance use under specific conditions.

This paper proposes a new Wi-Fi based indoor positioning method that is robust over unstable Wi-Fi access points (APs). Because Wi-Fi based indoor positioning relies on unstable and uncontrollable infrastructure (Wi-Fi APs), the positioning performance significantly decreases when such unstable APs are included in the localization system. This paper proposes a indoor positioning method by employing ensemble of weak position estimators, which permits us to construct a robust positioning model. Our proposed boosted position estimator has the following features. 1) The estimator does not overfit the training data and thus it is robust over unstable signals from APs. 2) Because each weak estimator employs a small number of APs for positioning, the estimator is not affected by the curse of dimensionality. 3) Our model can adaptively change the weight (importance) of each weak estimator according to a user's position in order to achieve a position-aware precise localization.

Recent advances in GPS and WiFi-based positioning technologies for mobile phones have triggered many location-based services. However, GPS positioning quickly drains a phone's battery and cannot be used indoors. On the other hand, WiFi positioning provides energy-efficient indoor and outdoor positioning with reasonable accuracy. However, WiFi positioning sometimes makes large errors caused by various reasons, e.g., the movement of reference WiFi access points. In this paper we attempt to detect and correct such errors automatically by performing outlier detection in time series.

This article proposes a new concept for a context-aware Web search method that automatically retrieves a webpage related to the daily activity that a user currently is engaged in and displays the page on nearby Internet-connected home appliances such as televisions. For example, when a user is washing a coffeemaker, a webpage is retrieved that includes tips such as “cleaning a coffee maker with vinegar removes stains well,” and the page is displayed on a nearby appliance. In this article, we design and implement a Web search method that employs ubiquitous sensors to monitor a user's daily life. Our proposed method automatically searches for a webpage related to a daily activity by using a query constructed from the use of daily objects employed in the activity that is detected with object-attached sensors. We evaluate the search method with real datasets collected from vast numbers of sensors and achieve very accurate webpage retrieval. We then investigate the usefulness and effectiveness of a daily life Web search with Wizard-of-Oz (WOz)-like experiments. We confirm that the presentation of webpages related to daily activities improves participants' future daily lives and triggers communication among the participants in the experiment.

This paper proposes an agent system that semi-automatically creates stories about daily events detected by ubiquitous sensors. These stories are knowledge of inhabitants' daily lives and it may be useful for human-friendly agent. Story flows in daily lives are extracted from interaction between sensor room inhabitants and a symbiotic agent. The agent asks causal relationships among daily events for inhabitants to create the story flow. Experimental results show that created stories let created stories perceive agent's intelligence.

The new concept proposed in this paper is a query free web search that automatically retrieves a web page including information related to the daily activity that we are currently engaged in for automatically displaying the page on Internet-connected domestic appliances around us such as televisions. When we are washing a coffee maker, for example, a web page is retrieved that includes tips such as `cleaning a coffee maker with vinegar removes stains well.' A method designed on the basis of this concept automatically searches for a web page by using a query constructed from the use of ordinary household objects that is detected by sensors attached to the objects. An in-situ experiment tests a variety of IR techniques and the experiment confirmed that our daily activities can produce related web pages with high accuracy.

"MADO Interface" is a tangible user interface consisting of a compact touch-screen display and physical blocks. "MADO" means "window" in Japanese, and MADO Interface is utilized as the real window into the virtual world. Users construct a physical object by simply combining electrical blocks. Then, by connecting MADO Interface to the physical object, they can watch the virtual model corresponding to the physical block configuration (shape, color, etc.) The size and the viewpoint of the virtual model seen by the user depend on the position of MADO Interface, maintaining the consistency between the physical and virtual worlds. In addition, users can interact with the virtual model by touching the display on MADO Interface. These features enable users to explore the virtual world intuitively and powerfully.

We propose a novel interface called Tearable that allows users to continuously experience the real sense of tearing paper. To provide such a real sense, we measured the actual vibration data of tearing a piece of real paper and analyzed them. Based on this data, we utilized hook-and-loop fasteners and a DC motor for representing the sense of tearing. We compared the force given by Tearable with that by a piece of real paper and recommended its reproducibility and usability. In addition, we evaluated Tearable with questionnaires after user experiences.

This paper proposes new concept of query free web search for daily living. We ordinarily benefit from additional information about our daily activities that we are currently engaged in. When washing a coffee maker, for example, we receive the benefit if we obtain such information as 'cleaning a coffee maker with vinegar removes its stain well.' Our proposed method automatically searches for a web page including such information relates to an activity of daily living when the activity is performed. We assume that wireless sensor nodes are attached to daily objects to detect object use; our method makes a query from the names of objects which are used. Then, the method retrieves a web page relates to the activity of daily living by using the query.

To build a cooperative logistics network covering multiple enterprises, a planning method that can build a long-distance transportation network is required. Many strict constraints are imposed on this type of problem. To solve these strict-constraint problems, a selfish-constraint-satisfaction type genetic algorithm (GA) is proposed. Moreover, a constraint pre-checking method is also applied. Our experimental result shows that the proposed method can obtain an accurate solution in a practical response time.

The environment generated media (EGM) are defined here as being generated from a massive amount of and/or incomprehensible environmental data by compressing them into averages or representative values and/or by converting them into such user-friendly media as text, figures, charts, and animations. As an application of EGM, an object-participation-type weblog is introduced, where anthropomorphic indoor objects with sensor nodes post weblog entries and comments about what happened to them in a sensor networked environment.

A method of solving the vehicle routing problem (VRP) that can obtain a highly optimal solution to achieve steady logistics operation is required to optimize a cooperative logistics network. To satisfy this requirement, a multi-stage GA and a method of multipoint evaluation are proposed. The multi-stage GA enables us to obtain an accurate solution under various conditions. Moreover, the method of multi-point evaluation can generate a delivery schedule that enables stable logistics operation. The experimental results revealed the method we propose can obtain accurate solutions to achieve stable operation.

In this paper, we introduce two our approaches to browse large Web pages designed for desktop PCs using mobile devices with small screens and poor input interfaces. In the first approach, multiple mobile users collaboratively browse large Web pages. We call this approach collaborative browsing. In the second approach, automatic scrolling in a large Web page is performed, where contents in the page are traversed. This enables users to rapidly view the page with the users’ minimum operations.

GeoNote.net is a social network system which allows users to share geographic information among their friends network. The system has various notable features and uses several key technologies such as social network construction methods, security for personal information, geographic information management on mobile devices including cellular phones, retrieval of geographic information according to the friends network, and so on. In this paper, we describe the system under design aspects and implementation aspects.

Recently, there has been an increasing interest in a social network. In a social network, nodes and links represent participants and their friendships, respectively. We have designed and implemented a query propagation mechanism and its applications to realize a social network composed by cellular phone users. In these applications, users can retrieve information from their friends or their friends’ friends by propagating the query in the network. To propagate a query in a wide range and improve the query success ratio, most users who receive the query must relay it to all their friends. However, this increases communication packets. In this paper, we propose a query routing method to decrease the number of communication packets by using user profiles.

In mobile computing environments, handheld devices with low functionality restrict the services provided for mobile users. We propose a new concept of collaborative browsing, where mobile users collaboratively browse Web pages designed for desktop PC. In collaborative browsing, a Web page is divided into multiple components, and each is distributed to a different device. In mobile computing environments, the number of handheld devices, their capabilities, and other conditions can vary widely amongst mobile users who want to browse content. Therefore, we developed a page partitioning method for collaborative browsing, which divides a Web page into multiple components. Moreover, we designed and implemented a collaborative Web browsing system in which users can search and browse their target information by discussing and watching partial pages displayed on multiple devices

It is difficult for users of mobile devices such as cellular phones equipped with a small screen and a poor input interface to browse Web pages designed for desktop PCs with large displays. Many studies and commercial products have tried to solve this problem. Web pages include images that have various roles such as site menus, line headers for itemization, and page titles. However, most studies of mobile Web browsing haven't paid much attention to the roles of Web images. In this paper, we define eleven Web image categories according to their roles and use these categories for proper Web image handling. We manually categorized 3,901 Web images collected from forty Web sites and extracted image features of each category according to the classification. By making use of the extracted features, we devised an automatic Web image classification method. Furthermore, we evaluated the automatic classification of real Web pages and achieved up to 83.1% classification accuracy. We also implemented an automatic Web page scrolling system as an application of our automatic image classification method.