Maintaining adequate situation awareness (SA) is crucial for the safe operation of conditionally automated vehicles (AVs), which requires drivers to regain control during takeover (TOR) events. This study developed a predictive model for real-time assessment of driver SA using multimodal data (e.g., galvanic skin response, heart rate and eye tracking data, and driver characteristics) collected in a simulated driving environment. Sixty-seven participants experienced automated driving scenarios with TORs, with conditions varying in risk perception and the presence of automation errors. Using data from forty-four participants (twenty-three of those had invalid data) a LightGBM (Light Gradient Boosting Machine) model trained on the top 12 predictors identified by SHAP (SHapley Additive exPlanations) achieved promising performance with RMSE = 0.89, M AE = 0.71, and Corr = 0.78. These findings have implications towards context-aware modeling of SA in conditionally automated driving, paving the way for safer and more seamless driver–AV interactions.

A particular concern with SAE Level 3 automation is the takeover transition from the automated vehicle to the human driver. In response, research has focused on investigating this transition. However, researchers have used a wide range of metrics to measure takeover performance. The lack of consistency in these metrics poses challenges for synthesizing findings. To address this issue, we conducted a systematic literature review of studies published between January 2009 and December 2019, focusing on the takeover performance metrics. Following prior research, we categorize these metrics into two dimensions: timeliness and quality. Additionally, we summarize the scenarios used to elicit takeover requests and analyze the corresponding maneuvers (braking, lane changing, and lane keeping). The results have shown inconsistencies in calculation and naming conventions of takeover performance metrics. Based on these findings, this study proposes several directions for standardizing definitions and terminology, and advancing toward a unified measure of takeover performance.

In mixed traffic environments that involve conventional vehicles (CVs) and autonomous vehicles (AVs), it is essential for CV drivers to maintain an appropriate level of situation awareness (SA) to ensure safe and efficient interactions with AVs. While previous research has established the benefits of external human–machine interfaces (HMIs) for communicating AV intent, this study extended this knowledge by focusing on the vital but underexplored interaction with CV drivers. Specifically, we investigated how AV communication through HMIs affected CV drivers by systematically comparing internal (iHMI) and external (eHMI) interfaces, and examined their impact on CV driver awareness, cognitive load, and behavior. Initially, we designed eight HMI concepts through a human-centered design process. The two highest-rated concepts were selected for implementation as eHMIs and iHMIs. Subsequently, we designed a within-subjects experiment with three conditions: a control condition without any communication HMI, and two treatment conditions using eHMIs and iHMIs as communication means. We investigated the effects of these conditions on 50 participants in a virtual environment (VR) driving simulator. Self-reported assessments and eye-tracking measures were employed to evaluate participants’ SA, trust, acceptance, and mental workload. Results indicated that the iHMI condition resulted in superior SA among participants and improved trust in the AV compared to the control and eHMI conditions. Additionally, iHMI led to a comparatively lower increase in mental workload compared to the other two conditions. Our study contributes to the development of effective AV-CV communications and has the potential to inform the design of future AV systems.

Brain disorder diagnosis via resting-state functional magnetic resonance imaging (rs-fMRI) is usually limited due to the complex imaging features and sample size. For brain disorder diagnosis, the graph convolutional network (GCN) has achieved remarkable success by capturing interactions between individuals and the population. However, there are mainly three limitations: 1) The previous GCN approaches consider the non-imaging information in edge construction but ignore the sensitivity differences of features to non-imaging information. 2) The previous GCN approaches solely focus on establishing interactions between subjects (i.e., individuals and the population), disregarding the essential relationship between features. 3) Multisite data increase the sample size to help classifier training, but the inter-site heterogeneity limits the performance to some extent. This paper proposes a knowledge-aware multisite adaptive graph Transformer to address the above problems. First, we evaluate the sensitivity of features to each piece of non-imaging information, and then construct feature-sensitive and feature-insensitive subgraphs. Second, after fusing the above subgraphs, we integrate a Transformer module to capture the intrinsic relationship between features. Third, we design a domain adaptive GCN using multiple loss function terms to relieve data heterogeneity and to produce the final classification results. Last, the proposed framework is validated on two brain disorder diagnostic tasks. Experimental results show that the proposed framework can achieve state-of-the-art performance.

Trust is crucial for ensuring the safety, security, and widespread adoption of automated vehicles (AVs), and if trust is lacking, drivers and the general public may hesitate to embrace this technology. This research seeks to investigate contextualized trust profiles in order to create personalized experiences for drivers in AVs with varying levels of reliability. A driving simulator experiment involving 70 participants revealed three distinct contextualized trust profiles (i.e.,confident copilots,myopic pragmatists, andreluctant automators) identified through K-means clustering, and analyzed in relation to drivers' dynamic trust, dispositional trust, initial learned trust, personality traits, and emotions. The experiment encompassed eight scenarios where participants were requested to take over control from the AV in three conditions: a control condition, a false alarm condition, and a miss condition. To validate the models, a multinomial logistic regression model was constructed using the shapley additive explanations explainer to determine the most influential features in predicting contextualized trust profiles, achieving an F1-score of 0.90 and an accuracy of 0.89. In addition, an examination of how individual factors impact contextualized trust profiles provided valuable insights into trust dynamics from a user-centric perspective. The outcomes of this research hold significant implications for the development of personalized in-vehicle trust monitoring and calibration systems to modulate drivers' trust levels, thereby enhancing safety and user experience in automated driving.

Image fusion technology aims to obtain a comprehensive image containing a specific target or detailed information by fusing data of different modalities. However, many deep learning-based algorithms consider edge texture information through loss functions instead of specifically constructing network modules. The influence of the middle layer features is ignored, which leads to the loss of detailed information between layers. In this article, we propose a multidiscriminator hierarchical wavelet generative adversarial network (MHW-GAN) for multimodal image fusion. First, we construct a hierarchical wavelet fusion (HWF) module as the generator of MHW-GAN to fuse feature information at different levels and scales, which avoids information loss in the middle layers of different modalities. Second, we design an edge perception module (EPM) to integrate edge information from different modalities to avoid the loss of edge information. Third, we leverage the adversarial learning relationship between the generator and three discriminators for constraining the generation of fusion images. The generator aims to generate a fusion image to fool the three discriminators, while the three discriminators aim to distinguish the fusion image and edge fusion image from two source images and the joint edge image, respectively. The final fusion image contains both intensity information and structure information via adversarial learning. Experiments on public and self-collected four types of multimodal image datasets show that the proposed algorithm is superior to the previous algorithms in terms of both subjective and objective evaluation.

Trust calibration poses a significant challenge in the interaction between drivers and automated vehicles (AVs) in the context of human-automation collaboration. To effectively calibrate trust, it becomes crucial to accurately measure drivers’ trust levels in real time, allowing for timely interventions or adjustments in the automated driving. One viable approach involves employing machine learning models and physiological measures to model the dynamic changes in trust. This study introduces a technique that leverages machine learning models to predict drivers’ real-time dynamic trust in conditional AVs using physiological measurements. We conducted the study in a driving simulator where participants were requested to take over control from automated driving in three conditions that included a control condition, a false alarm condition, and a miss condition. Each condition had eight takeover requests (TORs) in different scenarios. Drivers’ physiological measures were recorded during the experiment, including galvanic skin response (GSR), heart rate (HR) indices, and eye-tracking metrics. Using five machine learning models, we found that eXtreme Gradient Boosting (XGBoost) performed the best and was able to predict drivers’ trust in real time with an f1-score of 89.1% compared to a baseline model of$K$-nearest neighbor classifier of 84.5%. Our findings provide good implications on how to design an in-vehicle trust monitoring system to calibrate drivers’ trust to facilitate interaction between the driver and the AV in real time.

For significant memory concern (SMC) and mild cognitive impairment (MCI), their classification performance is limited by confounding features, diverse imaging protocols, and limited sample size. To address the above limitations, we introduce a dual-modality fused brain connectivity network combining resting-state functional magnetic resonance imaging (fMRI) and diffusion tensor imaging (DTI), and propose three mechanisms in the current graph convolutional network (GCN) to improve classifier performance. First, we introduce a DTI-strength penalty term for constructing functional connectivity networks. Stronger structural connectivity and bigger structural strength diversity between groups provide a higher opportunity for retaining connectivity information. Second, a multi-center attention graph with each node representing a subject is proposed to consider the influence of data source, gender, acquisition equipment, and disease status of those training samples in GCN. The attention mechanism captures their different impacts on edge weights. Third, we propose a multi-channel mechanism to improve filter performance, assigning different filters to features based on feature statistics. Applying those nodes with low-quality features to perform convolution would also deteriorate filter performance. Therefore, we further propose a pooling mechanism, which introduces the disease status information of those training samples to evaluate the quality of nodes. Finally, we obtain the final classification results by inputting the multi-center attention graph into the multi-channel pooling GCN. The proposed method is tested on three datasets (i.e., an ADNI 2 dataset, an ADNI 3 dataset, and an in-house dataset). Experimental results indicate that the proposed method is effective and superior to other related algorithms, with a mean classification accuracy of 93.05% in our binary classification tasks. Our code is available at: https://github.com/Xuegang-S.

Research indicates that monotonous automated driving increases the incidence of fatigued driving. Although many prediction models based on advanced machine learning techniques were proposed to monitor driver fatigue, especially in manual driving, little is known about how these black-box machine learning models work. In this paper, we proposed a combination of Gaussian Process Boosting (GPBoost) and SHapley Additive exPlanations (SHAP) to predict driver fatigue with explanations. First, in order to obtain the ground truth of driver fatigue, we used PERCLOS (percentage of eyelid closure over the pupil over time) between 0 and 100 as the response variable. Second, we built a driver fatigue regression model using both physiological and behavioral measures with GPBoost that was able to address the within-subjects correlations. This model outperformed other selected machine learning models with root-mean-squared error (RMSE) = 2.965, mean absolute error (MAE) = 1.407, and adjusted R2=0.996. Third, we employed SHAP to identify the most important predictor variables and uncovered the black-box GPBoost model by showing the main effects of the most important predictor variables globally and explaining individual predictions locally. Such an explainable driver fatigue prediction model offered insights into how to intervene in automated driving when necessary, such as during the takeover transition period from automated driving to manual driving.

Understanding how trust is built over time is essential, as trust plays an important role in the acceptance and adoption of automated vehicles (AVs). This study aims to investigate the effects of system performance and participants’ trust preconditions on dynamic situational trust during takeover transitions. We evaluate the dynamic situational trust of 42 participants using both self-reported and behavioral measures while watching 30 videos with takeover scenarios. The study is a 3 by 2 mixed-subjects design, where the within-subjects variable is the system performance (i.e., accuracy levels of 95%, 80%, and 70%) and the between-subjects variable is the preconditions of the participants’ trust (i.e., overtrust and undertrust). Our results showed that participants quickly adjusted their self-reported situational trust levels, which were consistent with different accuracy levels of system performance in both trust preconditions. However, participants’ behavioral situational trust was affected by their trust preconditions across different accuracy levels. For instance, the overtrust precondition significantly increased the agreement fraction compared to the undertrust precondition. The undertrust precondition significantly decreased the switch fraction compared to the overtrust precondition. These results have important implications for designing an invehicle trust calibration system for conditional AVs.

It is extremely important to ensure a safe takeover transition in conditionally automated driving. One of the critical factors that quantifies the safe takeover transition is takeover time. Previous studies identified the effects of many factors on takeover time, such as takeover lead time, non-driving tasks, modalities of the takeover requests, and scenario urgency. However, there is a lack of research to predict takeover time by considering these factors all at the same time. Toward this end, we used eXtreme Gradient Boosting (XGBoost) to predict the takeover time using a dataset from a meta-analysis study [Zhanget al.(2019)]. In addition, we used SHAP (SHapley Additive exPlanation) to analyze and explain the effects of the predictors on takeover time. We identified seven most critical predictors that resulted in the best prediction performance. Their main effects and interaction effects on takeover time were examined. The results showed that the proposed approach provided both good performance and explainability. Our findings have implications on the design of in-vehicle monitoring and alert systems to facilitate the interaction between the drivers and the automated vehicle.

The advancement in machine learning and artificial intelligence promotes the testing and deployment of autonomous vehicles (AVs) on public roads. The California Department of Motor Vehicles (CA DMV) has launched the Autonomous Vehicle Tester Program, which collects and releases reports related to Autonomous Vehicle Disengagement (AVD) from autonomous driving. Understanding the causes of AVD is critical to improving the AV system’s safety and stability and providing guidance for AV testing and deployment. In this work, we built a scalable end-to-end pipeline to collect, process, model, and analyze the disengagement reports released from 2014 to 2020 using natural language processing and deep transfer learning. The analysis of disengagement data using taxonomy, visualization, and statistical tests revealed the trends of AV testing, cause frequency, and significant relationships between causes and effects of AVD. We found that (1) manufacturers tested AVs intensively during the Spring and/or Winter, (2) test drivers initiated more than 80% of the disengagement while more than 75% of the disengagement were because of errors in perception, localization & mapping, planning and control of the AV system, and (3) there was a significant relationship between the initiator of AVD and the cause category. This study serves as a successful practice of deep transfer learning using pre-trained models and generates a consolidated disengagement database allowing further investigation for other researchers. The related code and data are available on github.1

Situation awareness (SA) is critical to improving takeover performance during the transition period from automated driving to manual driving. Although many studies measured SA during or after the driving task, few studies have attempted to predict SA in real time in automated driving. In this work, we propose to predict SA during the takeover transition period in conditionally automated driving using eye-tracking and self-reported data. First, a tree ensemble machine learning model, named LightGBM (Light Gradient Boosting Machine), was used to predict SA. Second, in order to understand what factors influenced SA and how, SHAP (SHapley Additive exPlanations) values of individual predictor variables in the LightGBM model were calculated. These SHAP values explained the prediction model by identifying the most important factors and their effects on SA, which further improved the model performance of LightGBM through feature selection. We standardized SA between 0 and 1 by aggregating three performance measures (i.e., placement, distance, and speed estimation of vehicles with regard to the ego-vehicle) of SA in recreating simulated driving scenarios, after 33 participants viewed 32 videos with six lengths between 1 and 20 s. Using only eye-tracking data, our proposed model outperformed other selected machine learning models, having a root-mean-squared error (RMSE) of 0.121, a mean absolute error (MAE) of 0.096, and a 0.719 correlation coefficient between the predicted SA and the ground truth. The code is available athttps://github.com/refengchou/Situation-awareness-prediction. Our proposed model provided important implications on how to monitor and predict SA in real time in automated driving using eye-tracking data.

In conditionally automated driving, in-vehicle alert systems can provide drivers with information to assist their takeovers from automated driving. This study investigated how display modality and information influenced drivers’ acceptance of the in-vehicle alert systems under different event criticality situations. We conducted an online video study with a 3 (information type) × 3 (display modality) × 2 (event criticality) mixed design involving 60 participants. The results showed that considering drivers’ perceived usefulness and ease of use, presenting why only information was not sufficient for takeovers as compared to what will only information and why + what will information. Participants reported higher ease of use in the combination of speech and augmented reality condition when compared to the speech only condition. High event criticality led to drivers’ lower perceived usefulness and more negative opinions of the displays. The findings have implications for the design of in-vehicle alert systems during takeover transitions.

Alzheimer's disease (AD) is a neurodegenerative disease with an irreversible and progressive process. To understand the brain functions and identify the biomarkers of AD and early stages of the disease [also known as, mild cognitive impairment (MCI)], it is crucial to build the brain functional connectivity network (BFCN) using resting-state functional magnetic resonance imaging (rs-fMRI). Existing methods have been mainly developed using only a single time-point rs-fMRI data for classification. In fact, multiple time-point data is more effective than a single time-point data in diagnosing brain diseases by monitoring the disease progression patterns using longitudinal analysis. In this article, we utilize multiple rs-fMRI time-point to identify early MCI (EMCI) and late MCI (LMCI), by integrating the fused sparse network (FSN) model with parameter-free centralized (PFC) learning. Specifically, we first construct the FSN framework by building multiple time-point BFCNs. The multitask learning via PFC is then leveraged for longitudinal analysis of EMCI and LMCI. Accordingly, we can jointly learn the multiple time-point features constructed from the BFCN model. The proposed PFC method can automatically balance the contributions of different time-point information via learned specific and common features. Finally, the selected multiple time-point features are fused by a similarity network fusion (SNF) method. Our proposed method is evaluated on the public AD neuroimaging initiative phase-2 (ADNI-2) database. The experimental results demonstrate that our method can achieve quite promising performance and outperform the state-of-the-art methods.

In conditionally automated driving, drivers engaged in non-driving related tasks (NDRTs) have difficulty taking over control of the vehicle when requested. This study aimed to examine the relationships between takeover performance and drivers’ cognitive load, takeover request (TOR) lead time, and traffic density. We conducted a driving simulation experiment with 80 participants, where they experienced 8 takeover events. For each takeover event, drivers’ subjective ratings of takeover readiness, objective measures of takeover timing and quality, and NDRT performance were collected. Results showed that drivers had lower takeover readiness and worse performance when they were in high cognitive load, short TOR lead time, and heavy oncoming traffic density conditions. Interestingly, if drivers had low cognitive load, they paid more attention to driving environments and responded more quickly to takeover requests in high oncoming traffic conditions. The results have implications for the design of in-vehicle alert systems to help improve takeover performance.

Automated driving has many potential benefits, such as improving driving safety and reducing drivers’ workload. However, from a human factors’ perspective, one concern is that drivers become increasingly out of the control loop once they start to engage in non-driving-related tasks, which makes it difficult for the drivers to take over control in some situations. In the present study, we examined reviewers’ comments of YouTube videos featuring takeover transitions on commercially available autonomous vehicles and categorized the comments into four topics: Non-driving related tasks, automation capability awareness, situation awareness, and warning effectiveness. Then we investigated people’ opinions on the design of the takeover mechanism of commercially available autonomous vehicles using topic mining and sentiment analysis, and we found that 1) the topic of automation capability awareness received many more positive comments than both negative and neutral comments while the distributions of positive, negative, and neutral comments were fairly even in other topics and 2) people had extreme positive and negative opinions in non-driving related tasks than other topics. Finally, we discussed possible design recommendations in order to facilitate takeover transitions.

This paper gives an overview of the ten-year development of the papers presented at the International ACM Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutoUI) from 2009 to 2018. We categorize the topics into two main groups, namely, manual driving-related research and automated driving-related research. Within manual driving, we mainly focus on studies on user interfaces (UIs), driver states, augmented reality and head-up displays, and methodology; Within automated driving, we discuss topics, such as takeover, acceptance and trust, interacting with road users, UIs, and methodology. We also discuss the main challenges and future directions for AutoUI and offer a roadmap for the research in this area.

Parkinson's disease (PD) is a neurodegenerative progressive disease that mainly affects the motor systems of patients. To slow this disease deterioration, early and accurate diagnosis of PD is an effective way, which alleviates mental and physical sufferings by clinical intervention. In this paper, we propose a joint regression and classification framework for PD diagnosis via magnetic resonance and diffusion tensor imaging data. Specifically, we devise a unified multitask feature selection model to explore multiple relationships among features, samples, and clinical scores. We regress four clinical variables of depression, sleep, olfaction, cognition scores, as well as perform the classification of PD disease from the multimodal data. The multitask model explores the relationships at the level of clinical scores, image features, and subjects, to select the most informative and diseased-related features for diagnosis. The proposed method is evaluated on the public Parkinson's progression markers initiative dataset. The extensive experimental results show that the multitask framework can effectively boost the performance of regression and classification and outperforms other state-of-the-art methods. The computerized predictions of clinical scores and label for PD diagnosis may offer quantitative reference for decision support as well.

Alzheimer's disease (AD) is a neurodegenerative and non-curable disease, with serious cognitive impairment, such as dementia. Clinically, it is critical to study the disease with multi-source data in order to capture a global picture of it. In this respect, an adaptive ensemble manifold learning (AEML) algorithm is proposed to retrieve multi-source neuroimaging data. Specifically, an objective function based on manifold learning is formulated to impose geometrical constraints by similarity learning. The complementary characteristics of various sources of brain disease data for disorder discovery are investigated by tuning weights from ensemble learning. In addition, a generalized norm is explicitly explored for adaptive sparseness degree control. The proposed AEML algorithm is evaluated by the public AD neuroimaging initiative database. Results obtained from the extensive experiments demonstrate that our algorithm outperforms the traditional methods.

Automatic delineation of skin lesion contours from dermoscopy images is a basic step in the process of diagnosis and treatment of skin lesions. However, it is a challenging task due to the high variation of appearances and sizes of skin lesions. In order to deal with such challenges, we propose a new dense deconvolutional network (DDN) for skin lesion segmentation based on residual learning. Specifically, the proposed network consists of dense deconvolutional layers (DDLs), chained residual pooling (CRP), and hierarchical supervision (HS). First, unlike traditional deconvolutional layers, DDLs are adopted to maintain the dimensions of the input and output images unchanged. The DDNs are trained in an end-to-end manner without the need of prior knowledge or complicated postprocessing procedures. Second, the CRP aims to capture rich contextual background information and to fuse multilevel features. By combining the local and global contextual information via multilevel feature fusion, the high-resolution prediction output is obtained. Third, HS is added to serve as an auxiliary loss and to refine the prediction mask. Extensive experiments based on the public ISBI 2016 and 2017 skin lesion challenge datasets demonstrate the superior segmentation results of our proposed method over the state-of-the-art methods.

Neuroimaging techniques have been widely applied to various neurodegenerative disease analysis to reveal the intricate brain structure. The high dimensional neuroimaging features and limited sample size are the main challenges for the diagnosis task due to the unbalanced input data. To handle it, a sparse low-rank learning framework is proposed, which unveils the underlying relationships between input data and output targets by building a matrix-regularized feature network. Then we obtain the feature weight from the network based on local clustering coefficients. By discarding the irrelevant features and preserving the discriminative structured features, our proposed method can select the most relevant features and identify different stages of Parkinson's disease (PD) from normal controls. Extensive experimental results evaluated on the Parkinson's progression markers initiative (PPMI) dataset demonstrate that the proposed method achieves promising classification performance and outperforms the conventional algorithms. Furthermore, it can detect potential brain regions related to PD for future medical analysis.

Dermoscopy imaging analysis is a routine procedure for diagnosis and treatment of skin lesions. Segmentation is the very first step to demarcate skin lesions for further quantitative analysis. However, it is a challenging task due to various changes from different viewpoints and scales of skin lesions. To handle these challenges, we devise a new dense deconvolutional network (DDN) for skin lesion segmentation based on encoding module and decoding module. Our devised network consists of convolution unit, dense deconvolutionallayer (DDL) and chained residual pooling block. DDL is adopted to restore the high resolution of the original input by upsampling, while the chained residual pooling is utilized to fuse multilevel features. Also, the hierarchical supervision is added to capture low level detailed boundary information. The DDN is trained in an end-to-end manner and free of prior knowledge and complicated post-processing procedures. With fusing the local and global contextual information, the high-resolution prediction output is obtained. The validation on the public ISBI 2016 and 2017 skin lesion challenge dataset demonstrates the effectiveness of our proposed method.

Quantitative analysis of bacterial morphotypes in the microscope images plays a vital role in diagnosis of bacterial vaginosis (BV) based on the Nugent score criterion. However, there are two main challenges for this task: 1) It is quite difficult to identify the bacterial regions due to various appearance, faint boundaries, heterogeneous shapes, low contrast with the background, and small bacteria sizes with regards to the image. 2) There are numerous bacteria overlapping each other, which hinder us to conduct accurate analysis on individual bacterium. To overcome these challenges, we propose an automatic method in this paper to diagnose BV by quantitative analysis of bacterial morphotypes, which consists of a three-step approach, i.e., bacteria regions segmentation, overlapping bacteria splitting, and bacterial morphotypes classification. Specifically, we first segment the bacteria regions via saliency cut, which simultaneously evaluates the global contrast and spatial weighted coherence. And then Markov random field model is applied for high-quality unsupervised segmentation of small object. We then decompose overlapping bacteria clumps into markers, and associate a pixel with markers to identify evidence for eventual individual bacterium splitting. Next, we extract morphotype features from each bacterium to learn the descriptors and to characterize the types of bacteria using an Adaptive Boosting machine learning framework. Finally, BV diagnosis is implemented based on the Nugent score criterion. Experiments demonstrate that our proposed method achieves high accuracy and efficiency in computation for BV diagnosis.

Viral product design involves sophisticated interactions between product portfolio planning and viral marketing. However, social network effects are mainly considered in marketing-related activities, and there is still limited investigation of the interplay between product design and viral marketing. In the context of social networks, it is important to jointly leverage both viral product attributes and viral influence attributes for product adoption maximization and product line performance optimization. In order to deal with the joint optimization problem, this paper presents a systematic formulation of a bilevel decision-making strategy for viral product design based on the Stackelberg game theory. The product adoption maximization problem with viral influence and product attributes is modeled as the leader and the product portfolio optimization problem with product attributes is modeled as the follower. The interaction and coupling of these two optimization problems are addressed with a coordinate-wise optimization strategy, in which adoption maximization is tackled with an improved greedy algorithm and a hybrid Taguchi genetic algorithm. A case study of Kindle Fire HD tablets demonstrates the feasibility and potential of the bilevel decision-making strategy for viral product design, which is advantageous over the existing viral marketing methods that only consider viral influence attributes.

Unlike visual stimuli, little attention has been paid to auditory stimuli in terms of emotion prediction with physiological signals. This paper aimed to investigate whether auditory stimuli can be used as an effective elicitor as visual stimuli for emotion prediction using physiological channels. For this purpose, a well-controlled experiment was designed, in which standardized visual and auditory stimuli were systematically selected and presented to participants to induce various emotions spontaneously in a laboratory setting. Numerous physiological signals, including facial electromyogram, electroencephalography, skin conductivity and respiration data, were recorded when participants were exposed to the stimulus presentation. Two data mining methods, namely decision rules and k-nearest neighbor based on the rough set technique, were applied to construct emotion prediction models based on the features extracted from the physiological data. Experimental results demonstrated that auditory stimuli were as effective as visual stimuli in eliciting emotions in terms of systematic physiological reactivity. This was evidenced by the best prediction accuracy quantified by the F1 measure (visual: 76.2% vs. auditory: 76.1%) among six emotion categories (excited, happy, neutral, sad, fearful and disgusted). Furthermore, we also constructed culture-specific (Chinese vs. Indian) prediction models. The results showed that model prediction accuracy was not significantly different between culture-specific models. Finally, the implications of affective auditory stimuli in human–computer interaction, limitations of the study and suggestions for further research are discussed.

In order to incorporate both affective and cognitive factors in the decision-making process, a user experience (UX) evaluation function based on cumulative prospect theory is proposed for three different affective states and two different types of products (affect-rich versus affect-poor). In order to tackle multiple parameters involved in the UX evaluation function, a hierarchical Bayesian model is proposed with a technique called “Markov chain Monte Carlo.” It estimates parameters that represent different cognitive tendencies and affective influences for customers at the individual and group levels by generating posterior probability density functions of the parameters to incorporate inherent uncertainty. An experiment with four hypotheses was designed to test the proposed model. We found that: 1) anxious participants tend to be more risk-averse than those in joy and excitement; 2) joyful and excited participants tend to be more risk-seeking than those in anxiety in UX-related choice decision making; 3) all participants tend to be averse to unpleasant UX; and 4) participants tend to value by feeling for affect-rich products and value by calculation for affect-poor products. Furthermore, the models of five different types can predict choice decision making between product profiles with around 80% accuracy. In summary, the results explain affective-cognitive decision-making behavior in the complex domain of UX design and, thus, illustrate the potential and feasibility of the proposed method.

Product ecosystem design entails complex user experience (UX) that involves interactions among multiple users, products, and the ambience. This paper aims to capture causal relationships between UX and design elements and in turn to provide decision support to product ecosystem analysis. A fuzzy reasoning Petri net is developed to deal with the uncertainty, complexity, and dynamics associated with UX modeling. Reasoning of diverse constructs of UX is embedded in the fuzzy production rules that are derived from self-report UX data based on rough set mining. A fuzzy reasoning algorithm is implemented to perform parallel inference by multicriteria rules and to simulate most likely UX under different ambient factors. A case study of subway station UX design demonstrates the potential of product ecosystem FRPN formulation.

Elderly in-home assistance (EHA) has traditionally been tackled by human caregivers to equip the elderly with homecare assistance in their daily living. The emerging ambience intelligence (AmI) technology suggests itself to be of great potential for EHA applications, owing to its effectiveness in building a context-aware environment that is sensitive and responsive to the presence of humans. This paper presents a case-driven AmI (C-AmI) system, aiming to sense, predict, reason, and act in response to the elderly activities of daily living (ADLs) at home. The C-AmI system architecture is developed by synthesizing various sensors, activity recognition, case-based reasoning, along with EHA-customized knowledge, within a coherent framework. An EHA information model is formulated through the activity recognition, case comprehension, and assistive action layers. The rough set theory is applied to model ADLs based on the sensor platform embedded in a smart home. Assistive actions are fulfilled with reference to a priori case solutions and implemented within the AmI system through human–object–environment interactions. Initial findings indicate the potential of C-AmI for enhancing context awareness of EHA applications.

Although Augmented Reality technology was first developed over forty years ago, there has been little survey work giving an overview of recent research in the field. This paper reviews the ten-year development of the work presented at the ISMAR conference and its predecessors with a particular focus on tracking, interaction and display research. It provides a roadmap for future augmented reality research which will be of great value to this relatively young field, and also for helping researchers decide which topics should be explored when they are beginning their own studies in the area.