Interactive point cloud segmentation has become a pivotal task for understanding 3D scenes, enabling users to guide segmentation models with simple interactions such as clicks, therefore significantly reducing the effort required to tailor models to diverse scenarios and new categories. However, in the realm of interactive segmentation, the meaning of instance diverges from that in instance segmentation, because users might desire to segment instances of both thing and stuff categories that vary greatly in scale. Existing methods have focused on thing categories, neglecting the segmentation of stuff categories and the difficulties arising from scale disparity. To bridge this gap, we propose ClickFormer, an innovative interactive point cloud segmentation model that accurately segments instances of both thing and stuff categories. We propose a query augmentation module to augment click queries by a global query sampling strategy, thus maintaining consistent performance across different instance scales. Additionally, we employ global attention in the query-voxel transformer to mitigate the risk of generating false positives, along with several other network structure improvements to further enhance the model’s segmentation performance. Experiments demonstrate that ClickFormer outperforms existing interactive point cloud segmentation methods across both indoor and outdoor datasets, providing more accurate segmentation results with fewer user clicks in an open-world setting. Project page: https://sites.google.com/view/clickformer/

Panoptic reconstruction is a challenging task in 3D scene understanding. However, most existing methods heavily rely on pre-trained semantic segmentation models and known 3D object bounding boxes for 3D panoptic segmentation, which is not available for in-the-wild scenes. In this paper, we propose a novel zero-shot panoptic reconstruction method from RGB-D images of scenes. For zero-shot segmentation, we leverage open-vocabulary instance segmentation, but it has to face partial labeling and instance association challenges. We tackle both challenges by propagating partial labels with the aid of dense generalized features and building a 3D instance graph for associating 2D instance IDs. Specifically, we exploit partial labels to learn a classifier for generalized semantic features to provide complete labels for scenes with dense distilled features. Moreover, we formulate instance association as a 3D instance graph segmentation problem, allowing us to fully utilize the scene geometry prior and all 2D instance masks to infer global unique pseudo 3D instance ID. Our method outperforms state-of-the-art methods on the indoor dataset ScanNet V2 and the outdoor dataset KITTI-360, demonstrating the effectiveness of our graph segmentation method and reconstruction network.

This article presents a computational model that predicts finger-drag gesture performance on touchscreen devices, by integrating the queueing network (QN) cognitive architecture and motion tracking. Specifically, the QN-based model was developed to predict two execution times: the finger movement time of drag-gesture (i.e., only the motion time of the finger touched and dragged on the surface of touchscreen) and the comprehensive process time of drag-gesture (i.e., the entire process time to complete the finger-drag task, including visual attention shift, memory storage and retrieval, and hand-finger movements). To develop predictive models for the finger movement time of drag-gesture, 11 participants’ motion data were collected and a regression analysis with parameters of hand-finger anthropometric data and eight angular directions was conducted. Human subject data from our previous study (Jeong & Liu, 2017a) were used to evaluate the QN-based model, generating similar outputs (R2 was more than 80% and root-mean square was less than 300 msec) for both execution times.

This article investigates the effects of the quantity and size of touch screen buttons and the task-interleaving strategies on drives’ eye glance behavior. An experiment was conducted on a fixed-base driving simulator with 20 participants. The participants were asked to perform a button search-and-press task on an in-vehicle touch screen while driving. A full-factorial within-subject design was used with three button quantities (4, 8, and 15) and three button sizes (14 mm, 24 mm, and 33 mm). Although a normal distribution was often assumed for the eye glance data in previous studies, our results show that the total eyes-off-road time (TEORT) and glance durations are generally not normally distributed (positively skewed) even after a log transformation. The results show that the number of buttons has an increasing effect on task completion time, TEORT, and long (2+ s) glances. However, in general, no such differences were found for button sizes. Further analysis shows that long glances were strongly associated with drivers completing the task with a single glance. It seems to suggest that a major cause of long glances is that drivers are reluctant to switch the task back to driving at subtask boundaries that are probably associated with the high cost of interruption. These findings confirm the importance of task resumability for in-vehicle user interfaces and have implications that careful task analysis needs to be conducted in the context of multitasking. Certain subtask combinations, such as a visual search followed by pressing the search target, may discourage task interleaving and ultimately compromise driving safety.

In this study, a linear antiwindup controller of an electrohydraulic system (EHS) is proposed to drive the joint motion of a two-degree-of-freedom robotic arm. Since the unknown external load disturbance is caused by the driven force/torque of the robotic arm and some hydraulic parametric uncertainties existing in an EHS model, perhaps the control saturation emerges in the controller, which will degrade the dynamic performance and the stable margin of the EHS. Thus, a model recovery antiwindup compensator embedded in an unconstrained controller is designed to suppress the control saturation of a servo valve. The antiwindup effectiveness of the proposed controller has been demonstrated by a comparative experiment study with both linear and nonlinear controllers, which indicates the proposed controller can achieve better dynamic performance under largely unknown load disturbance and model uncertainty.

In this paper, we propose a new approach of detecting emergency braking intention for brain-controlled vehicles by interpreting electroencephalography (EEG) signals of drivers. Regularization linear discriminant analysis with spatial-frequency features is applied to build the detection model. These spatial-frequency features are selected from the powers of frequency points across sixteen channels by using the sequential forward floating search. Experimental results from twelve subjects show that on average, the proposed method can detect emergency braking intentions 420 ms after the onset of emergency situations with the system accuracy of over 94%, showing the feasibility of developing a practical system of detecting driver emergency braking intention with the power spectra of EEG signals for brain-controlled vehicles.

Driver intention detection is an important component in human-centric driver assistance systems. This article proposes a novel method for detecting driver normal and emergency left- or right-lane-changing intentions by using driver models based on the queuing network cognitive architecture. Driver lane-changing and lane-keeping models are developed and used to simulate driver behavior data associated with 5 kinds of intentions (i.e., normal and emergency left- or right-lane-changing and lane-keeping intentions). The differences between 5 sets of simulated behavior data and the collected actual behavior data are computed, and the intention associated with the smallest difference is determined as the detection outcome. The experimental results from 14 drivers in a driving simulator show that the method can detect normal and emergency lane-changing intentions within 0.325 s and 0.268 s of the steering maneuver onset, respectively, with high accuracy (98.27% for normal lane changes and 90.98% for emergency lane changes) and low false alarm rate (0.294%).

Rapid mobile phone market growth has drastically shortened design cycle times forcing designers to identify user preferences quickly. Two studies test a novel method to quickly determine design preferences for touch screen and non–touch screen mobile phones. Interactive Genetic Algorithms (IGAs) are tested as an option for preference identification. IGAs efficiently explore design spaces, finding user preferences via an iterative system mimicking evolution. Each study tested six independent variables: horizontal and vertical button spacing; horizontal and vertical screen dimensions; corner radius; and user’s goal between aesthetically pleasing, functional, and both aesthetic and functional (first experiment), and aesthetic preference differences between touch screen and non–touch screen mobile phones for dialing phone numbers only (second experiment). Results showed IGAs to be a viable method; participant preference varied by goal (horizontal button spacing and screen width were most effected). Designers can employ IGAs to quickly determine aesthetic and other user preferences.

This paper describes the development of a novel driver lateral control model by integrating the driver's neuromuscular dynamics into the queuing network (QN)-based driver lateral control model. Experimental results from 16 participants in a driving simulator show that, compared to the QN-based model, the proposed model performs better, and its performance is closer to that of drivers when a vehicle runs at a relatively high speed. The proposed model not only has the advantages of the models based on a cognitive architecture but also captures the dynamic interaction between the vehicular steering system and the driver's neuromuscular system. Thus, it can better represent driver lateral control and has greater value in supporting the development of driver assistance systems.

In this paper, we propose a novel driver-vehicle interface for individuals with severe neuromuscular disabilities to use intelligent vehicles by using P300 and steady-state visual evoked potential (SSVEP) brain-computer interfaces (BCIs) to select a destination and test its performance in the laboratory and real driving conditions. The proposed interface consists of two components: the selection component based on a P300 BCI and the confirmation component based on an SSVEP BCI. Furthermore, the accuracy and selection time models of the interface are built to help analyze the performance of the entire system. Experimental results from 16 participants collected in the laboratory and real driving scenarios show that the average accuracy of the system in the real driving conditions is about 99% with an average selection time of about 26 s. More importantly, the proposed system improves the accuracy of destination selection compared with a single P300 BCI-based selection system, particularly for those participants with relatively low level of accuracy in using the P300 BCI. This study not only provides individuals with severe motor disabilities with an interface to use intelligent vehicles and thus improve their mobility, but also facilitates the research on driver-vehicle interface, multimodal interaction, and intelligent vehicles. Furthermore, it opens an avenue on how cognitive neuroscience may be applied to intelligent vehicles.

Computational dual-task models of driving with a secondary task can help compute, simulate, and predict driving behavior in dual task situations. These models can thus help improve the process of developing in-vehicle devices by reducing or eliminating the need for conducting driver experiments in the early stage of the development. Further, these models can help improve traffic flow simulation. This article develops a dual-task model of driving with a visual distraction task using the Queuing Network model of driver lateral control and a logistic regression model. The comparison between the model simulation data and the human data from drivers in a driving simulator shows that this computational model can perform driving with a secondary visual task well and its performance is consistent with the driver data.

Text-overlaid images abound in product design and human–computer interfaces (e.g., greeting cards and photo slideshows that overlay texts on large background images). This study proposes computational models of perceived aesthetics appeal of text-overlaid images as a function of their white-space fractions. Four models are proposed, including Character-based Page White-Space Fraction (WCP ), Character-based Background White-Space Fraction (WCB ), Bounding Box based Page White-Space Fraction (WBP ), and Bounding Box based Background White-Space Fraction (WBB ). Two experiments, using single-spaced and double-spaced texts, were conducted to investigate the relationships between the model predictions and the perceived aesthetic appeal of text-overlaid images. The results show that the Bounding Box based Background White-Space Fraction (WBB ), defined as the area ratio between the bounding rectangle of texts and the background region of image, is most valuable in modelling the subjective aesthetic appeal. The regression curves show that the optimal WBB is close to 0.8.

In this paper, we propose a new steady-state visually evoked potential (SSVEP) brain-computer interface (BCI) with visual stimuli presented on a windshield via a head-up display, and we apply this BCI in conjunction with an alpha rhythm to control a simulated vehicle with a 14-DOF vehicle dynamics model. A linear discriminant analysis classifier is applied to detect the alpha rhythm, which is used to control the starting and stopping of the vehicle. The classification models of the SSVEP BCI with three commands (i.e., turning left, turning right, and going forward) are built by using a support vector machine with frequency domain features. A real-time brain-controlled simulated vehicle is developed and tested by using four participants to perform a driving task online, including vehicle starting and stopping, lane keeping, avoiding obstacles, and curve negotiation. Experimental results show the feasibility of using the human “mind” alone to control a vehicle, at least for some users.

EEG-based brain-controlled mobile robots can serve as powerful aids for severely disabled people in their daily life, especially to help them move voluntarily. In this paper, we provide a comprehensive review of the complete systems, key techniques, and evaluation issues of brain-controlled mobile robots along with some insights into related future research and development issues. We first review and classify various complete systems of brain-controlled mobile robots into two categories from the perspective of their operational modes. We then describe key techniques that are used in these brain-controlled mobile robots including the brain-computer interface techniques and shared control techniques. This description is followed by an analysis of the evaluation issues of brain-controlled mobile robots including participants, tasks and environments, and evaluation metrics. We conclude this paper with a discussion of the current challenges and future research directions.

In this paper, we propose a P300 brain-computer interface (BCI) with visual stimuli presented on a head-up display and we apply this BCI for selecting destinations of a simulated vehicle in a virtual scene. To improve the usability of the selection system, we analyze the effects of the number of electroencephalogram (EEG) rounds on system performance. Experimental results from eight participants show that the BCI-based model of destination selection can be built with EEG data from eight channels, and participants can use this BCI to select a desired destination with an accuracy value of 93.6% ± 1.6% (mean value with standard error) in about 12 s of selection time. This paper lays a foundation for developing vehicles that use a BCI to select a desired destination from a list of predefined destinations and then use an autonomous navigation system to reach the desired destination.

In this paper, we propose a computational model of driver lateral control based on the queuing network cognitive architecture and the driver preview model about driver lateral control activities. This computational model was applied to model the dual tasks of driving with a cognitive distraction task. The comparison between human driver data and model simulation data shows that this computational model can perform vehicle lateral control well, and its performance is consistent with that of drivers under single- and dual-task driving conditions. Furthermore, we examine the effectiveness of some parameters of the model in representing different styles of driving and discuss the value of this computational model in facilitating the evaluation of vehicle dynamics and driver assistant systems and providing new insights into research on unmanned vehicle control techniques.

Human behavior models give insight into people's choices and actions and are tools for predicting performance and improving interface design. Most models focus on a task's cognitive aspects or its physical requirements. This research addresses the divide between cognitive and physical models by combining two models to produce an integrated cognitive-physical human model that enables studying of complex human-machine interactions. The capabilities of the integrated model are evaluated in a task scenario with both cognitive and physical components, i.e., driving while performing a secondary in-vehicle task. When applied in this way, the integrated model is called the Virtual Driver model and can replicate basic driving, in-vehicle tasks, and resource-sharing behaviors, providing a new way to study driver distraction. The model has applicability to interface design and predicting staffing requirements and performance.

This article reports two experiments and the corresponding modeling research on the effects of symmetry and the number of compositional elements on Chinese users' aesthetic ratings of interfaces composed with abstract black-and-white geometric images and realistic-looking web pages. Symmetry and the number of compositional elements are the two independent variables, each with three levels. The dependent variable is subjective ratings of aesthetic appeal. The results from both experiments show that symmetry of compositional elements significantly affects aesthetic ratings of Chinese users, whereas the number alone does not have a significant effect on aesthetic ratings. However, subjects preferred realistic web page images with few elements when they lack symmetry. We also describe our development and evaluation of a computational model of aesthetic ratings based on symmetry and the number of compositional elements to predict aesthetic ratings, which can be used to evaluate and support aesthetic design of interfaces.

Empirical data modeling can be used to model human performance and explore the relationships between diverse sets of variables. A major challenge of empirical data modeling is how to generalize or extrapolate the findings with a limited amount of observed data to a broader context. In this paper, we introduce an approach from machine learning, known as support vector regression (SVR), which can help address this challenge. To demonstrate the method and the value of modeling human performance with SVR, we apply SVR to a real-world human factors problem of night vision system design for passenger vehicles by modeling the probability of pedestrian detection as a function of image metrics. The results indicate that the SVR-based model of pedestrian detection shows good performance. Some suggestions on modeling human performance by using SVR are discussed.

This paper introduces a queueing network-based computational model to explain driver performance in a pedestrian-detection task assisted with night-vision-enhancement systems. The computational cognitive model simulated the pedestrian-detection task using images displayed by two night-vision systems as input stimuli. The system equipped with a far-infrared (FIR) sensor generated less-cluttered images than the system equipped with a near-infrared (NIR) sensor. Using a reinforcement learning process, the model developed eye-movement strategies for each night-vision system. The differences in eye-movement strategies generated different eye-movement behaviors, in accord with the empirical findings.

This paper describes two studies in which two night-vision enhancement systems were examined to compare nighttime driver performance in pedestrian detection. In the first study, the levels of clutter in the images displayed by the two types of night-vision enhancement systems were measured objectively and subjectively. The subjective ratings of clutter changed as a power function of the objective measure of clutter intensity. In the second study, the effect of clutter on glance behavior during pedestrian detection was examined in a driving simulator. Night-vision images with less clutter required shorter search times and fewer glances to detect the pedestrian, but the duration of each glance remained relatively constant.

The goal of this research is to examine the effectiveness of the interactive genetic algorithm (IGA) procedure in evaluating and improving interface aesthetics. Though the procedure has been adopted by a few other researchers in investigating preference or in design scenarios, its ability to improve subject's preference has not been demonstrated quantitatively, particularly for the outcome measure of aesthetic appeal. Two experiments are presented that use identical IGA characteristics to help subjects choose a good design quickly from a large design space. The first experiment uses abstract imageries as stimuli, and the second uses Web page templates for a Web log. The statistical evaluation of the IGA illustrates that it is effective in both cases, showing an 8% increase in aesthetic appeal for abstract imagery and a 15% increase for the Web pages over the duration of the IGA procedure.

Comprehensive and computational models of human performance have both scientific and practical importance to human–machine system design and human-centered computing. This article describes QN-ACES, a cognitive architecture that aims to integrate two complementary classes of cognitive architectures: Queueing network (QN) mathematical architecture and ACT–R, CAPS, EPIC, and Soar (ACES) symbolic architectures. QN-ACES represents the fourth major step along the QN architecture development for theoretical and methodological unification in cognitive and human–computer interaction modeling. The first three steps—QN architecture for response time, QN-RMD (Reflected Multidimensional Diffusions) for response time, response accuracy, and mental architecture, and QN-MHP (Model Human Processor) for mathematical analysis and real time simulation of procedural tasks—are summarized first, followed by a discussion of the rationale, importance and specific research issues of QN-ACES.

The primary purpose of night-vision systems in civilian vehicles is to help drivers detect pedestrians. Pedestrian detection distance with night-vision systems has been modeled based on image metrics. However, the probability of pedestrian detection, in particular considering the factor of distance, has not been modeled based on image metrics. In this paper, we first describe a model of the probability of pedestrian detection, which compares several combinations of image-based clutter, contrast, and pedestrian size metrics using a simple mathematical equation. Next, we describe a model of the probability of pedestrian detection as a function of distance and image-based metrics by combining the model of pedestrian-detection probability and a model that represents the relationship between the distance to a pedestrian and an image-based pedestrian size metric. In the final model, image-based metrics are used to predict pedestrian-detection performance and can also be used to evaluate and support the development of night-vision systems in vehicles.

Understanding and reproducing complex human oculomotor behaviors using computational models is a challenging task. In this paper, two studies are presented, which focus on the development and evaluation of a computational model to show the influences of cyclic top-down and bottom-up processes on eye movements. To explain these processes, reinforcement learning was used to control eye movements. The first study showed that, in a picture-viewing task, different policies obtained from different picture-viewing conditions produced different types of eye movement patterns. In another visual search task, the second study illustrated that feedback information from each saccadic eye movement could be used to update the model's eye movement policy, generating different patterns in the following saccade. These two studies demonstrate the value of an integrated reinforcement learning model in explaining both top-down and bottom-up processes of eye movements within one computational model.

This article describes two experiments investigating the effects of manipulating two compositional elements—symmetry and the number of compositional building blocks—on subjective appraisals of interface aesthetics. The experiments use stimuli with identical composition but varying subject matter. The first experiment uses abstract black-and-white geometric images, whereas the second uses realistic-looking Web pages as stimuli. Both experiments have three levels for each of the two independent variables, with the dependent measure being subjective ratings of aesthetic appeal. Results from both experiments show that the number of compositional elements influences aesthetic appeal ratings. For the abstract imagery, symmetry also plays a role such that participants find the more symmetric images appealing. Implications of these findings on interface design are discussed.

The risk of vehicle collisions significantly increases when drivers are overloaded with information from in-vehicle systems. One of the solutions to this problem is developing adaptive workload management systems (AWMSs) to dynamically control the rate of messages from these in-vehicle systems. However, existing AWMSs do not use a model of the driver cognitive system to estimate workload and only suppress or redirect in-vehicle system messages, without changing their rate based on driver workload. In this paper, we propose a prototype of a new queueing network-model human processor AWMS (QN-MHP AWMS), which includes a queueing network model of driver workload that estimates the driver workload in several driving situations and a message controller that determines the optimal delay times between messages and dynamically controls the rate of messages presented to drivers. Given the task information of a secondary task, the QN-MHP AWMS adapted the rate of messages to the driving conditions (i.e., speeds and curvatures) and driver characteristics (i.e., age). A corresponding experimental study was conducted to validate the potential effectiveness of this system in reducing driver workload and improving driver performance. Further development of the QN-MHP AWMS, including its use in in-vehicle system design and possible implementation in vehicles, is discussed.

Transcription typing is one of the basic and common activities in human-machine interaction and 34 transcription typing phenomena have been discovered involving many aspects of human performance including interkey time, typing units and spans, typing errors, concurrent task performance, eye movements, and skill effects. Based on the queuing network theory of human performance [Liu 1996; 1997] and current discoveries in cognitive and neural science, this article extends and applies the Queuing Network-Model Human Processor (QN-MHP [Liu et al. 2006]) to model 32 transcription typing phenomena. The queuing network model of transcription typing offers new insights into the mechanisms of cognition and human-computer interaction. Its value in proactive ergonomics design of user interfaces is illustrated and discussed.

Modeling and predicting of mental workload are among the most important issues in studying human performance in complex systems. Ample research has shown that the amplitude of the P300 component of event-related potential (ERP) is an effective real-time index of mental workload, yet no computational model exists that is able to account for the change of P300 amplitude in dual-task conditions compared with that in single-task situations. We describe the successful extension and application of a new computational modeling approach in modeling P300 and mental workload—a queuing network approach based on the queuing network theory of human performance and neuroscience discoveries. Based on the neurophysiological mechanisms underlying the generation of P300, the current modeling approach accurately accounts for P300 amplitude both in temporal and intensity dimensions. This approach not only has a basis in its biological plausibility but also has the ability to model and predict workload in real time and can be applied to other applied domains. Further model developments in simulating other dimensions of mental workload and its potential applications in adaptive system design are discussed.

Drivers overloaded with information significantly increase the chance of vehicle collisions. Driver workload, which is a multidimensional variable, is measured by both performance-based and subjective measurements and affected by driver age differences. Few existing computational models are able to cover these major properties of driver workload or simulate subjective mental workload and human performance at the same time. We describe a new computational approach in modeling driver performance and workload-a queuing network approach based on the queuing network theory of human performance and neuroscience discoveries. This modeling approach not only successfully models the mental workload measured by the six National Aeronautic and Space Administration Task Load Index workload scales in terms of subnetwork utilization but also simulates the driving performance, reflecting mental workload from both subjective- and performance-based measurements. In addition, it models age differences in workload and performance and allows us to visualize driver mental workload in real time. Further usage and implementation of the model in designing intelligent and adaptive in-vehicle systems are discussed.

Queueing Network-Model Human Processor (QN-MHP) is a computational architecture that integrates two complementary approaches to cognitive modeling: the queueing network approach and the symbolic approach (exemplified by the MHP/GOMS family of models, ACT-R, EPIC, and SOAR). Queueing networks are particularly suited for modeling parallel activities and complex structures. Symbolic models have particular strength in generating a person's actions in specific task situations. By integrating the two approaches, QN-MHP offers an architecture for mathematical modeling and real-time generation of concurrent activities in a truly concurrent manner. QN-MHP expands the three discrete serial stages of MHP, of perceptual, cognitive, and motor processing, into three continuous-transmission subnetworks of servers, each performing distinct psychological functions specified with a GOMS-style language. Multitask performance emerges as the behavior of multiple streams of information flowing through a network, with no need to devise complex, task-specific procedures to either interleave production rules into a serial program (ACT-R), or for an executive process to interactively control task processes (EPIC). Using QN-MHP, a driver performance model was created and interfaced with a driving simulator to perform a vehicle steering, and a map reading task concurrently and in real time. The performance data of the model are similar to human subjects performing the same tasks.

This article describes a three-node queueing network model of human multitask performance to account for interferences between concurrent spatial and verbal tasks. The model integrates considerations of single-channel queuing theoretic models of selective attention and parallel processing, multiple-resource models of divided attention, and provides a computational framework for modeling both the serial processing and the concurrent execution aspects of human multitask performance. The single-channel and the multiple-resource concepts and their applications in engineering models are reviewed. Experimental evidence in support of the queueing network model is summarized. The potential value of using queueing network methods to integrate currently isolated concepts of human multitask performance and in modeling human machine interaction in general is discussed.

This article examines two aspects of traffic management for highway networks. First, we introduce the term hortatory operations to describe highway traffic management as a type of human-machine system in which human operators have limited ability to directly control activities in the system. Instead, they must rely heavily on distributing messages of advice and encouragement as a means to achieve smooth traffic behavior. We discuss some of the features of this type of system. Second, we portray traffic management as a process that integrates both quantitative and qualitative information. We then describe the development and testing of a decision-aiding software tool that combines the two types of information. The tool supports traffic management by helping operators differentiate between demand-based and incident-based congestion. The decision aid's performance was investigated using real traffic data. The results of statistical analysis showed that the integrated process had superior performance compared to the "traditional" quantitative analysis running alone.