Robotic scope assistant systems allow surgeons to adjust the operative field view during surgery by robotically maneuvering laparoscopes. A Human-Robot Interface (HRI) is used for issuing commands to these systems, with an interaction mode mapping these commands to laparoscope movements. Optimizing the HRI and interaction mode can streamline laparoscope positioning as well as reduce cognitive workload, helping the surgeon focus on the surgical procedure. Comparing and assessing various HRIs and interaction modes is essential for efficient laparoscope maneuvering. This study evaluates HRIs based on head-motion, eye-motion, hand-motion, and voice-input operating under three interaction modes (namely: discrete, continuous, and threshold). The participants performed a user study comparing different HRIs under two simulated surgical scenarios (one in a real environment and the other in a virtual environment). The results indicated that head and eye-based HRIs performed well in continuous interaction mode, while the voice-based interface suffered from a delay. Conversely, hand-based HRIs demonstrated superior performance in both scenarios across all evaluation parameters. The study provides a benchmark for the comparison of different HRIs and provides insights into the effectiveness, limitations, and potential advantages of different HRIs.

We report on a naturalistic study investigating the effects of routine driving on cardiovascular activation. We recruited 21 healthy young adults from a broad geographic area in the Southwestern United States. Using the participants' own smartphones and smartwatches, we monitored for a week both their driving and non-driving activities. Monitoring included the continuous recording of a) heart rate throughout the day, b) hand motion during driving as a proxy of persistent texting, and c) contextualized driving data, complete with traffic and weather information. These high temporal resolution variables were complemented with the drivers' biographic and psychometric profiles. Our analysis suggests that anxiety predisposition and high speeds are associated with significant cardiovascular activation on drivers, likely linked to sympathetic arousal. Surprisingly, these associations hold true under good weather, normal traffic, and with experienced drivers behind the wheel. The said findings call for attention to insidious effects of apparently benign drives even for people in their prime. Accordingly, our research contributes to intriguing new discourses on driving affect and personal health informatics.

In the field of MRI super-resolution, training an image upscaling network under a pixel-oriented cost function (e.g., Mean-Intensity-Error) has proven to boost the signal-to-noise ratio. However, these types of cost functions tend to miss high-frequency details and fail to achieve an ideal sharpness, which is a pivotal image property for clinical applications to make diagnoses. To address this issue, the cost function of these upscaling networks typically includes a perceptual loss function, which is well recognized for the reconstruction of textures and enhancing sharpness, in addition to a pixel-oriented one. In this paper, we investigate the effect of perceptual loss on several MRI super-resolution metrics. We train UNet architecture under two loss function scenarios: One only including a pixel-oriented loss function, and the other a fusion of pixel-oriented and perceptual losses. We then employ an ablation study using a mixed effect model on a comprehensive set of evaluation criteria to measure the significance of change upon the inclusion of perceptual loss. Our results show that even though perceptual loss substantially shifts the networks towards outputting sharper images, it only causes negligible performance degradation in the accuracy of the reconstructed regions of interest, which can be alleviated using proper hyperparameter tuning.

The work assesses the efficacy of computer based remote tele-mentoring system (i.e. when the mentor and mentee are physically separated) for teaching minimally invasive surgical skills. The visual cues used for tele-mentoring comprises real-time virtual surgical instruments' motion augmented onto the operative field and remotely controlled by the mentor. In the feasibility study, the surgical task of laparoscopic intracorporeal suturing was simulated among 18 mentor-mentee pairs. Three modes of mentoring were used. Mode-I included traditional learning using pre-recorded videos (in absence of a mentor). Mode-II used traditional in-person hands-on mentoring. In Mode-III, a tele-mentoring prototype was used that connected a mentee with a remote mentor. Error count and duration were recorded for a learning stage followed by a testing stage for the three modes. The results show the error count for Mode-III reduces significantly as compared to Mode-I in the learning stage. Similarly, the error count for Mode-III also reduces significantly as compared to Mode-I in the testing stage. The errors count for Mode-III were equivalent to that of Mode-II for both learning and teaching stages. Furthermore, in Mode-III the duration reduces from learning to testing stage exhibiting the learning effect. Thus, computer based remote tele-mentoring is effective and more convenient to demonstrate surgical sub-steps consisting of tool-tissue interaction facilitating surgical skill transfer.

Operating machinery while distracted is a dangerous behavior, often habitual, which is the source of accidents. Distracted driving in particular has assumed the form of an epidemic, fueled by the ubiquity of smartphone usage and the tendency to slip into absent-mindedness in tedious commutes. Here we show that a method capable of detecting and communicating overarousal trends associated with the onset of distractions, can pull the driver out of a downward psychophysiological spiral. The method is reliable, unobtrusive, and subtle in its intervention-all important characteristics for real-time corrections on human handling of critical machinery. Arousal estimation is performed by a conservative statistical filter acting upon the driver's perinasal perspiration signal, as this is continuously extracted from a thermal imaging feed. Overarousal notices are communicated via a visual indicator placed in the driver's peripheral vision. Using this method, we conducted a parallel group experiment, where a control CLCL (n=23n=23) and a biofeedback BFBF (n=24n=24) cohort were distracted mentally and physically while driving, with only the biofeedback group receiving the benefit of overarousal notification. Results show that heeding biofeedback notices, cuts dramatically the time BFBF subjects are engaged in distractions with respect to the control group, significantly reducing their arousal levels and improving their driving behaviors in the context of a typical commute.

This work presents an interventional planning software to be used in conjunction with a robotic manipulator to perform transrectal MR guided prostate biopsies. The interventional software was designed taking in consideration a generic manipulator used under the two modes of operation: side-firing and end-firing of the biopsy needle. Studies were conducted with urologists using the software to plan virtual biopsies. The results show features of software relevant for operating efficiently under the two modes of operation.

Working in an environment with constant interruptions is known to affect stress, but how do interruptions affect emotional expression? Emotional expression can have significant impact on interactions among coworkers. We analyzed the video of 26 participants who performed an essay task in a laboratory while receiving either continual email interruptions or receiving a single batch of email. Facial videos of the participants were run through a convolutional neural network to determine the emotional mix via decoding of facial expressions. Using a novel co-occurrence matrix analysis, we showed that with batched email, a neutral emotional state is dominant with sadness being a distant second, and with continual interruptions, this pattern is reversed, and sadness is mixed with fear. We discuss the implications of these results for how interruptions can impact employees' well-being and organizational climate.

This work presents a novel deep learning architecture called BNU-Net for the purpose of cardiac segmentation based on short-axis MRI images. Its name is derived from the Batch Normalized (BN) U-Net architecture for medical image segmentation. New generations of deep neural networks (NN) are called convolutional NN (CNN). CNNs like U-Net have been widely used for image classification tasks. CNNs are supervised training models which are trained to learn hierarchies of features automatically and robustly perform classification. Our architecture consists of an encoding path for feature extraction and a decoding path that enables precise localization. We compare this approach with a parallel approach named U-Net. Both BNU-Net and U-Net are cardiac segmentation approaches: while BNU-Net employs batch normalization to the results of each convolutional layer and applies an exponential linear unit (ELU) approach that operates as activation function, U-Net does not apply batch normalization and is based on Rectified Linear Units (ReLU). The presented work (i) facilitates various image preprocessing techniques, which includes affine transformations and elastic deformations, and (ii) segments the preprocessed images using the new deep learning architecture. We evaluate our approach on a dataset containing 805 MRI images from 45 patients. The experimental results reveal that our approach accomplishes comparable or better performance than other state-of-the-art approaches in terms of the Dice coefficient and the average perpendicular distance.

Prostate biopsy is considered as a definitive way for diagnosing prostate malignancies. Urologists are currently moving towards MR-guided prostate biopsies over conventional transrectal ultrasound-guided biopsies for prostate cancer detection. Recently, robotic systems have started to emerge as an assistance tool for urologists to perform MR-guided prostate biopsies. However, these robotic assistance systems are designed for a specific clinical environment and cannot be adapted to modifications or changes applied to the clinical setting and/or workflow. This work presents the preliminary design of a cable-driven manipulator developed to be used in both MR scanners and MR-ultrasound fusion systems. The proposed manipulator design and functionality are evaluated on a simulated virtual environment. The simulation is created on an in-house developed interventional planning software to evaluate the ergonomics and usability. The results show that urologists can benefit from the proposed design of the manipulator and planning software to accurately perform biopsies of targeted areas in the prostate.

The emerging potential of augmented reality (AR) to improve 3D medical image visualization for diagnosis, by immersing the user into 3D morphology is further enhanced with the advent of wireless head-mounted displays (HMD). Such information-immersive capabilities may also enhance planning and visualization of interventional procedures. To this end, we introduce a computational platform to generate an augmented reality holographic scene that fuses pre-operative magnetic resonance imaging (MRI) sets, segmented anatomical structures, and an actuated model of an interventional robot for performing MRI-guided and robot-assisted interventions. The interface enables the operator to manipulate the presented images and rendered structures using voice and gestures, as well as to robot control. The software uses forbidden-region virtual fixtures that alerts the operator of collisions with vital structures. The platform was tested with a HoloLens HMD in silico. To address the limited computational power of the HMD, we deployed the platform on a desktop PC with two-way communication to the HMD. Operation studies demonstrated the functionality and underscored the importance of interface customization to fit a particular operator and/or procedure, as well as the need for on-site studies to assess its merit in the clinical realm.

Heart disease is highly prevalent in developed countries, causing 1 in 4 deaths. In this work we propose a method for a fully automated 4D reconstruction of the left ventricle of the heart. This can provide accurate information regarding the heart wall motion and in particular the hemodynamics of the ventricles. Such metrics are crucial for detecting heart function anomalies that can be an indication of heart disease. Our approach is fast, modular and extensible. In our testing, we found that generating the 4D reconstruction from a set of 250 MRI images takes less than a minute. The amount of time saved as a result of our work could greatly benefit physicians and cardiologist as they diagnose and treat patients.

This article documents thermophysiological patterns associated with migraine episodes, where the inner canthi and supraorbital temperatures drop significantly compared to normal conditions. These temperature drops are likely due to vasoconstriction of the ophthalmic arteries under the inner canthi and sympathetic activation of the eccrine glands in the supraorbital region, respectively. The thermal patterns were observed on eight migraine patients and meticulously quantified using advance computational methods, capable of delineating small anatomical structures in thermal imagery and tracking them automatically over time. These methods open the way for monitoring migraine episodes in nonclinical environments, where the patient maintains directional attention, such as his/her computer at home or at work. This development has the potential to significantly expand the operational envelope of migraine studies.

Electrodermal activity (EDA) is an important affective indicator, measured conventionally on the fingers with desktop sensing instruments. Recently, a new generation of wearable, battery-powered EDA devices came into being, encouraging the migration of EDA sensing to other body locations. To investigate the implications of such sensor/location shifts in psychophysiological studies we performed a validation experiment. In this experiment we used startle stimuli to instantaneously arouse the sympathetic system of n=23 subjects while sitting. Startle stimuli are standard but minimal stressors, and thus ideal for determining the sensor and location resolution limit. The experiment revealed that precise measurement of small EDA responses on the fingers and palm is feasible either with conventional or mobile EDA sensors. By contrast, precise measurement of small EDA responses on the sole is challenging, while on the wrist even detection of such responses is problematic for both EDA modalities. Given that affective wristbands have emerged as the dominant form of EDA sensing, researchers should beware of these limitations.

This study explored various user interface designs to transition a two dimensional (2D) questionnaire from its paper-and-pencil testing format to the mobile platform. The current administration of the test limits its usage beyond the lab environment. Creating a mobile version would facilitate ubiquitous administration of the test. Yet, the mobile design must be at least as good as its paper-based counterpart in terms of input accuracy and user interaction efforts. We developed four user interface designs, each of which featured a specific interaction approach. These approaches included displaying the 2D space of the questionnaire in its original form (M1), inputting one variable at a time on the 2D space (M2), dissolving the 2D space into two one-dimensional ordinal scales (M3), and orienting the input selections to the diagonal axes (M4). The designs were tested by a total of 34 participants, aged 18 to 52 years. The study results find the first three interaction approaches (M1-M3) effective but the fourth approach inefficient. Furthermore, the results indicate that the two-tap designs (M2 and M3) are equally as good as the one-tap design (M1).

In this paper, we present a novel framework for quantifying physiological stress at a distance via thermal imaging. The method captures stress-induced neurophysiological responses on the perinasal area that manifest as transient perspiration. We have developed two algorithms to extract the perspiratory signals from the thermophysiological imagery. One is based on morphology and is computationally efficient, while the other is based on spatial isotropic wavelets and is flexible; both require the support of a reliable facial tracker. We validated the two algorithms against the clinical standard in a controlled lab experiment where orienting responses were invoked on n=18 subjects via auditory stimuli. Then, we used the validated algorithms to quantify stress of surgeons (n=24) as they were performing suturing drills during inanimate laparoscopic training. This is a field application where the new methodology shines. It allows nonobtrusive monitoring of individuals who are naturally challenged with a task that is localized in space and requires directional attention. Both algorithms associate high stress levels with novice surgeons, while low stress levels are associated with experienced surgeons, raising the possibility for an affective measure (stress) to assist in efficacy determination. It is a clear indication of the methodology's promise and potential.

Unlike machines, we humans are prone to boredom when we perform routine activities for long periods of time. Workers' mental engagement in boring tasks diminishes, which eventually, compromises their performance. The result is a double-whammy because the workers do not get job satisfaction and their employers do not receive optimal return on investment. This paper proposes a novel way for improving workers' mental engagement and hence, enjoyment, in routine activities. Specifically, we propose to blend in routine tasks mild mental/physical challenges. To test our hypothesis, we chose to experiment on a monitoring task typical of security guard operations. We combined this routine task with an iPhone-based game to make it more enjoyable. The results from 10 participants show that their mental engagement and enjoyment were significantly higher during the combined task.

A novel template update method is proposed for facial tissue tracking in thermal clips. It is a smoothed matte approach, which provides the location along with the rate of updating. The template is able to adapt to abrupt orientation and physiological changes, while remaining robust to noise disturbances. Furthermore, it addresses successfully the difficult template drift problem. The new method was tested on tracking face regions in 40 thermal clips of individuals in varying psycho-physiological and environmental conditions. It demonstrated stability and accuracy, outperforming other template update strategies. The method promises improved performance in contact-free polygraphy and thus, it is of value in this emerging form of close range surveillance.

Thermal imaging techniques have been applied to detect and measure mental stress in polygraph screening and other applications. Mental stress is highly correlated with the activation of the corrugator muscle on the forehead. The vessels that supply blood to the corrugator muscle, proportionally to its degree of activation, are the supraorbital vessels. The rate of blood flow in these vessels can be indirectly measured via the intensity of heat emission from their segments. However, segmenting the thermal imprints of the supraorbital vessels is challenging because (1) they are fuzzy due to thermal diffusion, and (2) exhibit significant inter-individual and intra-individual variation. In this paper, a new segmentation method is proposed to extract the supraorbital vessels in thermal imagery. The new method features three steps: (1) automatic initialization of vessels; (2) automatic localization of the central lines of vessels; and (3) fast determination of vessel boundaries. The results show that the new method achieves high quality segmentation in both a simulated and a real dataset. The proposed method is expected to further increase the accuracy of stress measurements via thermal imaging.

The current dominant approaches to face recognition rely on facial characteristics that are on or over the skin. Some of these characteristics have low permanency can be altered, and their phenomenology varies significantly with environmental factors (e.g., lighting). Many methodologies have been developed to address these problems to various degrees. However, the current framework of face recognition research has a potential weakness due to its very nature. We present a novel framework for face recognition based on physiological information. The motivation behind this effort is to capitalize on the permanency of innate characteristics that are under the skin. To establish feasibility, we propose a specific methodology to capture facial physiological patterns using the bioheat information contained in thermal imagery. First, the algorithm delineates the human face from the background using the Bayesian framework. Then, it localizes the superficial blood vessel network using image morphology. The extracted vascular network produces contour shapes that are characteristic to each individual. The branching points of the skeletonized vascular network are referred to as Thermal Minutia Points (TMPs) and constitute the feature database. To render the method robust to facial pose variations, we collect for each subject to be stored in the database five different pose images (center, midleft profile, left profile, midright profile, and right profile). During the classification stage, the algorithm first estimates the pose of the test image. Then, it matches the local and global TMP structures extracted from the test image with those of the corresponding pose images in the database. We have conducted experiments on a multipose database of thermal facial images collected in our laboratory, as well as on the time-gap database of the University of Notre Dame. The good experimental results show that the proposed methodology has merit, especially with respect to the problem of low permanence over time. More importantly, the results demonstrate the feasibility of the physiological framework in face recognition and open the way for further methodological and experimental research in the area.

We present a novel approach for face recognition based on the physiological information extracted from thermal facial images. First, we delineate the human face from the background using a Bayesian method. Then, we extract the blood vessels present on the segmented facial tissue using image morphology. The extracted vascular network produces contour shapes that are unique for each individual. The branching points of the skeletonized vascular network are referred to as thermal minutia points (TMPs). These are reminiscent of the minutia points produced in fingerprint recognition techniques. During the classification stage, local and global structures of TMPs extracted from test images are matched with those of database images. We have conducted experiments on a large database of thermal facial images collected in our lab. The good experimental results show that our proposed approach has merit and promise.

An artificial vision system is presented for lie detection by analyzing face thermal image sequences.This system represents an alternative technique to the polygraph.Some of its features are: 1) it has no physical contact with the examinee, 2) it is non-intrusive, 3) it has a potential for private use, and 4) it can simultaneously analyze several persons.The proposed system is based on the detection of physiological changes in temperature in the lacrimal puncta area caused by the subtle increase in blood flow through the nearby vascular network.These changes take place when anxiety appears as a consequence of deception.Thus, the system segments the periorbital area, and tracks consecutive frames using the Kanade-Lucas-Tomasi algorithm.The results show a success rate of 79.2 % in detecting lies using a simple classification based on the comparison between the estimated temperatures in control questions, and the rest of the interrogation procedure.The performance of this system is comparable with previous works, where cameras with better specifications were used.

Research in the surveillance domain was confined for years in the military domain. Recently, as military spending for this kind of research was reduced and the technology matured, the attention of the research and development community turned to commercial applications of surveillance. In this paper we describe a state-of-the-art monitoring system developed by a corporate R&D lab in cooperation with the corresponding security business units. It represents a sizable effort to transfer some of the best results produced by computer vision research into a viable commercial product. Our description spans both practical and technical issues. From the practical point of view we analyze the state of the commercial security market, typical cultural differences between the research team and the business team and the perspective of the potential users of the technology. These are important issues that have to be dealt with or the surveillance technology will remain in the lab for a long time. From the technical point of view we analyze our algorithmic and implementation choices. We describe the improvements we introduced to the original algorithms reported in the literature in response to some problems that arose during field testing. We also provide extensive experimental results that highlight the strong points and some weaknesses of the prototype system.