Eye feature extraction from event-based data streams can be performed efficiently and with low energy consumption, offering great utility to real-world eye tracking pipelines. However, few eye feature extractors are designed to handle sudden changes in event density caused by the changes between gaze behaviors that vary in their kinematics, leading to degraded prediction performance. In this work, we address this problem by introducing the adaptive inference state space model (AISSM), a novel architecture for feature extraction that is capable of dynamically adjusting the relative weight placed on current versus recent information. This relative weighting is determined via estimates of the signal-to-noise ratio and event density produced by a complementary dynamic confidence network. Lastly, we craft and evaluate a novel learning technique that improves training efficiency. Experimental results demonstrate that the AISSM system outperforms state-of-the-art models for event-based eye feature extraction.

We report on a trans-Atlantic PhD student mobility program that connects two graduate research training initiatives in the US and Ireland, centered on developing future researchers in artificial intelligence (AI) and machine learning (ML). We discuss both the structure of the student exchange experiences and share key learnings from this international collaboration. The most important lesson learned is that providing a structured mobility program and matched visiting pairs is a highly effective way to improve learning outcomes compared to more typical ad-hoc individual visits.

This paper addresses an existing resource gap for studying complex emotional states when a speaker collaborates with a partner to solve a task. We present a novel dialogue resource — the MULTICOLLAB corpus — where two interlocutors, an instructor and builder, communicated through a Zoom call while sensors recorded eye gaze, facial action units, and galvanic skin response, with transcribed speech signals, resulting in a unique, heavily multimodal corpus. The builder received instructions from the instructor. Half of the builders were privately told to disobey the instructor’s directions. After the task, participants watched the Zoom recording and annotated their instances of frustration. In this study, we introduce this new corpus and perform computational experiments with time series transformers, using early fusion through time for sensor data and late fusion for speech transcripts. We then average predictions from both methods to recognize instructor frustration. Using sensor and speech data in a 4.5 second time window, we find that the fusion of both models yields 21% improvement in classification accuracy (with a precision of 79% and F1 of 63%) over a comparison baseline, demonstrating that complex emotions can be recognized when rich multimodal data from transcribed spoken dialogue and biophysical sensor data are fused.

Our AI-focused traineeships for graduate students integrate research and education components to contribute to diversifying the AI research workforce. We describe the program and introduce multiple strategies to achieve interdisciplinarity, diversity, equity, inclusion, and accessibility. Early evaluation results are included.

Were assessments of undergraduate researchers in a 10-week summer computing research experience impacted by the pandemic? We compare three cohort years: (1) pre-pandemic (in-person REU; prior to pandemic onset), (2) in-pandemic (remote REU, post-onset during ongoing pandemic), and (3) post-pandemic (in-person REU, post-onset with pandemic in the background). We discuss two forms of 5-point assessment ratings. First, we examine assessments of research skills on 34 questions, with a repeated measure of 3 assessments per cohort year at the beginning, middle, and end of their experience. Then, we examine assessment of presentation skills collected at the beginning vs. the end of the experience for pairs of students in all cohorts, considering 13 rating questions. Students' performance was assessed higher pre-pandemic. Also, being remote impacted completion performance. Lastly, effects linger after a return to in-person experiences, indicating adjustment challenges.

We provide an experience report about a remote framework for early undergraduate research experiences, which was thematically focused on sensing humans computationally. The framework included three complementary components. First, students experienced a team-based research cycle online, spanning formulating research questions, conducting literature review, performing fully remote human subject data collection experiments and data processing, analyzing and making inference over acquired data with computational experimentation, and disseminating findings. Second, the virtual program offered a set of professional development activities targeted to developing skills and knowledge for graduate school and research career trajectories. Third, it offered interactional and cohort-networking programming for community-building. We discuss not only the unique challenges of the virtual format and the steps put in place to address them but also the opportunities that being online afforded to innovate undergraduate research training remotely. We evaluate the remote training intervention through the organizing team's post-program reflection and the students' perceptions conveyed in exit interviews and a mid-program focus group. In addition to outlining lessons learned about more or less successful framework elements, we offer recommendations for applying the framework at other institutions as well as how to transfer activities to in-person formats.

The study of human gaze behavior in natural contexts requires algorithms for gaze estimation that are robust to a wide range of imaging conditions. However, algorithms often fail to identify features such as the iris and pupil centroid in the presence of reflective artifacts and occlusions. Previous work has shown that convolutional networks excel at extracting gaze features despite the presence of such artifacts. However, these networks often perform poorly on data unseen during training. This work follows the intuition that jointly training a convolutional network with multiple datasets learns a generalized representation of eye parts. We compare the performance of a single model trained with multiple datasets against a pool of models trained on individual datasets. Results indicate that models tested on datasets in which eye images exhibit higher appearance variability benefit from multiset training. In contrast, dataset-specific models generalize better onto eye images with lower appearance variability.

Current methods for segmenting eye imagery into skin, sclera, pupil, and iris cannot leverage information about eye motion. This is because the datasets on which models are trained are limited to temporally non-contiguous frames. We present Temporal RIT-Eyes, a Blender pipeline that draws data from real eye videos for the rendering of synthetic imagery depicting natural gaze dynamics. These sequences are accompanied by ground-truth segmentation maps that may be used for training image-segmentation networks. Temporal RIT-Eyes relies on a novel method for the extraction of 3D eyelid pose (top and bottom apex of eyelids/eyeball boundary) from raw eye images for the rendering of gaze-dependent eyelid pose and blink behavior. The pipeline is parameterized to vary in appearance, eye/head/camera/illuminant geometry, and environment settings (indoor/outdoor). We present two open-source datasets of synthetic eye imagery: sGiW is a set of synthetic-image sequences whose dynamics are modeled on those of the Gaze in Wild dataset, and sOpenEDS2 is a series of temporally non-contiguous eye images that approximate the OpenEDS-2019 dataset. We also analyze and demonstrate the quality of the rendered dataset qualitatively and show significant overlap between latent-space representations of the source and the rendered datasets.

To examine perceptions of faculty mentors of undergraduate research and their supervisors, this work discusses the results of surveys administered after 3 years of a summer CS-focused REU Site program. One survey was completed by administrators of faculty research mentors--deans and chairs--and the other was completed by faculty mentors. The surveys indicated a disconnect between how the groups assessed undergraduate research mentoring as an indicator of faculty productivity, and overt vs. covert recognition of undergraduate mentoring. Additional topics explored the effectiveness of internal communication of program outcomes and ways to improve it, as well as post-program continued mentoring engagement linking to perceptions of long-term student benefits.

Ellipse fitting, an essential component in pupil or iris tracking based video oculography, is performed on previously segmented eye parts generated using various computer vision techniques. Several factors, such as occlusions due to eyelid shape, camera position or eyelashes, frequently break ellipse fitting algorithms that rely on well-defined pupil or iris edge segments. In this work, we propose training a convolutional neural network to directly segment entire elliptical structures and demonstrate that such a framework is robust to occlusions and offers superior pupil and iris tracking performance (at least 10% and 24% increase in pupil and iris center detection rate respectively within a two-pixel error margin) compared to using standard eye parts segmentation for multiple publicly available synthetic segmentation datasets.

Deep neural networks for video-based eye tracking have demonstrated resilience to noisy environments, stray reflections, and low resolution. However, to train these networks, a large number of manually annotated images are required. To alleviate the cumbersome process of manual labeling, computer graphics rendering is employed to automatically generate a large corpus of annotated eye images under various conditions. In this work, we introduce a synthetic eye image generation platform that improves upon previous work by adding features such as an active deformable iris, an aspherical cornea, retinal retro-reflection, gaze-coordinated eye-lid deformations, and blinks. To demonstrate the utility of our platform, we render images reflecting the represented gaze distributions inherent in two publicly available datasets, NVGaze and OpenEDS. We also report on the performance of two semantic segmentation architectures (SegNet and RITnet) trained on rendered images and tested on the original datasets.

As large eye-tracking datasets are created, data privacy is a pressing concern for the eye-tracking community. De-identifying data does not guarantee privacy because multiple datasets can be linked for inferences. A common belief is that aggregating individuals' data into composite representations such as heatmaps protects the individual. However, we analytically examine the privacy of (noise-free) heatmaps and show that they do not guarantee privacy. We further propose two noise mechanisms that guarantee privacy and analyze their privacy-utility tradeoff. Analysis reveals that our Gaussian noise mechanism is an elegant solution to preserve privacy for heatmaps. Our results have implications for interdisciplinary research to create differentially private mechanisms for eye tracking.

Driven by significant investments from the gaming, film, advertising, and customer service industries among others, efforts across many different fields are converging to create realistic representations of humans that look like (computer graphics), sound like (natural language generation), move like (motion capture), and reason like (artificial intelligence) real humans. The ultimate goal of this work is to push the boundaries even further by exploring the development of realistic self-organized virtual humans that are capable of demonstrating coordinated behaviors across different modalities. Eye movements, for example, may be accompanied by changes in facial expression, head orientation, posture, gait properties, or speech. Traditionally however, these modalities are captured and modeled separately and this disconnect contributes to the well-known uncanny valley phenomenon. We focus initially on facial modalities, in particular, coordinated eye and head movements (and eventually facial expressions), but our proposed data-driven framework will be able to accommodate other modalities as well. transfer [Laine et al. 2017]. Despite these advances, the resulting renderings or animations are often still distinguishable from a real human, sometimes in unsettling ways - the so called uncanny valley phenomenon [Mori et al. 2012]. We argue that the traditional approach of capturing and modeling various human modalities separately contributes this effect. In this work, we focus on capturing, transferring, and generating realistic coordinated facial modalities (eye movements, head movements, and eventually facial expressions). We envision a flexible framework that can be extended to accommodate other modalities as well.

In this work, we aim to gain better insights into the underlying behaviors of people when they attempt to suppress amusement, the positive emotion experienced, specifically from finding something funny. We aim to better understand the different physiological manifestations that occur when this suppression happens. We investigate this phenomenon by observing the presence/absence of action units (AUs) during amusement expression and suppression. We also record galvanic skin responses (GSR) to more closely observe if there are major differences in physiological manifestations during amusement expression versus suppression. This study was performed as a part of a larger one on deceit detection since amusement suppression can also be viewed as a form deceit, based on the context. We showed that the overall facial expression signatures were unsurprisingly very different for amusement expression and suppression; the features associated with positive emotions were clearly dominant during free expression and sadness was especially dominant during suppression. In observing the GSR readings, free expression and suppression manifested quite differently across individuals but similarly within the same individual, suggesting that the internal state or emotional arousal level of the participants were not altered significantly when trying to suppress amusement. Finally, in more than 75% of the cases, we found that arousal induced by amusement could not readily be eliminated, even when the individuals succeeded in masking it on their faces. This further validates the claim in the social psychology literature that suppression decreases expressive behavior, but does not decrease the emotional experience (measured via GSR).

We present work-in-progress reflecting on the initial year of a distinctive summer Research Experiences for Undergraduates (REU) program. Our REU model combines fundamental research in computational sensing with a scholarly context that connects computer science with computational liberal arts. Students are intellectually stimulated to make sense of people's behaviors and cognitive processes with multimodal sensing hardware and software. In doing so, they explore the fundamental challenges found at the intersection of computing, the human experience, and scientific interrogation. The placement of the human experience at the core of the research theme enables an environment that stimulates and cultivates an innovative undergraduate research model. We highlight outcomes from the first year and discuss three emerging practices that are central to our REU framework: (1) team-based collaborative training; (2) transdisciplinary integration; and (3) systematic prioritization of inclusiveness. We also describe how these practices are incorporated into our overall undergraduate research framework and touch upon lessons learned from feedback collected.

Eye-tracking provides a mechanism for researchers to monitor where subjects deploy their visual attention. Eye-tracking has been used to gain insights into how humans scrutinize faces, however the majority of these studies were conducted using desktop-mounted eye-trackers where the subject sits and views a screen during the experiment. The stimuli in these experiments are typically photographs or videos of human faces. In this paper we present a novel approach using head-mounted eye-trackers which allows for automatic generation of gaze statistics for tasks performed in real-world environments. We use a trained hierarchy of Haar cascade classifiers to automatically detect and segment faces in the eye-tracker's scene camera video. We can then determine if fixations fall within the bounds of the face or other possible regions of interest and report relevant gaze statistics. Our method is easily adaptable to any feature-trained cascade to allow for rapid object detection and tracking. We compare our results with previous research on the perception of faces in social environments. We also explore correlations between gaze and confidence levels measured during a mock interview experiment.

Many computer vision algorithms are biologically inspired and designed based on the human visual system. Convolutional neural networks (CNNs) are similarly inspired by the primary visual cortex in the human brain. However, the key difference between current visual models and the human visual system is how the visual information is gathered and processed. We make eye movements to collect information from the environment for navigation and task performance. We also make specific eye movements to important regions in the stimulus to perform the task-at-hand quickly and efficiently. Researchers have used expert scanpaths to train novices for improving the accuracy of visual search tasks. One of the limitations of such a system is that we need an expert to examine each visual stimuli beforehand to generate the scanpaths. In order to extend the idea of gaze guidance to a new unseen stimulus, there is a need for a computational model that can automatically generate expert-like scanpaths. We propose a model for automatic scanpath generation using a convolutional neural network (CNN) and long short-term memory (LSTM) modules. Our model uses LSTMs due to the temporal nature of eye movement data (scanpaths) where the system makes fixation predictions based on previous locations examined.

Computer-based gaze guidance techniques have important applications in computer graphics, data visualization, image analysis, and training. Bailey et al. [2009] showed that it is possible to influence exactly where attention is allocated using a technique called Subtle Gaze Direction (SGD). The SGD approach combines eye tracking with brief image-space modulations in the peripheral regions of the field of view to guide viewer gaze about a scene. A fast eye-tracker is used to monitor gaze in real-time and the modulations are terminated before they can be scrutinized by the viewer's high acuity foveal vision. The SGD technique has been shown to improve spatial learning, visual search task performance, and problem solving in static digital imagery [Sridharan et al. 2012]. However, guiding attention in videos is challenging due to competing motion cues in the visual stimuli. We propose a novel method that uses scene saliency (spatial information) and optical flow (temporal information) to enable gaze guidance in dynamic scenes. The results of a user study show that the accuracy of responses to questions related to target regions in videos was higher among subjects who were gaze guided with our approach compared to a control group that was not actively guided.

The human gait cycle is incredibly efficient and stable largely because of the use of advance visual information to make intelligent selections of heading direction, foot placement, gait dynamics, and posture when faced with terrain complexity [Patla and Vickers 1997; Patla and Vickers 2003; Matthis and Fajen 2013; Matthis and Hayhoe 2015]. This is behaviorally demonstrated by a coupling between saccades and foot placement.

We present a novel pipeline for localizing a free roaming eye tracker within a LiDAR-based 3D reconstructed scene with high levels of accuracy. By utilizing a combination of reconstruction algorithms that leverage the strengths of global versus local capture methods and user-assisted refinement, we reduce drift errors associated with Dense-SLAM techniques. Our framework supports region-of-interest (ROI) annotation and gaze statistics generation and the ability to visualize gaze in 3D from an immersive first person or third person perspective. This approach gives unique insights into viewers' problem solving and search task strategies and has high applicability in complex static environments such as crime scenes.

Most computer systems require user authentication, which has led to an increase in the number of passwords one has to remember. In this paper we explore if spatial visual cues can be used to improve password recollection. Specifically, we consider if associating each character in a password to user-defined spatial regions in an image facilitates better recollection. We conduct a user study where participants were asked to recall randomly generated numeric passwords under the following conditions: no image association (No-Image), image association (Image-Only), image association combined with overt visual cues (Overt-Guidance), and image association combined with subtle visual cues (Subtle-Guidance). We measured the accuracy of password recollection and response time as well as average dwell-time at target locations for the gaze guided conditions. Subjects performed significantly better on password recollection when they were actively guided to regions in the associated image using overt visual cues. Accuracy of password recollection using subtle cues was also higher than the No-Image and Image-Only conditions, but the effect was not significant. No significant difference was observed in the average dwell-times between the overt and subtle guidance approaches.

No abstract available.

Humans rely heavily on spatial information to perform everyday tasks. Developing good spatial understanding is highly dependent on how the viewer's attention is deployed to specific locations in a scene. Bailey et al. [2009] showed that it is possible to influence exactly where attention is allocated using a technique called Subtle Gaze Direction (SGD). The SGD approach combines eye tracking with subtle image-space modulations to guide viewer gaze about a scene. The modulations are presented to peripheral regions of the field of view, in order to attract the viewer's attention, but are terminated before the viewer can scrutinize them with their high acuity foveal vision. It was observed that subjects who were guided using SGD performed significantly better in recollecting the count and location of target objects, however no significant performance improvement was observed in identifying the shape of the target objects [Bailey et al. 2012]. Also, in previous studies involving SGD, the target locations were manually chosen by researchers. This paper addresses these two limitations. We present a novel technique for automatically selecting target regions using visual saliency and key features in the image. The shape recollection issue is solved by modulating a rough outline of the target object obtained using an edge map composed from a pyramid of low spatial frequency maps of the original image. Results from a user study show that the influence of this approach significantly improved accuracy of target count recollection, location recollection, as well as shape recollection without any manual intervention. Furthermore our technique correctly predicted 81% of the target regions without any prior knowledge of the recollection task being assigned to the viewer. This work has implications for a wide range of applications including spatial learning in virtual environments as well as image search applications, virtual training and perceptually based rendering.

Virtual reality headsets and immersive head-mounted displays have become commonplace and have found their applications in digital gaming, film and education. An immersive perception is created by surrounding the user of the VR system with photo-realisitic scenes, sound or other stimuli (e.g. haptic) that provide an engrossing experience to the viewer. The ability to interact with the objects in the virtual environment have added greater interest for its use in learning and education. In this proposed work we plan to explore the ability to subtly guide viewers' attention to important regions in a controlled 3D virtual scene. Subtle gaze guidance [Bailey et al. 2009] approach combines eye-tracking and subtle imagespace modulations to guide viewer's attention about a scene. These modulations are terminated before the viewer can fixate on them using their high acuity foveal vision. This approach is preferred over other overt techniques that make permanent changes to the scene being viewed. This approach has also been tested in controlled realworld environments [Booth et al. 2013]. The key challenge to such a system, is the need for an external projector to present modulations on the scene objects to guide viewer's attention. However a VR system enables the user to view and interact in a 3D scene that is close to reality, thereby allowing researchers to digitally manipulate the 3D scene for active gaze guidance.

An ongoing challenge with head-mounted eye-trackers is how to analyze the data from multiple individuals looking at the same scene. Our work focuses on static scenes. Previous approaches involve capturing a high resolution panorama of the scene and then mapping the fixations from all viewers onto this panorama. However such approaches are limited as they typically restrict all viewers to observe the scene from the same stationary vantage point. We present a system which incorporates user-perspective gaze data with a 3D reconstruction of the scene. The system enables the visualization of gaze data from multiple viewers on a single 3D model of the scene instead of multiple 2D panoramas. The subjects are free to move about the scene as they see fit which leads to more natural task performance. Furthermore since it is not necessary to warp the scene camera video into a flat panorama, our system preserves the relative positions of the objects in the scene during the visualization process. This gives better insight into the viewer's problem solving and search task strategies. Our system has high applicability in complex static environments such as crime scenes and marketing studies in retail stores.

Modern media recommendation systems work based on the content of videos previously seen. While this may work for the habitual viewer, it is not always appropriate for many users whose tastes change based on their moods. A lot can be gathered from a person's facial expression while they are engaged in an activity, for example by observing someone as they view a film, we can likely tell if they enjoyed it or not. While a content providing company may not have the resources to employ human observers to watch audiences (or the inclination to do so, because it is not a very appealing idea), an automatic system that does this would be more feasible. Photoplethysmography is a field in which a person's physiological details can be gathered optically without requiring any physical contact with that person. By monitoring the intensity of light on a patch of the viewer's skin, we can accurately estimate their heart rate [Poh et al. 2011]. Our system combines heart rate measurement and facial expressions to quantify how much a user enjoys a video.

We present a framework for collaborative image analysis where gaze information is shared across all users. A server gathers and broadcasts fixation data from/to all clients and the clients visualize this information. Several visualization options are provided. The system can run in real-time or gaze information can be recorded and shared the next time an image is accessed. Our framework is scalable to large numbers of clients with different eye tracking devices. To evaluate our system we used it within the context of a spot-the-differences game. Subjects were presented with 10 image pairs each containing 5 differences. They were given one minute to detect the differences in each image. Our study was divided into three sessions. In session 1, subjects completed the task individually, in session 2, pairs of subjects completed the task without gaze sharing, and in session 3, pairs of subjects completed the task with gaze sharing. We measured accuracy, time-to-completion and visual coverage over each image to evaluate the performance of subjects in each session. We found that visualizing shared gaze information by graying out previously scrutinized regions of an image significantly increases the dwell time in the areas of the images that are relevant to the task (i.e. the regions where differences actually occurred). Furthermore, accuracy and time-to-completion also improved over collaboration without gaze sharing though the effects were not significant. Our framework is useful for a wide range of image analysis applications which can benefit from a collaborative approach.

The ability to direct a viewer's attention has important applications in computer graphics, data visualization, image analysis, and training. Existing computer-based gaze manipulation techniques, which direct a viewer's attention about a display, have been shown to be effective for spatial learning, search task completion, and medical training applications. In this work we extend the concept of gaze manipulation beyond digital imagery to include controlled, real-world environments. We address two main challenges in guiding attention to real-world objects: determining what object the viewer is currently paying attention to, and providing (projecting) a visual cue on a different part of the scene in order to draw the viewer's attention there. Our system consists of a pair of eye-tracking glasses to determine the viewer's gaze location, and a projector to create the visual cue in the physical environment. The results of a user study show that we can effectively direct the viewer's gaze in the real-world scene. Our technique has applicability in a wide range of instructional environments, including pilot training and driving simulators.

Narrative art tells a story, either as a moment in an ongoing story or as a sequence of events unfolding over time. In many works of art separate panels within the same frame are used to depict the sequence of events. Often, there is no clear delineation between these panels, or any indication of the optimal viewing order. To improve visual literacy we propose using Subtle Gaze Direction (SGD) to direct the viewers gaze across an image in a manner which reveals the story. SGD uses small image space modulations in the luminance channel to guide a viewer's gaze about an image without disrupting their normal visual experience. Using a simple ordering task we compared performance using no modulation and using subtle modulation with the correct order of narrative episodes as intended by the artist. Results from experiments show improved performance when SGD is employed. This experiment establishes the potential of the method as an aid to visual navigation in images where the viewing order is unclear.

Recent work in eye tracking and facial expression analysis has enabled new forms of hands-free user interaction with computer applications. There has been particular emphasis on using these mechanisms individually for drawing and other artwork related applications (e.g. [Shugrina et al. 2006] [Hornof et al. 2004]).In this paper we explore the combination of the two modalities and describe a system for hands-free drawing that integrates the use of the eyes and the face as a means for user control. We present a general architecture for incorporating eye tracking and facial expression analysis into a computer application and utilize this architecture in the design and implementation of a drawing application.

Contents of Visual Short-Term Memory depend highly on viewer attention. It is possible to influence where attention is allocated using a technique called Subtle Gaze Direction (SGD). SGD combines eye tracking with subtle image-space modulations to guide viewer gaze about a scene. Modulations are terminated before the viewer can scrutinize them with high acuity foveal vision. This approach is preferred to overt techniques that require permanent alterations to images to highlight areas of interest. In our study, participants were asked to recall the location of objects or regions in images. We investigated if using SGD to guide attention to these regions would improve recall. Results showed that the influence of SGD significantly improved accuracy of target count and spatial location recall. This has implications for a wide range of applications including spatial learning in virtual environments as well as image search applications, virtual training and perceptually based rendering.

In this work we present a framework for enabling the use of both eye gaze and hand gestures for interaction within a 3D virtual world. We define a set of natural interaction mechanisms for manipulation of objects within the 3D space and describe a prototype implementation based on Second Life that allows these mechanisms to be used in that world. We also explore how these mechanisms can be extended to other spatial tasks such as camera positioning and motion.

This paper presents strategies for providing successful Research Experiences for Undergraduates (REU). The authors have advised several undergraduates on research for the past few years, and have jointly supervised around twenty-five students, over two summers, on a project funded by an NSF-funded REU program in areas relating to the visualization of astrophysical data using high performance file systems. Several of these student projects have led to research publications. The paper briefly motivates the need for research in modern computing and engineering education. It then presents specific details about the development of summer REU programs including how to: secure funding and institutional support; plan a summer program including the design of scalable research projects; develop strategies to advertise and recruit students, especially from underrepresented groups; create a dynamic research and social environment through one-on-one mentoring; develop appropriate assessment and evaluation processes; and track student participants after they graduate from the program.

Co-clustering is the problem of deriving sub-matrices from the larger data matrix by simultaneously clustering rows and columns of the data matrix. Traditional co-clustering techniques are inapplicable to problems where the relationship between the instances (rows) and features (columns) evolve over time. Not only is it important for the clustering algorithm to adapt to the recent changes in the evolving data, but it also needs to take the historical relationship between the instances and features into consideration. We present ESCC, a general framework for evolutionary spectral co-clustering. We are able to efficiently co-cluster evolving data by incorporation of historical clustering results. Under the proposed framework, we present two approaches, Respect To the Current (RTC), and Respect To Historical (RTH). The two approaches differ in the way the historical cost is computed. In RTC, the present clustering quality is of most importance and historical cost is calculated with only one previous time-step. RTH, on the other hand, attempts to keep instances and features tied to the same clusters between time-steps. Extensive experiments performed on synthetic and real world data, demonstrate the effectiveness of the approach.

Projects funded by the National Science Foundation (NSF) Research Experiences for Undergraduates (REU) program aim to (a) enhance participation of students who otherwise might not have research opportunities, and (b) increase the number of students interested in graduate programs, thus expanding the pool of a well-trained scientific workforce. To provide meaningful experiences for these students, REU projects make use of a set of interesting, appropriate research problems that can be tackled in 8 to 10 weeks in summer.

A new experiment is presented that demonstrates the usefulness of an image space modulation technique called subtle gaze direction (SGD) for guiding the user in a simple searching task. SGD uses image space modulations in the luminance channel to guide a viewer's gaze about a scene without interrupting their visual experience. The goal of SGD is to direct a viewer's gaze to certain regions of a scene without introducing noticeable changes in the image. Using a simple searching task, we compared performance using no modulation, using subtle modulation, and using obvious modulation. Results from the experiments show improved performance when using subtle gaze direction, without affecting the user's perception of the image. We then extend the experiment to evaluate performance with the presence of distractors. The distractors took the form of extra modulations, which do not correspond to a target in the image. Experimentation shows, that, even in the presence of distractors, more accurate results are returned on a simple search task using SGD, as compared to results returned when no modulation at all is used. Results establish the potential of the method for a wide range of applications including gaming, perceptually based rendering, navigation in virtual environments, and medical search tasks.

This article presents a novel technique that combines eye-tracking with subtle image-space modulation to direct a viewer's gaze about a digital image. We call this paradigm subtle gaze direction . Subtle gaze direction exploits the fact that our peripheral vision has very poor acuity compared to our foveal vision. By presenting brief, subtle modulations to the peripheral regions of the field of view, the technique presented here draws the viewer's foveal vision to the modulated region. Additionally, by monitoring saccadic velocity and exploiting the visual phenomenon of saccadic masking, modulation is automatically terminated before the viewer's foveal vision enters the modulated region. Hence, the viewer is never actually allowed to scrutinize the stimuli that attracted her gaze. This new subtle gaze directing technique has potential application in many areas including large scale display systems, perceptually adaptive rendering, and complex visual search tasks.

We apply simplified image-based lighting methods to reduce the equipment, cost, time, and specialized skills required for high-quality photographic lighting of desktop-sized static objects such as museum artifacts. We place the object and a computer-steered moving-head spotlight inside a simple foam-core enclosure and use a camera to record photos as the light scans the box interior. Optimization, guided by interactive user sketching, selects a small set of these photos whose weighted sum best matches the user-defined target sketch. Unlike previous image-based relighting efforts, our method requires only a single area light source, yet it can achieve high-resolution light positioning to avoid multiple sharp shadows. A reduced version uses only a handheld light and may be suitable for battery-powered field photography equipment that fits into a backpack.

This paper presents several methods for shading meshes from scanned paint samples that represent dark to light transitions. Our techniques emphasize artistic control of brush stroke texture and color. We first demonstrate how the texture of the paint sample can be separated from its color gradient. We demonstrate three methods, two real-time and one off-line, for producing rendered, shaded images from the texture samples. All three techniques use texture synthesis to generate additional paint samples. Finally, we develop metrics for evaluating how well each method achieves our goal in terms of texture similarity, shading correctness and temporal coherence.