Phone scams pose a significant threat to individuals and communities, causing substantial financial losses and emotional distress. Despite ongoing efforts to combat these scams, scammers continue to adapt and refine their tactics, making it imperative to explore innovative countermeasures. This research explores the potential of large language models (LLMs) to provide detection of fraudulent phone calls. By analyzing the conversational dynamics between scammers and victims, LLM-based detectors can identify potential scams as they occur, offering immediate protection to users. While such approaches demonstrate promising results, we also acknowledge the challenges of biased datasets, relatively low recall, and hallucinations that must be addressed for further advancement in this field.

Phone scams remain a pervasive threat to both personal safety and financial security worldwide. Recent advances in large language models (LLMs) have demonstrated strong potential in detecting fraudulent behavior by analyzing transcribed phone conversations. However, these capabilities introduce notable privacy risks, as such conversations frequently contain sensitive personal information that may be exposed to third-party service providers during processing. In this work, we explore how to harness LLMs for phone scam detection while preserving user privacy. We propose MASK (Modular Adaptive Sanitization Kit), a trainable and extensible framework that enables dynamic privacy adjustment based on individual preferences. MASK provides a pluggable architecture that accommodates diverse sanitization methods-from traditional keyword-based techniques for high-privacy users to sophisticated neural approaches for those prioritizing accuracy. We also discuss potential modeling approaches and loss function designs for future development, enabling the creation of truly personalized, privacy-aware LLM-based detection systems that balance user trust and detection effectiveness, even beyond phone scam context.

Object detection can localize and identify objects in images, and it is extensively employed in critical multimedia applications such as security surveillance and autonomous driving. Despite the success of existing object detection models, they are often evaluated in ideal scenarios where captured images guarantee the accurate and complete representation of the detecting scenes. However, images captured by image sensors may be affected by different factors in real applications, including cyber-physical attacks. In particular, attackers can exploit hardware properties within the systems to inject electromagnetic interference so as to manipulate the images. Such attacks can cause noisy or incomplete information about the captured scene, leading to incorrect detection results, potentially granting attackers malicious control over critical functions of the systems. This paper presents a research work that comprehensively quantifies and analyzes the impacts of such attacks on state-of-the-art object detection models in practice. It also sheds light on the underlying reasons for the incorrect detection outcomes.

Automatic detection of backchannel has great potential to enhance artificial mediators, which indicate listeners' attention and agreement in human communication. It is often expressed by subtle non-verbal cues that occur briefly and sparsely. Focusing on identifying and locating these subtle cues (i.e., their occurrence moment and the involved body parts), this paper proposes a novel approach for backchannel detection. In particular, our model utilizes temporal- and modality-attention modules to determine and lead the model to pay more attention to both the indicative moment and the accompanying body parts at that specific time. It achieves an accuracy of 68.6% on the testing set in MultiMediate'23 backchannel detection challenge, outperforming the counterparts. Furthermore, we conducted an ablation study to thoroughly understand the contributions of our model. This study underscores the effectiveness of our selection of modality inputs and the importance of the two attention modules in our model.

Real-time engagement estimation holds significant potential across various research areas, particularly in the realm of human-computer interaction. It empowers artificial agents to dynamically adjust their responses based on user engagement levels, fostering more intuitive and immersive interactions. Despite the strides in automating real-time engagement estimation, the task remains challenging in real-world settings, especially when handling multi-modal human social signals. Capitalizing on human body and audio signals, this paper explores the appropriate feature representations of different modalities and effective modelling of dual conversations. This results in a novel and efficient multi-modal engagement detection model.We thoroughly evaluated our method in the MultiMediate'23 grand challenge. It performs consistently, with a notable improvement over the baseline model. Specifically, while the baseline achieves a concordance correlation coefficient (CCC) of 0.59, our approach yields a CCC of 0.70, suggesting its promising efficacy in real-life engagement detection.

As an impactful experiential learning pedagogy in higher education, service-learning (SL) can enhance students' academic learning and their sense of community and social responsibility by involving them in comprehensive community services. Much extant literature has justified the positive impacts of SL. However, the lack of quantitative analysis on identifying significant learning and course factors that strongly impact students' SL outcomes limits SL's further enhancement and adaptive development. This paper proposes to use machine learning approaches for modeling and identifying key learning factors in SL. We collect and study a large-scale dataset, including students' feedback on learning factors related to the different student experiences, course elements, and self-perceived learning outcomes. Machine learning algorithms are applied to model the various learning factors, contributing to effective classification models that predict students' learning outcomes using their evaluation on the learning factors. The most predictive model is then selected to identify a key set of important variables most indicative to students' SL outcomes. Our experiment results show that learning factors related to study challenges and interactions have significant positive impacts on students' learning gains. We believe that this paper will benefit future studies in this field.

Writing is one of the most common activities undertaken on a computer, and the activity of writing has been widely studied. Given that writing is an intensively cognitive process, it makes sense that the type of writing that is being produced would have an effect on the writer’s gaze and typing behaviors. However, only a few studies have explored this relationship. In this paper, we study the gaze-typing behaviors, specifically, the coordination between eye gaze and typing dynamics, of writers who are producing original articles in different genres: reminiscent, logical and creative. Our study focuses on Chinese typing, particularly via the Pinyin input method, which generates text via a two step method, and requires additional cognitive processes compared to typing in phonographic languages such as English. Our study involves 46 native Chinese speakers of varying ages from children to elderly. Our method deploys statistics- and sequence-based features to infer the mental state of the author during the writing process. The statistics-based features focus on modeling the overall gaze-typing behaviors during the process and the sequence-based features focus on the transition of the gaze-typing behaviors as the piece of writing progresses. Using a linear support-vector machine, we achieve an overall accuracy over 88% for the article-genre detection by using a leave-one-subject-out cross-validation evaluation.

Fire fighter fatalities and injuries in the U.S. remain too high and fire fighting too hazardous. Until now, fire fighters rely only on their experience to avoid life-threatening fire events, such as flashover. In this paper, we describe the development of a flashover prediction model which can be used to warn fire fighters before flashover occurs. Specifically, we consider the use of a fire simulation program to generate a set of synthetic data and an attention-based bidirectional long short-term memory to learn the complex relationships between temperature signals and flashover conditions. We first validate the fire simulation program with temperature measurements obtained from full-scale fire experiments. Then, we generate a set of synthetic temperature data which account for the realis-tic fire and vent opening conditions in a multi-compartment structure. Results show that our proposed method achieves promising performance for prediction of flashover even when temperature data is completely lost in the room of fire origin. It is believed that the flashover prediction model can facilitate the transformation of fire fighting tactics from traditional experience-based decision marking to data-driven decision marking and reduce fire fighter deaths and injuries.

The task of summarizing a document is a complex task that requires a person to multitask between reading and writing processes. Since a person's cognitive load during reading or writing is known to be dependent upon the level of comprehension or difficulty of the article, this suggests that it should be possible to analyze the cognitive process of the user when carrying out the task, as evidenced through their eye gaze and typing features, to obtain an insight into the different difficulty levels. In this paper, we categorize the summary writing process into different phases and extract different gaze and typing features from each phase according to characteristics of eye-gaze behaviors and typing dynamics. Combining these multimodal features, we build a classifier that achieves an accuracy of 91.0% for difficulty level detection, which is around 55% performance improvement above the baseline and at least 15% improvement above models built on a single modality. We also investigate the possible reasons for the superior performance of our multimodal features.

Refractive errors, such as myopia and astigmatism, can lead to severe visual impairment if not detected and corrected in time. Traditional methods of refractive error diagnosis rely on well-trained optometrists operating expensive and importable devices, constraining the vision screening process. Advance in smartphone camera has enabled novel low-cost ubiquitous vision screening to detect refractive error or ametropia through eye image processing, based on the principle of photorefraction. However, contemporary smartphone-based methods rely heavily on hand-crafted features and sufficiency of well-labeled data. To address these challenges, this paper exploits active learning methods with a set of Convolutional Neural Network features encoding information of human eyes from pre-trained gaze estimation model. This enables more effective training on refractive error detection models with less labeled data. Our experimental results demonstrate the encouraging effectiveness of our active learning approach. The new set of features is able to attain screening accuracy of more than 80% with mean absolute error less than 0.66, meeting the expectation of optometrists for 0.5 to 1. The proposed active learning also requires significantly fewer training samples of 18% in achieving satisfactory performance.

Uncorrected refractive errors can lead to permanent debilitating eye conditions if not corrected in a timely manner. Contemporary diagnostic methods rely on the professional acumen of optometrists and the use of expensive devices, which may not be easily accessible to all. According to the optical principle of photorefraction, refractive error can be estimated based on a relative pupil and crescent size of an eye image taken by a camera from a specified working distance. A low-cost approach would be to leverage smartphones with cameras for this purpose. However, the poor image quality generated from basic smartphones poses a challenge for the current approach as they often fail to accurately distinguish the crescent from the iris. We propose a novel method to detect and accurately measure the iris and crescent from smartphone photos. Based on this method, we further propose a set of features for machine learning to build our refractive error estimation model. The performance of our models are evaluated in an in-depth experiment.

This paper presents a general framework of an innovative TOUR model with four interrelated elements: Transformation, Outreach, Unification and Reinforcement, enhancing the learning experiences of computer science/computing students via a study tour. We brought our students on an overseas trip as an integral part of an academic course. The immediate goal of the study tour was to gear the students toward the program learning outcomes. Another goal was to take a first step in supporting the TOUR model, to immerse students within these four interrelated important learning and self-improvement elements, and to support each student in becoming a better person. The study tour comprises six core learning activities aligned with the TOUR elements. Evaluations were conducted based on a student survey and self-reflection. The results provide valuable insights into the design of the computer science student study tour, as well as the potential value of the TOUR model.

Typing is one of the most common activities that are undertaken on a computer. It would therefore be interesting to investigate whether it is possible to deduce characteristics of the user, such as their age or the type of the document that they are writing, just simply from typing dynamics. In this paper, we study the coordination between eye gaze and typing dynamics, or the gaze-typing behavior, of subjects who are producing original text. We focus upon the differences between different age groups (children vs elderly seniors) and different genres of writing (reminiscent, logical and creative). Using machine-learning, we achieve an accuracy of 93.5% for age detection and 61.1% for the article-category detection, using a leave-one-subject-out cross-validation evaluation, which is 44% and 28% higher than baselines.

Fall is a common cause of severe injuries that may lead to irreversible body damage and even death. A real-time fall monitoring system can reveal a fall in time for timely medical aid to a victim. This is particularly important in the context of mobile healthcare. Fall detection with most contemporary wearable devices relied solely on acceleration signals, often not flexible and robust enough. In this paper, we propose to deploy body signals in a multi-modality approach. Besides the common acceleration signals, we also make use of physiological signals returned by wearable devices for multiple modalities. Fall detection would not fail easily even if some acceleration signals become ineffective. Our experiment results indicate that we are able to attain an accuracy of more than 96%. An in-depth evaluation demonstrates that physiological signals can contribute in distinguishing falls from actions generating similar acceleration signals, such as jumps, sit-downs and walking-downstairs.

Internal thought refers to the process of directing attention away from a primary visual task to internal cognitive processing. It is pervasive and closely related to primary task performance. As such, automatic detection of internal thought has significant potential for user modeling in human-computer interaction and multimedia applications. Despite the close link between the eyes and the human mind, only few studies have investigated vergence behavior during internal thought and none has studied moment-to-moment detection of internal thought from gaze. While prior studies relied on long-term data analysis and required a large number of gaze characteristics, we describe a novel method that is user-independent, computationally light-weight and only requires eye vergence information readily available from binocular eye trackers. We further propose a novel paradigm to obtain ground truth internal thought annotations by exploiting human blur perception. We evaluated our method during natural viewing of lecture videos and achieved a 12.1% improvement over the state of the art. These results demonstrate the effectiveness and robustness of vergence-based detection of internal thought and, as such, open new research directions for attention-aware interfaces.

Advancement of micro-electromechanical systems enables easy daily activity and physiological data collection with a smart wristband and smartphone. Making use of those signals in various intelligent algorithm can contribute much to trending m-health applications. The ability of continuously monitoring physical activities and stress level can help users to better track their health condition. In this study, we propose to recognize different physical activities and detect long lasting stress level based on the 3-axis acceleration signals and physiological signals. We are able to achieve accuracy of around 97% for physical activities recognition and more than 80% for stress detection. We also discover that physiological signals alone cannot distinguish well between the high intensity activities and the stress condition.

User experience could be enhanced if the computer could understand human interaction intention. For instance, it could react to intercept and prevent interaction errors. This paper presents an approach to predicting users intention in interaction tasks based on past mouse movements. We adopt a long short-term memory (LSTM) model to predict the users» intention via their next mouse click interaction, upon being trained with past mouse interaction behaviors. To evaluate, we consider two scenarios in daily computer usage: a more structured crowdsourcing annotation task and a more free-form, open-ended web search task. Our results indicate that we could predict the next interaction event with reasonable accuracy. We also conducted a pilot study to investigate the possibility of applying our model for non-intentional mouse click detection. We believe that our findings would be beneficial towards the development of better intelligent agents.

Detecting human fight behavior from videos is important in social signal processing, especially in the context of surveillance. However, the uncommon occurrence of real human fight events generally restricts the data collection for fight detection in machine learning, and thus hampers the performance of contemporary data-driven approaches. To address this challenge, we present a novel cross-species learning method with a set of low-computational cost motion features for fight detection. It effectively circumvents the problem of limited human fight data for data-demaining approaches. Our method exploits the intrinsic commonality between human and animal fights, such as the physical acceleration of moving body parts. It also leverages an ensemble learning mechanism to adapt useful knowledge from similar source subsets across species. Our evaluation results demonstrate the effectiveness of the proposed feature representation for cross-species adaptation. We believe that cross-species learning is not only a promising solution to the data constraint issue, but it also sheds lights on the studies of other human mental and social behaviors in cross-disciplinary research.

Understanding human visual attention is essential for understanding human cognition, which in turn benefits human--computer interaction. Recent work has demonstrated a Personalized, Auto-Calibrating Eye-tracking (PACE) system, which makes it possible to achieve accurate gaze estimation using only an off-the-shelf webcam by identifying and collecting data implicitly from user interaction events. However, this method is constrained by the need for large amounts of well-annotated data. We thus present fast-PACE, an adaptation to PACE that exploits knowledge from existing data from different users to accelerate the learning speed of the personalized model. The result is an adaptive, data-driven approach that continuously “learns” its user and recalibrates, adapts, and improves with additional usage by a user. Experimental evaluations of fast-PACE demonstrate its competitive accuracy in iris localization, validity of alignment identification between gaze and interactions, and effectiveness of gaze transfer. In general, fast-PACE achieves an initial visual error of 3.98 degrees and then steadily improves to 2.52 degrees given incremental interaction-informed data. Our performance is comparable to state-of-the-art, but without the need for explicit training or calibration. Our technique addresses the data quality and quantity problems. It therefore has the potential to enable comprehensive gaze-aware applications in the wild.

A user-specific model generally performs better in facial affect recognition. Existing solutions, however, have usability issues since the annotation can be long and tedious for the end users (e.g., consumers). We address this critical issue by presenting a more user-friendly user-adaptive model to make the personalized approach more practical. This paper proposes a novel user-adaptive model, which we have called fast-Personal Affect Detection with Minimal Annotation (Fast-PADMA). Fast-PADMA integrates data from multiple source subjects with a small amount of data from the target subject. Collecting this target subject data is feasible since fast-PADMA requires only one self-reported affect annotation per facial video segment. To alleviate overfitting in this context of limited individual training data, we propose an efficient bootstrapping technique, which strengthens the contribution of multiple similar source subjects. Specifically, we employ an ensemble classifier to construct pretrained weak generic classifiers from data of multiple source subjects, which is weighted according to the available data from the target user. The result is a model that does not require expensive computation, such as distribution dissimilarity calculation or model retraining. We evaluate our method with in-depth experimental evaluations on five publicly available facial datasets, with results that compare favorably with the state-of-the-art performance on classifying pain, arousal, and valence. Our findings show that fast-PADMA is effective at rapidly constructing a user-adaptive model that outperforms both its generic and user-specific counterparts. This efficient technique has the potential to significantly improve user-adaptive facial affect recognition for personal use and, therefore, enable comprehensive affect-aware applications.

Stress is a fact of daily life. Stress can also deteriorate human's attention and memory, which, when a user is engaged in interactive applications, will negatively affect the user experience and downgrade the delivered performance. Traditional stress inference is mainly based on user physical features like Blood Volume Pulse, Galvanic Skin Response, often captured via devices that intrude on the user space. In contrast, this paper proposes a non-intrusive approach that exploits the consistency of users' behavioral patterns when interacting with a user interface, specifically, in terms of eye gaze and mouse movement. The relationship between the stress experienced by the user and his/her eye gaze and gaze-mouse coordination patterns are investigated. We show that both eye gaze and gaze-mouse coordination patterns can be exploited to distinguish whether a user is under stress. We also discover that a user's eye gaze behavior patterns are more consistent when he/she is under stress. This understanding of how a user's behavior differs under stress could be useful in the development of effective adaptive systems that can maximize user potential.

Gaze estimation has widespread applications. However, little work has explored gaze estimation on smartphones, even though they are fast becoming ubiquitous. This paper presents ScreenGlint, a novel approach which exploits the glint (reflection) of the screen on the user's cornea for gaze estimation, using only the image captured by the front-facing camera. We first conduct a user study on common postures during smartphone use. We then design an experiment to evaluate the accuracy of ScreenGlint under varying face-to-screen distances. An in-depth evaluation involving multiple users is conducted and the impact of head pose variations is investigated. ScreenGlint achieves an overall angular error of 2.44º without head pose variations, and 2.94º with head pose variations. Our technique compares favorably to state-of-the-art research works, indicating that the glint of the screen is an effective and practical cue to gaze estimation on the smartphone platform. We believe that this work can open up new possibilities for practical and ubiquitous gaze-aware applications.

This paper presents an investigation into user intention prediction in two common web-based tasks: crowdsourcing annotation and web search, based on human-mouse interaction information. User experience is gaining importance within the research area of human-centered computing, and is particularly useful for complex, multi-step tasks. To enhance user experience, the computer should be intelligent enough to be able to predict the user intention. For instance, an intelligent agent might be able to anticipate when the user is about to press a button, and helpfully enlarge or highlight it in advance. In this paper, we propose two prediction models on user intention: a classical model that considers only historical mouse activity sequence, and a multimodal model that utilizes mouse interaction signals as well as features extracted from mouse trajectory and clicking events. We evaluate our models and find that they achieve reasonable accuracy. Our preliminary results indicate that we can dynamically learn a multimodal model that can effectively predict a user's next activity from historical activity sequence and mouse interaction signals.

We present PACE, a Personalized, Automatically Calibrating Eye-tracking system that identifies and collects data unobtrusively from user interaction events on standard computing systems without the need for specialized equipment. PACE relies on eye/facial analysis of webcam data based on a set of robust geometric gaze features and a two-layer data validation mechanism to identify good training samples from daily interaction data. The design of the system is founded on an in-depth investigation of the relationship between gaze patterns and interaction cues, and takes into consideration user preferences and habits. The result is an adaptive, data-driven approach that continuously recalibrates, adapts and improves with additional use. Quantitative evaluation on 31 subjects across different interaction behaviors shows that training instances identified by the PACE data collection have higher gaze point-interaction cue consistency than those identified by conventional approaches. An in-situ study using real-life tasks on a diverse set of interactive applications demonstrates that the PACE gaze estimation achieves an average error of 2.56º, which is comparable to state-of-the-art, but without the need for explicit training or calibration. This demonstrates the effectiveness of both the gaze estimation method and the corresponding data collection mechanism.

Stress sensing is valuable in many applications, including online learning crowdsourcing and other daily human-computer interactions. Traditional affective computing techniques investigate affect inference based on different individual modalities, such as facial expression, vocal tones, and physiological signals or the aggregation of signals of these independent modalities, without explicitly exploiting their inter-connections. In contrast, this paper focuses on exploring the impact of mental stress on the coordination between two human nervous systems, the somatic and autonomic nervous systems. Specifically, we present the analysis of the subtle but indicative pattern of human gaze behaviors surrounding a mouse-click event, i.e. the gaze-click pattern. Our evaluation shows that mental stress affects the gaze-click pattern, and this influence has largely been ignored in previous work. This paper, therefore, further proposes a non-intrusive approach to inferring human stress level based on the gaze-click pattern, using only data collected from the common computer webcam and mouse. We conducted a human study on solving math questions under different stress levels to explore the validity of stress recognition based on this coordination pattern. Experimental results show the effectiveness of our technique and the generalizability of the proposed features for user-independent modeling. Our results suggest that it may be possible to detect stress non-intrusively in the wild, without the need for specialized equipment.

This paper presents Personalized Affect Detection with Minimal Annotation (PADMA), a user-dependent approach for identifying affective states from spontaneous facial expressions without the need for expert annotation. The conventional approach relies on the use of key frames in recorded affect sequences and requires an expert observer to identify and annotate the frames. It is susceptible to user variability and accommodating individual differences is difficult. The alternative is a user-dependent approach, but it would be prohibitively expensive to collect and annotate data for each user. PADMA uses a novel Association-based Multiple Instance Learning (AMIL) method, which learns a personal facial affect model through expression frequency analysis, and does not need expert input or frame-based annotation. PADMA involves a training/calibration phase in which the user watches short video segments and reports the affect that best describes his/her overall feeling throughout the segment. The most indicative facial gestures are identified and extracted from the facial response video, and the association between gesture and affect labels is determined by the distribution of the gesture over all reported affects. Hence both the geometric deformation and distribution of key facial gestures are specially adapted for each user. We show results that demonstrate the feasibility, effectiveness and extensibility of our approach.

The amount of data stored in databases is increasing at a tremendous rate. They are oftentimes stored over the web to be accessed by various clients. One useful and interesting query to a collection of databases is to capture a consistent snapshot of a set of interested attributes across the databases. While traditional snapshot algorithms for a distributed database could be adopted, they are mainly designed for database recovery and are costly to execute. We need efficient algorithms to return good-enough snapshots for querying purposes, in the absence of a consistent but costly snapshot as defined in distributed systems. In this paper, we propose the notion of an approximate consistent snapshot by considering and minimizing the deviation of the collected snapshot covering the interested attributes from a reference absolute snapshot. There are several variations of this kind of approximate snapshots. We propose effective algorithms to capture such snapshots in a collection of databases and then conduct performance evaluation on their efficiency. We believe that this notion of approximate consistency would be useful in practical situations.

Social signal processing is becoming an important topic in affective computing. In this paper, we focus on an important social interaction in real life, namely, fighting. Fight detection will be useful in public transportation, prisons, bars, or even sport. A robust mechanism in detecting fights from a video will be extremely useful, especially in applications relevant to surveillance systems. Recent research works focus on extracting visual features from high resolution video, leading to computationally expensive systems. In this paper, we propose an approach to detect fights in a natural and robust way based on motion analysis, which is not only intuitive, but also robust. Experimental results show that we can accurately detect fight activities in different video surveillance settings.

We present a vision-based, data-driven approach to identifying and measuring refractive errors in human subjects with low-cost, easily available equipment and no specialist training. Vision problems, such as refractive error (e.g. nearsightedness, astigmatism, etc) are common ocular problems, which, if uncorrected, may lead to serious visual impairment. The diagnosis of such defects conventionally requires expensive specialist equipment and trained personnel, which is a barrier in many parts of the developing world. Our approach aims to democratize optometric care by utilizing the computational power inherent in consumer-grade devices and the advances made possible by multimedia computing. We present results that show our system is able to match and outperform state-of-the-art medical devices under certain conditions.

Affect exchange is essential for healthy physical and social development [7], and friends and family communicate their emotions to each other instinctively. In particular, watching movies has always been a popular mode of socialization and video sharing is increasingly viewed as an effective way to facilitate communication of feelings and affects, even when the parties are not in the same location. We present an asynchronous video-sharing platform that uses Emotars to facilitate affect sharing in order to create and enhance the sense of togetherness through the experience of asynchronous movie watching. We investigate its potential impact and benefits, including a better viewing experience, supporting relationships, and strengthening engagement, connectedness and emotion awareness among individuals.

Service learning has been gaining attention in recent years. It has been established as an effective method to teach students a variety of concepts that are not easily taught in the classroom, and much effort has gone into making service learning accessible and relevant to computer science students. This paper investigates a popular mode of computing-related service learning -- offshore projects that seek to introduce information and computing technologies (ICTs) into a beneficiary population. Based on multiple years of experience working with ICTs in service learning, the authors examine the impact on students and beneficiaries through several critical questions, and draw conclusions and recommendations on good practices for designing offshore service learning projects for computing students.

This paper presents KID, an interactive app on a smart device, designed to facilitate and encourage young children to learn and practice Chinese characters. It relies on pen dynamics to extract the strokes and map the written character to the proper one. The stroke orientation is also analyzed for ordering and spatial alignment features that pinpoint common errors. A visual pictorial feedback is then provided to motivate children and to arouse their interest. We iterate the prototype design and implementation upon collecting feedback from focus group interviews, from where the system is greeted with positive comments.

Most eye gaze estimation systems rely on explicit calibration, which is inconvenient to the user, limits the amount of possible training data and consequently the performance. Since there is likely a strong correlation between gaze and interaction cues, such as cursor and caret locations, a supervised learning algorithm can learn the complex mapping between gaze features and the gaze point by training on incremental data collected implicitly from normal computer interactions. We develop a set of robust geometric gaze features and a corresponding data validation mechanism that identifies good training data from noisy interaction-informed data collected in real-use scenarios. Based on a study of gaze movement patterns, we apply behavior-informed validation to extract gaze features that correspond with the interaction cue, and data-driven validation provides another level of crosschecking using previous good data. Experimental evaluation shows that the proposed method achieves an average error of 4.06º, and demonstrates the effectiveness of the proposed gaze estimation method and corresponding validation mechanism.

As computer and interaction technologies mature, a much broader range of media is now used for input and output, each of which has its own rich repertoire of techniques, instruments, and cultural heritage. The combination of multiple media can produce novel multimedia human-computer interaction approaches which are more efficient and interesting than traditional single media methods. This paper presents CalliPaint, a system for cross-modal art generation that links together Chinese ink brush calligraphy writing and Chinese landscape painting. We investigate the mapping between the two modalities based on concepts of metaphoric congruence, and implement our findings into a prototype system. A multi-step evaluation experiment with real users suggests that CalliPaint provides a realistic and intuitive experience that allows even novice users to create attractive landscape paintings from writing. Comparison with a general-purpose digital painting software suggests that CalliPaint provides users with a more enjoyable experience. Finally, exhibiting CalliPaint in an open-access location for use by casual users without any training shows that the system is easy to learn.

This article presents the design and development of i*CATch, a construction kit for physical and wearable computing that was designed to be scalable, plug-and-play, and to provide support for iterative and exploratory learning. It consists of a standardized construction interface that can be adapted for a wide range of soft textiles or electronic boards, a set of functional components, and an easy-to-use hybrid text-graphical integrated development environment. The objective was to design an easily usable, manufacturable and extensible construction kit that can be used in a wide range of teaching tasks for a wide variety of student demographic profiles. We present detailed specifications of our construction kit and explain some of the major design decisions. Experiences in using the kit in multiple teaching environments, ranging from elementary school to postgraduate, demonstrate that the design objectives have been achieved.

MelodicBrush is a novel system that connects two ancient art forms: Chinese ink-brush calligraphy and Chinese music. Our system uses vision-based techniques to create a digitized ink-brush calligraphic writing surface with enhanced interaction functionalities. The music generation combines cross-modal stroke-note mapping and statistical language modeling techniques into a hybrid model that generates music as a real-time, auditory response and feedback to the user's calligraphic strokes.

There has been much recent work in wearable computing that is directed at democratization of the field, to make it more accessible to the general public and more easily used by the hobbyist user. As the field becomes more diversified, there has also been a shift away from the highly specialized functionality of earlier applications towards aesthetics, creativity, design and self-expression, as well as a push towards using wearable computing as an outreach tool to broaden interest and exposure in engineering and computing.

This paper introduces the Multi-modal Interface Framework (MIF). It is a system which allows developers to easily integrate interface devices of multiple modalities, such as voice, hand and finger gestures, and various tangible devices such as game controllers into a multi-modal input system. The integrated devices can then be used to control practically any computer application. The advantages offered by MIF are ease of use, flexibility and support for collaboration. Its design has been validated by applying it to integrate finger gestures, voice, a Wii mote and an iPhone to control applications such as Google Earth and Windows Media Player.

Controlling a team of robots is much more challenging than controlling an individual robot. Users desire high level commands that shield the users from too much detail, and yet still afford the desired precision of control. This project aims to design a method through which a team of robots can be controlled as easily and precisely as an individual robot. A simple language in the form of a set of finger gestures allows the user to give general motion commands to the team of robots. The gestures are supplemented by fine controls such as speed through tangible input gadgets. The gesture-based language has been implemented in a prototype user interface on a multi-touch screen. A number of test applications demonstrate the validity of the design.

The field of wearable computing and e-textiles has recently attracted much interest from the research and general community. Recent developments in this field raises the possibility of e-textile construction kits for hobbyists and novices alike. The unique nature of wearable computing and e-textiles also gives it a lot of potential as an educational computing topic, as it allows students to exercise their creativity and imagination while learning about computing and technology.

Advances in textile-friendly electronics devices, smart materials, sophisticated interfaces, and intelligent software have combined to make intelligent garments increasingly practical. This paper introduces Co-eTex, a framework designed to support collaborative development of intelligent garments. It is based on a newly-invented construction platform for eTextiles and Wearable Computing, which was designed to be robust, reliable, easy to construct and to program. It also includes a hybrid graphical-textual programming tool designed for novice programmers to program intelligent behaviour for intelligent garments. We describe the use of Co-eTex in rapidly developing prototypes for a variety of designs of intelligent garments. Based on our experiences in using the Co-eTex, we believe it is a possible direction leading to the development of mass customization or adaptive customization of intelligent garments.

This paper presents a design for a platform of collaborative robots and electronic devices. The platform design is based on the similarity between the type, functionality and characteristics of the robot or device. We categorize the participating devices by functionality rather than by architecture, therefore making it easy to support new robots or devices with similar functions but different architectures. This approach also allows users to develop, implement and port applications quickly and easily. We demonstrate the efficacy and correctness of our platform through a variety of robotic applications ranging from research to teaching.

To address the unique demands and challenges of educational computing, various kinds of environments, including graphics-rich and textual environments, have been proposed for use in introductory courses to provide students with a rich and interesting learning environment. In our experience, students in Grade 7 and younger respond best to the graphics environments while senior high school students prefer a conventional textual programming environment. Clearly, this leaves a gap at Grade 11-13, with students often on the one hand finding the graphics-based environments too limited and on the other finding the textual environments too difficult. In this paper, we propose a text-enhanced graphical programming environment which is innovative and interactive, and designed for junior high students with no programming experience. This environment allows students to design their own creative stories or programs. They build their programs using drag-and-drop iconic blocks, but unlike other, similar icon-based programming languages, they are also presented with the syntax of the actual program they are constructing in real-time. Once a particular icon block has been dropped in the programming area, the syntax statements corresponding to that block is immediately generated and presented to the user. The environment also allows them to modify the code without any limitations. Our results show that our textual-graphical hybrid environment has a positive impact on the learning experience of the students.

As enrollments in engineering and computer science programs around the world have fallen in recent years, those who wish to see this trend reversed take heart from findings that children are more likely to develop an abiding interest in technology if they are exposed to it at an early age [3, 9]. In line with this research, we now see more summer camps and workshops being offered to middle school students with the objective of teaching programming and computer technology [1, 6, 8, 12]. To offer students a stimulating and interesting environment while teaching computing subjects, the learning tools in these camps usually revolve around robots and graphical programming of animations or games. These tools tend to mainly attract youngsters who like robotics or game design. However, we believe that we can improve the diversity of the student pool by introducing other topics. In this paper, we describe our experience in designing and organizing a programming course that focuses on wearable computing, fashion and design for middle school students. We will show that 1) wearable computing is interesting and inspiring to the students, 2) wearable computing motivates both boys and girls to learn technology and computing, which implies that it may be able to increase the potential computer science population, 3) wearable computing can provide a space for students to exercise their creativity while at the same time, teaching them about technology and programming.

Abstract The use of middleware is one important approach in facilitating adaptation across wireless and mobile environments, where augmented service entities are composed and deployed to shield mobile clients from the effects of dynamic network characteristics. The MobiGate Coordination Language (MCL) system provides a language‐based approach to the building of mobile applications running in an adaptive middleware, MobiGATE. The concept of the separation of concerns forms the underlying and unifying principle in the provision of the adaptive composition of services. Specifically, a coordination language, MCL, is designed to provide rich constructs supporting the definition of compositions, with constrained type validation and checking. In particular, MCL is formalized by means of the design of a semantic model based on the Z language, which can be used to analyze architectural descriptions and detect possible composition errors such as feedback loops and open circuit problems. Copyright © 2006 John Wiley & Sons, Ltd.

The growing importance of multilingual information retrieval and machine translation has made multilingual ontologies extremely valuable resources. Since the construction of an ontology from scratch is a very expensive and time-consuming undertaking, it is attractive to consider ways of automatically aligning monolingual ontologies, which already exist for many of the world's major languages. Previous research exploited similarity in the structure of the ontologies to align, or manually created bilingual resources. These approaches cannot be used to align ontologies with vastly different structures and can only be applied to much studied language pairs for which expensive resources are already available. In this paper, we propose a novel approach to align the ontologies at the node level: Given a concept represented by a particular word sense in one ontology, our task is to find the best corresponding word sense in the second language ontology. To this end, we present a language-independent, corpus-based method that borrows from techniques used in information retrieval and machine translation. We show its efficiency by applying it to two very different ontologies in very different languages: the Mandarin Chinese HowNet and the American English WordNet. Moreover, we propose a methodology to measure bilingual corpora comparability and show that our method is robust enough to use noisy nonparallel bilingual corpora efficiently, when clean parallel corpora are not available.

Parsing, the task of identifying syntactic components, e.g., noun and verb phrases, in a sentence, is one of the fundamental tasks in natural language processing. Many natural language applications such as spoken-language understanding, machine translation, and information extraction, would benefit from, or even require, high accuracy parsing as a preprocessing step. Even though most state-of-the-art statistical parsers were initially constructed for parsing in English, most of them are not language-specific, in that they do not rely on properties of the language that are specific to English. Therefore, construction of a parser in a given language becomes a matter of retraining the statistical parameters with a Treebank in the corresponding language. The development of the Chinese treebank [Xia et al. 2000] spurred the construction of parsers for Chinese. However, Chinese as a language poses some unique problems for the development of a statistical parser, the most apparent being word segmentation. Since words in written Chinese are not delimited in the same way as in Western languages, the first problem that needs to be solved before an existing statistical method can be applied to Chinese is to identify the word boundaries. This is a step that is neglected by most pre-existing Chinese parsers, which assume that the input data has already been pre-segmented. This article describes a character-based statistical parser, which gives the best performance to-date on the Chinese treebank data. We augment an existing maximum entropy parser with transformation-based learning, creating a parser that can operate at the character level. We present experiments that show that our parser achieves results that are close to those achievable under perfect word segmentation conditions.

This paper presents a comprehensive empirical comparison between two approaches for developing a base noun phrase chunker: human rule writing and active learning using interactive real-time human annotation. Several novel variations on active learning are investigated, and underlying cost models for cross-modal machine learning comparison are presented and explored. Results show that it is more efficient and more successful by several measures to train a system using active learning annotation rather than hand-crafted rule writing at a comparable level of human labor investment.

Transformation-based learning has been successfully employed to solve many natural language processing problems.It has many positive features, but one drawback is that it does not provide estimates of class membership probabilities.In this paper, we present a novel method for obtaining class membership probabilities from a transformation-based rule list classifier.Three experiments are presented which measure the modeling accuracy and cross-entropy of the probabilistic classifier on unseen data and the degree to which the output probabilities from the classifier can be used to estimate confidences in its classification decisions.The results of these experiments show that, for the task of text chunking 1, the estimates produced by this technique are more informative than those generated by a state-of-the-art decision tree.

A great deal of work has been done demonstrating the ability of machine learning algorithms to automatically extract linguistic knowledge from annotated corpora.Very little work has gone into quantifying the difference in ability at this task between a person and a machine.This paper is a first step in that direction.