Appearance-based gaze estimation in 2-dimensional screen coordinates–the prediction of the users’ gaze from webcam footage–cannot yet compete in accuracy with infrared (IR) eye trackers. Yet by circumventing the constraints of requiring dedicated hardware, it shows great potential in many technological industries, as evidenced by some readily available commercial solutions, bringing democratization of eye tracking closer to the people. We present Residual Appearance-based Gaze Estimation network (RAGE-net), a novel convolutional neural network for gaze estimation without need of calibration, utilizing a fraction of computational resources required by similar networks, while also achieving competitive accuracy. The angular error is measured as 4.08°in the MPIIFaceGaze dataset (Max Planck Institute for Informatics Faze Gaze) and 3.96°in the MPIIGaze dataset. The architecture’s principles, covered by a comprehensive ablation study, include an attention branch, residual learning, weight sharing between eye channels, batch normalization and an eye image input normalization pipeline that removes dependence on full face input. With RAGE-net, we conduct an applicability study for gaze estimation approaches of similar accuracy for interpreting on-screen gaze in praxis. Findings demonstrate low heatmap validity, with coarse heatmaps as potential adaptation to approximate IR eye tracking. The effects of environmental factors such as camera position, illumination, distance and glasses are analyzed in-depth. • Proposed network with residual learning, attention mechanism and reduced input. • Angular error of 4.08°achieved with reduced network complexity 9.79 GFLOPs. • Ablation study of design modules, eye input cropping and normalization is presented. • Applicability of heatmaps from appearance-based gaze estimations is analyzed. • Impact of environment on webcam gaze tracking accuracy is investigated.

Follow-up questions during usability testing provide crucial insights into the user’s experience with a product or service. In unmoderated usability tests conducted online, artificial intelligence (AI) is emerging as a valuable tool. Large language model chatbots have the potential to intelligently ask follow-up questions automatically and in real time. Conversing with a chatbot during usability tests may uncover deeper qualitative insights. Our case study examines the implementation of GPT-4-generated follow-up questions to assess their impact on usability testing insights. Sixty participants took part in an experiment aimed at comparing the feedback they yield under different conditions: no follow-up questions, static questions prepared by researchers, real-time GPT-4 questions, and a blend of static and AI-generated questions. While GPT-4-generated questions effectively elaborated details about existing findings, they revealed fewer new usability issues. We discuss challenges encountered with GPT-4 follow-ups and propose enhancements to improve future models for generating effective follow-up questions.

Context: Tree testing is an established user testing method applied by software professionals to validate that an information architecture is logically navigable by users. We identify a methodological gap caused by previously unexamined non-uniformity between tree testing methods and software. Objective: To reveal the role of the user interface representations in tree testing, this research compares the results of 3 commonly-used tree testing variants. To assess how indicative they are of the user’s interaction with an information architecture implemented in an actual user interface, and to issue methodological recommendations, comparison with varied high-fidelity prototypes was performed. Methods: Two between-subject studies were conducted to obtain a new dataset of users navigating an information architecture in tree testing and in interactive user interface prototypes. Data from 180 participants and 1800 task completions between 6 experimental conditions—3 tree testing and 3 prototype user interface variants—was evaluated quantitatively and qualitatively. Results: Significant differences were found between results yielded by different tree testing method variants, and in how well they approximate user navigation in the same information architecture in high-fidelity prototypes. Implications for selection of the tree testing variant are proposed in the context of evaluated information architecture, with plausible broader applicability for tree testing methodology. Evidence supports the tree testing variant with highest visibility of previous navigation choices and direct controls over their reversal as the most accurate. Conclusion: Presented findings can contribute to the design of software information architecture based on more accurate early validation, owing to tree testing that simulates less artificial user behavior more reflective of the user’s navigation in the eventual user interface. We hope this will further the discussion and research leading to more holistic tree testing methodologies in the future.

In user experience design, prototypes are an indispensable tool for early diagnosis of usability issues. Designing usability testing to accommodate a prototype’s limited interactivity is essential to obtain relevant participant feedback. Hotspot Highlighting is a technique employed by all prominent prototyping tools to allow usability testers to see which areas of the prototype are clickable. The current body of knowledge lacks definite answers on how highlighting impacts usability testing results, compared to scenarios where participants complete tasks fully on their own. Can studies be treated the same, regardless of whether Hotspot Highlighting is enabled? What are the recommendations for how and when Hotspot Highlighting can or should be used? To investigate, we conduct a between-subjects experiment with 80 participants and 240 task completions in which we compare user behavior depending on the presence of Hotspot Highlighting. Its results indicate that Hotspot Highlighting can affect participant behavior before and after a highlight is displayed, leading to potentially different usability findings if left unaccounted for. The guidance of highlights changes the targets of clicks and encourages cognitively efficient finding of solutions by intentionally triggering the highlights. Considering the potential of Hotspot Highlighting to facilitate the usability testing of prototypes with limited interactivity, we discuss potential adaptations of the technique that address its current issues for more methodologically sound usability evaluation. • Highlighting of clickable prototype elements alters usability test results. • Users deliberately click outside of clickable elements to reveal hotspots. • Clicking behavior diverges from natural after hotspots shift user focus. • Testing of less interactive prototypes can be aided by Hotspot Highlighting.

Abstract With widespread use of online forms and questionnaires, detection of the user’s intent to lie has become increasingly important. In-lab studies have shown that mouse dynamics-information on how the user operates a mouse-can be valuable for automatically and unobtrusively distinguishing between faking and honest intent in personality questionnaires. However, data variability in the wild (e.g., mouse configurations, screen resolutions) could present a barrier for mouse dynamics in naturalistic conditions. Our aim is to design a data-driven faking good detection method operable under naturalistic conditions and evaluate its reliability. We conducted a between-subjects uncontrolled experiment where users’ mouse dynamics data was obtained with the BFI-2 personality questionnaire, yielding a sample of 5344 items from 112 participants. The proposed user model characterizes participants’ mouse dynamics, serving as input for prediction. Varied machine learning classification models were trained and evaluated to determine the best approach and feature set. XGBoost with LASSO feature selection achieved the highest F1-score of 86.92%, outperforming in a real-world web form and conditions the results achieved by related works utilizing specialized questions in laboratory conditions. Faking good prediction that leverages our user model can therefore be viable for online questionnaires in the wild.

Five-second testing is a method commonly used by user research professionals to assess users' first impressions of user interfaces or product designs.Its rule of thumb, that five seconds is generally the right amount of time for users to report realistic and relevant first impressions, misrepresents the reality of human cognition.Users possess disparate levels of cognitive ability for processing stimuli that possess varied visual complexity.We conducted a complex experiment where participants complete an evaluation of their cognitive abilityworking memory and perceptual speed.They are shown website stimuli of varied complexity over differing time periods (2/5/10 seconds) to answer a representative list of questions typical for evaluation of first impressions.We show that first impression feedback is rendered inconsistent by cognitive ability and visual complexity.Visually complex stimuli viewed for too short a time link to a problem with identifying the web page purpose.Participants with varied perceptual speed produce comparable results at 5 seconds, but at lower working memory, they provide less verbose answers and recall less information.These findings suggest that to receive relevant first impression feedback, the time for which participants are shown stimuli should be adapted to cognitive abilities and stimulus complexity.

Loyal customers play an important role in every store’s success. They tend to buy regularly and help stabilize incomes. Being able to identify potential repeat buyers allows marketers to act promptly and convince the users not to search for better offers elsewhere. Current customer behavior prediction approaches are based on non-interaction data (e.g., server logs). We see a gap in the research of the impact of low-level interaction data capturing user behavior more precisely (e.g., cursor movements, scrolls, inputs). We introduce three new datasets collected year-long from three different ecommerce stores. We evaluate the merit of low-level interactions for the task of repeat purchase prediction and study feature sets utilizing low-level interactions, together with non-interaction data. We compare their performance in the classification task to benchmark approaches relying solely on non-interaction data. Our experiments show inclusion of interaction data improves the prediction performance compared to the baseline non-interaction feature set.

Context: Despite the fact, that the various automated expertise metrics were proposed, we do not know which metrics the most reliably capture/reflect expertise. Goal: To define metrics for estimation of developer's expertise based on programming tasks, to evaluate which of them most reliably capture expertise, and to propose and evaluate an automatic process to compare the metrics. Method: We define three expertise metrics with respects to such characteristics as spent time, performed activities and complexity of source code. We evaluate Spearman's correlation between our expertise metrics and students' score obtained after completion of a programming course with 251 students. Results: The best (very strong) correlation is between the metrics based on complexity of source code and the student's qualification points. Conclusions: Very strong but not perfect correlation is between our estimation of student's expertise and his/her score in the second third of the course. Approximately in the middle of the course we might be able to predict students' grades.