As AI writing support becomes ubiquitous, the question of how disclosing its use affects reader perception remains critical and underexplored. We conducted a controlled study with 261 participants to examine how disclosing varying levels of AI involvement shifts perceptions of the author across six distinct communicative acts. Our analysis of 990 evaluations reveals that disclosure generally erodes perceived trustworthiness, caring, competence, and likability, with the most precipitous declines observed in social and interpersonal writing. A thematic analysis of participant feedback attributes these negative shifts to a perceived loss of human sincerity, diminished authorial effort, and the contextual inappropriateness of AI. Notably, however, we find that higher AI literacy mitigates these negative perceptions, leading to greater tolerance or even appreciation for AI assistance. Our results highlight the nuanced social dynamics of AI-mediated authorship and inform design implications for transparent, context-sensitive writing systems that better preserve trust and authenticity.

Interacting with the touchscreen of a mobile phone in virtual reality (VR) is challenging because users cannot see their fingers when aiming for targets. We propose using two mirrors reflecting the front camera of the phone and a purpose-built deep neural network to infer the 3D position of fingertips above the screen. Network training is self-supervised after only a few hundred initial labelled images and does not require any external sensor. The inferred fingertip positions can be used to control different hand models and objects in VR. Controlled experiments evaluate tracking performance for single-finger touch input, and compare several 3D hand representations with a flat 2D overlay used in previous work. The results confirm the suitability of our fingertip tracker to aid precise tapping of small targets on the phone screen and provide insights about the effect of various hand representations on control and presence. Finally, we provide several application examples showing how 3D fingertip input can complement and extend phone-based touch interaction in VR.

We identify and evaluate single-handed “dexterous gestures” to physically manipulate a phone using the fine motor skills of fingers. Four manipulations are defined: shift, spin (yaw axis), rotate (roll axis) and flip (pitch axis), with a formative survey showing all except flip have been performed for various reasons. A controlled experiment examines the speed, behaviour, and preference of manipulations in the form of dexterous gestures, by considering two directions and two movement magnitudes. Results show rotate is rated as easiest and most comfortable, while flip is rated lowest. Using a heuristic recognizer for spin, rotate, and flip, a one-week usability experiment finds increased practice and familiarity improve the speed and comfort of dexterous gestures. Design guidelines are developed to consider comfort, ability, and confidence when mapping dexterous gestures to interactions, and demonstrations show how such gestures can be used in smartphone applications.

Segmenting and determining the 3D bounding boxes of objects of interest in RGB videos is an important task for a variety of applications such as augmented reality, navigation, and robotics. Supervised machine learning techniques are commonly used for this, but they need training datasets: sets of images with associated 3D bounding boxes manually defined by human annotators using a labelling tool. However, precisely placing 3D bounding boxes can be difficult using conventional 3D manipulation tools on a 2D interface. To alleviate that burden, we propose a novel technique with which 3D bounding boxes can be created by simply drawing 2D bounding rectangles on multiple frames of a video sequence showing the object from different angles. The method uses reconstructed dense 3D point clouds from the video and computes tightly fitting 3D bounding boxes of desired objects selected by back-projecting the 2D rectangles. We show concrete application scenarios of our interface, including training dataset creation and editing 3D spaces and videos. An evaluation comparing our technique with a conventional 3D annotation tool shows that our method results in higher accuracy. We also confirm that the bounding boxes created with our interface have a lower variance, likely yielding more consistent labels and datasets.

Smartphone touch screens are potentially attractive for interaction in virtual reality (VR). However, the user cannot see the phone or their hands in a fully immersive VR setting, impeding their ability for precise touch input. We propose mounting a mirror above the phone screen such that the front-facing camera captures the thumbs on or near the screen. This enables the creation of semi-transparent overlays of thumb shadows and inference of fingertip hover points with deep learning, which help the user aim for targets on the phone. A study compares the effect of visual feedback on touch precision in a controlled task and qualitatively evaluates three example applications demonstrating the potential of the technique. The results show that the enabled style of feedback is effective for thumb-size targets, and that the VR experience can be enriched by using smartphones as VR controllers supporting precise touch input.

We propose a style of hand postures to trigger commands on a laptop. The key idea is to perform hand-postures while keeping the hands on, beside, or below the keyboard, to align with natural laptop usage. 36 hand-posture variations are explored considering three resting locations, left or right hand, open or closed hand, and three wrist rotation angles. A 30-participant formative study measures posture preferences and generates a dataset of nearly 350K images under different lighting conditions and backgrounds. A deep learning recognizer achieves over 97% accuracy when classifying all 36 postures with 2 additional non-posture classes for typing and non-typing. A second experiment with 20 participants validates the recognizer under real-time usage and compares posture invocation time with keyboard shortcuts. Results find low error rates and fast formation time, indicating postures are close to current typing and pointing postures. Finally, practical use case demonstrations are presented, and further extensions discussed.

We propose CursorTap, an extension of Forlines et al.'s mixed, absolute and relative "HybridPointing" to large wall-sized multitouch displays. Our technique uses a relative pointing quasimode activated with one hand, while the other hand controls a distant cursor similar to a large touchpad. A controlled experiment compares the technique to standard absolute touch input as a baseline and a whole-display "Drag" technique representing a common alternate approach. Results show CursorTap is fastest for the common usage scenario of reaching distant targets and then returning to nearby targets. Overall, median selection times across distances are similar with CursorTap, but linearly increase with the other techniques. As further validation, a second study explore show people use CursorTap in a two-person game. The results found just over half of the participants choose to use CursorTap for half of the primary interactions where "enemies" are eliminated using a tap, drag, or lasso "tool".

We propose mounting a downward-facing camera above the top end of a digital tablet pen. This creates a unique and practical viewing angle for capturing the pen-holding hand and the immediate surroundings which can include the other hand. The fabrication of a prototype device is described and the enabled interaction design space is explored, including dominant and non-dominant hand pose recognition, tablet grip detection, hand gestures, capturing physical content in the environment, and detecting users and pens. A deep learning computer vision pipeline is developed for classification, regression, and keypoint detection to enable these interactions. Example applications demonstrate usage scenarios and a qualitative user evaluation confirms the potential of the approach.

We present an empirical comparison of eleven bare hand, mid-air mode-switching techniques suitable for virtual reality in two experiments. The first evaluates seven techniques spanning dominant and non-dominant hand actions. Techniques represent common classes of actions selected by a methodical examination of 56 examples of prior art. The standard "subtraction method" protocol is adapted for 3D interfaces, with two baseline selection methods, bare hand pinch and device controller button. A second experiment with four techniques explores more subtle dominant-hand techniques and the effect of using a dominant hand device for selection. Results provide guidance to practitioners when choosing bare hand, mid-air mode-switching techniques, and for researchers when designing new mode-switching methods in VR.

We conduct a two-part study to better understand pen grip postures for general input like mode switching and com-mand invocation. The first part of the study asks participants what variations of their normal pen grip posture they might use, without any specific consideration for sensing capabilities. The second part evaluates three of their sug-gested postures with an additional set of six postures designed for the sensing capabilities of a consumer EMG armband. Results show that grips considered normal and mature, such as the dynamic tripod and the dynamic quadrupod, are the best candidates for pen-grip based interaction, followed by finger-on-pen postures and grips using pen tilt. A convolutional neural network trained on EMG data gathered during the study yields above 70% within-participant recognition accuracy for common sets of five postures and above 80% for three-posture subsets. Based on the results, we propose design guidelines for pen interaction using variations of grip postures.

We explore and evaluate a multi-finger raycasting design space that we call "multiray". Each finger projects a ray on to the display, so the user is interacting from a distance using a form of direct input. Specifically, we propose techniques, where patterns of ray intersections created by hand postures form 2D geometric shapes to trigger actions and perform direct manipulations that go beyond single-point selections. Two formative studies examine characteristics of multi-finger raycasting for different projection methods, shapes, and tasks. Based on the results of those investigations, we demonstrate a number of dynamic UI controls and operations that utilise multiray points and shapes.

ColourAIze is an interactive system that analyses black and white drawings on paper, automatically determines realistic colour fills using artificial intelligence (AI) and projects those colours onto the paper within the line art. In addition to selecting between multiple colouring styles, the user can specify local colour preferences to the AI via simple stylus strokes in desired areas of the drawing. This allows users to immediately and directly view potential colour fills for paper sketches or published black and white artwork such as comics. ColourAIze was demonstrated at the Winter 2017 Comic Market in Tokyo, where it was used by more than a thousand visitors. This short paper describes the design of the system and reports on usability observations gathered from demonstrators at the fair.

We introduce a new pen input space by forming postures with the same hand that also grips the pen while writing, drawing, or selecting. The postures contact the multitouch surface around the pen to enable detection without special sensors. A formative study investigates the effectiveness, accuracy, and comfort of 33 candidate postures in controlled tasks. The results indicate a useful subset of postures. Using raw capacitive sensor data captured in the study, a convolutional neural network is trained to recognize 10 postures in real time. This recognizer is used to create application demonstrations for pen-based document annotation and vector drawing. A small usability study shows the approach is feasible.

This paper presents the results of a 36 participant empirical comparison of touch mode-switching. Six techniques are evaluated, spanning current and future techniques: long press, non-dominant hand, two-fingers, hard press, knuckle, and thumb-on-finger. Two poses are controlled for, seated with the tablet on a desk and standing with the tablet held on the forearm. Findings indicate pose has no effect on mode switching time and little effect on error rate; using two-fingers is fastest while long press is much slower; non-preferred hand and thumb-on-finger also rate highly in subjective scores. The experiment protocol is based on Li et al.'s pen mode-switching study, enabling a comparison of touch and pen mode switching. Among the common techniques, the non-dominant hand is faster than pressure with touch, whereas no significant difference had been found for pen. Our work addresses the lack of empirical evidence comparing touch mode-switching techniques and provides guidance to practitioners when choosing techniques and to researchers when designing new mode-switching methods.

Tabletop interaction can be enriched by considering whole hands as input instead of only fingertips. We describe a generalised, reproducible computer vision algorithm to recognise hand contact shapes, with support for arm rejection, as well as dynamic properties like finger movement and hover. A controlled experiment shows the algorithm can detect seven different contact shapes with roughly 91% average accuracy. The effect of long sleeves and non-user specific templates is also explored. The algorithm is used to trigger, parameterise, and dynamically control menu and tool widgets, and the usability of a subset of these are qualitatively evaluated in a realistic application. Based on our findings, we formulate a number of design recommendations for hand shape-based interaction.

We explore grip and motion sensing to afford new techniques that leverage how users naturally manipulate tablet and stylus devices during pen + touch interaction. We can detect whether the user holds the pen in a writing grip or tucked between his fingers. We can distinguish bare-handed inputs, such as drag and pinch gestures produced by the nonpreferred hand, from touch gestures produced by the hand holding the pen, which necessarily impart a detectable motion signal to the stylus. We can sense which hand grips the tablet, and determine the screen's relative orientation to the pen. By selectively combining these signals and using them to complement one another, we can tailor interaction to the context, such as by ignoring unintentional touch inputs while writing, or supporting contextually-appropriate tools such as a magnifier for detailed stroke work that appears when the user pinches with the pen tucked between his fingers. These and other techniques can be used to impart new, previously unanticipated subtleties to pen + touch interaction on tablets.

The enormous increase of digital image collections urgently necessitates effective, efficient, and in particular highly flexible approaches to image retrieval. Different search paradigms such as text search, query-by-example, or query-by-sketch need to be seamlessly combined and integrated to support different information needs and to allow users to start (and subsequently refine) queries with any type of object. In this paper, we present QUEST (Query by Example, Sketch and Text), a novel flexible multi-modal content-based image retrieval (CBIR) framework. QUEST seamlessly integrates and blends multiple modes of image retrieval, thereby accumulating the strengths of each individual mode. Moreover, it provides several implementations of the different query modes and allows users to select, combine and even superimpose the mode(s) most appropriate for each search task. The combination of search paradigms is by itself done in a very flexible way: either sequentially, where one query mode starts with the result set of the previous one (i.e., for incrementally refining and/or extending a query) or by supporting different paradigms at the same time (e.g., creating an artificial query image by superimposing a query image with a sketch, thereby directly integrating query-by-example and query-by-sketch). We present the overall architecture of QUEST and the dynamic combination and integration of the query modes it supports. Furthermore, we provide first evaluation results that show the effectiveness and the gain in efficiency that can be achieved with the combination of different search modes in QUEST.

With the proliferation and sophistication of digital reading devices, new means to support the task of active reading (AR) have emerged. In this paper, we investigate the use of pen-and-touch-operated tabletops for performing essential processes of AR such as annotating, smooth navigation and rapid searching. We present an application to support these processes and then report on a user study designed to compare the suitability of our setup for three typical tasks against the use of paper media and Adobe Acrobat on a regular desktop PC. From this evaluation, we found out that pen and touch tabletops can successfully combine the advantages of paper and digital devices without their disadvantages. We however also learn from observations and participant feedback that there are still a number of hardware and software limitations that impede the user experience and hence need to be addressed in future systems.

Despite the fact that screen sizes and average screen resolutions have dramatically increased over the past few years, little attention has been paid to the design of web sites for large, high-resolution displays that are now becoming increasingly used both in enterprise and consumer spaces. We present a study of how the visual area of the browser window is currently utilised by news web sites at different widescreen resolutions. The analysis includes measurements of space taken up by the article content, embedded ads and the remaining components as they appear in the viewport of the web browser. The results show that the spatial distribution of page elements does not scale well with larger viewing sizes, which leads to an increasing amount of unused screen real estate and unnecessary scrolling. We derive a number of device-sensitive metrics to measure the quality of web page layout in different viewing contexts, which can guide the design of flexible layout templates that scale effectively on large screens.

Despite the fact that average screen size and resolution have dramatically increased, many of today's web sites still do not scale well in larger viewing contexts. The upcoming HTML5 and CSS3 standards propose features that can be used to build more flexible web page layouts, but their potential to accommodate a wider range of display environments is currently relatively unexplored. We examine the proposed standards to identify the most promising features and report on experiments with a number of adaptive layout mechanisms that support the required forms of adaptation to take advantage of greater screen real estates, such as automated scaling of text and media. Special attention is given to the effective use of multi-column layout, a brand new feature for web design that contributes to optimising the space occupied by text, but at the same time still poses problems in predominantly continuous vertical-scrolling browsing behaviours. The proposed solutions were integrated in a flexible layout template that was then applied to an existing news web site and tested on users to identify the adaptive features that best support reading comfort and efficiency.

The system described in this paper attempts to tackle the problem of finding online content based on paper documents through an intuitive touchscreen interface designed for modern scanners and multifunction printers. Touch Scan-n-Search allows the user to select elements of a scanned document (e.g. a newspaper article) and to seamlessly connect to common web search services in order to retrieve the online version of the document along with related content. This is achieved by automatically extracting keyphrases from text elements in the document (obtained by OCR) and creating "tappable" GUI widgets to allow the user to control and fine-tune the search requests. The retrieved content can then be printed, sent, or used to compose new documents.

SmartPublisher is a powerful, all-in-one application for pen-based devices with which users can quickly and intuitively create new documents by reusing individual image and text elements acquired from analogue and/or digital documents. The application is especially targeted at scanning devices with touch screen operating panels or tablet PCs connected to them (e.g. modern multifunction printers with large touch screen displays), as one of its main purposes is reuse of material obtained from scanned paper documents.