Always-on sensing of AI applications on AR glasses makes traditional permission techniques inefficient for context-dependent private visual data within home environments. Home presents a challenging privacy context due to massive sensitive objects and the intimate nature of daily routines. We propose VisGuardian, a fine-grained content-based visual permission technique for AR glasses. VisGuardian features a group-based control mechanism that enables users to efficiently manage permissions for multiple private objects. VisGuardian detects objects using YOLO and adopts a pre-classified schema to group them. By selecting a single object, users can obscure groups of related objects based on criteria including privacy sensitivity, object category, or spatial proximity. A technical evaluation shows VisGuardian achieves mAP50 of 0.6704 with only 14.0 ms latency and a 1.7% increase in battery consumption per hour. Furthermore, a user study (N=24) comparing VisGuardian to slider-based and object-based baselines found it to be significantly faster for setting permissions and was preferred by users for its efficiency, effectiveness, and ease of use.

In automated UI design generation, a key challenge is the lack of support for iterative processes, as most systems focus solely on end-to-end output. This stems from limited capabilities in interpreting design intent and a lack of transparency for refining intermediate results. To better understand these challenges, we conducted a formative study that identified concrete and actionable requirements for supporting iterative design with Generative Tools. Guided by these findings, we propose PrototypeFlow, a human-centered system for automated UI generation that leverages multi-modal inputs and models. PrototypeFlow takes natural language descriptions and layout preferences as input to generate the high-fidelity UI design. At its core is a theme design module that clarifies implicit design intent through prompt enhancement and orchestrates sub-modules for component-level generation. Designers retain full control over inputs, intermediate results, and final prototypes, enabling flexible and targeted refinement by steering generation and directly editing outputs. Our experiments and user studies confirmed the effectiveness and usefulness of our proposed PrototypeFlow.

Figure 1: We review and categorize VMIs aimed at enhancing context awareness.Our key contribution is a Macro-Micro-Macro level (whole-detail-whole) system design framework, providing actionable references from a Data Modality-Driven perspective: (1) Macro-level contextual factors: considerations for context understanding (Section 3); (2) Micro-level system foundations: input data modality (visual + other modalities), data integration stages, multimodal data processing and evaluation strategies (Sections 4, 5); (3) Macro-level design synthesis: application domains, design considerations and key challenges (Sections 6, 7).

Driver decision quality in take-overs is critical for effective human-Autonomous Driving System (ADS) collaboration.However, current research lacks detailed analysis of its variations.This paper

This work focuses on an active topic in the HCI community, namely tutorial creation by demonstration. We present a novel tool named SmartRecorder that facilitates people, without video editing skills, creating video tutorials for smartphone interaction tasks. As automatic interaction trace extraction is a key component to tutorial generation, we seek to tackle the challenges of automatically extracting user interaction traces on smartphones from screencasts. Uniquely, with respect to prior research in this field, we combine computer vision techniques with IMU-based sensing algorithms, and the technical evaluation results show the importance of smartphone IMU data in improving system performance. With the extracted key information of each step, SmartRecorder generates instructional content initially and provides tutorial creators with a tutorial refinement editor designed based on a high recall (99.38%) of key steps to revise the initial instructional content. Finally, SmartRecorder generates video tutorials based on refined instructional content. The results of the user study demonstrate that SmartRecorder allows non-experts to create smartphone usage video tutorials with less time and higher satisfaction from recipients.

Everyday, billions of people use footwear for walking, running, or exercise. Of emerging interest are “smart footwear”, which help users track gait, count steps or even analyse performance. However, such nascent footwear lack fine-grain ground surface context awareness, which could allow them to adapt to the conditions and create usable functions and experiences. Hence, this research aims to recognize the walking surface using a radar sensor embedded in a shoe, enabling ground context-awareness. Using data collected from 23 participants from an in-the-wild setting, we developed several classification models. We show that our model can detect five common terrain types with an accuracy of 80.0% and further ten terrain types with an accuracy of 66.3%, while moving. Importantly, it can detect the gait motion types such as ‘walking’, ‘stepping up’, ‘stepping down’, ‘still’, with an accuracy of 90%. Finally, we present potential use cases and insights for future work based on such ground-aware smart shoes.

Some people with upper body motor impairments but sound lower limbs usually use feet to interact with smartphones. However, touching the touchscreen with big toes is tiring, inefficient and easy to mistouch. Targeting at this pain point, we propose FootUI, which leverages the phone camera to track users’ feet and translates the foot gestures to smartphone operations. This technique enables users to interact with smartphones while reclining on the bed and improves the comfort of users. In this paper, we present the design and evaluation of the foot gestures through user studies as well as the development and evaluation of FootUI. Results show that toes-based foot gestures are not only less perceivable but also uncomfortable. FootUI avoid the use of toes-based gestures and is proved to be an easy, efficient and interesting input technique for people with upper body motor impairments but sound lower limbs.

We propose Auth+Track, a novel authentication model that aims to reduce redundant authentication in everyday smartphone usage. By sparse authentication and continuous tracking of the user’s status, Auth+Track eliminates the “gap” authentication between fragmented sessions and enables “Authentication Free when User is Around”. To instantiate the Auth+Track model, we present PanoTrack, a prototype that integrates body and near field hand information for user tracking. We install a fisheye camera on the top of the phone to achieve a panoramic vision that can capture both user’s body and on-screen hands. Based on the captured video stream, we develop an algorithm to extract 1) features for user tracking, including body keypoints and their temporal and spatial association, near field hand status, and 2) features for user identity assignment. The results of our user studies validate the feasibility of PanoTrack and demonstrate that Auth+Track not only improves the authentication efficiency but also enhances user experiences with better usability.

Artificial intelligence (AI) is enabling the automated analysis of large amounts of image/video data, boosting the speed of multimedia data processing remarkably. Meanwhile, Image Quality Assessment (IQA) plays an important role in developing automatic analysis methods. To ensure the effectiveness of AI, images in multimedia applications should be considered for visual examination by both human and machine. Therefore, it is significant to understand the differences between human's and AI's perception of semantic distortion. However, little work has been done due to the lack of data from human on the semantic level. In this paper, we first propose a semantic database (SID) based on the surveillance scenarios, by collecting subjective average recognition rates of 3 semantic targets (face, pedestrian, license plate) with 3 types of distortion (JPEG Compression, BPG Compression, Motion Blur). Then, we present a detailed analysis of how human and AI perceive semantic distortion differently. Experimental results show that AI is stronger in tolerance to distortion than human beings on average, while weaker at generalization and stability. It is also implied in the experiments that existing IQA methods are not effective enough at judging the semantic distortion.

Semantic information is important in video encryption. However, existing image quality assessment (IQA) methods, such as the peak signal to noise ratio (PSNR), are still widely applied to measure the encryption security. Generally, these traditional IQA methods aim to evaluate the image quality from the perspective of visual signal rather than semantic information. In this paper, we propose a novel semantic-level full-reference image quality assessment (FR-IQA) method named Semantic Distortion Measurement (SDM) to measure the degree of semantic distortion for video encryption. Then, based on a semantic saliency dataset, we verify that the proposed SDM method outperforms state-of-the-art algorithms. Furthermore, we construct a Region Of Semantic Saliency (ROSS) video encryption system to demonstrate the effectiveness of our proposed SDM method in the practical application.