Multi-person eyeblink detection in untrimmed in-the-wild videos is a recently emerged and challenging task. Due to its significant spatio-temporal fine-grained characteristics compared to general actions, we empirically find that general action detectors, though effective in general domains, struggle with this task (i.e., Blink-AP < 2%). Specialized eyeblink detection methods alleviate it through fine-grained spatio-temporal operations. SOTA method proposes a unified model combining instance-aware face localization and eyeblink detection through joint multi-task learning and feature sharing. While effective, it exhibits two critical limitations that may contribute to its unsatisfactory performance (i.e., Blink-AP=10.11%): (1) Face localization and eyeblink detection require distinct spatio-temporal feature granularities, making joint modeling in a unified feature space suboptimal. (2) Eyeblink task training could be largely affected by unstable face-eye feature learning under the joint training paradigm. To address this, we propose DeFB, a decomposed feature learning paradigm with favorable effectiveness and efficiency: (1) We model faces and eyes in granularity-specific feature spaces, which enhances fine-grained perception while reducing computational costs compared to a unified feature space. (2) To mitigate face-eye feature learning instability, we adopt an asynchronous learning mechanism where eye feature learning refines well-trained coarse face features, with shared queries acting as a bridge between stages to retain the efficient feature sharing of existing unified models. Compared with SOTA method, DeFB doubles the performance (Blink-AP: 24.65% v.s. 10.11%) while boosting efficiency by nearly 35%. DeFB can also be integrated as a plug-in to substantially augment the eyeblink detection capabilities of general action detectors.

In this paper, MoEG-HOI is proposed as a novel method for the challenging 3D hand-object interaction (HOI) motion generation task, by introducing Mixture-of-Experts (MoE) to this field for the first time. Almost all the mainstream approaches in HOI motion generation leverage diffusion model as its strong generative ability. Nevertheless, due to HOI’s fine-grained property, well training diffusion in one-stage way is actually not trivial. Existing state-of-the-art (SOTA) methods (e.g.,Text2HOI and MF-MDM) alleviate this mainly via a coarse-to-fine, multi-stage paradigm. Although effective and practical, this paradigm prevents end-to-end training for optimal performance. In contrast, MoEG-HOI applies MoE to address this in one-stage way, with end-to-end training ability. This allows each expert to specialize in certain distinct HOI patterns, which alleviates individual expert’s training difficulty. However, intuitively applying MoE is not optimal due to the issues of: (1) towards expert design, original MoE cannot well characterize hand’s articulated structure at the levels of hand, finger, and joint explicitly, and (2) for expert routing mechanism, the characteristics of variational HOI action classes and diffusion noise levels have not been concerned. Towards the first problem, MoE’s experts are designed into groups that correspond to motion generation for hand, finger, and joint respectively, under the semantic guidance from global to local. To facilitate this, HOI’s text description will be correspondingly refined at Hand-Finger-Joint levels using LLM. Secondly, during MoE routing, the information of HOI’s action label and diffusion noise level is concerned to select experts jointly, to better reveal actions’ inter-class variation and dynamics of diffusion generation. SOTA performance on ARCTIC, GRAB and H2O datasets demonstrates the effectiveness of our method.

In-depth analysis of competitive debates is essential for participants to develop argumentative skills and refine strategies, and further improve their debating performance. However, manual analysis of unstructured and unlabeled textual records of debating is time-consuming and ineffective, as it is challenging to reconstruct contextual semantics and track logical connections from raw data. To address this, we propose Conch, an interactive visualization system that systematically analyzes both what is debated and how it is debated. In particular, we propose a novel parallel spiral visualization that compactly traces the multidimensional evolution of clash points and participant interactions throughout debate process. In addition, we leverage large language models with well-designed prompts to automatically identify critical debate elements such as clash points, disagreements, viewpoints, and strategies, enabling participants to understand the debate context comprehensively. Finally, through two case studies on real-world debates and a carefully-designed user study, we demonstrate Conch's effectiveness and usability for competitive debate analysis.

3D human pose lifting from a single RGB image is a challenging task in 3D vision. Existing methods typically establish a direct joint-to-joint mapping from 2D to 3D poses based on 2D features. This formulation suffers from two fundamental limitations: inevitable error propagation from input predicted 2D pose to 3D predictions and inherent difficulties in handling self-occlusion cases. In this paper, we propose PandaPose, a 3D human pose lifting approach via propagating 2D pose prior to 3D anchor space as the unified intermediate representation. Specifically, our 3D anchor space comprises: (1) Joint-wise 3D anchors in the canonical coordinate system, providing accurate and robust priors to mitigate 2D pose estimation inaccuracies. (2) Depth-aware joint-wise feature lifting that hierarchically integrates depth information to resolve self-occlusion ambiguities. (3) The anchor-feature interaction decoder that incorporates 3D anchors with lifted features to generate unified anchor queries encapsulating joint-wise 3D anchor set, visual cues and geometric depth information. The anchor queries are further employed to facilitate anchor-to-joint ensemble prediction. Experiments on three well-established benchmarks (i.e., Human3.6M, MPI-INF-3DHP and 3DPW) demonstrate the superiority of our proposition. The substantial reduction in error by 14.7% compared to SOTA methods on the challenging conditions of Human3.6M and qualitative comparisons further showcase the effectiveness and robustness of our approach.

Micro lip reading, is characterized by tiny lip movements when someone is speaking. It is highly preferred in robotics and automation, such as a social robot for geriatric care, a patrol robot system for hospitals, and speech system for hearing impaired individuals, etc. However, it has not been well studied before. In this paper, we shed the light to this research topic. A labelled micro lip reading dataset (i.e. HUST-LMLR) of 399 video samples is first established. The samples are captured from the unconstrained movies. One key challenge for micro lip reading in the wild lies on extracting fine features of lip movements effectively and robustly. In this paper, we pay the research efforts to address this issue from two aspects: facial context attention and feature extraction. First, we propose a multi-task learning model for micro lip reading in the wild for the first time. It concerns micro lip reading and facial landmark detection jointly for capturing global face context with soft attention, to facilitate micro lip reading. Second, we propose that motion feature should be concerned and combined with appearance feature jointly to characterize tiny lip movements effectively. Finally, the experiments on HUST-LMLR demonstrate the challenges of our dataset and our proposed approach results in a remarkable improvement of the state-of-the-art by over 26% WER in HUST-LMLR. Furthermore, results in a well-known public dataset LRS2 also show the generalization and superiority of our approach. If accepted, we will publish our HUST-LMLR dataset and relative supporting matierials athttps://hust-dpkw.github.io/Micro-lip-reading/.Note to Practitioners—This paper aims to construct a challenging public dataset and propose a novel vision-based approach for automatically micro lip reading under the unconstrained conditions. A labelled dataset HUST-LMLR was first established to reveal the “micro” characteristics, with 399 video samples captured from 40 unconstrained movies or documentaries. In addition, two key manners in perspective of algorithm are proposed to address the problem of micro lip reading at sentence level in the wild: multi-task learning with facial landmark detection, feature extraction with appearance and motion combined. The extensive experiments demonstrate the challenges of our dataset HUST-LMLR as well as the superiority of our approach in micro lip reading tasks. Nevertheless, it is still somewhat sensitive to the extremely tiny variation of lip movements and the dramatic variation on human posture. It is worth noting that our proposition can be applied not only for medical support, smart health care but also for human-computer interaction, special education, information security, assistant driving, and virtual reality etc.

Online job advertisements on various job portals or websites have become the most popular way for people to find potential career opportunities nowadays. However, the majority of these job sites are limited to offering fundamental filters such as job titles, keywords, and compensation ranges. This often poses a challenge for job seekers in efficiently identifying relevant job advertisements that align with their unique skill sets amidst a vast sea of listings. Thus, we propose well-coordinated visualizations to provide job seekers with three levels of details of job information: a skill-job overview visualizes skill sets, employment posts as well as relationships between them with a hierarchical visualization design; a post exploration view leverages an augmented radar-chart glyph to represent job posts and further facilitates users’ swift comprehension of the pertinent skills necessitated by respective positions ; a post detail view lists the specifics of selected job posts for profound analysis and comparison. By using a real-world recruitment advertisement dataset collected from 51Job, one of the largest job websites in China, we conducted two case studies and user interviews to evaluate JobViz. The results demonstrated the usefulness and effectiveness of our approach.

Lines provide important information in images and line detection is crucial in many applications. Many line features can be used to detect line position while line width (i.e., thickness) is a more structured, higher-level feature compared to edge or other line features. Every point of the wide line structure has its own width in spite of the structure's asymmetry. In this paper, we use the parallel edges to get the line orientation and width. We do not need to know the actual width of the line, but rather recover it to get the region of interest (ROI). Then use a modified OTSU obtaining proper region threshold T to segment the wide line structures. By fusing feature of width and gray, a sequence of tests has been conducted on a variety of image samples obtained from simple natural scene and our experimental results demonstrate the practical and robust of the proposed method.