Social intelligence is vital for effective human-AI interaction. While LLMs demonstrate strong text-based social intelligence, the vision modality remains challenging due to the presence of non-verbal social cues. For example, gaze is the primary conveyor of social attention, yet it cannot be accurately perceived and understood by multimodal LLMs (MLLMs). Therefore, we propose GazeCoT, a pipeline using gaze estimation models to provide MLLMs with the attention of people in images or videos. The gaze information is provided as visual and text prompts compiled into a structured context to support MLLM social reasoning. Benchmark evaluation confirms that GazeCoT enhances MLLMs’ social intelligence by improving gaze perception. A user study in a challenging application involving parent-child interactions demonstrates that GazeCoT improves perceived explainability and trustworthiness by aligning MLLM social perception and social reasoning with human norms. We hope that GazeCoT, a versatile plug-and-play pipeline, can enable socially aware, MLLM-based HCI applications.

Assessing children’s cognitive development in early mathematics is vital for effective teaching. Compared to closed-ended questions, which may fail to capture nuanced developmental spectrum, open-ended elicitation tasks (e.g., asking students to manipulate objects or draw to represent numbers) serve as a promising approach to reveal deeper cognitive processes. However, their diverse and unstructured nature makes systematic analysis challenging for teachers. We present OpenCD, a teacher-facing system that automatically analyzes multimodal student responses to capture individualized insights. Based on Evidence-Centered Design, it combines Vision-Language Models (VLMs) and expert models to generate interactive diagnostic graphs and reports with traceability back to behavioral evidence. In our two-part evaluation, a validation study found 90.3% of the system’s diagnoses “completely reasonable,” and a user study showed that OpenCD reduced teachers’ analysis burden and enhanced their insights into student thinking. Our work contributes to scalable process-based assessment for mathematical literacy.

Web tracking is increasingly pervasive, raising serious concerns about user privacy and security. Among existing techniques, pixel tracking is particularly stealthy and cost-effective, embedding invisible images that exfiltrate user activities to third-party servers. Current defenses, including filter list blocking and conventional machine learning, often fail to capture the cross-site associations that enable pixel tracking to evade detection.

Machine Learning (ML) based malware classifiers are vulnerable to exploitation during the training phase due to the necessity of regular retraining with samples collected from the wild. Recent studies have highlighted the efficacy of backdoor attacks in the malware domain, where attackers can manipulate the model during training by injecting samples embedded with specific triggers, causing the model to establish an association between the trigger and a designated class, thereby achieving evasion of detection. While research on backdoor attacks has been extensively explored in the field of computer vision, it has been largely overlooked in the malware domain. Unlike in the computer vision domain, the threat model in the malware domain typically restricts attackers to employing clean-label attacks (i.e., attackers do not have control over the labeling of poisoned data). However, clean-label attack methods are generally less effective compared to those that involve embedding triggers and altering sample labels to the target class (called corrupted-label attacks). To address this limitation, we propose a simple yet effective method that involves Poisoning Malware-Similar Benignware (PMSB) instead of random selection, thereby approximating the scenario of corrupted-label attacks and enhancing the effectiveness of clean-label attacks. Additionally, we introduce three similarity measurement methods based on feature-based distance, distribution-based distance, and contribution-based difference to select malware-similar benignware. Comprehensive evaluations across three different trigger types and three datasets demonstrate the superiority and general applicability of PMSB.

Tracking fine-grained finger movements with IMUs for continuous 2D-cursor control poses significant challenges due to limited sensing capabilities. Our findings suggest that finger-motion patterns and the inherent structure of joints provide beneficial physical knowledge, which lead us to enhance motion perception accuracy by integrating physical priors into ML models. We propose MouseRing, a novel ring-shaped IMU device that enables continuous finger-sliding on unmodified physical surfaces like a touchpad. A motion dataset was created using infrared cameras, touchpads, and IMUs. We then identified several useful physical constraints, such as joint co-planarity, rigid constraints, and velocity consistency. These principles help refine the finger-tracking predictions from an RNN model. By incorporating touch state detection as a cursor movement switch, we achieved precise cursor control. In a Fitts’ Law study, MouseRing demonstrated input efficiency comparable to touchpads. In real-world applications, MouseRing ensured robust, efficient input and good usability across various surfaces and body postures.

In the ever-evolving digital landscape, the complexity of web technologies has significantly increased. This complexity highlights the limitations of traditional web defense mechanisms in offering complete protection. Web shells, especially, present a formidable challenge in the field of web security. Recognizing and addressing this challenge is of paramount importance. It necessitates innovative understandings/approaches that contribute to the collective knowledge in web security. To achieve this, our paper introduces a novel type of attack: the cross-file web shell. Alongside this, we propose a detection methodology utilizing Graph Neural Networks (GNNs). Our method leverages the Function Call Graph (FCG) to generate graph embedding, capturing both the structural and semantic nuances of code. By incorporating a variety of statistics features, our approach adeptly identifies the characteristic patterns of web shells. Utilizing deep learning, this technique allows for precise classification and detection. The efficacy of our method is demonstrated by its impressive performance in detecting cross-file web shells, achieving an accuracy of 96.65% and an F1-score of 96.63%. In addition, we simulate real-world cross-file web shell attack and successfully detecte them using our method. These results underscore the potential of our approach in significantly enhancing web security measures.

Accurate attribution and tracing of cyber attacks require a comprehensive understanding of the resources employed by malicious actors. However, Indicators of Compromise (IoCs) can only reveal a portion of the attacker’s assets. To enhance the capability of clue expansion, this study introduces a novel approach to associating attack sources, facilitating the identification of additional IP addresses and subnets that may correspond to a single malicious actor. We focus on the scenario of compromised email accounts and utilize login logs as foundational data. We employ Gaussian Mixture Models (GMM) to construct a reference model that captures known malicious behaviors. Then, we utilize a genetic algorithm to filter and select candidate subnets that exhibit the attack patterns outlined by the reference model. Through evaluation on real-world data, we demonstrate the effectiveness of our proposed method in successfully attributing multiple attack sources to a single attacker, thereby providing valuable insights for manual investigations.

We present ShadowTouch, a novel sensing method to recognize the subtle hand-to-surface touch state for independent fingers based on optical auxiliary. ShadowTouch mounts a forward-facing light source on the user’s wrist to construct shadows on the surface in front of the fingers when the corresponding fingers are close to the surface. With such an optical design, the subtle vertical movements of near-surface fingers are magnified and turned to shadow features cast on the surface, which are recognizable for computer vision algorithms. To efficiently recognize the touch state of each finger, we devised a two-stage CNN-based algorithm that first extracted all the fingertip regions from each frame and then classified the touch state of each region from the cropped consecutive frames. Evaluations showed our touch state detection algorithm achieved a recognition accuracy of 99.1% and an F-1 score of 96.8% in the leave-one-out cross-user evaluation setting. We further outlined the hand-to-surface interaction space enabled by ShadowTouch’s sensing capability from the aspects of touch-based interaction, stroke-based interaction, and out-of-surface information and developed four application prototypes to showcase ShadowTouch’s interaction potential. The usability evaluation study showed the advantages of ShadowTouch over threshold-based techniques in aspects of lower mental demand, lower effort, lower frustration, more willing to use, easier to use, better integrity, and higher confidence.

Soybean is a globally significant crop, playing a vital role in human nutrition and agriculture. Its complex genetic structure and wide trait variation, however, pose challenges for breeders and researchers aiming to optimize its yield and quality. Addressing this biological complexity requires innovative and accurate tools for trait prediction. In response to this challenge, we have developed SoyDNGP, a deep learning-based model that offers significant advancements in the field of soybean trait prediction. Compared to existing methods, such as DeepGS and DNNGP, SoyDNGP boasts a distinct advantage due to its minimal increase in parameter volume and superior predictive accuracy. Through rigorous performance comparison, including prediction accuracy and model complexity, SoyDNGP represents improved performance to its counterparts. Furthermore, it effectively predicted complex traits with remarkable precision, demonstrating robust performance across different sample sizes and trait complexities. We also tested the versatility of SoyDNGP across multiple crop species, including cotton, maize, rice and tomato. Our results showed its consistent and comparable performance, emphasizing SoyDNGP's potential as a versatile tool for genomic prediction across a broad range of crops. To enhance its accessibility to users without extensive programming experience, we designed a user-friendly web server, available at http://xtlab.hzau.edu.cn/SoyDNGP. The server provides two features: 'Trait Lookup', offering users the ability to access pre-existing trait predictions for over 500 soybean accessions, and 'Trait Prediction', allowing for the upload of VCF files for trait estimation. By providing a high-performing, accessible tool for trait prediction, SoyDNGP opens up new possibilities in the quest for optimized soybean breeding.

Machine learning (ML) based Malware classification provides excellent performance and has been deployed in various real-world applications. Training for malware classification often relies on crowdsourced threat feeds, which exposes a natural attack injection point. Considering a real-world threat model for backdoor poisoning attacks on a malware dataset, because attackers are generally considered to have no control over the sample-labeling process, they conduct a clean-label attack, a more realistic scenario, by generating backdoored benign binaries that will be disseminated through threat intelligence platforms and poison the datasets for downstream malware classifiers. To avoid the threat of backdoor poisoned datasets, we propose an explanation-powered defense methodology called make data reliable (MDR), which is a general and effective mitigation to ensure the reliability of datasets by removing backdoored samples. We use a surrogate model and explanation tool Shapley Additive exPlanations (SHAP) to filter suspicious samples, then perform watermark identification based on the filtered suspicious samples, and finally remove samples with the identified watermark to construct a reliable dataset. We conduct extensive experiments on two typical datasets that were manually poisoned using different attack strategies. Experimental results show that the MDR achieves backdoored samples removal rate greater than 99.0% for different datasets and attack conditions, while maintaining an extremely low false positive rate of less than 0.1%. Furthermore, to confirm the generality of MDR, we use different models to perform a model-agnostic evaluation. The results show that, MDR is a general methodology that does not rely on any specific model.