Shaking a cup of wine or other fluids in virtual environments is engaging but has been limited by challenges in delivering real-time haptic feedback for liquid collisions. ShakeSense is a haptic rendering system that integrates electrotactile stimulation with physics-based simulation to deliver immersive feedback for liquid dynamics in handheld containers. It employs a high-density electrode array to deliver dynamic tactile sensations, conveying friction and pressure changes on the user’s fingerpad. A dedicated end-to-end pipeline computes fingerpad forces from liquid-container-finger interactions, ensuring feedback aligns with natural fluid movement. Two studies evaluated ShakeSense’s performance and user perception. Study 1 showed that electrotactile patterns were distinguishable across directions, and synchronizing container movement with stimulation enhanced perceived force changes. Study 2 demonstrated that ShakeSense effectively simulated liquid motion, capturing multidimensional, coordinated interactions, and outperformed conventional Center-of-Mass approaches. Overall, ShakeSense provides clear, fine-grained tactile feedback for fluid interactions.

This workshop aims to explore the potential of large generative models to revolutionize the way we interact with multimodal information. A Large Language Model (LLM) represents a sophisticated form of artificial intelligence engineered to comprehend and produce natural language text, exemplified by technologies such as GPT, LLaMA, Flan-T5, ChatGLM, and Qwen, etc. These models undergo training on extensive text datasets, exhibiting commendable attributes including robust language generation, zeroshot transfer capabilities, and In-Context Learning (ICL). With the surge in multimodal content-encompassing images, videos, audio, and 3D models-over the recent period, Large MultiModal Models (LMMs) have seen significant enhancements. These improvements enable the augmentation of conventional LLMs to accommodate multimodal inputs or outputs, as seen in BLIP, Flamingo, KOSMOS, LLaVA, Gemini, GPT-4, etc. Concurrently, certain research initiatives have delved into generating specific modalities, with Kosmos2 and MiniGPT-5 focusing on image generation, and SpeechGPT on speech production. There are also endeavors to integrate LLMs with external tools to achieve a near 'any-to-any' multimodal comprehension and generation capacity, illustrated by projects like Visual-ChatGPT, ViperGPT, MMREACT, HuggingGPT, and AudioGPT. Collectively, these models, spanning not only text and image generation but also other modalities, are referred to as large generative models. This workshop will provide an opportunity for researchers, practitioners, and industry professionals to explore the latest trends and best practices in the field of multimodal applications of large generative models. We also remark that the submissions are not limited to the use of such models. The workshop will also focus on exploring the challenges and opportunities of integrating large language models with other AI technologies such as computer vision and speech recognition. Additionally, the workshop will provide a platform for participants to present their research, share their experiences, and discuss potential collaborations.

Pedestrian intent prediction is critical for autonomous driving, as accurately predicting crossing intentions helps prevent collisions and ensures the safety of both pedestrians and passengers. Recent research has focused on vision-based deep neural networks for this task, but challenges remain. First, current methods suffer from low efficiency in multi-feature fusion and unreliable predictions under challenging conditions. Additionally, real-time performance is essential in practical applications, so the efficiency of the algorithm is crucial. To address these issues, we propose a novel architecture, Dual-STGAT, which uses a dual-level spatio-temporal graph network to extract pedestrian pose and scene interaction features, reducing information loss and improving feature fusion efficiency. The model captures key features of pedestrian behavior and the surrounding environment through two modules: the Pedestrian Module and the Scene Module. The Pedestrian Module extracts pedestrian motion features using a spatio-temporal graph attention network, while the Scene Module models interactions between pedestrians and surrounding objects by integrating visual, semantic, and motion information through a graph network. Extensive experiments conducted on the PIE and JAAD datasets show that Dual-STGAT achieves over 90% accuracy in pedestrian crossing intention prediction, with inference latency close to 5ms, making it well-suited for large-scale production autonomous driving systems that demand both performance and computational efficiency.

Indoor scene modification has emerged as a prominent area within computer vision, particularly for its applications in Augmented Reality (AR) and Virtual Reality (VR). Traditional methods often rely on pre-existing object databases and predetermined object positions, limiting their flexibility and adaptability to new scenarios. In response to this challenge, we present a novel end-to-end multi-modal deep neural network capable of generating point cloud objects seamlessly integrated with their surroundings, driven by textual instructions. Our work proposes a novel approach in scene modification by enabling the creation of new environments with previously unseen object layouts, eliminating the need for pre-stored CAD models. Leveraging Point-E as our generative model, we introduce innovative techniques such as quantized position prediction and Top-K estimation to address the issue of false negatives resulting from ambiguous language descriptions. Furthermore, we conduct comprehensive evaluations to showcase the diversity of generated objects, the efficacy of textual instructions, and the quantitative metrics, affirming the realism and versatility of our model in generating indoor objects. To provide a holistic assessment, we incorporate visual grounding as an additional metric, ensuring the quality and coherence of the scenes produced by our model. Through these advancements, our approach not only advances the state-of-the-art in indoor scene modification but also lays the foundation for future innovations in immersive computing and digital environment creation. The project is available at https://ainnovatelab.github.io/Context-aware-Indoor-PCG.

To accurately understand engineering drawings, it is essential to establish the correspondence between images and their description tables within the drawings. Existing document understanding methods predominantly focus on text as the main modality, which is not suitable for documents containing substantial image information. In the field of visual relation detection, the structure of the task inherently limits its capacity to assess relationships among all entity pairs in the drawings. To address this issue, we propose a vision-based relation detection model, named ViRED, to identify the associations between tables and circuits in electrical engineering drawings. Our model mainly consists of three parts: a vision encoder, an object encoder, and a relation decoder. We implement ViRED using PyTorch to evaluate its performance. To validate the efficacy of ViRED, we conduct a series of experiments. The experimental results indicate that, within the engineering drawing dataset, our approach attained an accuracy of 96 % in the task of relation prediction, marking a substantial improvement over existing methodologies. The results also show that ViRED can inference at a fast speed even when there are numerous objects in a single engineering drawing.

Ke Lin, Yiyang Luo, Zijian Zhang, Luo Ping. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

Federated learning is a distributed machine learning paradigm that resolves the conflict between training requirements and client data privacy. There are some challenges for federated learning, such as data heterogeneity and communication load, which lead to the global model bias and slow convergence. In this work, we address the problem of data heterogeneity and communication load from a novel perspective, which is named FedSPARK. 1) We propose a new federated learning interaction training strategy, serial pipeline training (SPT). SPT changes the local training of a single client to serial training of multiple clients, which improves the performance of the global model on heterogeneous data. 2) We propose the global knowledge regularization (GKR) which is inspired by continuous learning. GKR can reduce the bias of the client local model by building global knowledge. Through theoretical analysis and experiments on multiple datasets, we show that our approaches greatly reduce the computation of clients and the amount of communication between clients and server, and improve the efficiency of federated learning compared to existing methods.

Despite that deep neural networks have achieved satisfactory performance, they rely on powerful hardware for training, which is expensive and not easy to get. Therefore, the compression and acceleration of deep neural network is important. In addition, lightweight design of complex model usually leads to the decrease of model accuracy. In order to solve the problems, we propose a novel ensemble learning model called FA-GAL-ResNet, which is a lightweight residual network integrated focused attention mechanism and generative adversarial learning by knowledge distillation. In the training of knowledge distillation, the traditional single teacher network is replaced by the multi-teacher hybrid network, which enhances the learning span of student network. Meanwhile, the distilled network with attention scoring module focuses on the notable features related to the target task and the irrelevant features would be filtered by grade of scores and weights. Experimental results show that the proposed network structure improves the accuracy of image classification task, in comparison with the separate complex as well as lightweight neural networks.