Lifting multi-view 2D instance segmentation to a radiance field has proven effective to enhance 3D understanding. Existing works rely on direct matching for end-to-end lifting, yielding inferior results, or employ a two-stage solution constrained by complex preor post-processing. In this work, we design Unified-Lift, a new end-to-end object-aware lifting approach that aims for high-quality 3D segmentation based on our object-aware 3D Gaussian representation. To start, we augment each Gaussian point with a Gaussian-level feature learned using a contrastive loss to encode instance information. Importantly, we introduce a learnable object-level codebook to account for individual objects in the scene for an explicit object-level understanding and associate the encoded object-level features with the Gaussian-level point features for segmentation predictions. While promising, achieving effective codebook learning is nontrivial and a naive solution leads to degraded performance. Hence, we formulate the association learning module and the noisy label filtering module for effective and robust codebook learning. We conduct experiments on three benchmarks LERF-Masked, Replica, and Messy Rooms. Both qualitative and quantitative results manifest that our Unified-Lift clearly outperforms existing methods in terms of segmentation quality and time efficiency.

Endoscopic reconstruction plays a crucial role in surgical robotics. The dynamic lighting conditions and integrated camera-light source in endoscopic scenes create a distinct reconstruction challenge: shape ambiguity. To mitigate this, we propose a Gaussian Splatting (GS) based framework for endoscopic scene reconstruction, enhanced with reflectance regularization. We embed every 3D Gaussian point with physical reflective attributes and combine this representation with a physically based inverse rendering framework. By jointly training 3DGS for view synthesis with this reflectance regularization, we are able to attain high-quality geometry without changing the volume rendering pipeline. Our experiments demonstrate the superiority in both geometry representation and rendering performance compared to existing GS approaches, making it a practical solution for endoscopic applications. Project is available at: https://med-air.github.io/GSR2.

This work introduces MindfulVerse, an AI-Generated Content (AIGC)-driven application for personalized mindfulness experiences in VR. Using fNIRS to observe brain activation, the study demonstrates that generative meditation improves neural activation in self-regulation regions and positively impacts emotional regulation and user participation compared to static content.

Open-vocabulary 3D segmentation is a fundamental yet challenging task, requiring a mutual understanding of both segmentation and language. However, existing Gaussian-splatting-based methods rely either on a single 3D language field, leading to inferior segmentation, or on pre-computed class-agnostic segmentations, suffering from error accumulation. To address these limitations, we present COS3D, a new collaborative prompt-segmentation framework that contributes to effectively integrating complementary language and segmentation cues throughout its entire pipeline. We first introduce the new concept of collaborative field, comprising an instance field and a language field, as the cornerstone for collaboration. During training, to effectively construct the collaborative field, our key idea is to capture the intrinsic relationship between the instance field and language field, through a novel instance-to-language feature mapping and designing an efficient two-stage training strategy. During inference, to bridge distinct characteristics of the two fields, we further design an adaptive language-to-instance prompt refinement, promoting high-quality prompt-segmentation inference. Extensive experiments not only demonstrate COS3D's leading performance over existing methods on two widely-used benchmarks but also show its high potential to various applications,~\ie, novel image-based 3D segmentation, hierarchical segmentation, and robotics.

Large Language Models (LLMs) have recently demonstrated remarkable capabilities in natural language processing tasks and beyond. This success of LLMs has led to a large influx of research contributions in this direction. These works encompass diverse topics such as architectural innovations, better training strategies, context length improvements, fine-tuning, multimodal LLMs, robotics, datasets, benchmarking, efficiency, and more. With the rapid development of techniques and regular breakthroughs in LLM research, it has become considerably challenging to perceive the bigger picture of the advances in this direction. Considering the rapidly emerging plethora of literature on LLMs, it is imperative that the research community is able to benefit from a concise yet comprehensive overview of the recent developments in this field. This article provides an overview of the literature on a broad range of LLM-related concepts. Our self-contained comprehensive overview of LLMs discusses relevant background concepts along with covering the advanced topics at the frontier of research in LLMs. This review article is intended to provide not only a systematic survey but also a quick, comprehensive reference for the researchers and practitioners to draw insights from extensive, informative summaries of the existing works to advance the LLM research.

Reconstructing 3D geometry from unoriented point clouds can benefit many downstream tasks. Recent shape modeling methods mostly adopt implicit neural representation to fit a signed distance field (SDF) and optimize the network by unsigned supervision. However, these methods occasionally have difficulty in finding the coarse shape for complicated objects, especially suffering from the "ghost" surfaces (i.e., fake surfaces that should not exist). To guide the network quickly fit the coarse shape, we propose to utilize the signed supervision in regions that are obviously outside the object and can be easily determined, resulting in our semi-signed supervision. To better recover high-fidelity details, a novel loss-based region sampling strategy and a progressive positional encoding (PE) method are applied to prioritize the optimization towards underfitting and complicated regions. Specifically, we voxelize and partition the object space into sign-known and sign-uncertain regions, in which different supervisions are applied. Besides, we adaptively adjust the sampling rate of each voxel according to the tracked reconstruction loss, so that the network can focus more on the complicated under-fitting regions. We conduct extensive experiments to demonstrate that our method achieves state-of-the-art performance compared to the existing fitting-based methods and comparable performance to learning-based methods on multiple datasets. The code is publicly available at https://github.com/Runsong123/SSP.

During the design, it is important to evaluate the user experience of representative users in many human product interactions. But, in some cases, it is difficult or even impossible to recruit representative users because they have disabilities that do not allow them to take part in such investigations. Thus, alternative populations are widely studied. The most common way to replace real blind people is to use sighted but blindfolded users when studying design solutions. To test whether such alternative or proxy users can be used to represent blind people in social interactions, we examined the communication quality of 20 blind-sighted pairs and 20 blindfolded-sighted pairs in two different experiments. A prototype named E-Gaze glasses was evaluated as the testing tool. Results clearly show that the blindfolded participants achieved significantly higher communication quality than the blind participants. In qualitative data analysis, the blindfolded participants also reported their user experience of being blindfolded in conversations. Our qualitative results strengthen the conclusion that blindfolded users’ behaviour is different from real blind users’ behaviour. We recommend that blind users should not be substituted for blindfolded users in human product evaluations when communication quality is measured.

As the population ages rapidly, there is a strong focus on the healthy aging of older adults. A central part of healthy aging is keeping people connected in later social life. Exergames are recommended as one of the coping strategies to help improve health and quality of life in older adults. In our study, we developed an exergame called Social Balance Ball to engage older and younger people to play together, encouraging social interaction between generations. From May to July 2021, we evaluated this exergame in Shanghai, China, performing a user experiment with 18 unfamiliar young-old pairs under three test conditions (virtual player, mediated human player, and co-located human player). To evaluate the exergame, our main findings demonstrated that participants felt significantly perceived social interaction in mediated play and co-located play than in virtual play. Overall, older participants perceived significantly higher social interaction than younger participants. In this study, we contribute (1) empirical research findings on how the Social Balance Ball exergame enhances social interaction in generations; (2) design implications for informing future design and development of social exergames.

Point cloud completion aims to recover the complete shape based on a partial observation. Existing methods require either complete point clouds or multiple partial observations of the same object for learning. In contrast to previous approaches, we present Partial2Complete (P2C), the first self-supervised framework that completes point cloud objects using training samples consisting of only a single incomplete point cloud per object. Specifically, our framework groups incomplete point clouds into local patches as input and predicts masked patches by learning prior information from different partial objects. We also propose Region-Aware Chamfer Distance to regularize shape mismatch without limiting completion capability, and devise the Normal Consistency Constraint to incorporate a local planarity assumption, encouraging the recovered shape surface to be continuous and complete. In this way, P2C no longer needs multiple observations or complete point clouds as ground truth. Instead, structural cues are learned from a category-specific dataset to complete partial point clouds of objects. We demonstrate the effectiveness of our approach on both synthetic ShapeNet data and real-world ScanNet data, showing that P2C produces comparable results to methods trained with complete shapes, and outperforms methods learned with multiple partial observations. Code is available at https://github.com/CuiRuikai/Partial2Complete.

With the help of the deep learning paradigm, many point cloud networks have been invented for visual analysis. However, there is great potential for development of these networks since the given information of point cloud data has not been fully exploited. To improve the effectiveness of existing networks in analyzing point cloud data, we propose a plug-and-play module, PnP-3D, aiming to refine the fundamental point cloud feature representations by involving more local context and global bilinear response from explicit 3D space and implicit feature space. To thoroughly evaluate our approach, we conduct experiments on three standard point cloud analysis tasks, including classification, semantic segmentation, and object detection, where we select three state-of-the-art networks from each task for evaluation. Serving as a plug-and-play module, PnP-3D can significantly boost the performances of established networks. In addition to achieving state-of-the-art results on four widely used point cloud benchmarks, we present comprehensive ablation studies and visualizations to demonstrate our approach's advantages. The code will be available at https://github.com/ShiQiu0419/pnp-3d.

As the basic task of point cloud analysis, classification is fundamental but always challenging. To address some unsolved problems of existing methods, we propose a network that captures geometric features of point clouds for better representations. To achieve this, on the one hand, we enrich the geometric information of points in low-level 3D space explicitly. On the other hand, we apply CNN-based structures in high-level feature spaces to learn local geometric context implicitly. Specifically, we leverage an idea of error-correcting feedback structure to capture the local features of point clouds comprehensively. Furthermore, an attention module based on channel affinity assists the feature map to avoid possible redundancy by emphasizing its distinct channels. The performance on both synthetic and real-world point clouds datasets demonstrate the superiority and applicability of our network. Comparing with other state-of-the-art methods, our approach balances accuracy and efficiency.

Object detection in three-dimensional (3D) space attracts much interest from academia and industry since it is an essential task in AI-driven applications such as robotics, autonomous driving, and augmented reality. As the basic format of 3D data, the point cloud can provide detailed geometric information about the objects in the original 3D space. However, due to 3D data s sparsity and unorderedness, specially designed networks and modules are needed to process this type of data. Attention mechanism has achieved impressive performance in diverse computer vision tasks; however, it is unclear how attention modules would affect the performance of 3D point cloud object detection and what sort of attention modules could fit with the inherent properties of 3D data. This work investigates the role of the attention mechanism in 3D point cloud object detection and provides insights into the potential of different attention modules. To achieve that, we comprehensively investigate classical 2D attentions, novel 3D attentions, including the latest point cloud transformers on SUN RGB-D and ScanNetV2 datasets. Based on the detailed experiments and analysis, we conclude the effects of different attention modules. This paper is expected to serve as a reference source for benefiting attention-embedded 3D point cloud object detection. The code and trained models are available at: https://qithub.com/SkiQiu0419/attentions_in_3D_detection.

Given the prominence of current 3D sensors, a fine-grained analysis on the basic point cloud data is worthy of further investigation. Particularly, real point cloud scenes can intuitively capture complex surroundings in the real world, but due to 3D data’s raw nature, it is very challenging for machine perception. In this work, we concentrate on the essential visual task, semantic segmentation, for large-scale point cloud data collected in reality. On the one hand, to reduce the ambiguity in nearby points, we augment their local context by fully utilizing both geometric and semantic features in a bilateral structure. On the other hand, we comprehensively interpret the distinctness of the points from multiple resolutions and represent the feature map following an adaptive fusion method at point-level for accurate semantic segmentation. Further, we provide specific ablation studies and intuitive visualizations to validate our key modules. By comparing with state-of-the-art networks on three different benchmarks, we demonstrate the effectiveness of our network.

Point cloud analysis is attracting attention from Artificial Intelligence research since it can be widely used in applications such as robotics, Augmented Reality, self-driving. However, it is always challenging due to irregularities, unorderedness, and sparsity. In this article, we propose a novel network named Dense-Resolution Network (DRNet) for point cloud analysis. Our DRNet is designed to learn local point features from the point cloud in different resolutions. In order to learn local point groups more effectively, we present a novel grouping method for local neighborhood searching and an error-minimizing module for capturing local features. In addition to validating the network on widely used point cloud segmentation and classification benchmarks, we also test and visualize the performance of the components. Comparing with other state-of-the-art methods, our network shows superiority on ModelNet40, ShapeNet synthetic and ScanObjectNN real point cloud datasets.

Eye contact is crucial in social interactions, linking with sincerity and friendliness. However, blind people cannot see and make eye contact when they communicate with sighted people. It influences the involvement of blind people in blind-sighted conversations. Based on this context, we implemented Social glasses with an eye-tracking system, aiming to improve the communication quality between blind and sighted people in face-to-face conversations. Social glasses attempts to simulate the appropriate gaze for blind people, especially establishing the “eye contact” in blind-sighted conversations. To evaluate the impact of the interactive gaze displayed on the Social glasses, we performed dyadic-conversation tests under four experimental conditions (No Gaze, Constant Gaze, Random Gaze, and Interactive Gaze) for 40 participants. Quantitative results showed that the Interactive gaze has a positive impact on improving the communication quality between blind and sighted people, which were consistent with a qualitative analysis of the participants’ comments.

In face-to-face communication, eye gaze is integral to a conversation to supplement verbal language. The sighted often uses eye gaze to convey nonverbal information in social interactions, which a blind conversation partner cannot access and react. My doctoral research is to design gaze simulation for the blind person, to improve the conversation quality between sighted and blind people in face-to-face communication. Designing gaze simulation will consist of two primary parts: to help the blind person feel the gaze from the sighted and to simulate the natural gaze for the blind person as a visual reaction. This paper outlines the motivation, context, research methods, and completed and future steps of my research.

Eye gaze plays an essential role in social interaction which influences our perception of others. It is most likely that we can perceive the existence of another intentional subject through the act of cathing one another's eyes. Based on the notion of perceptual crossing, we aim to establish a meaningful social interaction that emerges out of the perceptual crossing between a person and an everyday object by exploiting the gazing behavior of the person as the input modality for the system. We investigated in literature the experiments that adopt the perceptual crossing as their foundation, lessons learned from literature were used as input for a concept to create meaningful social interaction. We used an eye-tracker to measure gaze behavior that allows the participant to interact with the object by using their eyes through active exploration. It creates a situation where both of them mutually becoming aware of each other's existence. Further, we discuss the motivation for this research, present a preliminary experiment that influences our decision and our directions for future work.

Our eye gaze is important during social interactions. It can generate significant social cues in nonverbal communication. The feeling of being look back when we are gazing at someone influences our social behaviour. In this paper, we propose an interactive coffee cup that is responsive whenever a person is fixating on it. Taking the user's gazing behaviour as our system input modality, we want to create an environment where a person may establish social interaction with an everyday object whenever he/she is looking at it. To make an object visible to the user's eyes and for the user to feel connected with the object, it is expected that the object to possess distinctive characteristics that can acknowledge the user that it is aware of being look at. By combining the recent technology of eye trackers, mechanical design and embedded electronics, we want to explore the possibility of nonverbal social interaction between a person and inanimate object that will respond when a person is looking at it to allow social interaction and to create a sense of emotional bond between the two.

In anthropomorphic design, there has been increasing interests in using kinetic motion and shape changing of the physical objects as a medium to communicate with people. In this paper, we introduce an interactive installation named Whispering Bubbles to explore anthropomorphism through shape-changing interfaces embedded in a physical space. It aims to provide a poetic place for people to whisper with the organically shaped objects (bubbles), to help people release mental stress in their modern lives. When a person approaches bubbles within a given distance, slight up-and-down movements of the bubbles will be activated by infrared sensors embedded in the space; when a person stands nearby a bubble and whispers to it, the bubble will "hear" with its sound detector and be triggered to bend towards the person, indicating engagement in listening. A scale model is implemented to explore and demonstrate interactions.

In face-to-face communication, eye gaze is integral to a conversation to supplement verbal language. The sighted often uses eye gaze to convey nonverbal information in social interactions, which a blind conversation partner cannot access and react to them. In this paper, we present E-Gaze glasses (E-Gaze), an assistive device based on an eye tracking system. It simulates gaze for the blind person to react and engage the sighted in face-to-face conversations. It is designed based on a model that combines eye-contact mechanism and turn-taking strategy. We further propose an experimental design to test the E-Gaze and hypothesize that the model-driven gaze simulation can enhance the conversation quality between the sighted and the blind person in face-to-face communication.

Gaze and eye contact are frequently in social occasions used among sighted people. Gaze is considered as a predictor of attention and engagement between interlocutors in conversations. However, gaze signals from the sighted are not accessible for the blind person in face-to-face communication. In this paper, we present functional work-in-progress prototype, E-Gaze glasses, an assistive device based on an eye tracking system. E-Gaze simulates natural gaze for blind people, especially establishing the "eye contact" between blind and sighted people to enhance their engagement in face-to-face conversations. The gaze behavior is designed based on a turn-taking model, which interprets the corresponding relationship between the conclusive gaze behavior and the interlocutors' conversation flow.

Gaze signals, frequently used by the sighted in social interactions as visual cues, are hardly accessible for low-vision and blind people. A concept is proposed to help the blind people access and react to gaze signals in face-to-face communication. 20 blind and low-vision participants were interviewed to discuss the features of this concept. One feature of the concept is further developed into a prototype, namely Tactile Band, to aim at testing the hypothesis that tactile feedback can enable the blind person to feel attention (gaze signals) from the sighted, enhancing the level of engagement in face-to-face communication. We tested our hypothesis with 30 participants with a face-to-face conversation scenario, in which the blindfolded and the sighted participants talked about a given daily topic. Comments from the participants and the reflection on the experiment provided useful insights for improvements and further research.

Gaze signals are frequently used by the sighted in social interactions as visual cues. However, these signals and cues are hardly accessible for people with visual disability. A conceptual design of E-Gaze glasses is proposed, assistive to create gaze communication between blind and sighted people in face-to-face conversations. We interviewed 20 totally blind and low vision participants to envision the use of the E-Gaze. We explained four features of E-Gaze to participants using persona and use scenarios. Participants discussed the features on their usefulness, efficiency and interest. The results helped us clarify the design direction and further research.