Point cloud quality frequently degrades during various processes, such as scanning, compression, and transmission. Hence, reliable Point Cloud Quality Assessment (PCQA) methods are essential for detecting and mitigating the degradation in 3D applications. This paper proposes an accurate full-reference PCQA method that leverages Multimodal Large Language Models (MLLMs). The proposed method utilizes responses generated by MLLMs to assess point cloud quality. We introduce three innovative PCQA metrics derived from MLLMs: 1) response similarity score, 2) relative quality response score, and 3) absolute quality response score. In addition, we integrate these MLLM-based scores with conventional PCQA metrics using support vector regression to improve accuracy. Experimental results demonstrate that the average Pearson’s Linear Correlation Coefficient (PLCC) and Spearman’s Rank-Order Correlation Coefficient (SROCC) across three datasets improved by 0.046 (from 0.871 to 0.917) and 0.055 (from 0.842 to 0.897), respectively, compared to the state-of-the-art FR-PCQA method.

3D mesh models with texture maps play an important role in various 3D applications. Since No-Reference Textured Mesh Quality Assessment (NR-TMQA) methods are essential to identify low-quality mesh models and enhance their quality, this paper presents an accurate NR-TMQA method. Our key contributions are as follows: 1) We propose a novel NR-TMQA scheme that leverages both point cloud and rendered image representations from distorted mesh models. 2) We introduce a novel point cloud representation derived from the faces of a 3D mesh model. It is referred to as a face-based point cloud, which effectively captures face-level features. 3) We present novel graph-based features to estimate mesh quality using the face-based point cloud. Our experimental results demonstrate that Pearson’s linear correlation coefficient and Spearman’s rank-order correlation coefficient in two large-scale TMQA datasets improved by 0.148 (from 0.572 to 0.720) and 0.18 (from 0.525 to 0.705), respectively, compared to the conventional state-of-the-art TMQA method.

In this research, we propose a directional sound source representation technique for 3D contents such as volumetric video in metaverse and digital twin. Our proposed technique enables us to have a novel 3D audio-visual experience which is derived from immersive audio presentation expressing the radiation characteristics of sound source. To realize such an experience, we configure the spaced placement of paired microphone array and capture sound source signals completely without obstacles for volumetric video capture. Then, we synthesize the directional sound source signal using our technique which conducts signal processing to capture sound signals based on the positional and directional information of an object relative to a user. We developed and demonstrated a VR application using this technique to evaluate the change of sound with the object or user's movement in accordance with visual rendering. In our user study, we received lots of positive feedback for a novel audio-visual experience.

Lifestyle changes and digitalization have reduced opportunities for face-to-face, intimate communication, which is an indispensable activity for human beings. We have realized a method that allows intimate communication between two persons even though they are faraway from each other by integrating multimodal technologies. Sync Sofa is a new sofa-type communication tool. It senses the partner with a camera, microphones, and accelerometers. The sensed data are cross-modally integrated and then presented to the user through a life-size display, multichannel loudspeakers, and multichannel vibrotactile actuators. We have received positive feedback from many users who have experienced Sync Sofa, such as, "I felt like the other person was really sitting next to me".

Foreground segmentation technologies play an important role in applications such as free-viewpoint video (FVV) and sports video analysis. In this situation, we propose a new method that achieves accurate foreground silhouette extraction method using dynamic object presence probability (DOPP). Our main contributions are as follows. 1) Object presence probability for each pixel is calculated from the object recognition results based on deep learning. After that, background subtraction is implemented by changing the threshold and the update rate of the background model in response to the object presence probability. Parameter tuning of background subtraction is executed by using the object recognition results to improve the silhouette extraction quality. 2) To calculate more accurate silhouette images, parameters of background subtraction are adjusted by monitoring optical flows between consecutive frames. The object presence probability of the current frame is dynamically updated by using the object presence probability of the previous frame with optical flows. In the experiments, we confirmed that the proposed method achieved more accurate silhouette extraction than conventional methods in three sports sequences.

In this paper, we report on a parallel free-viewpoint video synthesis algorithm that can efficiently reconstruct a high-quality 3D scene representation of sports scenes. The proposed method focuses on a scene that is captured by multiple synchronized cameras featuring wide-baselines. The following strategies are introduced to accelerate the production of a free-viewpoint video taking the improvement of visual quality into account: (1) a sparse point cloud is reconstructed using a volumetric visual hull approach, and an exact 3D ROI is found for each object using an efficient connected components labeling algorithm. Next, the reconstruction of a dense point cloud is accelerated by implementing visual hull only in the ROIs; (2) an accurate polyhedral surface mesh is built by estimating the exact intersections between grid cells and the visual hull; (3) the appearance of the reconstructed presentation is reproduced in a view-dependent manner that respectively renders the non-occluded and occluded region with the nearest camera and its neighboring cameras. The production for volleyball and judo sequences demonstrates the effectiveness of our method in terms of both execution time and visual quality.