Point cloud quality frequently degrades during various processes, such as scanning, compression, and transmission. Hence, reliable Point Cloud Quality Assessment (PCQA) methods are essential for detecting and mitigating the degradation in 3D applications. This paper proposes an accurate full-reference PCQA method that leverages Multimodal Large Language Models (MLLMs). The proposed method utilizes responses generated by MLLMs to assess point cloud quality. We introduce three innovative PCQA metrics derived from MLLMs: 1) response similarity score, 2) relative quality response score, and 3) absolute quality response score. In addition, we integrate these MLLM-based scores with conventional PCQA metrics using support vector regression to improve accuracy. Experimental results demonstrate that the average Pearson’s Linear Correlation Coefficient (PLCC) and Spearman’s Rank-Order Correlation Coefficient (SROCC) across three datasets improved by 0.046 (from 0.871 to 0.917) and 0.055 (from 0.842 to 0.897), respectively, compared to the state-of-the-art FR-PCQA method.

In real-time applications utilizing point clouds, no-reference point cloud quality assessment (NR-PCQA) methods are essential to improve the accuracy of downstream tasks. For example, in point cloud denoising, NR-PCQA results can be benchmarks for determining the optimal parameters when reference data are unavailable. This paper presents an accurate and fast NR-PCQA method based on graph signal processing. First, we propose new features derived from graph signal variation (GSV) to train a support vector regression model. These features improve the correlation with subjective scores and the robustness against inaccurate graph construction. Second, we present a point selection technique based on graph edge weights that allows us to exclude less relevant points, which results in a precise PCQA. Third, we propose a diagonal scan-line graph (DSLG) construction with a superior tradeoff between accurate and fast graph construction. Our experiments demonstrate improved accuracy and computation time compared with conventional methods with three types of open datasets.

This paper presents an automated classification system for object detection in multi-LiDAR sensor networks. The system leverages a deep learning model for an initial coarse classification and subsequently employs feature-based extraction of secondary classes from point-cloud data. Additional labeling and deep learning-based processing on these secondary classes facilitate a more fine-grained classification. The efficacy of the presented approach is demonstrated with real-world data.

We introduce Painting Inferno, a novel method for controlling heat and stiffness using highly electrically conductive carbon nanomaterial heating paint. Heat has found widespread applications in thermochromic displays, shape-changing interfaces, haptic devices, and materials with adjustable stiffness. Although Joule heaters based on heating circuits using electrically conductive materials have been widely used, the complex design and fabrication processes limit the freedom to create custom heaters in scale, shape, and material. As an alternative Joule heating method, we explore the potential of carbon nanomaterial heating paints, which enable the rapid generation of uniform heat at low voltages. We present simple fabrication methods for creating handmade heaters using off-the-shelf materials and cutting machines and demonstrate the feasibility of crafting heaters with complex shapes and grid-array configurations. Leveraging the heating paint’s compatibility with various materials, we showcase the versatile applications for interactive thermal displays, stiffness modulation devices, and pneumatic interfaces for stiffness-shape transformations.

Point clouds are utilized in various 3D applications such as cross-reality (XR) and realistic 3D displays. In some applications, e.g., for live streaming using a 3D point cloud, real-time point cloud denoising methods are required to enhance the visual quality. However, conventional high-precision denoising methods cannot be executed in real time for large-scale point clouds owing to the complexity of graph constructions with K nearest neighbors and noise level estimation. This paper proposes a fast graph-based denoising (FGBD) for a large-scale point cloud. First, high-speed graph construction is achieved by scanning a point cloud in various directions and searching adjacent neighborhoods on the scanning lines. Second, we propose a fast noise level estimation method using eigenvalues of the covariance matrix on a graph. Finally, we also propose a new low-cost filter selection method to enhance denoising accuracy to compensate for the degradation caused by the acceleration algorithms. In our experiments, we succeeded in reducing the processing time dramatically while maintaining accuracy relative to conventional denoising methods. Denoising was performed at 30fps, with frames containing approximately 1 million points.

Point clouds in 3D applications frequently experience quality degradation during processing, e.g., scanning and compression. Reliable point cloud quality assessment (PCQA) is important for developing compression algorithms with good bitrate-quality trade-offs and techniques for quality improvement (e.g., denoising). This paper introduces a full-reference (FR) PCQA method utilizing spectral graph wavelets (SGWs). First, we propose novel SGW-based PCQA metrics that compare SGW coefficients of coordinate and color signals between reference and distorted point clouds. Second, we achieve accurate PCQA by integrating several conventional FR metrics and our SGW-based metrics using support vector regression. To our knowledge, this is the first study to introduce SGWs for PCQA. Experimental results demonstrate the proposed PCQA metric is more accurately correlated with subjective quality scores compared to conventional PCQA metrics.

Point clouds are utilized in many 3-D applications such as cross-reality (XR) and realistic 3-D display. They consist of a set of points with 3-D coordinates and associated color signals. These color signals are often perturbed by noise induced by the measurement errors of scanning devices. In this paper, we propose a point cloud denoising method for color signals. Since many conventional methods for point cloud color denoising are based on a low-pass filter in the graph spectral domain, denoising accuracy is affected by the choice of graph. We propose a graph construction method using 3-D patch-based similarity, in which the similarity is calculated with small 3-D patches around the connected points. This is in contrast with conventional graph construction methods for denoising, which are based on point properties such as pairwise point distances and differences in color. Second, we propose a low-pass filtering method where the frequency response is chosen automatically depending on the estimated noise level. Our experimental results show that our proposed method, 3-D patch-based similarity (3DPBS), achieves the best denoising accuracy compared with graph-based state-of-the-art methods.

Point cloud applications suffer from geometric noise caused by measurement errors induced by the point cloud acquisition system. We propose a novel graph construction method, surface-consistent non-negative kernel regression (SC-NNK), that can achieve more accurate denoising of geometry information in combination with spectral graph wavelet transforms (SGWTs). Unlike conventional graph construction methods such as the K-nearest neighbor (KNN), which have been adopted in previous SGWT-based geometry denoising methods, SC-NNK graphs consider geometrical and frequency characteristics to remove redundant edge connections from a KNN graph. In addition, we propose a novel noise level estimation method that achieves improved accuracy by detecting flat surfaces in point clouds, resulting in better wavelet shrinkage thresholds for denoising. Our experimental results show that the proposed method outperforms recent deep-learning-based and graph-based state-of-the-art denoising methods.

In this research, we propose a directional sound source representation technique for 3D contents such as volumetric video in metaverse and digital twin. Our proposed technique enables us to have a novel 3D audio-visual experience which is derived from immersive audio presentation expressing the radiation characteristics of sound source. To realize such an experience, we configure the spaced placement of paired microphone array and capture sound source signals completely without obstacles for volumetric video capture. Then, we synthesize the directional sound source signal using our technique which conducts signal processing to capture sound signals based on the positional and directional information of an object relative to a user. We developed and demonstrated a VR application using this technique to evaluate the change of sound with the object or user's movement in accordance with visual rendering. In our user study, we received lots of positive feedback for a novel audio-visual experience.

We propose a novel graph-based denoising method to correct the quantization error (step noise) arising in the process of generating the visual hull, a commonly used technique to synthesize free-viewpoint video. To reduce this step noise effectively, we propose two new notions of consistency, pixel value consistency and normal vector consistency. The resulting denoising method involves a first step of graph construction using the proposed consistency metrics, followed by graph filtering of the 3D point cloud coordinates. Our experiments show that our approach provides visually and quantitatively better performance than state-of-the-art methods.

Many applications that use point clouds, such as 3D immersive telepresence, suffer from geometric quality degradation. This noise may be caused by measurement errors of the capturing device or by the point cloud estimation method. In this paper, we propose a novel graph-based point cloud denoising approach using the spectral graph wavelet transform (SGWT) and graph wavelet shrinkage. Unlike conventional SGWT-based denoising methods, the proposed wavelet shrinkage thresholds are determined based on the normal vector at each point and are thus based on the local geometric structure of the point cloud. This approach avoids excessive wavelet shrinkage, which can lead to the loss of complex geometric structure. Experimental results show that the proposed method achieves the best accuracy as compared with recent deep-learning-based and graph-based state-of-the-art denoising methods.

We propose a novel rendering method for computer-generated holograms (CGHs) which enables low memory usage and low computational complexity. Although the conventional elementary hologram (EH) method realizes smooth motion parallax and rendering for realistic expressions with low complexity, it incurs large memory usage. This is because point light sources (PLSs) must be acquired and stored independently for different perspectives to reconstruct multi-view images. Our method effectively reduces memory usage by integrating PLSs whose 3D coordinates are extremely close to each other. Furthermore, to reduce the complexity of this integration, our method skips some of the PLS acquisition processes under the assumption that the loss of PLSs for the EH does not significantly affect the final reconstructed image quality. Our experimental results show a reduction in memory usage of 93% with a calculation time only 1.2-fold longer than compared with the EH method.

Lifestyle changes and digitalization have reduced opportunities for face-to-face, intimate communication, which is an indispensable activity for human beings. We have realized a method that allows intimate communication between two persons even though they are faraway from each other by integrating multimodal technologies. Sync Sofa is a new sofa-type communication tool. It senses the partner with a camera, microphones, and accelerometers. The sensed data are cross-modally integrated and then presented to the user through a life-size display, multichannel loudspeakers, and multichannel vibrotactile actuators. We have received positive feedback from many users who have experienced Sync Sofa, such as, "I felt like the other person was really sitting next to me".

We are the first to apply split rendering to AR to improve the quality of the transparent channel. The proposed method evaluates a cloud-based AR streaming system that offloads photorealistic rendering to a cloud server and splits the rendering between the cloud server and the mobile device (split rendering). Server-side rendering is capable of rendering photorealistic images, but the quality of the image is degraded by coding when the video is compressed and transmitted. In particular, degradation of the transparent channel significantly reduces the subjective image quality. The proposed method avoids this degradation by rendering the transparent channel at the mobile terminal. In addition, the server improves the coding efficiency by padding the transparent areas. Nonlinear quantization of the difference images contributes to improved subjective image quality. Experimentally, we confirmed that the proposed method has a SSIM gain of 0.022 compared to the conventional method.

We present a novel pose detection method that can be used in mobile augmented reality (AR) services. Making 3D object pose detection robust against changes in viewpoint is a vitally important but quite difficult task because 3D objects often change their appearance significantly with changes in viewpoint, and the possible range of viewpoints is wide compared with planar targets. 3D-Ferns, which is a keypoint classifier for 3D object pose detection, performs direct 2D-3D matching and handles a wide range of detectable viewpoints, including all rotations. However, many difficult viewpoints still exist for pose detection because of the unevenness of matching performance over all viewpoints. In this paper, we propose a novel class selection strategy that evens out matching performance over all possible viewpoints and improves detection performance from difficult viewpoints by focusing on the per-viewpoint repeatability (PVR) of class 3D points. Experimental results demonstrate the impact of stability of 2D-3D matching on detection performance and the effect of our method, which reduces the detection failures in conventional approaches by over 23% for 3D targets that have various shapes and textures.

Optimal sharpness differs from image to image, de-pending on the content. In general, human observer prefers images of artifacts sharper and those of natural-objects less sharper. We have developed a content-adaptive automatic image sharpening algorithm that relies on the length of lines extracted from the image. It is applicable to images with various regions such as those contain natural and artificial objects. The proposed algorithm is expected to be used in image processing modules of image input/output devices, e.g. digital cameras, printers, etc.

Fractional sampling (FS) and Doppler diversity equalization in OFDM receivers can achieve two types of diversity (path diversity and Doppler diversity) simultaneously on time-varying multipath channels. However FS with a higher sampling rate requires the large amount of complexity in demodulation. In this paper, a novel sampling point selection (SPS) scheme with MMSE equalization in FS-OFDM receivers is proposed. On fast time-varying multipath channels, the proposed scheme selects the appropriate samples from the fractionally sampled signals. Through the computer simulation, it is demonstrated that with the proposed scheme, both path diversity gain and Doppler diversity gain can increase as compared to a conventional non-SPS scheme.

In this paper, a metric weighting scheme on a coded fractional sampling orthogonal frequency division multiplexing (FS OFDM) system is investigated. FS achieves path diversity with a single antenna through oversampling and subcarrier-based maximal ratio combining (MRC). Though the oversampling increases diversity order, correlation among noise components may deteriorate bit error rate (BER) performance. To clarify the relationship between the impulse response of the pulse shaping filter and the BER performance, five different pulse shaping filters are evaluated in the coded FS OFDM system. Numerical results obtained through computer simulation show that the metric weighting based on the Frobenius norm improves BER performance of the coded FS OFDM system.

A diversity scheme with fractional sampling (FS) in OFDM receivers is investigated recently. When a sharp filter is employed, the correlation of noise samples among the adjacent subcarriers increases, and the performance of this scheme is deteriorated. Therefore, low-complexity subcarrier based MMSE combining scheme with the subblock of the noise covariance matrix is proposed in this paper. Numerical results through computer simulation show that the MMSE combining scheme proposed in this paper can outperforms a conventional MRC scheme and a subcarrier based MMSE combining scheme, when the sharp filter is employed.