Interaction with 3D objects in Augmented Reality (AR) on tablets is increasingly common, yet most existing systems rely on single-handed input, causing occlusion, fatigue, and device instability. Using both hands to hold the tablet alleviates these issues. This article introduces a dual-thumb interaction framework that enhances both selection and manipulation of 3D AR content. We first propose the 3D Parabolic Ray Pointing technique, where thumb movements along the X, Y, and Z axes control the landing point for distant pointing. Comparative experiments show that this method reduces fatigue and is preferred over existing techniques. We then develop two dual-thumb techniques for 3D object selection and manipulation: (1) Planar WiM Selection with Steering Wheel Manipulation and (2) WiM-Based Selection and Manipulation. Evaluations across various tasks reveal complementary advantages. Finally, we implement these techniques in an AR sandbox application, demonstrating the framework's potential for broader adoption.

ABSTRACT Recent advances in Score Distillation Sampling (SDS) have enabled text‐driven 3D human generation, yet the standard classifier‐free guidance (CFG) framework struggles with semantic misalignment and texture oversaturation due to limited model capacity. We propose a novel framework that decouples conditional and unconditional guidance via a dual‐model strategy: A pretrained diffusion model ensures geometric stability, while a preference‐tuned latent reward model enhances semantic fidelity. To further refine noise estimation, we introduce a lightweight U‐shaped Swin Transformer (U‐Swin) that regularizes predicted noise against the reward model, reducing gradient bias and local artifacts. Additionally, we design a time‐varying noise weighting mechanism to dynamically balance the two guidance signals during denoising, improving stability and texture realism. Extensive experiments show that our method significantly improves alignment with textual descriptions, enhances texture details, and outperforms state‐of‐the‐art baselines in both visual quality and semantic consistency.

Reconstructing accurate urban building models remains challenging due to large-scale variability and complex topology. We introduce a practical multi-step framework that reconstructs a single building from a dense triangle mesh by explicitly exploiting its block-wise composition. The pipeline first partitions the mesh into spatially coherent regions, then extracts height-aware contours and classifies geometry into facades, roofs, and appurtenances. For each part, we perform contour-guided vectorized modeling with profile fitting and consistency constraints, producing watertight, semantically structured models at Level-of-Detail (LOD) 2.3. An interactive refinement module further supports user adjustments to resolve rare failure cases and enforce design regularity. The proposed decomposition and layered fitting yield compact outputs while preserving salient geometry, and make the reconstruction robust across scales and moderate noise. We also analyze modeling assumptions and implementation details to ensure reproducibility. Overall, this work offers a component-aware, engineering-ready solution for converting unstructured meshes into structured building models that are amenable to downstream urban modeling and visualization. buildingreconstruction; block-wisesegmentation; contour-guidedvectorization; LOD;interactiverefinement; watertight modeling

Hands are a primary interface in daily interactions, making high-quality, controllable hand modeling and realistic real-time rendering crucial. Thus, we propose JGHand (Joint-driven 3D Gaussian Hand), a novel 3D Gaussian Splatting (3DGS)-based hand representation that renders high-fidelity hand images in real time across diverse poses and characters. Unlike existing articulated neural rendering techniques, we introduce a differentiable spatial transformation process based on 3D key points, enabling flexible deformations for varying bone lengths and poses. Additionally, we propose a real-time shadow simulation method based on per-pixel depth to simulate self-occlusion shadows from finger movements. Finally, we incorporate hand priors to develop an animatable 3DGS hand representation driven solely by 3D key points. We validate the effectiveness of each component through comprehensive ablation studies. Experimental results on public datasets demonstrate that JGHand achieves real-time rendering speeds with enhanced quality, surpassing state-of-the-art methods.

Abstract The utilization of neural implicit surface as a geometry representation has proven to be an effective multi‐view surface reconstruction method. Despite the promising results achieved, reconstructing geometry from objects in real‐world scenes remains challenging due to the interaction between surface materials and complex ambient light, as well as shadow effects caused by self‐occlusion, making it a highly ill‐posed problem. To address this challenge, we propose MF‐SDF, a method that use a hybrid neural network and spherical gaussian representation to model environmental lighting, so that the model can express the situation of multiple light sources including directional light (such as outdoor sunlight) in real‐world scenarios. Benefit from this, our method effectively reconstructs coherent surfaces and accurately locates the shadow location on the surface. Furthermore, we adopt a shadow aware multi‐view photometric consistency loss, which mitigates the erroneous reconstruction results of previous methods on surfaces containing shadows, thereby improve the overall smoothness of the surface. Additionally, unlike previous approaches that directly optimize spatial features, we propose a Fourier feature optimization method that directly optimizes the tensorial feature in the frequency domain. By optimizing the high‐frequency components, this approach further enhances the details of surface reconstruction. Finally, through experiments, we demonstrate that our method outperforms existing methods in terms of reconstruction accuracy on real captured data.

Densely captured real-world materials require effective compression for rendering, material generation and reconstruction. Neural networks with high compression rates and the ability to fit complex functions can encode each BRDF into the corresponding network. However, current works that take advantage of single implicit neural representations are incapable of effectively modeling the high-frequency details of the highlight region. In this paper, we propose an improved compact neural network representation of BRDF data based on the sinusoidal activation. The lightweight network and the periodic activation function improve the fidelity of the reproduction material appearance under the condition of a high compression rate. Furthermore, the method of building a unified model using neural networks can decode all materials from latent space. However, the deep structure of the network model increases memory consumption. To overcome this challenge, we propose a hypernetwork framework that compresses measured BRDFs to latent space and generates weights for the neural network-based representation of materials. The lightweight implicit representation of BRDF generated by training directly from original materials shows the characteristics of a low memory footprint and high-precision reproduction of appearance. Additionally, we apply the hypernetwork to reconstruct materials from a single image. Thanks to implicit representation of BRDF that can reproduce the appearance with high fidelity, the reflectance properties can be accurately recovered.

User interface (UI) in head-worn mixed reality (MR) can be divided into two categories: user-centered and object-centered. Many studies have focused on user-centered UI, but few have explored object-centered UI. In this research, we explored the design of object-centered UI in head-worn MR environment. First, (b) we proposed three design considerations for object-centered UI in head-worn MR environment, and then verified them through two user studies with four UIs, where one is pure object-centered UI, one is pure user-centered UI, the others are hybrid UIs. In the first study, we conducted a user experience (UX) research on the access comfort, convenience and unobstructed sight degree of each UI, and demonstrated users’ preferences for the object-centered UI. In the second study, we designed a real application scenario to further evaluate the design considerations in a complex MR environment, and assessed its usability. We found positive user experience qualities of object-centered UI. Finally, we summarized four design recommendations, which could guide the UI design for the interaction with real-world objects in the future of everyday life with head-worn MR.

This paper presents a learning-based method for detailed 3D face reconstruction from a single unconstrained image. The core of our method is an end-to-end multi-task network architecture. The purpose of the proposed network is to predict a geometric representation of 3D face from a given facial image. Unlike most existing reconstruction methods using low-dimension morphable models, we propose a pixel-based multi-scale representation of a detailed 3D face to ensure that our reconstruction results are not limited by the expressiveness of linear models. We break the task of high-fidelity face reconstruction into three subtasks, which are face region segmentation, coarse-scale reconstruction and detail recovery. So the end-to-end network is constructed as a multitask mode, which contains three subtask networks to deal with different subtasks respectively. A backbone network with feature pyramid structure is proposed as well to provide different levels of feature maps required by the three subtask networks. We train our end-to-end network in the spirit of the recent photo-realistic data generation approach. The experimental results demonstrate that our method can work with totally unconstrained images and produce high-quality reconstruction but with less runtime compared to the state-of-the-art.

Estimating the pose of two hands is a crucial problem for many human-computer interaction applications. Since most of the existing works utilize cropped images to predict the hand pose, they require a hand detection stage before pose estimation or input cropped images directly. In this paper, we propose the first real-time one-stage method for pose estimation from a single RGB image without hand tracking. Combining the self-attention mechanism with convolutional layers, the network we proposed is able to predict the 2.5D hand joints coordinate while locating the two hands regions. And to reduce the extra memory and computational consumption caused by self-attention, we proposed a linear attention structure with a spatial reduction attention block called SRAN block. We demonstrate the effectiveness of each component in our network through the ablation study. And experiments on public datasets showed the competitive result with the state-of-the-art method.

Abstract We specifically present a general method for monocular fluid videos to reconstruct and edit 3D fluid volume. Although researchers have developed many monocular video‐based methods, the reconstructed results are merely one layer of geometry surface, lack of accurate physical attributes of fluids, and challenging to edit fluid. We obtain a high‐quality 3D fluid volume by extending the smoothed particle hydrodynamics (SPH) model with external force guidance. For reconstructing fluid, we design target particles that are recovered from the shape from shading (SFS) method and initialize fluid particles that are spatially consistent with target particles. For editing fluid, we translate the deformation of target particles into the 3D fluid volume by merging user‐specified features of interest. Separating the low‐ and high‐frequency height field allows us to efficiently solve the motion equations for a liquid while retaining enough details to obtain realistic‐looking behaviours. Our experimental results compare favourably to the state‐of‐the‐art in terms of global fluid volume motion features and fluid surface details and demonstrate our model can achieve desirable and pleasing effects.

Abstract This paper offers a fast multi‐graphics processing unit (GPU) parallel simulation framework to the problem of real‐time and nonlinear finite element computation of brain deformation. A load balancing strategy is proposed to ensure the efficient distribution of nonlinear finite element computation on multi‐GPU. A data storage structure is designed to minimize the amount of data transfer and make full use of the overlay technique of GPU to reduce the transferring latency between multi‐GPUs. We further present a fast central processing unit (CPU)–GPU parallel continuous collision detection and response method, which not only can deal with the collision between the brain and skull but also can handle the self‐collision of the brain. Our method can make full use of CPU and GPU to implement a parallel computation about deformation and collision detection. Our experimental results show that our method is able to handle a brain geometric model with high detail gyrus composed of more than 40,000 tetrahedron elements. This can facilitate the fidelity of the current virtual brain surgery simulator. We evaluate our approach qualitatively and quantitatively and compare it with related works.

We present a deep learning-based method for removing makeup effects (de-makeup) in a face image. This problem poses a major challenge due to obscuring of the underlying facial features by cosmetics, which is very important in multimedia applications in the field of security, entertainment, and social networking. To address this task, we propose the bidirectional tunable de-makeup network (BTD-Net), which jointly learns the makeup process to aid in learning the de-makeup process. For tractable learning of the makeup process, which is a one-to-many mapping determined by the cosmetics that are applied, we introduce a latent variable that reflects the makeup style. This latent variable is extracted in the de-makeup process and used as a condition on the makeup process to constrain the one-to-many mapping to a specific solution. Through extensive experiments, our proposed BTD-Net is found to surpass the state-of-art techniques in estimating realistic non-makeup faces that correspond to the input makeup images. We additionally show that applications such as tuning the amount of makeup can be enhanced through the use of this method.

This paper presents a method for reconstructing 3D face expressions from monocular video sequences. Unlike previous approaches we don't require any prior face models, nor a large collection of images with diverse variation of poses and illuminations. Instead, we leverage a monocular video sequence without any restrictions. We formulate the 3D face reconstruction as an energy minimization problem integrated with dense optical flow variation, as rigid as possible(ARAP) constraint, spatial and temporal constraints. This paper offers the first dense optical flow variational approach to the problem of 3D reconstruction of non-rigid face expressions from a monocular video. Dense optical flow variation cost substitutes for photo consistency cost to enhance the reconstruction of exaggerated expressions. A generic 3D face template mesh and a simple 3D warping algorithm allow us to reconstruct a true 3D face mesh, relax the constraints of diverse views or illuminations and also avoid the dependency of the quality of prior face models, such as the facial expressions or face races varieties limitation. Finally, we use a per-pixel shape-from-shading(SFS) algorithm to estimate the fine-scale geometry details such as wrinkles to further improve the reconstruction fidelity. Given unconstrained monocular RGB videos, our method reconstructs wrinkle-level 3D face model, without the need for any prior models or diverse capture conditions.

The main problems of complex-scene understanding and semantic scene segmentation are caused by mismatched relationships, confusion categories, and inconspicuous classes. Towards above issues, we propose a global refinement atrous convolutional neural network (GRANet) for semantic scene segmentation. To enlarge the receptive field of filters, we use atrous convolution instead of the downsampling operators. To handle the challenge caused by the existence of objects at multiple scales in a scene, we adopt multiple rates atrous convolution structure. And to overcome the problem that the current semantic segmentation architecture can not make good use of global information, we propose a multiple pooling module schemes to utilize the global context information to boost the performance of our GRANet. The proposed GRANet achieves state-of-the-art performance on the SiftFlow Dataset and attains comparable performance with other state-of-the-art works on Cityscapes dataset.

Displays that can portray environments that are perceivable from multiple views are known as multiscopic displays. Some multiscopic displays enable realistic perception of 3D environments without the need for cumbersome mounts or fragile head-tracking algorithms. These automultiscopic displays carefully control the distribution of emitted light over space, direction (angle) and time so that even a static image displayed can encode parallax across viewing directions (Iightfield). This allows simultaneous observation by multiple viewers, each perceiving 3D from their own (correct) perspective. Currently, the illusion can only be effectively maintained over a narrow range of viewing angles. In this paper, we propose and analyze a simple solution to widen the range of viewing angles for automultiscopic displays that use parallax barriers. We propose the use of a refractive medium, with a high refractive index, between the display and parallax barriers. The inserted medium warps the exitant lightfield in a way that increases the potential viewing angle. We analyze the consequences of this warp and build a prototype with a 93% increase in the effective viewing angle.

Nowadays, in order to achieve an immersive experience, virtual reality systems usually require vivid 3D models and a good understanding of particular scenes. The limitations of separately optimizing image segmentation and 3D modeling from images have gradually been seen by more and more researchers, so plenty of novel methods on how to combine them for a better result begin to be put forward widely. In this paper, we propose a new hybrid framework to generate semantic 3D dense models from monocular images. Based on the available hierarchical CRFs model, we make full use of the correlation between voxels and their corresponding pixels from different images. Naturally, valuable information from 3D space can be added as one of the important energy items in the model. Either pixels, segments or voxles are all regarded as a node in the huge graph we build. Our ultimate goal is to realize a joint optimization for both 3D dense reconstruction and image segmentation. Experiments have been done on four real challenging datasets and all of the results prove the efficiency of our proposed hybrid framework.

This paper proposes a novel algorithm to automatically detect the repetitive elements with accurate shapes, locations and sizes from single façade image. Unlike other algorithms, our algorithm is not entirely dependent on the extracted feature points, edges and symmetric information. Our algorithm mainly includes following steps: First, we combine the clustering method with the repetitive characteristic curve to derive templates and to detect repetitive elements matched with derived templates. Moreover, a global repetition-based optimization framework is proposed to derive occluded repetitive elements and determine the number of all the repetitive elements with the accurate locations, shapes and sizes. Experiment results demonstrate that the proposed algorithm improves the accuracy, robustness and efficiency on façade databases compared with the state-of-the-art methods.

This paper presents a unified variational formulation for joint object segmentation and stereo matching, which takes both accuracy and efficiency into account. In our approach, depth-map consists of compact objects, each object is represented through three different aspects: 1) the perimeter in image space; 2) the slanted object depth plane; and 3) the planar bias, which is to add an additional level of detail on top of each object plane in order to model depth variations within an object. Compared with traditional high quality solving methods in low level, we use a convex formulation of the multilabel Potts Model with PatchMatch stereo techniques to generate depth-map at each image in object level and show that accurate multiple view reconstruction can be achieved with our formulation by means of induced homography without discretization or staircasing artifacts. Our model is formulated as an energy minimization that is optimized via a fast primal-dual algorithm, which can handle several hundred object depth segments efficiently. Performance evaluations in the Middlebury benchmark data sets show that our method outperforms the traditional integer-valued disparity strategy as well as the original PatchMatch algorithm and its variants in subpixel accurate disparity estimation. The proposed algorithm is also evaluated and shown to produce consistently good results for various real-world data sets (KITTI benchmark data sets and multiview benchmark data sets).

In this paper, we propose the Interval-valued Matrix Factorization (IMF) framework. Matrix Factorization (MF) is a fundamental building block of data mining. MF techniques, such as Nonnegative Matrix Factorization (NMF) and Probabilistic Matrix Factorization (PMF), are widely used in applications of data mining. For example, NMF has shown its advantage in Face Analysis (FA) while PMF has been successfully applied to Collaborative Filtering (CF). In this paper, we analyze the data approximation in FA as well as CF applications and construct interval-valued matrices to capture these approximation phenomenons. We adapt basic NMF and PMF models to the interval-valued matrices and propose Interval-valued NMF (I-NMF) as well as Interval-valued PMF (I-PMF). We conduct extensive experiments to show that proposed I-NMF and I-PMF significantly outperform their single-valued counterparts in FA and CF applications.

In this paper we propose a framework of topic modeling ensembles, a novel solution to combine the models learned by topic modeling over each partition of the whole corpus. It has the potentials for applications such as distributed topic modeling for large corpora, and incremental topic modeling for rapidly growing corpora. Since only the base models, not the original documents, are required in the ensemble, all these applications can be performed in a privacy preserving manner. We explore the theoretical foundation of the proposed framework, give its geometric interpretation, and implement it for both PLSA and LDA. The evaluation of the implementations over the synthetic and real-life data sets shows that the proposed framework is much more efficient than modeling the original corpus directly while achieves comparable effectiveness in terms of perplexity and classification accuracy.

Component-based software reuse and development is considered to be the best way to improve the efficiency and quality of development. To meet the demands of aircraft instrument exhibition in an aviation virtual library, we present a modeling and simulation tool for cockpit instruments on component methods, achieving display functionality and providing a visual interaction platform in the aviation virtual library. For the rapid model construction and reuse of virtual library resources, we classify entities in the cockpit into several categories by their behavior, and design a template for each category to describe its attributes. This efficiently reduces the modeling cycle, and thus largely helps users to take advantage of the library resource.

This paper presents a rapid and robust method to align large sets of range scans captured by a 3D scanner automatically. The method incorporates the color information from the range data into the pairwise registration. Firstly, it detects the features using SIFT (Scale-Invariant Feature Transform) on grayscale images generated from two range scans to align. Then a quasi-dense matching algorithm, based on the match propagation principle, is applied to specify the matching pixel pairs between two images. All matches obtained are mapped to 3D space but in different world coordinates, and fitered by the 3D geometry constraint discovered from the range data. The remaining set of point correspondences is used to estimate the rigid transformation. Finally, a modified ICP (Iterative Closest Point) algorithm is applied to refine the result. The paper also describes a framework to use this alignment method for object reconstruction. The reconstruction proceeds by acquiring several range scans with color information from different directions, following which pair-wise of range data are aligned with the above method selectively and iteratively. Then a model graph containing the correct pair-wise matches is created and a span tree specifying a complete model is constructed. Finally a global optimization is performed to refine the result. This reconstruction technique achieves a robust and high performance in the application of rebuilding the 3D models of culture heritages for virtual museum automatically.

Image inpainting technique has been widely used for reconstructing damaged old photographs and removing unwanted objects from images. In this paper, we present an image inpainting method based on the existing exemplar-based image inpainting idea. Our method improves the robustness and effectiveness by rational confidence computing method, matching strategy and filling scheme. Therefore, our method effectively prevents "growing garbage", which is a common problem in other methods. With our method, we can obtain preferable results to those obtained by other similar methods.