The rapid advancement of information technology has brought multimodal information retrieval into the research spotlight. Neural networks, particularly Transformers, have emerged as the dominant solution for extracting multimodal feature vectors. While neural network acceleration has been extensively explored, the subsequent retrieval stage in multimodal scenarios remains under-optimized. Conventional retrieval approaches, such as cosine similarity sorting on von Neumann architectures, suffer from significant data migration and computational inefficiencies. Hashing methods enhance storage and computation efficiency but encounter challenges in energy-efficient implementation and mitigating accuracy losses due to modal heterogeneity. This paper presents a hybrid architecture that integrates in-memory processing (PIM) and content-addressable memory (CAM) to address these challenges. Transformer-extracted features are processed via in-memory random hashing leveraging device-intrinsic properties, with CAM facilitating parallel search space reduction. A final cosine similarity reranking stage refines the results while balancing accuracy with energy efficiency. Experimental evaluations validate that the proposed method, when compared to the baseline traditional CPU-based cosine similarity retrieval, 1) achieves almost identical level of accuracy, dramatically outperforming other pure CAMbased Hamming distance retrieval approaches; and 2) reduces latency by $9.45 \times$ and energy consumption by $30.20 \times$.

Real-time simulation of hyperelastic membranes like cloth still faces a lot of challenges, such as hyperplasticity modeling and contact handling. In this study, we propose projective peridynamics that uses a local-global strategy to enable fast and robust simulation of hyperelastic membranes with contact. In the global step, we propose a semi-implicit strategy to linearize the governing equation for hyperelastic materials that are modeled with peridynamics. By decomposing the first Piola-Kirchhoff stress tensor into a positive and a negative part, successive substitutions can be taken to solve the nonlinear problems. Convergence is guaranteed by further addressing the overshooting problem. Since our global step solve requires no energy summation and dot product operations over the entire problem, it fits into GPU implementation perfectly. In the local step, we further present a GPU-friendly gradient descent method to prevent interpenetration by solving an optimization problem independently. Putting the global and local solves together, experiments show that our method is robust and efficient in simulating complex models of membranes involving hyperelastic materials and contact.

We propose a semi-Lagrangian method for multiphase interface tracking. In contrast to previous methods, our method maintains an explicit polygonal mesh, which is reconstructed from an unsigned distance function and an indicator function, to track the interface of arbitrary number of phases. The surface mesh is reconstructed at each step using an efficient multiphase polygonization procedure with precomputed stencils while the distance and indicator function are updated with an accurate semi-Lagrangian path tracing from the meshes of the last step. Furthermore, we provide an adaptive data structure, multiphase distance tree, to accelerate the updating of both the distance function and the indicator function. In addition, the adaptive structure also enables us to contour the distance tree accurately with simple bisection techniques. The major advantage of our method is that it can easily handle topological changes without ambiguities and preserve both the sharp features and the volume well. We will evaluate its efficiency, accuracy and robustness in the results part with several examples.

Multi-task learning has been proposed to improve the generalization performance by learning multiple tasks jointly. One challenge for this learning paradigm is to effectively seek the shared information across multiple tasks. In this paper, we propose a novel multi-task learning method to adaptively share information. Unlike many existing multi-task learning methods which impose strong assumptions on task related-ness, our method captures the relationships among tasks and identifies the disparities of each task simultaneously, thus can flexibly exploit the shared information. Moreover, we apply it to fine-grained categorization problem, which usually suffers from the difficulties of insufficient training data and high inter-class similarity. The experimental results on two widely used datasets show the superiority of our method compared with some state-of-the-art methods.

We introduce an adaptive hybrid level set surface tracking for fluid simulation in this paper. To improve the accuracy of the convectional level set method, we propose a method that combines the higher accuracy interpolation and octree grid to improve the accuracy while still maintain good efficiency. Adaptive gradient-augmented level set is adopted for higher accuracy interpolation and two strategies are proposed to guide the building of the octree structure. We demonstrate our method with several level set tests, and couple it to several existing fluid simulators. The results show that our method facilitates significant improvements in accuracy and efficiency, and is effective for free surface fluid simulation in capturing fluid details.

We present an efficient interactive system for dressing a naked bird with feathers. In our system, a skeleton associated with guide feathers is used to describe the distribution of the body feathers. The special skeleton can be easily built by the user, given a 3D bird model as input. To address the problem of interpenetrations among feathers, the growth priority between the feather roots is defined, with which we obtain the growth order from a greedily constructed directed acyclic graph. Each feather is then adjusted in that order by a height field based collision resolution process. The height field not only provides an efficient way to detect the collision but also enables us to finely control the degree of collision during feather adjustments. The results show that our approach is capable of resolving the collisions among thousands of feathers in a few seconds. If model animation is desired, the feathers can be adjusted on the fly at interactive framerates. Details of our implementation are provided with several examples to demonstrate the effectiveness of our system.

We introduce a novel framework for tracking multiphase interfaces with explicit contouring technique. In our framework, an unsigned distance function and an additional indicator function are used to represent the multiphase system. Our method maintains the explicit polygonal meshes that define the multiphase interfaces. At each step, distance function and indicator function are updated via semi-Lagrangian path tracing from the meshes of the last step. Interface surfaces are then reconstructed by polygonization procedures with precomputed stencils and further smoothed with a feature-preserving non-manifold smoothing algorithm to stay in good quality. Our method is easy to be implemented and incorporated into multiphase simulation, such as immiscible fluids, crystal grain growth and geometric flows. We demonstrate our method with several level set tests, including advection, propagation, etc., and couple it to some existing fluid simulators. The results show that our approach is stable, flexible, and effective for tracking multiphase interfaces.

Shadow mapping is a fast and easy to use method to produce hard shadows. However, it introduces aliasing due to its uniform sampling strategy and limited shadow map resolution. In this paper, we propose a memory efficient algorithm to render high quality shadows. Our algorithm is based on a multi-resolution shadow map structure, which includes a conventional shadow map for scene regions where a low-resolution shadow map is sufficient, and a high-resolution patch buffer to capture scene regions that are susceptible to aliasing. With this data structure, we are able to capture shadow details with far less memory footprint than conventional shadow mapping. In order to maintain an appropriate performance compared to conventional shadow mapping, we designed a customized CUDA rasterizer to render the high-resolution patches.

ABSTRACT This paper presents a novel approach for simulating the dynamic coupling between a tree and raindrops based on physical deformation and fluid simulation. By the approach, tree animation in the rain can be simulated in a two‐resolution way: branch motion and leaf motion. The branch is represented by the Euler–Bernoulli beam model, and the leaf petiole is represented by the three‐prism elastic model. Interaction coupling liquid motion on the hydrophilic surface with a flexible petiole is well implemented by a special design. To simplify the computation process, instead of the computation‐intensive three‐dimensional Navier–Stokes equations, shallow water equations are used to simulate the water dynamics together with the whole leaf deformation. Simulation has been also made to various phenomena incurred from the interactive coupling. These include, among others, part of impacting raindrops splashing into the air with the remaining flowing along the slant of the leaf and merging into larger ones or hanging on the blade boundary, with the leaf rebounding and vibrating after the drops fall off the leaf. A level‐of‐detail approach is exploited to accelerate rendering in views of different distances. The experimental results illustrate that the approach can be applied to efficiently generate realistic details of the interactive coupling between a tree and raindrops. Copyright © 2012 John Wiley & Sons, Ltd.

In the past decade, modern GPUs have provided increasing programmability with vertex, geometry and fragment shaders. However, many classical problems have not been efficiently solved using the current graphics pipeline where some stages are still fixed functions on chip. In particular, multi-fragment effects, especially order-independent transparency, require programmability of the blending stage, that makes it difficult to be solved in a single geometry pass. In this paper we present FreePipe, a system for programmable parallel rendering that can run entirely on current graphics hardware and has performance comparable with the traditional graphics pipeline. Within this framework, two schemes for the efficient rendering of multi-fragment effects in a single geometry pass have been developed by exploiting CUDA atomic operations. Both schemes have achieved significant speedups compared to the state-of-the-art methods that are based on traditional graphics pipelines.

Both exponential shadow maps and variance shadow maps are efficient shadow maps anti-aliasing algorithms based on pre-filtering, but exponential shadow maps will cause light bleeding, float overflow and penumbra discontinuity problems and variance shadow maps also have serious light bleeding problem. All these prevent their individual use in practice especially in large virtual environment such as games. This paper presents a hybrid method by storing three values in the depth texture: depth, squared depth and exponential depth. From these values we calculate two occlusion values using exponential and variance shadow maps respectively. A minimum of the two occlusion values is taken as the final result. This can efficiently eliminate the problems of the two methods and can be widely used in practice.

Modern GPUs provide gradually increasing programmability on vertex shader, geometry shader and fragment shader in the past decade. However, many classical problems such as order-independent transparency (OIT), occlusion culling have not yet been efficiently solved using the traditional graphics pipeline. The main reason is that the behavior of the current stage of the pipeline is hard to be determined due to the unpredictable future data. Since the rasterization and blending stage are still largely fixed functions on chip, previous improvements on these problems always require hardware modifications thus remain on the theoretical level. In this paper we propose CUDA Renderer, a fully programmable graphics pipeline using compute unified device architecture (CUDA) [NVIDIA 2008] which can completely run on current graphics hardware. Our experimental results have demonstrated significant speedup to traditional graphics pipeline especially on OIT. We believe many other problems can also benefit from this flexible architecture.

In this article, we present an efficient algorithm for encoding the connectivity information of triangular meshes. By the method, all triangles are traversed first to obtain operator series. Then an arithmetic coder based on variable code-mode is applied to encode the operator series. According to the operator last encoded, the property of triangular mesh and the method of mesh traversal, a code-mode is calculated for each operator being encoded currently, where the operator with higher prediction probability is given a shorter binary strand. Then we can obtain the binary strand according to its code-mode and encode every bit of this binary strand by adaptive arithmetic coding method. Testing result shows that the compression ratio of our algorithm is very high and even higher than the compression ratio by using TG algorithm, which is commonly regarded as one of the best in terms of compression ratio.

With in skeleton based deformation, two issues are waiting for investigation, large angle rotation and artifacts near joint regions. In this paper we propose rotation angle decomposition and detail preserving method to conquer them. A skeleton with multiple connected bones is prior computed for a triangle mesh. While deforming, large angle rotation is simulated by several progressive small angle rotations. We also introduce a distortion metric function E to quantify the difference between mesh edge vectors before and after deformation, preserving the detail of mesh surfaces. Nearby vertex position blending is enforced. We use sparse conjugate gradient method for the minimization of metric function E. Results show our method is suitable for 3D modeling and character animation applications, making the deformation near joint regions more natural.

In our surrounding environment, we may often see many various miscible liquid-liquid mixture phenomena, like pouring honey or ink into water, Coca Cola into strong wine etc., while few papers have devoted to the simulation of the phenomena. In this paper, we use a two-fluid lattice Boltzmann method (TFLBM) to simulate the underlying dynamics of miscible mixtures. By the method, a subgrid model is applied to improve its numerical stability so that the free surface of the mixture, accompanying with higher Reynolds number, can be simulated. We also apply control forces to the mixture with interesting animation created. By optimizing the memory structure and taking the advantage of dual-core or multi-core systems, we achieve real time computation for a domain in 643 cells full of fluid mixtures.

Through interaction with surroundings, the fluids may change their properties such as shapes, temperature vastly, and the same would happen to the surroundings simultaneously. On the other hand, different surroundings characterize different interactions, and may change the shapes and motions of the fluids in different ways. Therefore, it is of importance in physically-based simulation of fluids to build physically correct models to represent the varying interactions between fluids and the environments. In this paper, we make a simple summation on the interactions, and in particular focus on those most interesting to us, and model them with various physical solutions. In some of the methods, advantage is taken with the graphics processing unit (GPU) to achieve real-time computation for medial-scale simulation

Abstract Miscible fluid mixtures, like pouring honey into water, Coca Cola into strong wine, are common phenomena in our daily life. While two miscible fluids are mixed together, their appearances in terms of colors and shapes will change due to their mixing interaction. The interaction between the mixture components could be regarded as a combination of the diffusing process and demixing process. If the former dominates the interaction, it is miscible; otherwise, it is immiscible. The complex microscopic interplay between the mixture components makes the simulation highly challenging. So far, there have been some dedicated research in computer graphics dealing with immiscible mixtures, but few works have been done focusing on miscible mixtures. In this paper, for the first time, we introduce a two‐fluid lattice Boltzmann method (LBM), called TFLBM, applied to miscible binary mixtures. Different from other similar methods, the viscous and diffusing properties of the fluid in our work are considered separately, so that the physical insight is exposed more clearly and rationally. In addition, the operation of LBM is mostly a linear local computation, and graphics processing unit (GPU) has been utilized to achieve real‐time simulation. Copyright © 2006 John Wiley & Sons, Ltd.

This paper presents a novel approach to implementing dynamic LOD on GPU. For our purpose, a quadtree structure is created based on seamless geometry image atlas, which is a 3D surface representation in parameter space by combining the features of geometry images and poly-cube maps. All the nodes in the quadtree are packed into the atlas textures. There are two rendering passes in our approach. In the first pass, the LOD selection is performed in the fragment shaders. The resultant buffer is taken as the input texture to the second rendering pass by vertex texturing, and thus the node culling and triangulation can be performed in the vertex shaders. Our LOD algorithm can generate adaptive meshes dynamically, and can be fully implemented on GPU. It improves the efficiency of LOD selection, and alleviates the computing load on CPU.

Point and polygon representations have their respective merits in rendering objects. In this paper, we propose a novel hybrid multi-resolution approach, PQuadtrees, to efficiently render highly detailed objects. PQuadtrees are constructed from geometry images. Both point and polygon are tightly integrated into a uniform structure. While traversing the P-Quadtrees in rendering, the part of surface that face the viewer can be rendered by coarser quad mesh to reduce the numbers of rendering primitives. The shading details can be enhanced by hardware accelerated normal mapping. The view dependent LOD selects the finer hierarchy on silhouette, which is rendered by points. The rendering of large-scale model is greatly accelerated while the visual effect both at the surfaces and the silhouette is guaranteed.

In this paper, we solve the 3D fluid dynamics problem in a complex environment by taking advantage of the parallelism and programmability of GPU. In difference from other methods, innovation is made in two aspects. Firstly, more general boundary conditions could be processed on GPU in our method. By the method, we generate the boundary from a 3D scene with solid clipping, making the computation run on GPU despite of the complexity of the whole geometry scene. Then by grouping the voxels into different types according to their positions relative to the obstacles and locating the voxel that determines the value of the current voxel, we modify the values on the boundaries according to the boundary conditions. Secondly, more compact structure in data packing with flat 3D textures is designed at the fragment processing level to enhance parallelism and reduce execution passes. The scalar variables including density and temperature are packed into four channels of texels to accelerate the computation of 3D Navier-Stokes equations (NSEs). The test results prove the efficiency of our method, and as a result, it is feasible to run middle-scale problems of 3D fluid dynamics in an interactive speed for more general environment with complex geometry on PC platform.

Abstract Taking advantage of the parallelism and programmability of GPU, we solve the fluid dynamics problem completely on GPU. Different from previous methods, the whole computation is accelerated in our method by packing the scalar and vector variables into four channels of texels. In order to be adaptive to the arbitrary boundary conditions, we group the grid nodes into different types according to their positions relative to obstacles and search the node that determines the value of the current node. Then we compute the texture coordinates offsets according to the type of the boundary condition of each node to determine the corresponding variables and achieve the interaction of flows with obstacles set freely by users. The test results prove the efficiency of our method and exhibit the potential of GPU for general‐purpose computations. Copyright © 2004 John Wiley & Sons, Ltd.

View-dependent level-of-detail (LOD) and visibility culling are two powerful tools for accelerating the rendering of very large models in a real-time visualization, especially in walkthrough of a large-scale terrain environment. In this paper, we propose a visibility-driven Continuous LOD (CLOD) framework for terrain, which takes advantage of both techniques. The visibility determination is based on the well-known occlusion horizon algorithm. By making use of the features of the terrain extracted in pre-processing stage, a new cascading occlusion culling (COC) algorithm is proposed to cull those regions classified as invisible to current viewpoint in real time. The time consumption and storage overheads that we spend on visibility preprocessing are quite small. Visibility-driven CLOD enhances culling efficiency and improves the frame rates significantly for walkthrough of a terrain environment.

In this paper we propose a hybrid representation of environment models by a point representation augmented to geometric representation of surface polygons reconstructed from multiple reference images, through which a real time walkthrough of a complex environment can be achieved. By the method, we start from classification of pixels of the source images into two categories, corresponding respectively to the planar and non-planar surfaces in 3D space. For the pixels corresponding to the planar surfaces, the plane coefficients are reconstructed and all their appearances in the reference images are merged to form uniformly sampled texture images by a comparison of sampling rate and resampling. For the pixels corresponding to the non-planar surfaces, a point representation is applied and the redundant pixels are removed again through sampling rate comparison. The remained pixels are organized by OBB-tree according to their space coordinates. At the same time, the holes that are unable to be captured by all the reference images are pre-filled in the preprocessing phase so that the probability of hole appearance in walkthrough is greatly reduced. Under this hybrid representation, texture mapping and point warping are employed to render the novel views, to take full advantages of the acceleration utility of graphics hardware.

In this paper, a hybrid representation of environment models using a combination of points and polygons is proposed. Through the model, a real time walkthrough of a complex environment can be achieved. We start from multiple depth reference images and classify the pixels of images into two categories, corresponding respectively to planar and non-planar surfaces in 3D space, then all the redundant points are removed by a comparison algorithm of sampling rate. For pixels corresponding to nonplanar surfaces, the point representation is maintained, and a local reconstruction and resampling process is employed to re-sample a set of points that distribute more uniformly on the surfaces. The re-sampled points are organized by an OBB-tree according to their space coordinates and a multiresolution structure is built to improve the rendering efficiency. For pixels corresponding to planar surfaces, the plane coefficients are reconstructed and all their appearances in the reference images are merged to texture maps. Under this hybrid representation, texture mapping and point-based rendering are employed to render the novel views, to take full advantage of the acceleration utility of graphics hardware. The algorithm demonstrates the combined advantages of approaches in traditional computer graphics and PBR/IBR, texture mapping for plane surfaces and point-based rendering for high detail surfaces and shapes.

Due to visibility change and surface enlargement in producing a novel view from a new viewpoint, 3D re-projection from one reference image in IBMR inevitably produces holes in the destination image. Even worse, exposure errors occur when a background region occluded is visible in a desired image because of the absence of some background elements in the reference image. The general solution to this kind of problems is to use multiple images from different viewpoints as input source. By doing so however, the rendering cost would increase with the number of reference images and the composition algorithm has to rely on the z-buffer processing.In fact, plenty of redundant information exists among different reference images. Seeking for a nice way to extract the information needed in the novel view from the reference images is the key issue in solving the problem. In this paper, we propose a new method of image synthesis from multiple reference images. The method combines forward warping and backward warping to fulfil the image composition task for a novel viewpoint. The primary inspiration behind the development of our image synthesis method comes from a fact that the polygon edge geometry may indicate where an exposure and, possible an exposure error occur in the destination image if object silhouettes are prior known. The feature that intersection between scanline and polygons must be in pairs is employed to distinguish holes caused by enlargement of surfaces from those by visibility change. Different heuristic methods are used to choose one image as a primary reference image which shares the most resemblance with the destination image, and other reference images for filling different kinds of holes. Depth continuity along scanline and the depth information already present in the destination image are employed to accelerate the searching process of the corresponding pixels for filling holes.