Invariance against rotations of 3D objects is an important property in analyzing 3D point set data. Conventional 3D point set DNNs having rotation invariance typically obtain accurate 3D shape features via supervised learning by using labeled 3D point sets as training samples. However, due to the rapid increase in 3D point set data and the high cost of labeling, a framework to learn rotation-invariant 3D shape features from numerous unlabeled 3D point sets is required. This paper proposes a novel self-supervised learning framework for acquiring accurate and rotation-invariant 3D point set features at object-level. Our proposed lightweight DNN architecture decomposes an input 3D point set into multiple global-scale regions, called tokens, that preserve the spatial layout of partial shapes composing the 3D object. We employ a self-attention mechanism to refine the tokens and aggregate them into an expressive rotation-invariant feature per 3D point set. Our DNN is effectively trained by using pseudo-labels generated by a self-distillation framework. To facilitate the learning of accurate features, we propose to combine multi-crop and cut-mix data augmentation techniques to diversify 3D point sets for training. Through a comprehensive evaluation, we empirically demonstrate that, (1) existing rotation-invariant DNN architectures designed for supervised learning do not necessarily learn accurate 3D shape features under a self-supervised learning scenario, and (2) our proposed algorithm learns rotation-invariant 3D point set features that are more accurate than those learned by existing algorithms.

Abstract 3D point set reconstruction is an important and challenging 3D shape analysis task. Current state-of-the-art algorithms for 3D point set reconstruction employ a deep neural network (DNN) having an encoder–decoder architecture. Recently, the decoder DNNs that transform multiple 2D planar patches to reconstruct a 3D shape have seen some success. These “patch-folding” decoders are adept at approximating smooth surfaces in 3D objects. However, 3D point sets generated by these decoders often lack local geometrical details, as 2D planar patches tend to overly constrain the patch folding process. In this paper, we propose a novel decoder DNN for 3D point sets called Hyperplane Mixing and Folding Net (HMF-Net). HMF-Net uses less constrained hyperplane, not 2D plane, patches as its input to the folding process. HMF-Net has, as its core building block, a stack of token-mixing layers to effectively learn global consistency among the hyperplane patches. In addition to HMF-Net, we also propose a novel loss for 3D point set reconstruction called Weighted Chamfer Distance (WCD). WCD tries to weight, or amplify, loss from parts of shape that are highly variable across training samples by emphasizing higher point-pair distance values between a generated point set and a groundtruth point set. This helps the decoder DNN learn shape details better. We comprehensively evaluate our algorithm under three 3D point set reconstruction scenarios, that are, shape completion, shape upsampling, and shape reconstruction from 2D images. Experimental results demonstrate that our algorithm yields accuracies higher than the existing algorithms for 3D point set reconstruction.

Hand-drawn sketch is a powerful modality to query 3D shape models. However, specifying a detailed 3D shape by a sketch on the first try without reference (i.e., 3D model or real object) is difficult. In this paper, we aim at a sketch-based 3D shape retrieval system that tolerates coarsely drawn or incomplete sketches having small number of strokes. Such a system could be used to start a sketch-retrieve-refine interactive loop that could lead to a 3D shape having required shape details. Proposed algorithm uses deep feature embedding into common feature embedding space to compare sketches and 3D shape models. To handle coarse or incomplete sketches, a sketch, which is a sequence of strokes, is augmented by removing stroke for training a pair of DNNs to extract sketch features. A sketch feature is a fusion of an image based feature extracted by a convolutional neural network (CNN) and a 2D point sequence feature extracted by using a recurrent neural network (RNN). Embedding of 3D shape feature and the sketch feature is learned by using triplet loss. Experimental evaluation of the proposed method is performed using (simulated) incomplete sketches created by removing part of their strokes. The experiments show that sketch stroke removal augmentation significantly improved retrieval accuracy if queried by using such incomplete sketches.

As the scale of 3D model databases increase, speed, in addition to accuracy, of its search becomes very important. One way to achieve fast search is to use a compact 3D shape feature whose cost of comparison is very small. Binarization of a real-valued feature vector, via hashing, is a way to obtain such compact feature vector. Previous algorithms for producing binarized 3D model features via hashing consisted of two disconnected stages; handcrafted real-valued 3D shape feature extraction followed by hashing into binary code. This compartmentalized approach, however, leads to less-than optimal binary codes, as these two stages are optimized independently. This paper proposes a deep semantic hashing algorithm called Binarized Deep Local feature Aggregation Network (BDLAN) which jointly optimizes real-valued feature extraction per 3D model and its banarization via hashing. BDLAN training minimizes quantization error caused by binarization. However, this constraint alone often maps real-valued features to their nearest binary codes in Hamming space, which are nonoptimal local minima. To alleviate the issue, we add a simple regularization called Probabilistic Bit Inversion (PBI) of binary codes. Experimental evaluation of the proposed algorithms demonstrates superior efficiency and competitive accuracy to the existing 3D model retrieval algorithms employing real-valued features.

Aggregating a set of local features has been used widely to realize recognition or retrieval of multimedia data including 2D images and 3D models. A number of feature aggregation algorithms (e.g., Bag-of-Features, Locality-constrained Linear coding, or Fisher Vector coding) have been proposed. They first learn a codebook, or a set of codewords, by clustering the local features and then encode these local features by using the learned codebook. Despite the great success of these feature aggregation algorithms, we argue that they are not necessarily optimal in terms of accuracy since their codebook learning and feature encoding are computed separately. In this paper, we propose two novel feature aggregation algorithms based on k-Sparse Autoencoder (kSA) that realize more accurate local feature aggregation. Our proposed algorithms, called Database-adaptive kSA (DkSA) aggregation and Per-data-adaptive kSA (PkSA) aggregation, jointly optimize codebook learning and feature encoding. In addition, the kSA-based feature encoding enhances saliency of local features due to k-sparseness constraints and non-negativity constraints. Of the two proposed algorithms, the PkSA aggregation exploits reconstruction error of a local feature derived from the kSA for more accurate aggregated feature. Experimental evaluation using a shape-based 3D model retrieval scenario showed that the retrieval accuracy of our proposed algorithms are superior to the existing feature aggregation algorithms we have compared against.

Aggregating a set of local features has become one of the most common approaches for representing a multi-media data such as 2D image and 3D model. The success of Bag-of-Features (BF) aggregation [2] prompted several extensions to BF, that are, VLAD [12], Fisher Vector (FV) coding [22] and Super Vector (SV) coding [34]. They all learn small number of codewords, or representative local features, by clustering a set of large number of local features. The set of local features extracted from a media data (e.g., an image) is encoded by considering distribution of features around the codewords; BF uses frequency, VLAD and FV uses displacement vector, and SV uses a combination of both. In doing so, these encoding algorithms assume linearity of feature space about a codeword. Consequently, even if the set of features form a non-linear manifold, its non-linearity would be ignored, potentially degrading quality of aggregated features. In this paper, we propose a novel feature aggregation algorithm called Diffusion-on-Manifold (DM) that tries to take into account, via diffusion distance, structure of non-linear manifold formed by the set of local features. In view of 3D shape retrieval, we also propose a local 3D shape feature defined for oriented point set. Experiments using shape-based 3D model retrieval scenario show that the DM aggregation results in better retrieval accuracy than the existing aggregation algorithms we've compared against, that are, VLAD, FV, and SV, etc..

This paper proposes a novel sketch-based 3D model retrieval algorithm that is scalable as well as accurate. Accuracy is achieved by a combination of (1) a set of state-of-the-art visual features for comparing sketches and 3D models, and (2) an efficient algorithm to learn data-driven similarity across heterogeneous domains of sketches and 3D models. For the latter, we adopted the algorithm [18] by Furuya et al., which fuses, for more accurate similarity computation, three kinds of similarities, i.e., Those among sketches, those among 3D models, and those between sketches and 3D models. While the algorithm by Furuya et al. [18] does improve accuracy, it does not scale. We accelerate, without loss of accuracy, retrieval result ranking stage of [18] by embedding its cross-modal similarity graph into Hamming space. The embedding is performed by a combination of spectral embedding and hashing into compact binary codes. Experiments show that our proposed algorithm is more accurate and much faster than previous sketch-based 3D model retrieval algorithms.

Sketch-based 3D model retrieval algorithms compare a query, a line drawing sketch, and 3D models for similarity by rendering the 3D models into line drawing-like images. Still, retrieval accuracies of previous algorithms remained low, as sets of features, one of sketches and the other of rendered images of 3D models, are quite different, they are said to lie in different domains. A previous approach used semantic labels to establish correspondence between features across inter-domain gap. This approach, however, is prone to over learning if dataset is difficult to learn, i.e., if labeling is sparse and/or if only a small subset of each class is labeled. This paper proposes Cross-Domain Manifold Ranking (CDMR), an algorithm that effectively compares two sets of features that lie in different domains. The proposed algorithm first establishes feature subspace, or manifold, separately in each of the domains. Then, the two feature manifolds are interrelated to form a unified Cross-Domain Manifold (CDM) by using both feature similarity and semantic label correspondence across the domains. Given a query sketch, similarity ranks of 3D models are computed by diffusing relevance value from the sketch over the CDM. Experimental evaluation by using sketch-based 3D model retrieval benchmarks showed that the CDMR is more accurate than state-of-the-art sketch-based 3D model retrieval algorithms.

Retrieval of 3D models by using 2D sketch query has recently become an active area of research. However, retrieval accuracy of the modality is still much lower than using a 3D model example as query. In this paper, we propose an algorithm that employs manifold-learning based dimension reduction for sketch-based 3D model retrieval. The algorithm compares multi-view rendering of 3D models with the 2D sketch. To improve distance computation, a distance metric adapted to data sample distribution is learned by using a manifold-learning algorithm, that is, Locally Linear Embedding (LLE). In order to lower the cost of training the LLE, we reduce number of training samples by clustering, either in feature space or in view space. Experimental evaluation has shown that both view space clustering and feature space clustering lowers training cost by more than 10 times while significantly improving retrieval accuracy. A compact 50 dimensional feature after the dimension reduction is much faster to compare, and its retrieval accuracy is 40% better than the original 30k dimensional feature. In terms of training cost, view clustering approach costs 20 times less than the one using full set of features without clustering.

Retrieval accuracy in content-based multimedia retrieval can be improved by using distance metric learned from distribution of features in input feature space. One way to achieve this is by dimension reduction via manifold-learning, such as Locally Linear Embedding [8]. While effective in improving retrieval accuracy, these algorithms have high computational cost that depends on feature dimensionality d and number of training samples N. In this paper, we explore a clustering-based approach to reduce number of training samples; it uses L cluster centers (L≪N) computed from N input features as training samples. We propose to use extremely randomized clustering tree [3] for clustering. Experiments showed that the proposed approach produces better retrieval performance than random sampling, and that the randomized tree is much faster than the k-means algorithm.

This paper proposes an improvement to Manifold Ranking algorithm used for search results ranking in the context of shape-based 3D model retrieval. Manifold Ranking algorithm by Zhou et al estimates, given a set of high-dimensional feature vectors, a lower-dimensional manifold on which the features lie. It then computes diffusion-based distances from a feature vector (or feature vectors) to the other feature vectors on the manifold. When applied to content-based retrieval, overall retrieval accuracy is significantly better than a "simple" fixed distance metric. However, in a small neighborhood of query, retrieval ranks obtained by a "simple" distance metric (e.g., L1-norm) performs better than those obtained by Manifold Ranking. Proposed re-ranking algorithm tries to combine ranking results due to both simple distance metric and Manifold Ranking in an automatic query expansion framework for better ranking results. Experimental evaluation has shown that the proposed method is effective in improving retrieval accuracy.

The first of the SHREC series of 3D model retrieval contests, SHREC 2006 [5] organized by Prof. Veltkamp et al. has made an impact in the way researchers compare performances of their 3D model retrieval methods. The task was to retrieve polygon soup models found in the Princeton Shape Benchmark database [5] having diverse shape and semantics. While many researchers used the SHREC 2006 as their benchmark, there has been no “official” contest since 2006 that used the same SHREC 2006 format but with up-to-date algorithms and methods. The SHREC 2007 added new tracks, e.g., for 3D face models, watertight models, protein models, CAD models, partial matching, and relevance feedback. However, the format of SHREC 2006 was missing. This SHREC 2008 Generic Models Track (GMT) tries to repeat the SHREC 2006 so that we can compare state-of-the-art methods for polygon soup models by using a stable benchmark dataset and ground truth classifications. A change from the SHREC 2006 to the SHREC 2008 GMT is the acknowledgement of learning based algorithms for 3D model retrieval. The SHREC 2008 GMT has two entry categories depending on if supervised learning is used or not. We wanted to encourage various forms of learning algorithms, as we believe learning algorithms are as essential as features themselves for effective 3D model retrieval. At the same time, we do not want to discourage methods without supervised learning. So we created two sub-tracks, one for unsupervised methods and the other for supervised methods. To test the behavior of supervised method for the queries having “unseen” ground truth classifications, we added a new set of queries, in addition to the original set of queries used in the SHREC 2006.

In this paper, we describe a shape-based 3D model retrieval method based on multi-scale local visual features. The features are extracted from 2D range images of the model viewed from uniformly sampled locations on a view sphere. The method is appearance-based, and accepts all the models that can be rendered as a range image. For each range image, a set of 2D multi-scale local visual features is computed by using the Scale Invariant Feature Transform [22] algorithm. To reduce cost of distance computation and feature storage, a set of local features describing a 3D model is integrated into a histogram using the Bag-Of-Features approach. Our experiments using two standard benchmarks, one for articulated shapes and the other for rigid shapes, showed that the methods achieved the performance comparable or superior to some of the most powerful 3D shape retrieval methods.

A local shape feature has an advantage in dealing with deformable or articulated 3D models. We evaluate the performance of our local, 2D visual features and their integration method based on the bag-of-features approach using the SHREC'08 CAD model track. The evaluation showed that, it performed very well, winning the 2ndplace in the contest, although it lost to a method that employs supervised learning of classes in the benchmark dataset.

In this paper, we describe a method of shape-based 3D model retrieval that employs a set of 3D, local, multi-scale features extracted from a voxel representation of a 3D model to be compared. The method first convert a surface based 3D model into a voxel model. Then, a novel 3D extension of the popular 2D image feature, the Scale Invariant Feature Transform by David Lowe, is applied to extract a set of 3D local features. The 3D feature is invariant to rotation, uniform scaling, and translation in 3D space. A 3D model typically yields thousands of such local 3D features. Our method extracts thousands of 3D local features from a model to compare its shape. Our evaluation showed that the method is quite effective, achieving Means First Tier of 58% for the Query Set 1 of the SHREC’08 Generic Models Track.

In this paper, we evaluate the effect of learning algorithms, unsupervised and supervised, for 3D face model retrieval using a global shape feature. We used the dataset and protocol of SHREC 2007 3D face models track (SHREC 2007 3DFMT) for the evaluation. Unlike the entrants for the track, we used global shape features to capture overall geometric shape of faces, e.g., that of foreheads. One of the global features, as it is, produced mean average precision highly relevant (MAPH) figure of 0.84, outperforming the top finisher of the SHREC 2007 3DFMT whose MAPH=0.66. Learning was quite effective; for the same global feature, an unsupervised learning method produced MAPH=0.90, and a simple supervised learning method produced an "ideal" performance of MAPH=1.0..

A shape feature by itself is not sufficient for effective 3D model retrieval. Long-lasting semantics shared by a community as well as a short-lived intention of a user determines the similarity of 3D models. In this paper, we describe a method of shape-based 3D model retrieval that employs off-line, semi-supervised learning of multiple classes in the database to capture long-lasting, shared semantic knowledge. The method performs two learning based dimension reductions, first one to accommodate distribution of features in the feature space and the second one to accommodate the semantic knowledge embodied in a set of user-defined semantic labels. We evaluate the method by using the SHREC’08 3D Generic and CAD Models Track.

This paper discusses a method to watermark a 3D shape model defined as a set of unoriented points. Our approach is to perform frequency domain analysis of the shape of the 3D point set for watermarking. Our method applies the mesh-spectral analysis technique proposed by Zachi Karni, et al. to the point set model for frequency domain shape analysis. As the technique requires connectivity of vertices for the analysis, our method generates a nonmanifold mesh from the point set and uses its connectivity for the analysis. The watermarks embedded by using the method can be detected, to some extent, after such attacks as cropping, (simulated) simplification, similarity transformation, and additive random noise applied on the watermarked point set model.

This paper presents an approach for interactive and intuitive deformation of 3D shapes. The approach combines a fast and efficient 3D free form deformation algorithm with a high degrees-offreedom input device of hand joint angles, namely, the CyberGlove. The deformation algorithm employs compactly supported radial basis functions (CSRBFs) to produce reasonable shape deformations quickly. However, positioning and moving CSRBFs properly to create a desired shape deformation is not a trivial task. In our approach, the 10 CSRBFs are virtually attached to the fingertips of the CyberGloves worn on both hands of the user. This allows the user an intuitive manipulation of 3D shapes in a manner somewhat similar to clay modeling. While the shape deformation algorithm is not physically-based, the shape deformation created by the algorithm is plausible and quite useful. In terms of speed, deformations involving all the 15k vertices of a 32k triangle mesh model can be produced at speed of about 30 frames-per-second.

As the number of in-house and public-domain 3D shape models increase, importance of shape-similarity based search and retrieval for 3D shapes models has increased rapidly. In this paper, we describe our preliminary findings in applying a multiresolution analysis technique to the task of shape similarity comparison of polygon soup models. We used the 3D alpha shapes algorithm to create a multiresolution hierarchy of shapes from the given 3D models. We then applied a (single resolution) shape descriptor to each of the models at multiple resolution levels to derive a multiresolution shape descriptor. According to our evaluation experiments, the retrieval performance of our resolution descriptor outperformed its single resolution counterpart, proving the effectiveness of the basic approach.

The paper proposes a digital watermarking algorithm for 2D vector digital maps. The watermark is a robust, informed-detection watermark to be used to prevent such abuses as an intellectual property rights violation. The algorithm proposed in the paper embeds watermarks in the frequency-domain representation of a 2D vector digital map. Our method treats vertices in the map as a point set, and imposes connectivity among the points by using Delaunay triangulation. The method then computes the mesh-spectral coefficients (Karni, 2000) from the mesh created. Modifications of the coefficients according to the message bits, and inverse transforming the coefficients back into the coordinate domain produces the watermarked map. Our evaluation experiments showed that the watermark produced by the method is resistant against additive random noise, similarity transformation, vertex insertion and removal. It is also resistant, to some extent, against cropping. Compared to our previous algorithm (Ohbuchi, 2002), the algorithm described in this paper showed significantly improved attack resiliency.

Digital maps are used, for example, in car navigation systems and Web-based map services. As digital data, digital maps are easy to update, duplicate, and distribute. At the same time, illegal duplication and distribution or forgery of the maps is also easy. This paper proposes a digital watermarking algorithm for vector digital maps as a method to counter such abuses of the maps. A watermark bit is embedded by displacing an average of coordinates of a set of vertices that lies in a rectangular area created on a map by adaptively subdividing the map. The watermark is resistant against additive random noise, similarity transformation, and vertex insertion/removal, and, to some extent, cropping.

In this paper, we propose a method for shape-similarity search of 3D polygonal-mesh models. The system accepts triangular meshes, but tolerates degenerated polygons, disconnected component, and other anomalies. As the feature vector, the method uses a combination of three vectors, (1) the moment of inertia, (2) the average distance of surface from the axis, and (3) the variance of distance of the surface from the axis. Values in each vector are discretely parameterized along each of the three principal axes of inertia of the model. We employed the Euclidean distance and the elastic-matching distance as the measures of distance between pairs of feature vectors. Experiments showed that the proposed shape features and distance measures perform fairly well in retrieving models having similar shape from a database of VRML models.

This paper presents a robust watermarking algorithm with informed detection for 3D polygonal meshes. The algorithm is based on our previous algorithm [22] that employs mesh-spectral analysis to modify mesh shapes in their transformed domain. This paper presents extensions to our previous algorithm so that (1) much larger meshes can be watermarked within a reasonable time, and that (2) the watermark is robust against connectivity alteration (e.g., mesh simplification), and that (3) the watermark is robust against attacks that combine similarity transformation with such other attacks as cropping, mesh simplification, and smoothing. Experiment showed that our new watermarks are resistant against mesh simplification and remeshing combined with resection, similarity transformation, and other operations..

Existing methods of morphing 3D meshes are often limited to cases in which 3D input meshes to be morphed are topologically equivalent. The paper presents a new method for morphing 3D meshes having different surface topological types. The most significant feature of the method is that it allows explicit control of topological transitions that occur during the morph. Transitions of topological types are specified by means of a compact formalism that resulted from a rigorous examination of singularities of 4D hypersurfaces and embeddings of meshes in 3D space. Using the formalism, every plausible path of topological transitions can be classified into a small set of cases. In order to guide a topological transition during the morph, our method employs a key frame that binds two distinct surface topological types. The key frame consists of a pair of "faces", each of which is homeomorphic to one of the source (input) 3D meshes. Interpolating the source meshes and the key frame by using a tetrahedral 4D mesh and then intersecting the interpolating mesh with another 4D hypersurface creates a morphed 3D mesh. We demonstrate the power of our methodology by using several examples of topology transcending morphing.

Existing data embedding algorithms for polygonal meshes and their attributes cannot be applied to the majority of (geometric) computer aided design (CAD) applications for two major reasons. First, these CAD systems employ parametric curves and surfaces, not polygonal meshes, as their main shape-defining primitives. Second, most CAD applications do not tolerate modifications of model topology and/or geometry that are introduced by existing data embedding algorithms. We propose a new data embedding algorithm for non-uniform rational B-spline (NURBS) curves and surfaces, which employs rational linear reparameterization for embedding messages. The algorithm exactly preserves the shape, that is, the geometry and topology of its embedding targets. Furthermore, it preserves the data size of the model. We consider how these two properties, preservation of shape and preservation of data size, can be significant with regard to the use of data embedding in CAD applications. In addition to the shape- and data size-preserving data embedding algorithm for NURBS curves and surfaces, we outline additional methods for embedding data in various types of parametric curves and surfaces.

This paper discusses techniques for embedding data into three-dimensional (3-D) polygonal models of geometry. Given objects consisting of points, lines, (connected) polygons, or curved surfaces, the algorithms described in produce polygonal models with data embedded into either their vertex coordinates, their vertex topology (connectivity), or both. Such data embedding can be used, for example, for copyright notification, copyright protection, theft deterrence, and inventory of 3-D polygonal models. A description of the background and requirements is followed by a discussion of where, and by what fundamental methods, data can be embedded into 3-D polygonal models. The paper then presents several data embedding algorithms, with examples, based on these fundamental methods. By means of these algorithms and examples, we show that the embedding of data into 3-D polygonal models is a practicable technique.

The advantages of digital media such as the Internet and CD-ROMs lie in the fact that their contents are easy to duplicate, edit, and distribute.These advantages, however, are double-edged swords, because they also facilitate unauthorized use of such contents.Data embedding, which places information into the contents themselves, is an approach to address this issue.Embedded information can be used, for example, for copyright protection, theft deterrence, and inventory.This paper discusses our work on embedding data into three-dimensional (3D) polygonal models of geometry.Given objects consisting of points, lines, polygons, or curved surfaces, the data embedding algorithms described in this paper produce polygonal models with data embedded.Data are placed into 3D polygonal models by modifying either their vertex coordinates, their vertex topology (connectivity), or both.A brief review of related work and a description of the requirements of data embedding is followed by a discussion of where, and by what fundamental methods, data can be embedded into 3D polygonal models.The paper then presents data-embedding algorithms, with examples, based on these fundamental methods.

Augmented reality systems with see-through headmounted displays have been used primarily for applications that are possible with today's computational capabilities. We explore possibilities for a particular application-in-place, real-time 3D ultrasound visualization-without concern for such limitations. The question is not "How well could we currently visualize the fetus in real time," but "How well could we see the fetus if we had sufficient compute power?" Our video sequence shows a 3D fetus within a pregnant woman's abdomen-the way this would look to a HMD user. Technical problems in making the sequence are discussed. This experience exposed limitations of current augmented reality systems; it may help define the capabilities of future systems needed for applications as demanding as real-time medical visualization.>

We describe initial results which show "live" ultrasound echography data visualized within a pregnant human subject.The visualization is achieved by using a small video camera mounted in front of a conventional head-mounted display worn by an observer.The camera's video images are composite with computer-generated ones that contain one or more 2D ultrasound images properly transformed to the observer's current viewing position.As the observer walks around the subject.the ultrasound images appear stationary in 3-space within the subject.This kind of enhancement of the observer's vision may have many other applications, e.g., image guided surgical procedures and on location 3D interactive architecture preview.