3D surface reconstruction from unorganized point clouds is a fundamental task in visual computing with numerous applications in areas such as robotics, virtual reality, augmented reality, and animation. To date, many deep learning-based surface reconstruction methods have been proposed, demonstrating great performance on many benchmark datasets. Among these, neural implicit field learning-based methods have gained popularity for their capability of representing complex structures in a continuous implicit distance field. Existing neural implicit field learning methods either utilize voxelized point cloud then feed them to a deep network, or directly take points as input. In this paper, we propose an implicit surface reconstruction framework based on point voxel geometric-aware transformer PVT to seamlessly integrate point-based convolution with voxel-based convolution using bidirectional transformers. Experiments show that the proposed PVT framework can better encode local geometry details and provide a significant performance boost over existing state-of-the-art methods.

Depth estimation is the fundamental computer vision task for scene analysis. With the emergence of the deep learning era supervised monocular image depth estimation (MDE) became a popular choice for the task. Predominantly, MDE methods utilize 360 images as ideal input due to their comprehensive field of view scene content compared to perspective images, but they suffer from distortions in polar regions making it a more challenging illposed problem to date. Over the years, methods using CNNs, and/or large transformers taking 360 and/or projected perspective patch inputs have been proposed to solve the 360 MDE problem by formulating it as a regression or a classification task. Nevertheless, their performance still suffers from global discrepancy, inaccuracy, poor details, and generalizability. Lately, diffusion-generating models have shown state-of-the-art performance in image synthesis that captures exceptionally rich knowledge of the visual world. However, their ability to perform omnidirectional perception tasks is still unexplored. In this paper, we explore a new approach called OmniDiffusion that reformulates the 360 MDE task as a diffusion denoising process. We present a diffusion-based framework to learn an iterative denoising process that denoises random depth distribution into the required depths. The diffusion process is performed in the latent space and uses the guidance of encoded RGB image visual as a condition. Furthermore, to advance the image latent in a geometrically meaningful direction we leverage semantic segmentation and surface normal information to provide a more detailed contextual assistance to the denoising process. The performed experiments on the multiple real-world datasets show that our diffusion-denoising approach with the proposed conditions more appropriately refines depths outperforming the existing MDE and diffusion-based methods with state-of-the-art generalization ability while generating more accurate, high-quality, and detailed 360 depths.

3D surface reconstruction from unorganized point clouds is a fundamental task in visual computing and has numerous applications in areas such as robotics, virtual reality, augmented reality, and animation. To date, many deep learning-based surface reconstruction methods have been proposed with outstanding performance on various benchmark datasets. Among them, neural implicit field learning-based methods have been particularly popular because they can represent both complex inner structures and open surfaces in a continuous implicit distance field. Existing implicit distance field-based methods either utilize voxels with 3D convolutions or rely on point-based convolutions directly. In this paper, we propose Bifusion, a bi-directional point-voxel fusion framework that aims to seamlessly fuse point and voxel-based implicit fields. Experiments demonstrate that the proposed Bifusion can better encode local geometry details and provide a significant performance boost over existing state-of-the-art methods.

Panorama images are popularly used for comprehensive scene understanding due to their integrated field of view. To overcome the spherical image distortions observed in commonly used Equirectangular Projection (ERP) 360-format images, the existing 360 monocular deep learning-based depth estimation networks propose using distortion-free tangent patch images projected from ERP to predict perspective depths which are merged to get the final ERP depth map. These methods show improved performance over previous methods; however, they produce depth maps that are inconsistent, and uneven, have merging artifacts, and miss fine structure details due to the missing holistic contextual information in the learned local tangent patch image features. To address this problem, we propose a novel multi-scale 360 monocular depth estimation framework, MS360, which focuses on guiding the local tangent perspective image features with coarse integrated image features. Specifically, our method first extracts coarse comprehensive features with perspective tangent patches from downsampled ERP as input to the coarse UNet structure. Secondly, we use a fine branch network to capture local geometric information using perspective tangent images from high-resolution ERP. Furthermore, we present a Multi-Scale Feature Fusion (MSFF) bottleneck module to fuse and guide the fine local features with coarse holistic features via an attention mechanism. Lastly, we predict a low-resolution depth map using coarse features and a final high-resolution depth map using coarse-guided fine image features as input to the coarse and fine decoder networks. Our method greatly reduces the discrepancies, and local patch merging artifacts in the depth maps. Performed experiments on multiple real-world depth estimation benchmark datasets show that our network outperforms the existing models both quantitatively and qualitatively while producing smooth and high-quality depth maps.

Deep learning (DL) in orthopaedics has gained significant attention in recent years. Previous studies have shown that DL can be applied to a wide variety of orthopaedic tasks, including fracture detection, bone tumour diagnosis, implant recognition, and evaluation of osteoarthritis severity. The utilisation of DL is expected to increase, owing to its ability to present accurate diagnoses more efficiently than traditional methods in many scenarios. This reduces the time and cost of diagnosis for patients and orthopaedic surgeons. To our knowledge, no exclusive study has comprehensively reviewed all aspects of DL currently used in orthopaedic practice. This review addresses this knowledge gap using articles from Science Direct, Scopus, IEEE Xplore, and Web of Science between 2017 and 2023. The authors begin with the motivation for using DL in orthopaedics, including its ability to enhance diagnosis and treatment planning. The review then covers various applications of DL in orthopaedics, including fracture detection, detection of supraspinatus tears using MRI, osteoarthritis, prediction of types of arthroplasty implants, bone age assessment, and detection of joint-specific soft tissue disease. We also examine the challenges for implementing DL in orthopaedics, including the scarcity of data to train DL and the lack of interpretability, as well as possible solutions to these common pitfalls. Our work highlights the requirements to achieve trustworthiness in the outcomes generated by DL, including the need for accuracy, explainability, and fairness in the DL models. We pay particular attention to fusion techniques as one of the ways to increase trustworthiness, which have also been used to address the common multimodality in orthopaedics. Finally, we have reviewed the approval requirements set forth by the US Food and Drug Administration to enable the use of DL applications. As such, we aim to have this review function as a guide for researchers to develop a reliable DL application for orthopaedic tasks from scratch for use in the market.

Deep learning has emerged as a powerful tool in various domains, revolutionising machine learning research. However, one persistent challenge is the scarcity of labelled training data, which hampers the performance and generalisation of deep learning models. To address this limitation, researchers have developed innovative methods to overcome data scarcity and enhance deep model learning capabilities. Two prevalent techniques that have gained significant attention are transfer learning and self-supervised learning. Transfer learning leverages knowledge learned from pre-training on a large-scale dataset, such as ImageNet, and applies it to a target task with limited labelled data. This approach allows models to benefit from the learned representations and effectively transfer knowledge to new tasks, resulting in improved learning performance and generalisation. On the other hand, self-supervised learning focuses on training models using pretext tasks that do not require manual annotation, allowing them to learn valuable representations from large amounts of unlabelled data. These learned representations can then be fine-tuned for downstream tasks, mitigating the need for extensive labelled data. In recent years, transfer and self-supervised learning have found applications in various fields, including medical image processing, video recognition, and natural language processing. These approaches have demonstrated remarkable achievements, enabling breakthroughs in areas such as disease diagnosis, object recognition, and language understanding. However, while these methods offer numerous advantages, they also have limitations. For example, transfer learning may face domain mismatch issues between the pre-training and target domains, while self-supervised learning requires careful design of pretext tasks to ensure meaningful representations. This review paper explores the recent applications of these pre-training methods in various fields within the past three years. It delves into the advantages and limitations of each approach, assesses the performance of models employing these techniques, and identifies potential directions for future research. By providing a comprehensive review of current pre-training methods, this article offers guidance for selecting the best technique for specific deep learning applications to address the data scarcity issue.

Abstract Meeting the rising global demand for healthcare diagnostic tools is crucial, especially with a shortage of medical professionals. This issue has increased interest in utilizing deep learning (DL) and telemedicine technologies. DL, a branch of artificial intelligence, has progressed due to advancements in digital technology and data availability and has proven to be effective in solving previously challenging learning problems. Convolutional neural networks (CNNs) show potential in image detection and recognition, particularly in healthcare applications. However, due to their resource-intensiveness, they surpass the capabilities of general-purpose CPUs. Therefore, hardware accelerators such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and graphics processing units (GPUs) have been developed. With their parallelism efficiency and energy-saving capabilities, FPGAs have gained popularity for DL networks. This research aims to automate the classification of normal and abnormal (specifically Diabetic Foot Ulcer—DFU) classes using various parallel hardware accelerators. The study introduces two CNN models, namely DFU_FNet and DFU_TFNet. DFU_FNet is a simple model that extracts features used to train classifiers like SVM and KNN. On the other hand, DFU_TFNet is a deeper model that employs transfer learning to test hardware efficiency on both shallow and deep models. DFU_TFNet has outperformed AlexNet, VGG16, and GoogleNet benchmarks with an accuracy 99.81%, precision 99.38% and F1-Score 99.25%. In addition, the study evaluated two high-performance computing platforms, GPUs and FPGAs, for real-time system requirements. The comparison of processing time and power consumption revealed that while GPUs outpace FPGAs in processing speed, FPGAs exhibit significantly lower power consumption than GPUs.

Given the tremendous potential and influence of artificial intelligence (AI) and algorithmic decision‐making (DM), these systems have found wide‐ranging applications across diverse fields, including education, business, healthcare industries, government, and justice sectors. While AI and DM offer significant benefits, they also carry the risk of unfavourable outcomes for users and society. As a result, ensuring the safety, reliability, and trustworthiness of these systems becomes crucial. This article aims to provide a comprehensive review of the synergy between AI and DM, focussing on the importance of trustworthiness. The review addresses the following four key questions, guiding readers towards a deeper understanding of this topic: (i) why do we need trustworthy AI? (ii) what are the requirements for trustworthy AI? In line with this second question, the key requirements that establish the trustworthiness of these systems have been explained, including explainability, accountability, robustness, fairness, acceptance of AI, privacy, accuracy, reproducibility, and human agency, and oversight. (iii) how can we have trustworthy data? and (iv) what are the priorities in terms of trustworthy requirements for challenging applications? Regarding this last question, six different applications have been discussed, including trustworthy AI in education, environmental science, 5G‐based IoT networks, robotics for architecture, engineering and construction, financial technology, and healthcare. The review emphasises the need to address trustworthiness in AI systems before their deployment in order to achieve the AI goal for good. An example is provided that demonstrates how trustworthy AI can be employed to eliminate bias in human resources management systems. The insights and recommendations presented in this paper will serve as a valuable guide for AI researchers seeking to achieve trustworthiness in their applications.

3D city-scale models are useful in a number of applications, including education, city planning, navigation systems, artificial intelligence training, and simulations. However, final models need to be immersive and interactive, which requires a mixed reality (XR) environment design that combines e.g., a Cave Automatic Virtual Environment (CAVE) VR system with the Microsoft Hololens2 in a networked and multimodal setting. In this paper, we propose a pipeline to convert a city-scale point cloud into a finalized city-scale textured mesh in which, a number of XR devices can share the same environment and co-exist in a shared space for model interactions. Specifically, we use input point clouds obtained from wide area motion imagery systems or off-the-shelf drones pertaining to Albuquerque, New Mexico, but the pipeline is generalized so that other input can be used. Using four different traditional algorithms and an additional deep learning method, we create meshes for the model interactions. For each mesh produced, we map high-resolution textures onto them, producing a more accurate city, which is then passed into the shared/networked Unity environment. Ten participants provided their assessment of mesh quality and interactivity of the networked environment during exploration of different city reconstructions with the CAVE and laptop device modalities. Results on the perceptual immersive quality of the Point2Mesh deep learning meshes highlights the need for improvements to handle large city scale point clouds.

Modeling and simulation of large urban regions is beneficial for a range of applications including intelligent transportation, smart cities, infrastructure planning, and training artificial intelligence for autonomous navigation systems including ground vehicles and aerial drones. Immersive environments including virtual reality (VR), augmented reality (AR), mixed reality (MR or XR) can be used to explore city scale regions for planning, design, training and operations. Virtual environments are in the midst of rapid change as innovations in display tech-nologies, graphics processors and game engine software present new opportunities for incorporating modeling and simulation into engineering workflows. Game engine software like Unity with photorealistic rendering and realistic physics have plug-in support for a variety of virtual environments. In this paper, we explore the visualization of urban scale real world accurate meshes in virtual environments, including the Microsoft HoloLens head mounted display or the CAVE VR for multi-user interaction.

A well-known challenge in applying deep-learning methods to omnidirectional images is spherical distortion. In dense regression tasks such as depth estimation, where structural details are required, using a vanilla CNN layer on the distorted 360 image results in undesired information loss. In this paper, we propose a 360 monocular depth estimation pipeline, OmniFusion, to tackle the spherical distortion issue. Our pipeline transforms a 360 image into less-distorted perspective patches (i.e. tangent images) to obtain patch-wise predictions via CNN, and then merge the patch-wise results for final output. To handle the discrepancy between patch-wise predictions which is a major issue affecting the merging quality, we propose a new framework with the following key components. First, we propose a geometry-aware feature fusion mechanism that combines 3D geometric features with 2D image features to compensate for the patch-wise discrepancy. Second, we employ the self-attention-based transformer architecture to conduct a global aggregation of patch-wise information, which further improves the consistency. Last, we introduce an iterative depth refinement mechanism, to further refine the estimated depth based on the more accurate geometric features. Experiments show that our method greatly mitigates the distortion issue, and achieves state-of-the-art performances on several 360 monocular depth estimation benchmark datasets. Our code is available at https://github.com/yuyanli0831/OmniFusion.

In this paper, we present a comprehensive point cloud semantic segmentation network that aggregates both local and global multi-scale information. First, we propose an Angle Correlation Point Convolution (ACPConv) module to effectively learn the local shapes of points. Second, based upon ACPConv, we introduce a local multi-scale split (MSS) block that hierarchically connects features within one single block and gradually enlarges the receptive field which is beneficial for exploiting local context. Third, inspired by HRNet that has excellent performance on 2D image vision tasks, we build an HRNet customized for point cloud to learn global multi-scale context. Lastly, we introduce a point-wise attention fusion approach that fuses multi-resolution predictions and further improves point cloud semantic segmentation performance. Our experimental results and ablations on several benchmark datasets show that our proposed method is effective and able to achieve state-of-the-art performances compared to existing methods.

received valuable feedback from readers in the field regarding our paper that was published in a special issue of BMC Bioinformatics in 2020 (BMC Bioinformatics, 21(S21):534, 2020), we discovered some visualization errors and typos in two Figures (Figs. 17 and 18) in the published manuscript.The Fourier Shell Correlation (FSC) values in Figs.17b and 18b in the published paper were plotted in the incorrect order by mistake.The error only occurred in visualizing the data, while the original raw data are still the same as before.Therefore, we replot the data to create a new version of Figs. 17 and 18 and fix some typos (e.g., the title of the x-axis of Figs.17b and 18b) according to the feedback.Figures 17 and 18 are given below.

Omnidirectional 3D information is essential for a wide range of applications such as Virtual Reality, Autonomous Driving, Robotics, etc. In this paper, we propose a novel, model-agnostic, two-stage pipeline for omnidirectional monocular depth estimation. Our proposed framework PanoDepth takes one 360 image as input, produces one or more synthesized views in the first stage, and feeds the original image and the synthesized images into the subsequent stereo matching stage. In the second stage, we propose a differentiable Spherical Warping Layer to handle omnidirectional stereo geometry efficiently and effectively. By utilizing the explicit stereo-based geometric constraints in the stereo matching stage, PanoDepth can generate dense high-quality depth. We conducted extensive experiments and ablation studies to evaluate PanoDepth with both the full pipeline as well as the individual modules in each stage. Our results show that PanoDepth outperforms the state-of-the-art approaches by a large margin for 360 monocular depth estimation. Our code is available at https:// github.com/yuyanli0831/PanoDepth_3dv.

Modeling and simulation of large urban regions is beneficial for a range of applications including intelligent transportation, smart cities, infrastructure planning, and training artificial intelligence for autonomous navigation systems including ground vehicles and aerial drones. Immersive environments including virtual reality (VR), augmented reality (AR), mixed reality (MR or XR) can be used to explore city scale regions for planning, design, training and operations. Virtual environments are in the midst of rapid change as innovations in display technologies, graphics processors and game engine software present new opportunities for incorporating modeling and simulation into engineering workflows. Game engine software like Unity with photorealistic rendering and realistic physics have plug-in support for a variety of virtual environments and typically model the scene as meshes. In this paper, we develop an end-to-end workflow for creating urban scale real world accurate synthetic environments that can be visualized in virtual environments including the Microsoft HoloLens head mounted display or the CAVE VR for multi-user interaction. Four meshing algorithms are evaluated for representation accuracy and city-scale meshes imported into Unity for assessing the quality of the immersive experience.

BACKGROUND: Cryo-EM data generated by electron tomography (ET) contains images for individual protein particles in different orientations and tilted angles. Individual cryo-EM particles can be aligned to reconstruct a 3D density map of a protein structure. However, low contrast and high noise in particle images make it challenging to build 3D density maps at intermediate to high resolution (1-3 Å). To overcome this problem, we propose a fully automated cryo-EM 3D density map reconstruction approach based on deep learning particle picking. RESULTS: A perfect 2D particle mask is fully automatically generated for every single particle. Then, it uses a computer vision image alignment algorithm (image registration) to fully automatically align the particle masks. It calculates the difference of the particle image orientation angles to align the original particle image. Finally, it reconstructs a localized 3D density map between every two single-particle images that have the largest number of corresponding features. The localized 3D density maps are then averaged to reconstruct a final 3D density map. The constructed 3D density map results illustrate the potential to determine the structures of the molecules using a few samples of good particles. Also, using the localized particle samples (with no background) to generate the localized 3D density maps can improve the process of the resolution evaluation in experimental maps of cryo-EM. Tested on two widely used datasets, Auto3DCryoMap is able to reconstruct good 3D density maps using only a few thousand protein particle images, which is much smaller than hundreds of thousands of particles required by the existing methods. CONCLUSIONS: We design a fully automated approach for cryo-EM 3D density maps reconstruction (Auto3DCryoMap). Instead of increasing the signal-to-noise ratio by using 2D class averaging, our approach uses 2D particle masks to produce locally aligned particle images. Auto3DCryoMap is able to accurately align structural particle shapes. Also, it is able to construct a decent 3D density map from only a few thousand aligned particle images while the existing tools require hundreds of thousands of particle images. Finally, by using the pre-processed particle images, Auto3DCryoMap reconstructs a better 3D density map than using the original particle images.

BACKGROUND: Cryo-electron microscopy (Cryo-EM) is widely used in the determination of the three-dimensional (3D) structures of macromolecules. Particle picking from 2D micrographs remains a challenging early step in the Cryo-EM pipeline due to the diversity of particle shapes and the extremely low signal-to-noise ratio of micrographs. Because of these issues, significant human intervention is often required to generate a high-quality set of particles for input to the downstream structure determination steps. RESULTS: Here we propose a fully automated approach (DeepCryoPicker) for single particle picking based on deep learning. It first uses automated unsupervised learning to generate particle training datasets. Then it trains a deep neural network to classify particles automatically. Results indicate that the DeepCryoPicker compares favorably with semi-automated methods such as DeepEM, DeepPicker, and RELION, with the significant advantage of not requiring human intervention. CONCLUSIONS: Our framework combing supervised deep learning classification with automated un-supervised clustering for generating training data provides an effective approach to pick particles in cryo-EM images automatically and accurately.

Volumetric grid is widely used for 3D deep learning due to its regularity. However the use of relatively lower order local approximation functions such as piece-wise constant function (occupancy grid) or piece-wise linear function (distance field) to approximate 3D shape means that it needs a very high-resolution grid to represent finer geometry details, which could be memory and computationally inefficient. In this work, we propose the PointGrid, a 3D convolutional network that incorporates a constant number of points within each grid cell thus allowing the network to learn higher order local approximation functions that could better represent the local geometry shape details. With experiments on popular shape recognition benchmarks, PointGrid demonstrates state-of-the-art performance over existing deep learning methods on both classification and segmentation.

Rock detachment events along roadways pose public safety concerns but can be predicted and safely handled using geological measurements of discontinuities. With modern sensing technology, these measurements can be taken on 3-D point clouds and 2-D optical images that provide a high level of structural accuracy and visual detail. Doing so allows engineers to obtain the needed data with relative ease while eliminating the biases and hazards inherent in taking manual measurements. This letter presents an approach for fusing the 2-D and 3-D data in natural and unstructured scenes. This includes a novel method for visualizing imagery obtained with very different sensors to maximize their visual similarity making registration a more tangible task. To show the effectiveness of our registration methodology, we evaluate measurements taken manually and digitally on rock facet and cut discontinuity orientations in Rolla, MO. Our method is able to align the 2-D and 3-D data with an accuracy of under 2 cm. The median difference between measurements manually obtained by a geological engineer and those obtained with our proposed software is 3.65.

In the event of natural or man-made disasters, providing rapid situational awareness through video/image data collected at salient incident scenes is often critical to the first responders. However, computer vision techniques that can process the media-rich and data-intensive content obtained from civilian smartphones or surveillance cameras require large amounts of computational resources or ancillary data sources that may not be available at the geographical location of the incident. In this paper, we propose an incident-supporting visual cloud computing solution by defining a collection, computation, and consumption (3C) architecture supporting fog computing at the network edge close to the collection/consumption sites, which is coupled with cloud offloading to a core computation, utilizing software-defined networking (SDN). We evaluate our 3C architecture and algorithms using realistic virtual environment test beds. We also describe our insights in preparing the cloud provisioning and thin-client desktop fogs to handle the elasticity and user mobility demands in a theater-scale application. In addition, we demonstrate the use of SDN for on-demand compute offload with congestion-avoiding traffic steering to enhance remote user quality of experience in a regional-scale application. The optimization between fogs computing at the network edge with core cloud computing for managing visual analytics reduces latency, congestion, and increases throughput.

Recently, pathological diagnosis plays a crucial role in many areas of medicine, and some researchers have proposed many models and algorithms for improving classification accuracy by extracting excellent feature or modifying the classifier. They have also achieved excellent results on pathological diagnosis using tongue images. However, pixel values can't express intuitive features of tongue images and different classifiers for training samples have different adaptability. Accordingly, this paper presents a robust approach to infer the pathological characteristics by observing tongue images. Our proposed method makes full use of the local information and similarity of tongue images. Firstly, tongue images in RGB color space are converted to Lab. Then, we compute tongue statistics information. In the calculation process, Lab space dictionary is created at first, through it, we compute statistic value for each dictionary value. After that, a method based on Doublets is taken for feature optimization. At last, we use XGBOOST classifier to predict the categories of tongue images. We achieve classification accuracy of 95.39% using statistics feature and the improved classifier, which is helpful for TCM (Traditional Chinese Medicine) diagnosis.

With the improvement of people's living standards, there is no doubt that people are paying more and more attention to their health. However, shortage of medical resources is a critical global problem. As a result, an intelligent prognostics system has a great potential to play important roles in computer aided diagnosis. Numerous papers reported that tongue features have been closely related to a human's state. Among them, the majority of the existing tongue image analyses and classification methods are based on the low-level features, which may not provide a holistic view of the tongue. Inspired by a deep convolutional neural network (CNN), we propose a deep tongue image feature analysis system to extract unbiased features and reduce human labor for tongue diagnosis. With the unbalanced sample distribution, it is hard to form a balanced classification model based on feature representations obtained by existing low-level and high-level methods. Our proposed deep tongue image feature analysis model learns high-level features and provide more classification information during training time, which may result in higher accuracy when predicting testing samples. We tested the proposed system on a set of 267 gastritis patients, and a control group of 48 healthy volunteers (labeled according to Western medical practices). Test results show that the proposed deep tongue image feature analysis model can classify a given tongue image into healthy and diseased state with an average accuracy of 91.49%, which demonstrates the relationship between human body's state and its deep tongue image features.

Circle detection from digital images is a necessary operation in many robotics and computer vision tasks to facilitate shape and object recognition. We propose and analyze a novel method, based on line segment detection and circle completeness verification, to detect circles in images. The key idea is to use line segments instead of raw edge pixels to get the circle candidates followed by a verification step to measure the circle's completeness. Experimental results on several synthesized and hand-sketched as well as natural images with various complication favor the accuracy, robustness and efficiency of our approach against other well-known techniques. Our method can deal with incomplete, cocentric, discontinuous and occluded circles with noise and deformation. Moreover, in this paper, we create CDBD, the first benchmark dataset for circle detection with ground truth circles labeled by human, which will establish standard quantitative results in future research regarding circle detection.

LIDAR range scans can be used to quickly create accurate 3D models for virtual reality and as a basis to visualize sets of photographs, videos, and virtual objects in a cohesive environment. The number of existing virtual reality programs that use LIDAR data as input has motivated our group to develop methods for fusing images with 3D scans and for augmenting the scans with both dynamic objects present in videos and virtual models. Bringing together as many data sources as possible increases users' abilities to present related information in one, intuitive venue. We demonstrate how to register a variety of 2D imagery with a range scan to construct photo-realistic models and to extract walking people captured in videos and model them in a 3D space. We also present a method for determining the sun position from a set of stitched photographs in order to apply correct lighting to virtual objects placed amongst real world data. Naturally lit objects can be inserted into original photographs using our 2D-3D registration information. These methods are all combined to display and study photos, videos, and virtual objects in a complete 3D environment.

Matching and aligning architectural imagery is an important step for many applications but can be a difficult task due to repetitive elements often present in buildings. Many keypoint descriptor and matching methods will fail to produce distinctive descriptors for each region of man-made structures, which causes ambiguity when attempting to match areas between images. In this paper, we outline a technique for reducing the search space for matching by taking a two-step approach, aligning pairs one dimension at a time and by abstracting images that originally contain many repetitive elements into a set of distinct, representative patches. We also present a simple, but very effective method for computing the intra-image saliency for a single image that allows us to directly identify unique areas in an image without machine learning. We use this information to find distinctive keypoint matches across image pairs. We show that our pipeline is able to overcome many of the pitfalls encountered when using traditional keypoint and regional matching techniques on commonly encountered images of urban scenes.

In the event of natural or man-made disasters, many videos may be collected by civilians and surveillance cameras that can be extremely useful for first responders trying to ascertain the extent of the damage. However, watching and analyzing numerous videos on separate screens can be a cumbersome task. Registering a set of 2D videos with a 3D model can provide an intuitive venue for viewing multiple videos simultaneously. In such a setup, it is likely that the user will want to work with the dynamic 3D environment from a remote location, requiring that videos be transferred over a network to be registered with a 3D model. In this paper, we propose combining the fields of computer vision, cloud computing, and high-speed networking to create a system that takes in HD videos, streams the data to a server where a dynamic 3D model is constructed, and provides a virtual scene navigation program for viewing the videos in a 3D scene from a mobile device. We test transferring the data of interest over different types of networks and processing the videos on various server configurations to determine the capabilities of such a system and the necessary requirements for it to provide a high-quality user experience.

We propose a contextual framework for 2D image matching and registration using an ensemble feature. Our system is beneficial for registering image pairs that have captured the same scene but have large visual discrepancies between them. It is common to encounter challenging visual variations in image sets with artistic rendering differences or in those collected over a period of time during which the lighting conditions and scene content may have changed. Differences between images may also be caused using a variety of cameras with different sensors, focal lengths, and exposure values. Local feature matching techniques cannot always handle these difficulties, so we have developed an approach that builds on traditional methods to consider linear and histogram of gradient information over a larger, more stable region. We also present a technique for using linear features to estimate corner keypoints, or pseudo corners, that can be used for matching. Our pipeline follows this unique matching stage with homography refinement methods using edge and gradient information. Our goal is to increase the size of accurate keypoint match sets and align photographs containing a combination of man-made and natural imagery. We show that incorporating contextual information can provide complimentary information for scale invariant feature transform and boost local keypoint matching performance, as well as be used to describe corner feature points.

Parallel vectors (PV), the loci where two vector fields are parallel, are commonly used to represent curvilinear features in 3D for data visualization. Methods for extracting PV usually operate on a 3D grid and start with detecting seed points on a cell face. We propose, to the best of our knowledge, the first provably correct test that determines the parity of the number of PV points on a cell face. The test only needs to sample along the face boundary and works for any choice of the two vector fields. A discretization of the test is described, validated, and compared with existing tests that are also based on boundary sampling. The test can guide PV-extraction algorithms to ensure closed curves wherever the input fields are continuous, which we exemplify in extracting ridges and valleys of scalar functions.

In order to study the characteristics of electroencephalogram (EEG) induced by different types of music, we selected 58 volunteers (college students) with no musical training experience. Half of them are males and the rest are females. We finally adopted 49 experimental samples for further data processing. The results showed that: (1) music caused the changes of energy intensity in specific area of the brain and related EEG waveforms; (2) different types of music caused the changes of energy intensity in different regions and different EEG waveforms.

Radial artery pulse wave is a very important biological signal in the Traditional Chinese Medical (TCM) theory. It also plays an important role in the field of health informatics. In this paper, we proposed a new shiatsu three-part pulse collection method based on TCM to collect three-part pulses objectively. We established a feasible method for three-part pulse collection, and provided an objective process for radial artery pulse collection. The result of this research can be applied to disease diagnosis and health informatics.

In this paper we propose a hybrid approach for tree classification in airborne LIDAR (Light Detection and Ranging) data by integrating the point based supervised classification with region-based unsupervised clustering method. Furthermore we propose a novel 3D robust statistics-based shape feature that can overcome the limitations of existing methods in separating building boundary points from tree points. Experimental results show the new algorithm is very effective and can achieve very high accuracy.

Airborne LIDAR data often consumes hundreds of gigabytes. Existing LIDAR data compression schemes can compress the file to 5%-23% of the original size. Even after compression, the compressed data size is still in the order of gigabyte, which makes it impractical for many applications. This paper proposes a novel geometry based compression scheme. It first introduces a LIDAR classification method that accurately classifies airborne LIDAR data into tree and non-tree points; different geometry based compression schemes are then applied for different types of data. The proposed method can not only compress LIDAR data significantly, but also extract useful semantic information from the data. Experimental results show that the new approach achieves very high compression ratio, making applications that were not practical before feasible.

In this paper, we propose a novel method to register frames from multiple cameras into a consistent global scale. Assuming a moving object is observed in multiple camera setups, we use initial frames to create a global reference structure where the pose variation of each new frame is estimated using a RANSAC-based registration algorithm. We further combine the registration method with other state-of the-art techniques to build a high quality 3D reconstruction system with a smaller number of cameras than used by more traditional methods. Experimental results show that our method performs better and is more economical than the registration of separate monocular structures from motion methods. 3D reconstruction results on various challenging real world multi-camera video datasets also illustrate the feasibility and robustness of our method.

ABSTRACT In this paper, we propose a feature‐preserving surface reconstruction method from sparse noisy 3D measurements such as range scanning or passive multiview stereo. In contrast to earlier methods, we define a novel type of explicit 3D filter—regularized weighted least squares filter—to characterize the detail features such as surface wrinkles and sharp features. To account for noise, we rasterize input‐oriented points into a probabilistic volume (base volume) and then create a guidance volume by Gaussian filtering. Both the base volume and the guidance volume are further filtered by regularized weighted least squares filter to detect and recover detail features. After the two‐stage filtering, a global minimal surface is computed by graph cut and meshed as a geometric model. Experimental results on various datasets show that our method is robust to noise, outliers, and missing parts, which makes it more suitable to fit indoor/outdoor multiview stereo data. Unlike other methods, our method can completely recover scene structures and preserve detail features from noisy point samples. Copyright © 2012 John Wiley & Sons, Ltd.

In this paper we propose a new hybrid image segmentation algorithm that integrate the region-based method with the boundary-based method. More specifically we take an information fusion approach based on the Tensor Voting framework that seamlessly fuse the information from the region-based Mean Shift method with the boundary-based Canny Edge Detection algorithm. We have tested our algorithm on several images from the Caltech 101 database [18]. Experiments results show the new algorithm is very efficient and can achieve very good segmentation results.

Medical image segmentation has become an essential technique in clinical and research-oriented applications. Because manual segmentation methods are tedious, and fully automatic segmentation lacks the flexibility of human intervention or correction, semi-automatic methods have become the preferred type of medical image segmentation. We present a hybrid, semi-automatic segmentation method in 3D that integrates both region-based and boundary-based procedures. Our method differs from previous hybrid methods in that we perform region-based and boundary-based approaches separately, which allows for more efficient segmentation. A region-based technique is used to generate an initial seed contour that roughly represents the boundary of a target brain structure, alleviating the local minima problem in the subsequent model deformation phase. The contour is deformed under a unique force equation independent of image edges. Experiments on MRI data show that this method can achieve high accuracy and efficiency primarily due to the unique seed initialization technique.

3D point data acquired from laser scan or stereo vision can be quite noisy. A preprocessing step is often needed before a surface reconstruction algorithm can be applied. In this paper, we propose a nonparametric approach for noisy point data preprocessing. In particular, we proposed an anisotropic kernel based nonparametric density estimation method for outlier removal, and a hill-climbing line search approach for projecting data points onto the real surface boundary. Our approach is simple, robust and efficient. We demonstrate our method on both real and synthetic point datasets.

A photometric stereo method is optimized in view of Signal-to-Noise-Ratio(SNR) in both the spectral and spatial domain to accurately reconstruct the shapes for wide spectral band objects. Optical polarization filters are used to capture images free from specular reflection components so that the reflection can be approximated to Lambertian. The Jacobi iterative method is applied to solve the equalities between the images and the reflection model for shape. The reconstruction is optimized in view of SNR using the albedos as spatial weights and average albedo values as spectral weights. It is further optimized doubly using the albedos to suppress effects of noise most noticeable in black regions of objects and using a spatial weight dependent on the degree of saturations. Experimental results show that the optimization is very effective to obtain good shapes for wide spectral band objects with minimal effects of reflection models different from the Lambertian.

A new color photometric stereo method is presented, which reconstructs both albedo and shape from three color images. Optical polarization filters are placed in front of the light source and a camera to capture images free from specular reflection components. First, the albedo maps are derived from three color images. Then, the shape is reconstructed based on the Lambertian reflection from both the images and the albedo maps, using the Jacobi iterative method. The reconstruction is optimized using local and global weights, to minimize degradations due to shadows and saturations in images and color-dependent reflection characteristics of objects. Experimental results are given to evaluate the method.

Brain imaging studies of the corpus callosum (CC) in autism have yielded inconsistent results. In this paper, we explore the three-dimensional profile of CC abnormalities in autism. The CC is segmented from mid-sagittal MRI and four adjacent slices on both sides, using our newly developed semiautomatic method. A subsequent contour stitching is performed to create the 3D surface of the CC, and the point correspondence problem can be simplified by our segmentation scheme. After alignment, differences from each surface to a template are computed to create a signed distance map of each subject. The group difference in the distance map is analyzed using two sample t-test, which results in a significance map. The statistical results reveal significant difference between patients and controls in the body of the CC.

In this paper, we propose a new region-growing based iso-surface extraction algorithm that can generate high-quality curvature-adaptive semi-regular meshes, preserve sharp features and will extract all the disjoint components of the iso-surface. More importantly, in this paper, we propose a novel normal consistency constraint that ensures the intersection of the Delaunay sphere of the new triangle and the iso-surface is a topological disk, an important property that makes the new algorithm very robust when dealing with large scale of volumetric datasets of complex topology and geometry.

This paper presents a flexible, scalable framework for interactive hands-on shape design in a haptic, stereoscopic virtual environment. The framework is founded upon the concept of PDE-based geometric surface flow. Given an input polygonal mesh, a user can interactively define implicit functions around regions of interest of the mesh model, and the locally or globally affected regions of the model will automatically deform according to the underlying partial differential equations and reconstruct the implicitly defined shape. During the model deformation process, the model can always maintain its regularity and can properly modify its topology when collisions between different parts of the model occur. With augmented haptics functionality and stereoscopic display, our system provides a more intuitive interface, which allows users to directly manipulate 3D polygonal objects with hands.

Abstract This paper presents HapticFlow, a haptics‐based direct mesh editing system founded upon the concept of PDE‐based geometric surface flow. The proposed flow‐based approach for direct geometric manipulation offers a unified design paradigm that can seamlessly integrate implicit, distance‐field based shape modeling with dynamic, physics‐based shape design. HapticFlow provides an intuitive haptic interface and allows users to directly manipulate 3D polygonal objects with ease. To demonstrate the effectiveness of our new approach, we developed a variety of haptics‐based mesh editing operations such as embossing, engraving, sketching as well as force‐based shape manipulation operations. Copyright © 2004 John Wiley & Sons, Ltd.