3D window details for buildings are important in many 3D simulation and visualization applications. However, they are not easy to acquire or reconstruct. Thus, many 3D city models have no 3D windows, but only 2D planar textures for their façades (i.e., textured LoD2 CityGML models). Many procedural methods have been proposed to generate 3D façade details from images. However, they usually require tedious efforts to create a procedural grammar to achieve desired results, and lack consideration of window semantics which is a useful building property. In this paper, we propose a novel semantics-aware method for adding 3D window details to textured LoD2 CityGML models. We propose a two-level deep learning-based windowpane detection followed by processing and adjusting the detection results then generating and adding 3D windows to the building models. Different from existing methods, we focus on adding window details considering the semantics (i.e., frames and panes). Moreover, our method does not require tedious reconstruction or grammar creation efforts. It extracts the information present in the texture itself only, finds and adjusts the patterns and shapes from the detection results in an unsupervised and efficient manner to achieve neat window parsing results. Specifically, we propose clustering-based window/pane alignment, neatness-based window image voting, grid-based symmetry and thickness filtering, and fitting-based window-top modeling. Experiments on representative 3D city datasets and illustrative applications demonstrate the effectiveness and usefulness of our method.

Abstract Line drawing with colorization is a popular art format and tool for architectural illustration. The goal of this research is toward generating a high-quality and natural-looking colorization based on an architectural line drawing. This paper presents a new Generative Adversarial Network (GAN)-based method, named ArchGANs, including ArchColGAN and ArchShdGAN. ArchColGAN is a GAN-based line-feature-aware network for stylized colorization generation. ArchShdGAN is a lighting effects generation network, from which the building depiction in 3D can benefit. In particular, ArchColGAN is able to maintain the important line features and the correlation property of building parts as well as reduce the uneven colorization caused by sparse lines. Moreover, we proposed a color enhancement method to further improve ArchColGAN. Besides the single line drawing images, we also extend our method to handle line drawing image sequences and achieve rotation animation. Experiments and studies demonstrate the effectiveness and usefulness of our proposed method for colorization prototyping.

Abstract We propose a high-performance texture streaming system for real-time rendering of large 3D cities with millions of textures. Our main contribution is a texture streaming system that automatically adjusts the streaming workload at runtime based on measured frame latencies, specifically addressing the high memory binding costs of hardware virtual texturing which causes frame rate stuttering. Our system streams textures in parallel with prioritization based on GPU computed mesh perceptibility, and these textures are cached in a sparse partially resident image at runtime without the need for a texture preprocessing step. In addition, we improve rendering quality by minimizing texture pop-in artifacts using a color blending scheme based on mipmap levels. We evaluate our texture streaming system using three structurally distinct datasets with many textures and compared it to a baseline, a game engine, and our prior method. Results show an 8X improvement in rendering performance and 7X improvement in rendering quality compared to the baseline.

Anomaly detection is a challenging task in data analysis, especially when it comes to unsupervised pixel-level segmentation of anomalies in images. In this paper, we present a novel multi-stage defect repaired image resynthesis framework for the detection and segmentation of anomalies in images. In contrast to the existing reconstruction-based approaches, our reconstruction is free from artifacts caused by defective regions so that the defects can be identified from the residual map between input samples and their resynthesized defect-eliminated outputs. Our method outperforms the state-of-art benchmarks in most categories using the publicly available MVTec dataset. Besides, the method also demonstrates an excellent capability of repairing defects in abnormal samples.

Artificial intelligence is being increasingly applied in manufacturing to maximize industrial productivity. Image clustering, as a fundamental research direction in unsupervised learning, has been used in various fields. Since no label information is required in clustering, it can perform a preliminary analysis of the data while saving lots of manpower. In this paper, we propose a novel end-to-end clustering network called Self-supervised Pairing Image Clustering (SPIC) for industrial application, which produces clustering prediction for input images in an advanced pair classification network. For training this network, a self-supervised pairing module is built to form balanced pairs accurately and efficiently without label information. Since the existence of trivial solutions cannot be avoided in most of unsupervised learning methods, two additional information theoretic-constraints regularize the training that ensures the clustering prediction to be unambiguous and close to the real data distribution during training. Experimental results indicate that the proposed SPIC outperforms the state-of-art approaches on manufacturing datasets-NEU and DAGM. It also shows the execellent generalization capability on other genral public datasets, such as MNIST, Omniglot, CIFAR10, and CIFAR100.

Architectural illustration using line drawing with colorization is an important tool and art format. In this paper, in order to generate a natural-looking and high quality watercolorlike colorization for architectural line drawing, we propose a novel Generative Adversarial Network (GAN) approach, namely ArchGANs. The proposed ArchGANs unifies a line-feature-aware stylized colorization network (ArchColGAN), which can learn, predict and generate the coloring based on a dataset, as well as a shading generation network (ArchShdGAN), which augments the illustration with controllable lighting effects for better depicting building in 3D. Specifically, ArchColGAN can preserve the essential line features and building part correlation property, it also tackles the uneven colorization problem caused by the sparse lines. Experimental results demonstrate our proposed method is effective and suitable for colorization prototyping.

We introduce a novel, high performing, bandwidth-aware texture streaming system for progressive texturing of buildings in large 3D cities, with optional texture pre-processing. We seek to maintain high and consistent texture streaming performance across different city datasets, and to address the high memory binding latency in hardware virtual textures. We adopt the sparse partially-resident image to cache mesh textures at runtime and propose to allocate memory persistently, based on mesh visibility weightings and estimated GPU bandwidth. We also retain high quality rendering by minimizing texture pop-ins when transitioning between texture mipmaps. We evaluate our texture streaming system on large city datasets, including a tile-based dataset with 56K large atlases and a dataset containing 5.7M individual textures. Results indicate fast and robust streaming and rendering performance with minimal pop-in artifacts suitable for real-time rendering of large 3D cities.

In many 3D applications, building models in polygon-soup representation are commonly used for the purposes of visualization, for example, in movies and games. Their appearances are fine, however geometry-wise, they may have limited information of connectivity and may have internal intersections between their parts. Therefore, they are not well-suited to be directly used in 3D geospatial applications, which usually require geometric analysis. For an input building model in polygon-soup representation, we propose a novel appearance-driven approach to interactively convert it to a two-manifold model, which is more well-suited for 3D geospatial applications. In addition, the level of detail (LOD) can be controlled interactively during the conversion. Because a model in polygon-soup representation is not well-suited for geometric analysis, the main idea of the proposed method is extracting the visual appearance of the input building model and utilizing it to facilitate the conversion and LODs generation. The silhouettes are extracted and used to identify the features of the building. After this, according to the locations of these features, horizontal cross-sections are generated. We then connect two adjacent horizontal cross-sections to reconstruct the building. We control the LOD by processing the features on the silhouettes and horizontal cross-sections using a 2D approach. We also propose facilitating the conversion and LOD control by integrating a variety of rasterization methods. The results of our experiments demonstrate the effectiveness of our method.

This paper proposes an integrated detection framework of solder joint defects in the context of Automatic Optical Inspection (AOI) of Printed Circuit Boards (PCBs). Both localization and classifications tasks were considered. For the localization part, in contrast to the existing methods that are highly specified for particular PCBs, we used a generic deep learning method which can be easily ported to different configurations of PCBs and soldering technologies and also gives real-time speed and high accuracy. For the classification part, an active learning method was proposed to reduce the labeling workload when a large labeled training database is not easily available because it requires domain-specified knowledge. The experiments show that the localization method is fast and accurate. In addition, high accuracy with only minimal user input was achieved in the classification framework on two different datasets. The results also demonstrated that our method outperforms three other active learning benchmarks.

Automatic optical inspection for manufacturing traditionally was based on computer vision. However, there are emerging attempts to do it using deep learning approach. Deep convolutional neural network allows to learn semantic image features which could be used for defect detection in products. In contrast to the existing approaches where supervised or semi-supervised training is done on thousands of images of defects, we investigate whether unsupervised deep learning model for defect detection could be trained with orders of magnitude smaller amount of representative defect-free samples (tenths rather than thousands). This research is motivated by the fact that collection of large amounts of defective samples is difficult and expensive. Our model undergoes only one-class training and aims to extract distinctive semantic features from the normal samples in an unsupervised manner. We propose a variant of transfer learning, that consists of combination of unsupervised learning used upon VGG16 with pre-trained on ImageNet weight coefficients. To demonstrate a defect detection, we used a set of Printed Circuit Boards (PCBs) with different types of defects - scratch, missing washer/extra hole, abrasion, broken PCB edge. The trained model allows us to make clusters of normal internal representations of features of PCB in high-dimensional feature space, and to localize defective patches in PCB image based on distance from normal clusters. Initial results show that more than 90% of defects were detected.

Detecting defects is an integral part of any manufacturing process. Most works still utilize traditional image processing algorithms to detect defects owing to the complexity and variety of products and manufacturing environments . In this paper, we propose an approach based on deep learning which uses autoencoders for extraction of discriminative features . It can detect different defects without using any defect samples during training. This method, where samples of only one class (i.e. defect-free samples) are available for training, is called One Class Classification (OCC). This OCC method can also be used for training a neural network when only one golden sample is available by generating many copies of the reference image by data augmentation . The trained model is then able to generate a descriptor—a unique feature vector of an input image. A test image captured by an Automatic Optical Inspection (AOI) camera is sent to the trained model to generate a test descriptor, which is compared with a reference descriptor to obtain a similarity score. After comparing the results of this method with a popular traditional similarity matching method SIFT , we find that in the most cases this approach is more effective and more flexible than the traditional image processing-based methods, and it can be used to detect different types of defects with minimum customization .

Polygon soup building models are fine for visualization purposes such as in games and movies. They, however, are not suitable for 3D geospatial applications which require geometrical analysis, since they lack connectivity information and may contain intersections internally between their parts. In this paper, we propose an appearance-driven method to interactively convert an input polygon soup building model to a two-manifold mesh, which is more suitable for 3D geospatial applications. Since a polygon soup model is not suitable for geometrical analysis, our key idea is to extract and utilize the visual appearance of the input building model for the conversion. We extract the silhouettes and use them to identify the features of the building. We then generate horizontal cross sections based on the locations of the features and then reconstruct the building by connecting two neighbouring cross sections. We propose to integrate various rasterization techniques to facilitate the conversion. Experimental results show the effectiveness of the proposed method.

This paper proposes a method for automatic image-based classification of solder joint defects in the context of Automatic Optical Inspection (AOI) of Printed Circuit Boards (PCBs). Machine learning-based approaches are frequently used for image-based inspection. However, a main challenge is to manually create sufficiently large labeled training databases to allow for high accuracy of defect detection. Creating such large training databases is time-consuming, expensive, and often unfeasible in industrial production settings. In order to address this problem, an active learning framework is proposed which starts with only a small labeled subset of training data. The labeled dataset is then enlarged step-by-step by combining K-means clustering with active user input to provide representative samples for the training of an SVM classifier. Evaluations on two databases with insufficient and shifting solder joints samples have shown that the proposed method achieved high accuracy while requiring only minimal user input. The results also demonstrated that the proposed method outperforms random and representative sampling by ~ 3.2% and ~ 2.7%, respectively, and it outperforms the uncertainty sampling method by ~ 0.5%.

Surface level defect detection, such as detecting missing components, misalignments and physical damages, is an important step in any manufacturing process. In this paper, similarity matching techniques for manufacturing defect detection are discussed. We are proposing an algorithm which detects surface level defects without relying on the availability of defect samples for training. Furthermore, we are also proposing a method which works when only one or a few reference images are available. It implements a deep autoencoder network and trains input reference image(s) along with various copies automatically generated by data augmentation. The trained network is then able to generate a descriptor-a unique signature of the reference image. After training, a test image of the same product is sent to the trained network to generate a test image descriptor. By matching the reference and test descriptors, a similarity score is generated which indicates if a defect is found. Our experiments show that this approach is more generic than traditional hand-engineered feature extraction methods and it can be applied to detect multiple type of defects.

This paper outlines the major components and function of the technologically integrated oncosimulator developed primarily within the Advancing Clinico Genomic Trials on Cancer (ACGT) project. The Oncosimulator is defined as an information technology system simulating in vivo tumor response to therapeutic modalities within the clinical trial context. Chemotherapy in the neoadjuvant setting, according to two real clinical trials concerning nephroblastoma and breast cancer, has been considered. The spatiotemporal simulation module embedded in the Oncosimulator is based on the multiscale, predominantly top-down, discrete entity-discrete event cancer simulation technique developed by the In Silico Oncology Group, National Technical University of Athens. The technology modules include multiscale data handling, image processing, invocation of code execution via a spreadsheet-inspired environment portal, execution of the code on the grid, and the visualization of the predictions. A refining scenario for the eventual coupling of the oncosimulator with immunological models is also presented. Parameter values have been adapted to multiscale clinical trial data in a consistent way, thus supporting the predictive potential of the oncosimulator. Indicative results demonstrating various aspects of the clinical adaptation and validation process are presented. Completion of these processes is expected to pave the way for the clinical translation of the system.

In this work, we present a system based on open-source toolkits that can utilize multiple phases of a liver CT dataset to plan a surgery before the patient enters the operation room. The contributions are an optimized deformable registration of arterial and venous phases of the liver and a methodology to assess surgical risk utilizing both phases. It is shown by the example of two clinical cases, how our system enables the fusion of complementary information into a 3D representation of the patient anatomy. Processing times of single processing steps on a modern machine are quite low, which allows for an integration into clinical routine.

We present a novel statistical shape model approach for fully automatic CT liver segmentation. Unlike previous techniques, our method combines learned local shape priors with constraints that are directly derived from the current curvature of the model in order to restrict adaptation to regions where large deformations are expected and observed. Our approach is based on a multi-tiered framework that is more robust against model initialization errors than existing methods, because the model's degrees of freedom are step-wise increased. We evaluated our method on a large data base of 86 CT liver scans from different vendors, protocols, varying resolution and contrast enhancement. For comparison, 50 of the scans were taken from 2 public data bases, one of it being the MICCAI'07 liver segmentation challenge data base. Evaluation shows state of the art results with an average mean surface distance between 1.3 mm and 1.85 mm compared to ground truth depending on the image resolution. With an average segmentation time of 45 seconds our approach outperforms other automatic methods.

The assessment of bone mineral density (BMD) in vertebrae is critical for the diagnosis of osteoporosis. Recent developments in dual-source CT allow for the simultaneous acquisition of two image data sets with different X-ray tube energies — dual-energy CT (DECT). We present a comprehensive approach for assessing the density of the trabecular bone in vertebrae of the spine based on DECT image data. For this, we apply and combine methods from different areas: the deformation of a template mesh for delineating the structures of interest, a biophysical model of the trabecular bone for the computation of BMD values, and different visualization approaches for the display of the results. In addition, we investigate the correlation between the computed BMD values with concurrently measured pull-out forces for pedicle screws. We show that there is a linear correlation between both measures and thus, DECT provides correct BMD values for the trabecular bone. We conclude that our approach enables the radiologist to diagnose osteoporosis based on DECT image data which has the potential to replace the current gold standard dual-energy X-ray absorp-tiometry.