Despite significant advancements in deep classification techniques and in-lab automatic optical inspection (AOI) models for long-tailed or highly imbalanced data, applying these approaches to real-world IC defect classification tasks remains challenging. This difficulty stems from two primary factors. First, real-world conditions, such as the high yield-rate requirements in the IC industry, result in data distributions that are far more skewed than those found in general public imbalanced datasets. Consequently, classifiers designed for open imbalanced datasets often fail to perform effectively in real-world scenarios. Second, real-world samples exhibit a mix of class-specific attributes (e.g., defect types) and class-agnostic, domain-related features (e.g., design characteristics of product lines). This complexity adds significant difficulty to the classification process, particularly for highly imbalanced datasets. To address these challenges, this paper introduces the IC-Defect-14 dataset, a large, highly imbalanced IC defect image dataset sourced from AOI systems deployed in real-world IC production lines. This dataset is characterized by its unique “intra-class clusters” property, which presents two major challenges: large intra-class diversity and high inter-class similarity. These characteristics, rarely found simultaneously in existing public datasets, significantly degrade the performance of current state-of-the-art classifiers for highly imbalanced data. To tackle this challenge, we propose the Regional Channel Attention-based Multi-Expert Network (ReCAME-Net). This network follows a multi-expert classifier framework and integrates a regional channel attention module, metric learning losses, a hard category mining strategy, and a knowledge distillation procedure. Extensive experimental evaluations demonstrate that ReCAME-Net outperforms previous state-of-the-art models on the IC-Defect-14 dataset while maintaining comparable performance and competitiveness on general public datasets. Our resources can be found at https://github.com/YoursEver/ReCAME-Net.

With increasing revelations of academic fraud, detecting forged experimental images in the biomedical field has become a public concern. The challenge lies in the fact that copy-move targets can include background tissue, small foreground objects, or both, which may be out of the training domain and subject to unseen attacks, rendering standard object-detection-based approaches less effective. To address this, we reformulate the problem of detecting biomedical copy-move forgery regions as an intra-image co-saliency detection task and propose CMSeg-Net, a copy-move forgery segmentation network capable of identifying unseen duplicated areas. Built on a multi-resolution encoder-decoder architecture, CMSeg-Net incorporates self-correlation and correlation-assisted spatial-attention modules to detect intra-image regional similarities within feature tensors at each observation scale. This design helps distinguish even small copy-move targets in complex microscopic images from other similar objects. Furthermore, we created a copy-move forgery dataset of optical microscopic images, named FakeParaEgg, using open data from the ICIP 2022 Challenge to support CMSeg-Net's development and verify its performance. Extensive experiments demonstrate that our approach outperforms previous state-of-the-art methods on the FakeParaEgg dataset and other open copy-move detection datasets, includingCASIA-CMFD,CoMoFoD, andCMF.

Recent advances in VLSI fabrication technology have led to die shrinkage and increased layout density, creating an urgent demand for advanced hotspot detection techniques. However, by taking an object detection network as the backbone, recent learning-based hotspot detectors learn to recognize only the problematic layout patterns in the training data. This fact makes these hotspot detectors difficult to generalize to real-world scenarios. We propose a novel lithography simulator-powered hotspot detection framework to overcome this difficulty. Our framework integrates a lithography simulator with an object detection backbone, merging the extracted latent features from both the simulator and the object detector via well-designed cross-attention blocks. Consequently, the proposed framework can be used to detect potential hotspot regions based on 1) the variation of possible circuit shape deformation estimated by the lithography simulator and 2) the problematic layout patterns already known. To this end, we utilize RetinaNet with a feature pyramid network as the object detection backbone and leverage LithoNet as the lithography simulator. Extensive experiments demonstrate that our proposed simulator-guided hotspot detection framework outperforms the previous state-of-the-art methods on real-world data.

In this paper, we proposed a fine-grained attribute pre-labeling method based on the multi-label recovery techniques. Given a fine-grained image dataset with overlooked attributes in its annotation vectors, our method can predict those missing attribute labels by learning the between-label dependency based on the estimated similarity between known attributes and the similarity of extracted deep image features. Furthermore, to prevent the learnable label dependency matrix from converging to a trivial solution, we designed a trace-loss to penalize the self-dependency of attributes. Comprehensive experiments on the CUB-200-2011 dataset show that, given a training set with 40% of attribute labels randomly dropped: i) our approach achieves a pre-labeling performance with an mAP value of 30.7 on a blind testing set, and ii) the missing attributes in the training set can be corrected with an accuracy of 89%. Our method can effectively and robustly perform the fine-grained pre-labeling task.

Continual revelations of academic fraud have raised concerns regarding the detection of forged experimental images in the public domain. To address this issue, we introduce a multiscale attention-based deep (MAD) descriptor scheme for detecting copy-move image forgery in biomedical research scenarios. Our method utilizes the common object detection network as a backbone and incorporates the positional embedding module, the channel-attention module, and the self-attention module to generate a dense feature field for input images. The proposed method demonstrates robustness against common attacks encountered during research manuscript preparation, low-contrast biomedical images featuring small foreground objects, and unseen or unlearned object patterns. Extensive experiments substantiate that our approach outperforms previous copy-move forgery methods when applied to real-world cases across various domains. Our proposed method can serve as an efficient screening tool for the rapid identification of biomedical image forgeries.

With continuous shrinking of process nodes, semiconductor manufacturing encounters more and more serious inconsistency between designed layout patterns and resulted wafer images. Conventionally, examining how a layout pattern can deviate from its original after complicated process steps, such as optical lithography and subsequent etching, relies on computationally expensive process simulation, which suffers from incredibly long runtime for large-scale circuit layouts, especially in advanced nodes. In addition, being one of the most important and commonly adopted resolution enhancement techniques, optical proximity correction (OPC) corrects image errors due to process effects by moving segment edges or adding extra polygons to mask patterns, while it is generally driven by simulation or time-consuming inverse lithography techniques (ILTs) to achieve acceptable accuracy. As a result, more and more state-of-the-art works on process simulation or/and OPC resort to the fast inference characteristic of machine/deep learning. This paper reviews these data-driven approaches to highlight the challenges in various aspects, explore preliminary solutions, and reveal possible future directions to push forward the frontiers of the research in design for manufacturability.

Learning-based presimulation (i.e., layout-to-fabrication) models have been proposed to predict the fabrication-induced shape deformation from an IC layout to its fabricated circuit. Such models are usually driven by pairwise learning, involving a training set of layout patterns and their reference shape images after fabrication. However, it is expensive and time consuming to collect the reference shape images of all layout clips for model training and updating. To address the problem, we propose a deep-learning-based layout novelty detection scheme to identify novel (unseen) layout patterns, which cannot be well predicted by a pretrained presimulation model. We devise a global–local novelty scoring mechanism to assess the potential novelty of a layout by exploiting two subnetworks: 1) an autoencoder and 2) a pretrained presimulation model. The former characterizes the global structural dissimilarity between a given layout and training samples, whereas the latter extracts a latent code representing the fabrication-induced local deformation. By integrating the global dissimilarity with the local deformation boosted by a self-attention mechanism, our model can accurately detect novelties without the ground-truth circuit shapes of test samples. Based on the detected novelties, we further propose two active-learning strategies to sample a reduced amount of representative layouts most worthy to be fabricated for acquiring their ground-truth circuit shapes. Experimental results demonstrate: 1) the effectiveness of our layout novelty detection algorithm and 2) the ability of our active-learning strategies in selecting representative novel layouts for keeping a learning-based presimulation model updated.

Due to the high labor cost of physicians, it is difficult to collect a rich amount of manually-labeled medical images for developing learning-based computer-aided diagnosis (CADx) systems or segmentation algorithms. To tackle this issue, we reshape the image segmentation task as an image-to-image (I2I) translation problem and propose a retinal vascular segmentation network, which can achieve good cross-domain generalizability even with a small amount of training data. We devise primarily two components to facilitate this I2I-based segmentation method. The first is the constraints provided by the proposed gradient-vector-flow (GVF) loss, and, the second is a two-stage Unet (2Unet) generator with a skip connection. This configuration makes 2Unet's first-stage play a role similar to conventional Unet, but forces 2Unet's second stage to learn to be a refinement module. Extensive experiments show that by re-casting retinal vessel segmentation as an image-to-image translation problem, our I2I translator-based segmentation subnetwork achieves better cross-domain generalizability than existing segmentation methods. Our model, trained on one dataset, e.g., DRIVE, can produce segmentation results stably on datasets of other domains, e.g., CHASE-DB1, STARE, HRF, and DIARETDB1, even in low-shot circumstances.

We propose an adversarial training framework to simultaneously address the vessel segmentation and dirt/reflection removal problems in fundus photographs used for diabetic retinopathy diagnosis. This framework contains two primary subnetworks, each triggered by a set of loss terms, i.e., one for segmentation and the other for reconstruction. These two subnetworks act as inverse functions of each other so that they form an autoencoder framework with a 2-dimensional latent code, which can be a vessel segmentation mask after binarization. To further improve the segmentation and reconstruction performance, we devise a loss function based on gradient vector flow (GVF) and reorganize the generator network. Experimental results show that the proposed method has a good generalization capability. Trained on DRIVE’s training set, our model can produce segmentation and reconstruction-based artifact removal results stably on other datasets like CHASE-DB1 and STARE. The average F1-score of our segmentation results of DRIVE’s testing set reaches 0.7964, and the artifact-free reconstruction results can achieve an average PSNR of 24 dB.

Few-shot learning strategies are developed for training a reliable model on even a limited amount of data, but few-shot learning tasks usually lead to the over-fitting dilemma and result in a task-level inductive bias. In contrast to conventional few-shot learning techniques following the meta-learning framework design, recent few-shot learning studies aim to derive a reliable feature extractors via a self-supervised learning mechanism for solving the dilemma. Therefore, we proposed in this paper a task-aware few-shot visual classification framework by articulating meta-learning, traditional supervised classification, and self-supervised learning schemes. The proposed mechanism learns to transform an initial feature embedding into a more general and representative space so that classification performance can be boosted. Extensive experiments show that the proposed method can solve the over-fitting dilemma and outperforms previous state-of-the-art few-shot learning methods.

Label noise in training data can significantly degrade a model’s generalization performance for supervised learning tasks. Here we focus on the problem that noisy labels are primarily caused by mislabeled confusing samples, which tend to be concentrated near decision boundaries rather than uniformly distributed, and whose features should be equivocal. To address the problem, we propose an ensemble learning method to correct noisy labels by exploiting the local structures of feature manifolds. Different from typical ensemble strategies that increase the prediction diversity among sub-models via certain loss terms, our method trains sub-models on disjoint subsets, each being a union of randomly selected seed samples’ nearest-neighbors of the same class on the data manifold. As a result, only a limited number of sub-models will be affected by locally-concentrated noisy labels, and each sub-model can learn a coarse representation of the data manifold along with a corresponding graph. The constructed graphs are used to suggest a set of label correction candidates, and accordingly, our method determines label correction results by majority decisions. Our experiments on real-world noisy label datasets demonstrate the superiority of the proposed method over existing state-of-the-arts.

We propose a deep learning-based data-driven framework consisting of two convolutional neural networks: 1) LithoNet that predicts the shape deformations on a circuit due to IC fabrication and 2) OPCNet that suggests IC layout corrections to compensate for such shape deformations. By learning the shape correspondences between pairs of layout design patterns and their scanning electron microscope (SEM) images of the product wafer thereof, given an IC layout pattern, LithoNet can mimic the fabrication process to predict its fabricated circuit shape. Furthermore, LithoNet can take the wafer fabrication parameters as a latent vector to model the parametric product variations that can be inspected on SEM images. Besides, traditional optical proximity correction (OPC) methods used to suggest a correction on a lithographic photomask is computationally expensive. Our proposed OPCNet mimics the OPC procedure and efficiently generates a corrected photomask by collaborating with LithoNet to examine if the shape of a fabricated circuit optimally matches its original layout design. As a result, the proposed LithoNet-OPCNet framework can not only predict the shape of a fabricated IC from its layout pattern but also suggests a layout correction according to the consistency between the predicted shape and the given layout. Experimental results with several benchmark layout patterns demonstrate the effectiveness of the proposed method.

The performance of a convolutional neural network (CNN) based face recognition model largely relies on the richness of labeled training data. However, it is expensive to collect a training set with large variations of a face identity under different poses and illumination changes, so the diversity of within-class face images becomes a critical issue in practice. In this paper, we propose a 3D model-assisted domain-transferred face augmentation network (DotFAN) that can generate a series of variants of an input face based on the knowledge distilled from existing rich face datasets of other domains. Extending from StarGAN's architecture, DotFAN integrates with two additional subnetworks, i.e., face expert model (FEM) and face shape regressor (FSR), for latent facial code control. While FSR aims to extract face attributes, FEM is designed to capture a face identity. With their aid, DotFAN can separately learn facial feature codes and effectively generate face images of various facial attributes while keeping the identity of augmented faces unaltered. Experiments show that DotFAN is beneficial for augmenting small face datasets to improve their within-class diversity so that a better face recognition model can be learned from the augmented dataset.

In this paper, we propose a method of reconstructing the depth map of a set of multiview images from the epipolar plane images (EPIs) of multiview Images. Our method involves two steps: finding support points and estimating depth. First, we propose to include a consistency term and a smoothness term in the objective function for edge point detection, where the consistency term is used to identify edge points and the smoothness term is applied to mitigate false edge detection due to light density variations caused by viewpoint changes. Then, based on the detected edge points, a depth map can be estimated by solving a energy minimization problem, in which a line uniformness term and a matching error term are introduced to ensure the line traces estimated from EPIs for depth estimation match the colors of edge points well. The depths of non-edge points are then estimated by introducing an additional prior term. In order to speed up our algorithm, the depth estimation problem is aggregated by a winner-take-all strategy. Experiments show that our method outperforms the state-of-the-art schemes in reconstructing depth map with fine details.

In order to understand olfactory coding and functional connectome within the fly brain, scientists need not only a 3D stereotypical brain atlas but also anatomical local landmarks to describe hard-wiring circuits of olfactory neurons. Olfactory glomeruli are such indispensable local landmarks; however, it is hard to segment them from confocal microscopy images. We propose in this paper a systematic approach for semi-automatic olfactory glomerulus segmentation. This method consists of two phases. The former phase aims to highlight within-glomerulus regions, appraise the amount of recognizable glomeruli in each z-slice, and to suggest a seed used to generate an initial contour for every recognizable glomerulus. The latter phase, starting from the initial contour of each recognizable glomerulus, propagates the initial contour information to adjacent slices recursively until the glomerulus' contours on all z-slices are obtained. Experiment results shows that our results are similar to those of manual segmentation. The proposed strategy is effective.

Multi-resolution and wavelet analysis have generated considerable interest in the field of mesh surface representation. In this paper, we propose a backward, coarse-to-fine framework that derives a semi-regular approximation of an original mesh, and demonstrate its effectiveness on level-of-detail and scalable coding applications. The framework is flexible and simple because the position of a new vertex at a finer resolution can be derived in a closed form, based on the affine combination of a subdivision scheme, the original mesh, and “new” information about the wavelet coefficients. We report the results of experiments on both applications; and also compare the scalable coding results with those of other methods.

With the progress of model averaging algorithms, scientists in the field of brain research have an increasing demand for methods capable to register and warp source data to the pre-registered standard atlas. We here propose a thin-plate spline (TPS) based surface registration method to facilitate the registration and warping process of Drosophila brain data. Our contributions are twofold. First, the proposed method performs TPS-based registration in the parameterization domain, and hence it no longer needs a rigid transformation to globally align and scale the input models. Second, the obtained well-registered surface model can act as boundary constraints for further volumetric registration schemes. Experiments show that the proposed method is effective. For models with a 750-voxel-long bounding box diagonal, the average surface-to-surface distance is reduced to about 0.1-voxel-long after registration.

Recently developed were the Brainbow and Flybow techniques that can image and visualize a large number of neurons simultaneously; however, scientists still lack adequate tools to process this kind of colored multi-neuron image volumes. Due to dozens of colorized neuron fibers spreading densely in a very intricate structure, it is difficult to trace them by existing algorithms designed for single-neuron images. We proposed a framework to formulate and solve this issue, and the experimental results show that our method can successfully extract independent neurons from Flybow images. Consequently, the proposed procedure contributes to neuroscience by increasing the efficiency of collecting neuron information from Flybow images.