This paper is directed towards the food crystal quality control area for manufacturing, focusing on efficiently predicting food crystal counts and size distributions. Previously, manufacturers used the manual counting method on microscopic images of food liquid products, which requires substantial human effort and suffers from inconsistency issues. Food crystal segmentation is a challenging problem due to the diverse shapes of crystals and their surrounding hard mimics. To address this challenge, we propose an efficient instance segmentation method based on object detection. Experimental results show that the predicted crystal counting accuracy of our method is comparable with existing segmentation methods, while being five times faster. Based on our experiments, we also define objective criteria for separating hard mimics and food crystals, which could benefit manual annotation tasks on similar dataset.

We introduce a novel method for interpolating views between two fixed cameras to create a free-viewpoint experience comparable to advanced telepresence systems. Inspired by Video Frame Interpolation (VFI) works, our proposed method spatially interpolates between camera views, treating our setup as a unique frame interpolation scenario. Due to the occlusion that exists in areas like the face contour from two cameras with a long baseline, which results in serious artifacts with previous VFI methods. To mitigate this issue, we integrate face pose information of pitch, yaw and roll angles in our method as a robust guidance prior to improve pixel mapping in occluded regions. Furthermore, we synthesize a large portrait-focused multi-view dataset facilitating training of our proposed model. Our multi-stage flow refinement model progressively refines the quality of the bi-directional flows which eventually results in improved interpolated view details. With the flexible setup and fast inference speed of our proposed method, a practical and cost-effective solution can be implemented for telepresence systems without the need for expensive hardware or extensive computational resources.

Denoising and super-resolution are two important tasks for video enhancement. Despite recent progress for each task, there are very few works that target both tasks simultaneously. In this paper, we propose an efficient noise-robust video super-resolution method that is trained end-to-end for an input video containing observable noises. We investigate current approaches to address this joint denoising and super-resolution task and compare them to our proposed method. Experimental results show that our method achieves competitive reconstruction performance with existing solutions on various datasets while maintaining a low computation cost and a small model size which prove the effectiveness of our joint model design and training. Our code is available at "https://github.com/Eventhyn/EVDSRNet.".

This paper looks at semi-supervised learning (SSL) for image-based text recognition. One of the most popular SSL approaches is pseudo-labeling (PL). PL approaches assign labels to unlabeled data before re-training the model with a combination of labeled and pseudo-labeled data. However, PL methods are severely degraded by noise and are prone to over-fitting to noisy labels, due to the inclusion of erroneous high confidence pseudo-labels generated from poorly calibrated models, thus, rendering threshold-based selection ineffective. Moreover, the combinatorial complexity of the hypothesis space and the error accumulation due to multiple incorrect autoregressive steps posit pseudo-labeling challenging for sequence models. To this end, we propose a pseudo-label generation and an uncertainty-based data selection framework for semi-supervised text recognition. We first use Beam-Search inference to yield highly probable hypotheses to assign pseudo-labels to the unlabelled examples. Then we adopt an ensemble of models, sampled by applying dropout, to obtain a robust estimate of the uncertainty associated with the prediction, considering both the character-level and word-level predictive distribution to select good quality pseudo-labels. Extensive experiments on several benchmark handwriting and scene-text datasets show that our method outperforms the baseline approaches and the previous state-of- the-art semi-supervised text-recognition methods.

A promising direction for recovering the lost information in low-resolution headshot images is utilizing a set of high-resolution exemplars from the same identity. Complementary images in the reference set can improve the generated headshot quality across many different views and poses. However, it is challenging to make the best use of multiple exemplars: the quality and alignment of each exemplar cannot be guaranteed. Using low-quality and mismatched images as references will impair the output results. To overcome these issues, we propose the Headshot Image Super-Resolution with Multiple Exemplars network (HIME) method. Compared with previous methods, our network can effectively handle the misalignment between the input and the reference without requiring facial priors and learn the aggregated reference set representation in an end-to-end manner. Furthermore, to reconstruct more detailed facial features, we propose a correlation loss that provides a rich representation of the local texture in a controllable spatial range. Experimental results demonstrate that the proposed framework not only has significantly fewer computation cost than recent exemplar-guided methods but also achieves better qualitative and quantitative performance.

A novel statistical ink drop displacement (IDD) printer model for the direct binary search (DBS) halftoning algorithm is proposed. It is intended primarily for pagewide inkjet printers that exhibit dot displacement errors. The tabular approach in the literature predicts the gray value of a printed pixel based on the halftone pattern in some neighborhood of that pixel. However, memory retrieval time and the complexity of memory requirements hamper its feasibility in printers that have a very large number of nozzles and produce ink drops that affect a large neighborhood. To avoid this problem, our IDD model embodies dot displacements by moving each perceived ink drop in the image from its nominal location to its actual location, rather than manipulating the average gray values. This enables DBS to directly compute the appearance of the final printout without retrieving values from a table. In so doing, the memory issue is eliminated and the computation efficiency is enhanced. The deterministic cost function of DBS is replaced by the expectation over the ensemble of the displacements for the proposed model such that the statistical behavior of the ink drops is accounted for. Experimental results show significant improvement in the quality of the printed image over the original DBS. Besides, the image quality obtained by the proposed approach appears to be slightly better than that obtained by the tabular approach.

Label-free cell imaging, where the cell is not "labeled" or modified by fluorescent chemicals, is an important research area in the field of biology. It avoids altering the cell’s properties which typically happens in the process of chemical labeling. However, without the contrast enhancement from the label, the analysis of label-free imaging is more challenging than label-based imaging. In addition, it provides few human interpretable features, and thus needs machine learning approaches to help with the identification and tracking of specific cells. We are interested in label-free phase contrast imaging to track cells flowing in a cell sorting device where images are acquired at 500 frames/s. Existing Multiple Object Tracking (MOT) methods face four major challenges when used for tracking cells in a microfluidic sorting device: (i) most of the cells have large displacements between frames without any overlap; (ii) it is difficult to distinguish between cells as they are visually similar to each other; (iii) the velocities of cells vary with the location in the device; (iv) the appearance of cells may change as they move in and out of the focal plane of the imaging sensor that observes the isolation process. In this paper, we introduce a method for tracking cells in a predefined flow in the sorting device via phase contrast microscopy. Our proposed method is based on DeepSORT and YOLOv4 and exploits prior knowledge of a cell’s velocity to assist tracking. We modify the Kalman filter in DeepSORT to accommodate a non-constant velocity motion model and integrate a representative velocity field obtained from fluid dynamics into the Kalman filter. The experimental results show that our proposed method outperforms several MOT methods for tracking cells in the sorting device.

Image re-composition has always been regarded as one of the most important steps during the post-processing of a photo. The quality of an image re-composition mainly depends on a person’s taste in aesthetics, which is not an effortless task for those who have no abundant experience in photography. Besides, while re-composing one image does not require much of a person’s time, it could be quite time-consuming when there are hundreds of images to be recomposed. To solve these problems, we propose a method that automates the process of re-composing an image to the desired aspect ratio. Although there already exist many image re-composition methods, they only provide a score to their predicted best crop but fail to explain why the score is high or low. Conversely, we succeed in designing an explainable method by introducing a novel 10-layer aesthetic score map, which represents how the position of the saliency in the original uncropped image, relative to that of the crop region, contributes to the overall score of the crop, so that the crop is not just represented by a single score. We conducted experiments to show that the proposed score map boosts the performance of our algorithm, which achieves a state-of-the-art performance on both public and our own datasets.

In this paper, we explore open-domain sketch-to-photo translation, which aims to synthesize a realistic photo from a freehand sketch with its class label, even if the sketches of that class are missing in the training data. It is challenging due to the lack of training supervision and the large geometric distortion between the freehand sketch and photo domains. To synthesize the absent freehand sketches from photos, we propose a framework that jointly learns sketch-to-photo and photo-to-sketch generation. However, the generator trained from fake sketches might lead to unsatisfying results when dealing with sketches of missing classes, due to the domain gap between synthesized sketches and real ones. To alleviate this issue, we further propose a simple yet effective open-domain sampling and optimization strategy to "fool" the generator into treating fake sketches as real ones. Our method takes advantage of the learned sketch-to-photo and photo-to-sketch mapping of in-domain data and generalizes it to the open-domain classes. We validate our method on the Scribble and SketchyCOCO datasets. Compared with the recent competing methods, our approach shows impressive results in synthesizing realistic color, texture, and maintaining the geometric composition for various categories of open-domain sketches.

Aperiodic, clustered-dot, halftone patterns have recently become popular for commercial printing of continuous-tone images with laser, electrophotographic presses, because of their inherent stability and resistance to moiré artifacts. Halftone screens designed using the multistage, multipass, clustered direct binary search (MS-MP-CLU-DBS) algorithm can yield halftone patterns with very high visual quality. But the characteristics of these halftone patterns depend on three input parameters for which there are no known formulas to choose their values to yield halftone patterns of a certain quality level and scale. Using machine learning methods, two predictors are developed that take as input these three parameters. One predicts the quality level of the halftone pattern. The other one predicts the scale of the halftone pattern. To provide ground truth information for training these predictors, human subjects viewed a large number of halftone patches generated from MS-MP-CLU-DBS-designed screens and assigned each patch to one of four quality levels. For each patch, the location of the peak in the radially averaged power spectrum (RAPS) is calculated as a measure of the scale or effective line frequency of the pattern. Experimental results demonstrate the accuracy of the two predictors and the effectiveness of screen design procedures based on these predictors to generate both monochrome and color high quality halftone images.

Aperiodic, clustered-dot halftoning can overcome the inherent instability of electrophotographic (EP) printing technologies. In our previous work, we described a screen design algorithm based on Direct Binary Search (DBS) that yields aperiodic, clustered-dot halftone textures [1]. In the present paper, we introduce a new microscreen design algorithm that allows the embedding of binary metadata in an aperiodic, clustered-dot halftone image with no loss of visual quality. We also describe a simple method for decoding the embedded metadata with only a knowledge of the two microscreens used to halftone the continuous-tone image. Our results with one image demonstrate 100% recovery of the embedded metadata.

In this paper, we explore the space-time video super-resolution task, which aims to generate a high-resolution (HR) slow-motion video from a low frame rate (LFR), low-resolution (LR) video. A simple solution is to split it into two sub-tasks: video frame interpolation (VFI) and video super-resolution (VSR). However, temporal interpolation and spatial super-resolution are intra-related in this task. Two-stage methods cannot fully take advantage of the natural property. In addition, state-of-the-art VFI or VSR networks require a large frame-synthesis or reconstruction module for predicting high-quality video frames, which makes the two-stage methods have large model sizes and thus be time-consuming. To overcome the problems, we propose a one-stage space-time video super-resolution framework, which directly synthesizes an HR slow-motion video from an LFR, LR video. Rather than synthesizing missing LR video frames as VFI networks do, we firstly temporally interpolate LR frame features in missing LR video frames capturing local temporal contexts by the proposed feature temporal interpolation network. Then, we propose a deformable ConvLSTM to align and aggregate temporal information simultaneously for better leveraging global temporal contexts. Finally, a deep reconstruction network is adopted to predict HR slow-motion video frames. Extensive experiments on benchmark datasets demonstrate that the proposed method not only achieves better quantitative and qualitative performance but also is more than three times faster than recent two-stage state-of-the-art methods, e.g., DAIN+EDVR and DAIN+RBPN.

Graininess noise is a common artifact in inkjet printing. While current inkjet printing technologies attempt to control graininess in single color images, the results are often less than optimal for multi-color images. This is due to fluidic interactions between inks of different colors. This paper will describe a color decomposition methodology that can be used to study ink flow patterns in multi-color inkjet printed images at a microscopic scale. This technique is used to decompose multi-color images into several independent color components. The ink patterns in these components are analyzed to relate them to visually perceptible graininess noise.

Digital halftoning is an essential part of the process for printing color, continuous-tone content. Traditionally, the highest quality has been achieved with analog, offset lithographic presses, using color screen sets that yield periodic, clustered-dot halftone patterns. Increasingly, these systems are being supplanted by digital presses that are based on either electrophotographic or inkjet marking processes. Due to the inherent instability of the electrophotographic marking process, periodic, clustered-dot halftone patterns are also widely used with such presses. However, digital presses have much lower resolution than their analog counterparts. Simply mimicking the traditional screen designs used with commercial, offset presses will result in halftone patterns that are more susceptible to moire due to the interaction between the periodic patterns used to render the different color channels. The moire can be reduced by increasing the frequency of the halftone patterns. But this may make the print appear grainier than its analog counterpart. In this paper, we introduce a principled design procedure that allows one to design color screen sets that generate periodic, clustered-dot halftone patterns that improve color stability without increasing graininess.

Due to the limits of bandwidth and storage space, digital images are usually down-scaled and compressed when transmitted over networks, resulting in loss of details and jarring artifacts that can lower the performance of high-level visual tasks. In this paper, we aim to generate an artifact-free high-resolution image from a low-resolution one compressed with an arbitrary quality factor by exploring joint compression artifacts reduction (CAR) and super-resolution (SR) tasks. First, we propose a context-aware joint CAR and SR neural network (CAJNN) that integrates both local and non-local features to solve CAR and SR in one-stage. Finally, a deep reconstruction network is adopted to predict high quality and high-resolution images. Evaluation on CAR and SR benchmark datasets shows that our CAJNN model outperforms previous methods and also takes 26.2% shorter runtime. Based on this model, we explore addressing two critical challenges in high-level computer vision: optical character recognition of low-resolution texts, and extremely tiny face detection. We demonstrate that CAJNN can serve as an effective image preprocessing method and improve the accuracy for real-scene text recognition (from 85.30% to 85.75%) and the average precision for tiny face detection (from 0.317 to 0.611).

Digital halftoning is an essential part of the process for printing color, continuous-tone content. Traditionally, the highest quality has been achieved with analog, offset lithographic presses, using color screen sets that yield periodic, clustereddot halftone patterns. Increasingly, these systems are being supplanted by digital presses that are based on either electrophotographic or inkjet marking processes. Due to the inherent instability of the electrophotographic marking process, periodic, clustered-dot halftone patterns are also widely used with such presses. However, digital presses have much lower resolution than their analog counterparts. Simply mimicking the traditional screen designs used with commercial, offset presses will result in halftone patterns that are more susceptible to moire due to the interaction between the periodic patterns used to render the different color channels. This causes instability in the printed colors. The moire can be reduced by increasing the frequency of the halftone patterns. But this may make the print appear grainier than its analog counterpart. In this paper, we introduce a principled design procedure that allows one to design color screen sets that generate periodic, clustered-dot halftone patterns that improve color stability without increasing graininess. We present experimental results to support the benefits of our new color screen set design framework.

Pedestrian detection in the embedded system, such as video surveillance equipment, usually involves low-resolution pedestrian samples and requires a low computational cost. Many pedestrian detectors rely on a large feature pool and suffer in their efficiency and performance for real-time monitoring. In this paper, a set of light-weight features is proposed to enhance the pedestrian detection performance when a small-medium scale of training data with low-resolution images is available. To address this issue, a difference matrix projection (DMP) is developed to compute aggregated multi-oriented pixel differences using global matrix operations. Both the pixel differences and aggregation are computed using global matrix projection to avoid the laborious iterative operations. We tested our method on the INRIA, Daimler Chrysler classification (Daimler-CB), NICTA, and Caltech Pedestrian datasets. The experiments on these benchmark data sets show encouraging results in terms of detection performance, particularly for image datasets with low-resolution pedestrians.

In this work, we propose a high-efficiency person segmentation system that achieves high segmentation accuracy with a much smaller CNN network. In this approach, key-point detection annotation is incorporated for the first time and a novel spatial saliency map, in which the intensity of each pixel indicates the likelihood of forming a part of the human and reflects the distance from the body, is generated to provide more spatial information. Additionally, a lightweight automatic person segmentation network is proposed, which is small and efficient for person segmentation by leveraging atrous convolution. The experimental results prove that an image pyramid resizing augmentation can also improve efficiency. Our proposed segmentation method achieves an accuracy of 94.06% on the person segmentation dataset built in this work, which exceeds the results of previous state-of-the-art methods in accuracy and efficiency.

Error diffusion is a popular technique widely used in desktop printers, especially inkjet printers. However, since the conventional error diffusion is computed in raster order, it produces worm artifacts in the highlights and shadows. In addition, as a serial algorithm, it limits the efficiency and flexibility of hardware implementations. To address these two issues, we propose a novel serpentine based error diffusion algorithm that uses tone dependent error weights and thresholds. We also propose an expanded error weight location matrix to improve the halftone quality in the extreme tones. With this new algorithm, we achieve better halftones comparing to the original tone dependent fast error diffusion, especially in the quarter tones.

The inherent noise in the observed (e.g., scanned) binary document image degrades the image quality and harms the compression ratio through breaking the pattern repentance and adding entropy to the document images. In this paper, we design a cost function in Bayesian framework with dictionary learning. Minimizing our cost function produces a restored image which has better quality than that of the observed noisy image, and a dictionary for representing and encoding the image. After the restoration, we use this dictionary (from the same cost function) to encode the restored image following the symbol-dictionary framework by JBIG2 standard with the lossless mode. Experimental results with a variety of document images demonstrate that our method improves the image quality compared with the observed image, and simultaneously improves the compression ratio. For the test images with synthetic noise, our method reduces the number of flipped pixels by 48.2% and improves the compression ratio by 36.36% as compared with the best encoding methods. For the test images with real noise, our method visually improves the image quality, and outperforms the cutting-edge method by 28.27% in terms of the compression ratio.

We study the concept of color semantics by modeling a dataset of magazine cover designs, evaluating the model via crowdsourcing, and demonstrating several prototypes that facilitate color-related design tasks. We investigate a probabilistic generative modeling framework that expresses semantic concepts as a combination of color and word distributions -- color-word topics. We adopt an extension to Latent Dirichlet Allocation (LDA) topic modeling, called LDA-dual, to infer a set of color-word topics over a corpus of 2,654 magazine covers spanning 71 distinct titles and 12 genres. Although LDA models text documents as distributions over word topics, we model magazine covers as distributions over color-word topics. The results of our crowdsourcing experiments confirm that the model is able to successfully discover the associations between colors and linguistic concepts. Finally, we demonstrate several prototype applications that use the learned model to enable more meaningful interactions in color palette recommendation, design example retrieval, pattern recoloring, image retrieval, and image color selection.

Low-rank matrix factorization serves as a key technique in learning latent factor models for many applications in machine learning. However, in many applications, observed data often exhibits different levels of noise. To address this issue, we propose a Robust Probabilistic Discriminative Matrix Factorization (RPDMF) method for binary matrix factorization on noise polluted data. We illustrate the benefits of our approach in real examples, and show how our method significantly outperforms Probabilistic Discriminative Matrix Factorization (PDMF) and classical method Weighted Nonnegative Matrix Factorization (WNMF) in the application of image tag completion.

Most electro-photographic printers prefer clustered-dot halftone textures for rendering smooth and stable prints. Clustered-dot halftone patterns can be periodic or aperiodic. As periodic clustered-dot halftone can lead to undesirable moiré patterns, stochastic clustered-dot halftone textures are more preferred. There are available different screening methods to generate stochastic clustered-dot halftone textures but there are no standard print quality assessment measures that can be easily used for quantitatively evaluating and comparing different stochastic clustered-dot halftoning methods. We explore the use of compactness measures for this purpose, and also propose a new compactness measure that seems good metric to quantitatively compare and assess the print quality of different stochastic clustered-dot halftoning methods. Using the proposed metric, we compare three different stochastic clustered-dot halftoning methods, and our results are almost in agreement with psychophysical experiments results reported earlier.

The goal of this paper is to investigate the effect of choosing a certain halftone screen on image graininess development and to establish the metrics for image graininess in high-end digital printing technologies. With the obtained knowledge, our model helps us choose the optimal periodicity matrices for designing regular or irregular clustered dot halftones. The main advantage of the proposed model lies in predicting the graininess ratios based solely on the periodicity matrix. We conduct an in-depth Fourier analysis, which exposes to us all the details regarding the spectrum of a desired halftone screen. Hence, at this stage of the design process, there is no need to generate the halftone screens themselves.

Perceptual distortion prediction at near-threshold level has many applications in general image/video processing tasks. This paper presents a computational model to predict the near-threshold perceptual distortions based on optimal structure classification. This model accounts for contrast sensitivity, light adaptation, and various masking effects of the human visual system (HVS), and automatically adapts to local image structures by a soft classification scheme using a Gaussian Mixture Model (GMM). The proposed model is trained and verified on the public CSIQ local masking database. We demonstrate a superior prediction performance of the proposed model compared to previous research.

A novel framework using direct binary search (DBS) is proposed for printed image watermarking. Watermarks are embedded in the halftone by forcing pairs of pixels to be positively or negatively correlated, halftone image quality and watermark detection are two goals to be jointly maximized, and a modified toggle and swap strategy of DBS is employed to find the optimal halftone. Experiment results are presented for watermark rate (WMR) equal to 4.62%, 9.77%, 27.81%, and 46.73%. Excellent performance is demonstrated, e.g. when the WMR is below 10%, the image quality degradation is negligible, and the error rate (BER) is about 1%.

Periodic clustered-dot screens are widely used for electrophotographic printers due to their print stability. However, moiré is a ubiquitous problem that arises in color printing due to the beating together of the clustered-dot, periodic halftone patterns that are used to represent different colorants. The traditional solution in the graphic arts and printing industry is to rotate identical square screens to angles that are maximally separated from each other. However, the effectiveness of this approach is limited when printing with more than four colorants, i.e., N -color printing, where N > 4 . Moreover, accurately achieving the angles that have maximum angular separation requires a very high-resolution plate writer, as is used in commercial offset printing. Commercially available high-end digital printers cannot achieve this resolution. In this paper, we propose a systematic way to design color screen sets for periodic, clustered-dot screens that offer more explicit control of the moiré properties of the resulting screens when used in color printing. We develop a principled approach for the moiré-free screen design that is called lattice-based screen design. The basic concept behind our approach is the creation of the screen set on a 2D lattice in the frequency domain, and then picking each fundamental frequency vector of the individual colorant planes in the created spectral lattice according to the desired properties. The lattice-based screen design offers more flexibility in designing N -color screen sets with different halftone geometries, and all of them are guaranteed to be all-orders moiré-free. We demonstrate the efficacy of our proposed method by introducing several new screen designs, and a comparison with published screen designs.

In this paper, we consider a dual-mode process for the electrophotographic laser printer--a low-frequency halftoning for smooth regions and a high-frequency halftoning for detail regions. These regions are described by an object map that is extracted from the page description language version of the document. This manner of switching screens depending on the local content provides a stable halftone without artifacts in smooth areas and preserves the detail rendering in detail or texture areas. However, when switching between halftones with two different frequencies, jaggies may occur along the boundaries between areas halftoned with low- and high-frequency screens. To reduce the jaggies, our screens obey a harmonic relationship. In addition, we implement a blending process based on a transition region. We propose a nonlinear blending process in which at each pixel, we choose the maximum of the two weighted halftones, where the weights vary according to the position in the transition region. Moreover, we describe an online tone-mapping for the boundary blending process, based on an offline calibration procedure that effectively assures the desired tone values within the transition region.

Office machines (such as printers, scanners, facsimile machines, and copiers) can consume significant amounts of power. Most office machines have sleep modes to save power. Power management of these machines is usually timeout-based: a machine sleeps after being idle long enough. Setting the time-out duration can be difficult: if it is too long, the machine wastes power during idleness. If it is too short, the machine sleeps too soon and too often—the wake-up delay can significantly degrade productivity. Thus, power management is a tradeoff between saving energy and keeping response time short. Many power management policies have been published and one policy may outperform another in some scenarios. There is no definite conclusion regarding which policy is always better. This article describes two methods for office equipment power management. The first method adaptively reduces power based on a constraint of the wake-up delay. The second is a hybrid method with multiple candidate policies and it selects the most appropriate power management policy. Using 6 months of request traces from 18 different printers, we demonstrate that the hybrid policy outperforms individual policies. We also discover that power management based on business hours does not produce consistent energy savings.

As one of the most important state-of-the-art classification techniques, Support Vector Machine (SVM) has been widely adopted in many real-world applications, such as object detection, face recognition, text categorization, etc., due to its competitive practical performance and elegant theoretical interpretation. However, it treats all samples independently, and ignores the fact that, in many real situations especially when data are in high dimensional space, samples typically lie on low dimensional manifolds of the feature space and thus a sample can be related to its neighbors by being represented as a linear combination of other samples on the same manifold. This linear representation, which is usually sparse, reflects the structure of underlying manifolds. It has been extensively explored in the recent literature and proven to be critical for the performance of classification. To benefit from both the underlying low dimensional manifold structure and the large margin classifier, this paper proposes a novel method called Sparsity Preserving Support Vector Machine(SP-SVM), which explicitly considers the sparse representation of samples while maximizing the margin between different classes. Consequently, SP-SVM inherits both the discriminative power of support vector machine and the merits of sparsity. A set of experiments on real-world benchmark data sets show that SP-SVM achieves significantly higher precision on recognition task than various competitive baselines including the traditional SVM, the sparse representation based method and the classical nearest neighbor classifier.

Images can both express and affect people's emotions. It is intriguing and important to understand what emotions are conveyed and how they are implied by the visual content of images. Inspired by the recent success of deep convolutional neural networks (CNN) in visual recognition, we explore two simple, yet effective deep learning-based methods for image emotion analysis. The first method uses off-the-shelf CNN features directly for classification. For the second method, we fine-tune a CNN that is pre-trained on a large dataset, i.e. ImageNet, on our target dataset first. Then we extract features using the fine-tuned CNN at different location at multiple levels to capture both the global and local information. The features at different location are aggregated using the Fisher Vector for each level and concatenated to form a compact representation. From our experimental results, both the deep learning-based methods outperforms traditional methods based on generic image descriptors and hand-crafted features.

Photo aesthetic quality assessment is a challenging task. In this paper we propose a framework to automatically assess the aesthetic quality of online shopping photos. Novel image features that indicate photo aesthetic quality are introduced. We further investigate the relevance between our image features and photo aesthetic quality with the elastic net. A ranking of features in the order of their relevance to photo aesthetic quality is thus obtained. Moreover, we apply the wrapper feature selection methodology with the best-first searching algorithm to establish an optimal feature subset that yields best prediction accuracy. With a photo database, we adopt the support vector regression (SVR) technique to train an aesthetic quality predictor using the selected optimal feature subset. Promising prediction accuracy is obtained with cross-validation.

Quantifying image quality through subjective evaluation is very critical to image quality evaluation. Using the image quality ruler method, an average score per stimulus can be easily obtained in the unit of Just Noticeable Differences (JNDs). However, it requires a large number of subjects, since pure averaging does not consider the different judging quality of different subjects. In this paper, we propose an image quality evaluation framework using the image quality ruler method with a statistical model. By incorporating this model, we consider the quality score, the expertise of the subjects, and the difficulty of image rating task as three hidden variables. Then we use expectation-maximization (EM) to estimate these hidden variables. From our experimental results, we show that our method provides reliable results without using a large number of subjects. Preliminary results also demonstrate that the estimates of the parameters can guide us to better distribute the valuable human resources used to conduct psychophysical experiments.

In this work, we discuss the use of depth information to correct the distortion due to the curved shape of the pages of an open book in captured images. This work is relevant to camera-based capture devices that can use a projector to cast structured light patterns to provide depth information. In order to improve the visual quality of captured documents, we use 3D shape reconstruction methods and geometric rectification to flatten the curvature of an open book. Shading correction is applied to the captured image, as well. Our models exploit specific prior assumptions about the nature of the printed material that is captured. The improvement in captured open book images obtained by using our method can be observed in the included experimental results.

Support vector machine is one of the most successful machine learning methods in image processing and computer vision in the past decades. However, its performance strongly depends on the training data, which are sometimes expensive and of low quality. Specifically, in many real applications, such as face recognition, the images are rarely perfectly aligned, thus the misalignment between training and testing data impairs the performance. In this paper, we propose a strategy to compensate the misalignment between images while learning the classifier without looking at the testing samples. Specifically, some certain critical transformations are inferred and applied to training samples to alleviate the effect of the worst case of possible misalignment. The resulted large margin classifier generalizes better than traditional SVM, especially when there is misalignment. Experimental results on real image data sets show the efficacy of the proposed algorithm.

Kernel methods have been regarded as an effective approach in image processing. However, when calculating the similarity induced by kernels, existing kernel methods usually incorporate irrelevant features (e.g., the background features of an object in a image), which are then inherited to kernel learning methods and thus lead to suboptimal performance. To attack this problem, we introduce a framework of kernel tapering, which is a simple and effective approach to reduce the effects of irrelevant features while keeping the positive semi-definiteness of kernel matrices. In theory, it can be demonstrated that the tapered kernels asymptotically approximate the original kernel functions. In practical image applications where noises or irrelevant features are widely observed, we have further shown that the introduced kernel tapering framework can greatly enhance the performance of their original kernel partners for kernel k-means and kernel nonnegative matrix factorization.

Relief printing, also known as 2.5D printing, is capable of representing texture in a more appealing way to the human observer due to the fact that it can reproduce the tactile details and structures of a texture. This paper describes a novel method to reproduce a given textured area of an image based on the semantic information linked to that area. A texture detector is trained offline with a non-parametric test in order to produce a printing mask which labels pixels more likely to belong to a specific texture-related semantic concept. The height map needed to print pixels in relief is calculated automatically from the printing mask and the gray-scale version of the original image.

Dot gain is a classic problem in digital printing that causes printed halftones and text to appear darker than desired. For printing of text, we propose a method to preprocess the image sent to the printer in order to compensate for dot gain. It is based on an accurate model that predicts the printed absorptance for given local neighborhood in the digital image, a cost function to penalize lack of fidelity to the desired target text image, and the use of direct binary search (DBS) to minimize the cost.

Many imaging applications require the implementation of space-varying convolution for accurate restoration and reconstruction of images. Here, we use the term space-varying convolution to refer to linear operators whose impulse response has slow spatial variation. In addition, these space-varying convolution operators are often dense, so direct implementation of the convolution operator is typically computationally impractical. One such example is the problem of stray light reduction in digital cameras, which requires the implementation of a dense space-varying deconvolution operator. However, other inverse problems, such as iterative tomographic reconstruction, can also depend on the implementation of dense space-varying convolution. While space-invariant convolution can be efficiently implemented with the fast Fourier transform, this approach does not work for space-varying operators. So direct convolution is often the only option for implementing space-varying convolution. In this paper, we develop a general approach to the efficient implementation of space-varying convolution, and demonstrate its use in the application of stray light reduction. Our approach, which we call matrix source coding, is based on lossy source coding of the dense space-varying convolution matrix. Importantly, by coding the transformation matrix, we not only reduce the memory required to store it; we also dramatically reduce the computation required to implement matrix-vector products. Our algorithm is able to reduce computation by approximately factoring the dense space-varying convolution operator into a product of sparse transforms. Experimental results show that our method can dramatically reduce the computation required for stray light reduction while maintaining high accuracy.

Automated publishing systems require large databases containing document page layout templates. Most of these layout templates are created manually. A lower cost alternative is to extract document page layouts from existing documents. In order to extract the layout from a scanned document image, it is necessary to perform Optical Font Recognition (OFR) since the font is an important element in layout design. In this paper, we use the Conditional Random Field (CRF) model to perform OFR. First, we extract typographical features of the text. Then, we train the probabilistic model using a log-linear parameterization of CRF. The advantage of using CRF is that it does not assume that the typographical features are independent of each other. We demonstrate the effectiveness of this approach on a set of 616 fonts.

The JBIG2 standard is widely used for binary document image compression primarily because it achieves much higher compression ratios than conventional facsimile encoding standards. In this paper, we propose a dynamic hierarchical dictionary design method (DH) for multi-page binary document image compression with JBIG2. Our DH method outperforms other methods for multi-page compression by utilizing the information redundancy among pages with the following technologies. First, we build a hierarchical dictionary to keep more information per page for future usage. Second, we dynamically update the dictionary in memory to keep as much information as possible subject to the memory constraint. Third, we incorporate our conditional entropy estimation algorithm to utilize the saved information more effectively. Our experimental results show that the compression ratio improvement by our DH method is about 15% compared to the best existing multi-page encoding method.

3D shape reconstruction is one of the most important topics in computer vision due to its wide field of application. Among various technologies, structured light is considered to be one of the most reliable techniques. This paper addresses the problem of finding correspondences in a structured-light 3D shape reconstruction system. The work is based on the 3-symbol binary M-array encoded pattern proposed by Albitar. A new 6-symbol M-array pattern is designed with guaranteed minimum Hamming distance of three among all 3 × 3 windows. Improvements to the decoding algorithm are made, which allows successful identification of most symbols, and which corrects one possible error or missing symbol in each 3 × 3 window. Finally, refinements are made to the algorithm for finding correspondences, including back-projection of the captured structured light pattern to the projector plane to eliminate projective distortion.

Binarization algorithms are used to create a binary representation of a raster document image, typically with the intent of identifying text and separating it from background content. In this paper, we propose a binarization algorithm via one-pass local classification. The algorithm first generates the initial binarization results by local thresholding, then corrects the results by a one-pass local classification strategy, followed by the process of component inversion. The experimental results demonstrate that our algorithm achieves a somewhat lower binarization error rate than the state-of-the-art algorithm COS [1], while requiring significantly less computation.

Determining the aesthetics of photographs has recently become a research topic of considerable interest. In this project, we focus on constructing meaningful features to model the aesthetic quality of photos with faces. Utilizing face information, color, composition features, as well as novel saliency-based spatial features, we construct an aesthetic inference model, which is more accurate than a state-of-the-art method. Further, we show that this model can be improved by applying different sets of features for single-face and multiple-face photos. Third, we demonstrate by combining low-level generic features with handcrafted features, that the model can be made to achieve even lower error rates.

In this paper, we present a recommendation system for the automatic design of magazine covers. Our users are non-designer designers: individuals or small and medium businesses who want to design without hiring a professional designer while still wanting to create aesthetically compelling designs. Because a design should have a purpose, we suggest a number of semantic features to the user, e.g., "clean and clear," "dynamic and active," or "formal," to describe the color mood for the purpose of his/her design. Based on these high level features and a number of low level features, such as the complexity of the visual balance in a photo, our system selects the best photos from the user's album for his/her design. Our system then generates several alternative designs that can be rated by the user. Consequently, our system generates future designs based on the user's style. In this fashion, our system personalizes the designs of a user based on his/her preferences.

In this paper, we present a new algorithm for aperiodic clustered-dot halftoning based on direct binary search (DBS). The DBS optimization framework has been modified for designing clustered-dot texture, by using filters with different sizes in the initialization and update steps of the algorithm. Following an intuitive explanation of how the clustered-dot texture results from this modified framework, we derive a closed-form cost metric which, when minimized, equivalently generates stochastic clustered-dot texture. An analysis of the cost metric and its influence on the texture quality is presented, which is followed by a modification to the cost metric to reduce computational cost and to make it more suitable for screen design.

Multilevel halftoning algorithms are becoming increasingly important as the capabilities of image output devices improve. The traditional approach to multilevel halftoning is hampered by the appearance of contouring in the vicinity of native tones of the output device. To overcome this limitation, we propose a novel framework based on maintaining a consistent periodic, clustered-dot halftone texture across the tone scale. We develop metrics for granularity and structure dissimilarity, and show how these can be used to guide the manner in which the halftone texture evolves from native tone to native tone, across the tone scale. Experimental results confirm the benefits of our new approach.

Managing large document databases has become an important task. Being able to automatically compare document layouts and classify and search documents with respect to their visual appearance proves to be desirable in many applications. We propose a new algorithm that approximates a metric function between documents based on their visual similarity. The comparison is based only on the visual appearance of the document without taking into consideration its text content. We measure the similarity of single page documents with respect to distance functions between three document components: background, text, and saliency. Each document component is represented as a Gaussian mixture distribution; and distances between the components of different documents are calculated as an approximation of the Hellinger distance between corresponding distributions. Since the Hellinger distance obeys the triangle inequality, it proves to be favorable in the task of nearest neighbor search in a document database. Thus, the computation required to find similar documents in a document database can be significantly reduced.

Most electrophotographic printers use periodic, clustered-dot screening for rendering smooth and stable prints. However, when used for color printing, this approach suffers from the problem of periodic moire resulting from interference between the periodic halftones of individual color planes. There has been proposed an approach, called CLU-DBS for stochastic, clustered-dot halftoning and screen design based on direct binary search. We propose a methodology to embed a printer model within this halftoning algorithm to account for dot-gain and dot-loss effects. Without accounting for these effects, the printed image will not have the appearance predicted by the halftoning algorithm. We incorporate a measurement-based stochastic model for dot interactions of an electro-photographic printer within the iterative CLU-DBS binary halftoning algorithm. The stochastic model developed is based on microscopic absorptance and variance measurements. The experimental results show that electrophotography-model based stochastic clustered dot halftoning improves the homogeneity and reduces the graininess of printed halftone images.

Periodic clustered-dot screens are widely used for electrophotographic printers due to their homogeneous halftone texture and their robustness to dot gain. However, when applied to color printing, there are two important phenomena that limit the quality of printed color halftones generated using a screening technology: (1) moire´ due to the superposition halftone patterns corresponding to different periodicity matrices, and (2) appearance changes due to misregistration between different colorant planes. This paper focuses on analyzing the registration sensitivity of periodic, clustered-dot screens. To quantitatively measure the effect of registration errors, we introduce two new functions: (1) cost, and (2) risk of registration errors. We propose the notion of “visual equivalence”, and derive three propositions under which visual equivalence can be achieved, even when registration errors occur.

Printer identification based on printed documents can provide forensic information to protect copyright and verify authenticity. In addition to intrinsic features (intrinsic signatures) of the printer, modulating the printing process to embed specific features (extrinsic signatures) will further extend the encoding capacity and decoding accuracy. One of the key issues with embedding extrinsic signatures is that the embedding should not degrade the image quality, but needs to be detectable by a detection algorithm. In this paper, we will demonstrate the feasibility of embedding code sequences in electrophotographic halftone images by modulating dot size through laser intensity modulation. We have developed corresponding embedding and detection algorithms to embed and extract information. Experimental results indicate that using a 600 dpi native resolution printer's default halftone algorithm, we can encode 5 bits of information in every 310 printer scan-lines or approximately every 0.5 inches.

The hybrid screen is a halftoning method that generates stochastic dispersed dot textures in highlights and periodic clustered dot textures in midtones. Each tone level is sequentially designed from highlight to midtone by applying an iterative halftoning algorithm such as direct binary search (DBS). By allowing random seeding followed by swap-only DBS in a predefined core region within each microcell, each level can be designed while satisfying the stacking constraint and guaranteeing a smooth transition between levels. This paper describes a complete design process for the hybrid screen, introduces a number of enhancements to the original hybrid screen, and evaluates their impact on print quality. These enhancements include a multilevel screen design based on either extending a bilevel screen or directly generating a multilevel screen on a high resolution grid, and extending the hybrid screen design procedure to color by jointly optimizing the color screens using color DBS. For the multilevel screen, we show that the best choice for the core size critically depends on the bit depth and screen frequency. For the color hybrid screen, we demonstrate a significant improvement in the highlights over halftones generated by independently designed screens.

We present a framework for developing a color transformation between two display devices to achieve a desired perceptual match. The framework is realized by first developing an optimal color transformation between the two devices, and then formulating an optimization problem to develop a hardware resource-constrained approximation to the optimal transformation. We employ the framework to investigate how well the system performs when the transformation is constrained to be a single 3×3 linear transformation directly between the non-linear RGB spaces of two devices. The motivation for this constraint is to ease resource requirements in a real-time hardware implementation.

We describe an automatic method for beautifying digital portraits by smoothing the skin of the face. The method builds on existing face detection and face feature alignment technology to automatically segment the face and neck areas to be smoothed. A smoothing filter is then applied to these areas. The resulting portraits are enhanced in a subtle and natural fashion.

Color space transformations are very common in digital imaging and display systems. Their complex mathematics requires hardware friendly implementations to make them practical. In this paper, look-up tables (LUT) are described for implementation of the color transformations. We show that this is possible because an analytical model of the transform is available for training and cross-validation. We identify parameters that affect the performance of a LUT-based system and then formulate the solution as an optimization problem.

Digital images can be captured or generated by a variety of sources including digital cameras, scanners and computer graphics softwares. In many cases it is important to be able to determine the source of a digital image such as for criminal and forensic investigation. This paper presents methods for distinguishing between an image captured using a digital camera, a computer generated image and an image captured using a scanner. The method proposed here is based on the differences in the image generation processes used in these devices and is independent of the image content. The method is based on using features of the residual pattern noise that exist in images obtained from digital cameras and scanners. The residual noise present in computer generated images does not have structures similar to the pattern noise of cameras and scanners. The experiments show that a feature based approach using an SVM classifier gives high accuracy.

With the introduction of the digital press in recent years, the cost of short-run printing jobs has decreased. The “print-on-the-fly” capability of these digital presses makes it possible to compose documents based on the contents extracted from various databases. The traditional proofing method in print shops focuses on human visual inspection of one hard copy among thousands of print-outs. However, in variable data printing (VDP), each instance of the same template could use different texts, fonts, and images obtained through various capturing devices. Visual inspection of every copy is not cost efficient in the VDP environment. In this paper, we present a scheme to automatically predict the presence of moiré artifacts in the print-out before the actual printing take place. Only the images that may induce moiré artifact with respect to the targeted press will be identified and adaptive image processing can be applied to eliminate the potential artifact.

In this paper, we propose an adaptive interpolation method for low quality still images such as single video frames. Our method chooses different interpolation schemes according to the local image context. First, we segment the image into three types of regions: smooth regions which contain no edges, well-defined edges such as object boundaries, and textured regions which are covered by dense spurious edges. For smooth regions, we simply apply bilinear interpolation. For the well-defined edges, we develop an edge-directed interpolation scheme. For the remaining textured regions, we apply a residual error estimation method to enhance the high frequency components, hence improving the visual sharpness of these regions. Our results show a visible improvement compared to images generated by traditional interpolation methods, with sharp edges, enhanced textures, and low noise level in smooth regions.

In this paper, we present the adaptive bilateral filter (ABF) for sharpness enhancement and noise removal. The ABF sharpens an image by increasing the slope of the edges without producing overshoot or undershoot. It is an approach to sharpness enhancement that is fundamentally different from the unsharp mask (USM). This new approach to slope restoration also differs significantly from previous slope restoration algorithms in that the ABF does not involve detection of edges or their orientation, or extraction of edge profiles. In the ABF, the edge slope is enhanced by transforming the histogram via a range filter with adaptive offset and width. The ABF is able to smooth the noise, while enhancing edges and textures in the image. The parameters of the ABF are optimized with a training procedure. ABF restored images are significantly sharper than those restored by the bilateral filter. Compared with an USM based sharpening method-the optimal unsharp mask (OUM), ABF restored edges are as sharp as those rendered by the OUM, but without the halo artifacts that appear in the OUM restored image. In terms of noise removal, ABF also outperforms the bilateral filter and the OUM. We demonstrate that ABF works well for both natural images and text images.

This paper describes a real-time, strip-based, low-complexity document page classification algorithm, which can be used as a copy mode selector in the copy pipeline. The benefits of such a copy mode selector include improving copy quality, simplifying user interaction, and increasing copy rate.

In this paper, we present an adaptive bilateral filter (ABF) for sharpness enhancement and noise removal. ABF sharpens an image by increasing the slope of the edges without producing overshoot or undershoot. Our new approach to slope restoration significantly differs from the previous slope restoration algorithms in that ABF does not involve detecting edge orientations or edge profiles. Compared with the bilateral filter, ABF restored images are significantly sharper. Compared with an unsharp mask (USM) based sharpening method the optimal USM (OUM), ABF restored edges are as sharp as those rendered by the OUM, but without halo. ABF also outperforms the bilateral filter and the OUM in noise removal.

Forensic characterization of devices is important in many situations such as establishing the trust and verifying authenticity of data and the device that created it. Current forensic identification techniques for digital cameras, scanners and printers are highly reliable due to the fact that each of these devices cannot escape inherent electro-mechanical properties which add "signatures" to the data they produce. In this paper we will describe the sensor forensics work going on at Purdue University.

It is critical to understand user requirements in Web site development. As a method of user requirements analysis for a self-help technical support Web site, focus group interviews can be a very efficient and effective approach both before the interface has been designed and after it has been in use for some time. This article shows how focus group interviews were used to develop a self-help technical support Web site for print quality troubleshooting. It also shows how focus group interviews led to improvements in the efficiency and effectiveness of the Web site. In addition, the article classifies the critical usability issues identified and shows how the feedback from focus group interviews is reflected in the Web site development. Finally, experimental results demonstrate that the use of focus group interviews significantly improved user performance and preference.

Characterization of halftone texture is important for quantitative assessment of halftone quality. In this paper, we develop a new framework based on directional local sequency analysis and a filter bank structure. We decompose a halftone image into subband images, from which we can easily reconstruct the original halftone. Based on these subband images, we define the directional sequency spectrum which is analogous to the two-dimensional Fourier spectrum, and formulate several texture measures. Two test image sets are used to justify these measures.

In electrophotographic printing, a periodic clustered-dot halftone pattern is preferred for a smooth and stable result. In addition, the screen frequency should be high enough to minimize the visibility of the halftone textures and to ensure good detail rendition. However, at these frequencies, the halftone cell may contain too few pixels to provide a sufficient number of distinct gray levels. This will result in contouring and posterization. The traditional solution is to grow the clusters asynchronously within a repeating block of clusters known as a supercell. The growth of each individual cluster is governed by a microscreen. The order in which the clusters grow within the supercell is determined by a macroscreen. Typically, the macroscreen is a recursive pattern due to Bayer. In highlights and shadows, this ordering results in visible artifacts. Replacing the Bayer screen by a stochastic macroscreen eliminates these artifacts, but results in new artifacts. In this paper, we propose a new composite screen architecture that employs multiple microscreens and multiple macroscreens in the highlights and shadows. These screens are jointly designed by using the direct binary search (DBS) algorithm.

In this paper, we present a methodology to design filters for an imaging system to improve the accuracy of the spectral measurements for families of reflective surfaces. We derive the necessary and sufficient conditions that the sensor space of the system must obey in order to measure the spectral reflectance of the surfaces accurately. Through simulations, we show how these conditions can be applied to design filters using a set of sample spectral data acquired from extracted teeth. For this set of data, we also compare our results to those of Wolski's method, a conventional filter design method which produces filters that recover tristimulus values of surfaces accurately under several illuminants. We show that our method produces filters that capture the spectral reflectance better given the same number of measurements. The errors in predicting the color of the sample data are much lower under every test illuminant when the filters designed with our method are used.

Characterization of halftone textures is important for quantitative assessment of halftone quality. We develop a new framework based on directional local sequency analysis and a filter bank structure. We decompose a halftone image into subband images from which we can easily reconstruct the original halftone. Based on these subband images, we define the directional sequency spectrum which is analogous to the 2D Fourier spectrum, and formulate several texture measures. Three test images are used to justify these measures.

In this paper, we develop a model-based color halftoning method using the direct binary search (DBS) algorithm. Our method strives to minimize the perceived error between the continuous tone original color image and the color halftone image. We exploit the differences in how the human viewers respond to luminance and chrominance information and use the total squared error in a luminance/chrominance based space as our metric. Starting with an initial halftone, we minimize this error metric using the DBS algorithm. Our method also incorporates a measurement based color printer dot interaction model to prevent the artifacts due to dot overlap and to improve color texture quality. We calibrate our halftoning algorithm to ensure accurate colorant distributions in resulting halftones. We present the color halftones which demonstrate the efficacy of our method.

Due to its high image quality and moderate computational complexity, error diffusion is a popular halftoning algorithm for use with inkjet printers. However, error diffusion is an inherently serial algorithm that requires buffering a full row of accumulated diffused error (ADE) samples. For the best performance when the algorithm is implemented in hardware, the ADE data should be stored on the chip on which the error diffusion algorithm is implemented. However, this may result in an unacceptable hardware cost. In this paper, we examine the use of quantization of the ADE to reduce the amount of data that must be stored. We consider both uniform and nonuniform quantizers. For the nonuniform quantizers, we build on the concept of tone-dependency in error diffusion, by proposing several novel feature-dependent quantizers that yield improved image quality at a given bit rate, compared to memoryless quantizers. The optimal design of these quantizers is coupled with the design of the tone-dependent parameters associated with error diffusion. This is done via a combination of the classical Lloyd-Max algorithm and the training framework for tone-dependent error diffusion. Our results show that 4-bit uniform quantization of the ADE yields the same halftone quality as error diffusion without quantization of the ADE. At rates that vary from 2 to 3 bits per pixel, depending on the selectivity of the feature on which the quantizer depends, the feature-dependent quantizers achieve essentially the same quality as 4-bit uniform quantization.

The quality of halftone prints produced by inkjet (IJ) printers can be limited by random dot-placement errors. While a large literature addresses model-based halftoning for electrophotographic printers, little work has been done on model-based halftoning for IJ printers. In this paper, we propose model-based approaches to both iterative least-squares halftoning and tone-dependent error diffusion (TDED). The particular approach to iterative least-squares halftoning that we use is direct binary search (DBS). For DBS, we use a stochastic model for the equivalent gray-scale image, based on measured dot statistics of printed IJ halftone patterns. For TDED, we train the tone-dependent weights and thresholds to mimic the spectrum of halftone textures generated by model-based DBS. We do this under a metric that enforces both the correct radially averaged spectral profile and angular symmetry at each radial frequency. Experimental results generated with simulated printers and a real printer show that both IJ model-based DBS and IJ model-based TDED very effectively suppress IJ printer-induced artifacts.

We present an enhanced error diffusion halftoning algorithm for which the filter weights and the quantizer thresholds vary depending on input pixel value. The weights and thresholds are optimized based on a human visual system model. Based on an analysis of the edge behavior, a tone dependent threshold is designed to reduce edge effects and start-up delay. We also propose an error diffusion system with parallel scan that uses variable weight locations to reduce worms.

An improved controller design and implementation technique for electrophotographic process (EP) was proposed. The new controller was modified from a previous design to address two additional issues for generic EP platforms, i.e., reducing position-dependent disturbances and reducing system sensitivity to manufacturing variations in EP engine and consumables. To handle position-dependent periodic disturbances, a digital repetitive controller was developed and implemented using spatial sampling. The result is a control algorithm that will take into account the variation of the nominal operating speed. Second, system variations due to manufacturing variations as well as consumable changes were incorporated into the design of a two degree of freedom (TDOF) robust controller. The controller is optimal in the sense that it minimizes the size of the sensitivity function from a set of disturbance signals to a set of measurable signals critical to print quality, e.g. photoconductor drum velocity or scan line spacing. A suitable trade-off between system performance and robustness to system modeling uncertainties was considered in the synthesis and optimization formulation. The effectiveness of the proposed controller design and implementation technique was numerically and experimentally verified. Printed samples demonstrated significant reduction in visible banding that was verified by reflectance measurement.

We incorporate models for printer dot interactions within the iterative direct binary search (DBS) halftoning algorithm. Monochrome electro-photographic printers are considered. Both analytical and stochastic models are studied. In the analytical model it is assumed that the printer can generate a circular spot with constant absorptance at each printer addressable location, whereas the stochastic model is based on microscopic absorptance and variance measurements. We also present an efficient strategy for evaluating the change in computational cost as the search progresses. With our scheme, updating the change in error only involves a few fetches from two look-up-tables and some scalar multiplications and additions. Experimental results are provided that show that DBS with an appropriate model for printer dot interactions yields enhanced detail rendition, and improved tonal gradation in shadow areas.

Because of its good image quality and moderate computational requirements, error diffusion has become a popular halftoning solution for desktop printers, especially inkjet printers. By making the weights and thresholds tone-dependent and using a predesigned halftone bitmap for tone-dependent threshold modulation, it is possible to achieve image quality very close to that obtained with far more computationally complex iterative methods. However, the ability to implement error diffusion in very low cost or large format products is hampered by the requirement to store the tone-dependent parameters and halftone bitmap, and also the need to store error information for an entire row of the image at any given point during the halftoning process. For the first problem, we replace the halftone bitmap by deterministic bit flipping, which has been previously applied to halftoning, and we linearly interpolate the tone-dependent weights and thresholds from a small set of knot points. We call this implementation a reduced lookup table. For the second problem, we introduce a new serial block-based approach to error diffusion. This approach depends on a novel intrablock scan path and the use of different parameter sets at different points along that path. We show that serial block-based error diffusion reduces off-chip memory access by a factor equal to the block height. With both these solutions, satisfactory image quality can only be obtained with new cost functions that we have developed for the training process. With these new cost functions and moderate block size, we can obtain image quality that is very close to that of the original tone-dependent error diffusion algorithm.

Proposes a novel method of hybrid LUT/screen halftoning which achieves both high quality and compressibility using only point processes. Experimental results show that the halftone images from our hybrid LUT/screen algorithm have quality comparable to those from the direct binary search (DBS) algorithm with excellent performance of lossless halftone compression.

There have been many advances in printer model-based halftoning. However, virtually all of this work has focused on electrophotographic (EP) printers. The lack of comparable work for inkjet (IJ) printers may be due to the fact that IJ printers do render dots that have a nearly hard, ideal profile, and are much more stable than dots rendered by EP printers. However, as we shall see, IJ printers do exhibit dot displacement errors that are caused by misaligned nozzles in the printhead, and which appear random in nature. These error can cause significant artifacts in the printed image. The goal of this paper is to model the IJ printer, and then to incorporate this model into halftoning algorithms so that high print quality can be achieved. Here, we develop a stochastic printer model based on the dot statistics of the IJ printer.

We present a clustered minority pixel error diffusion halftoning algorithm for which the quantizer threshold is modified based on the past output and a dot activation map. Dot size, dot shape, and dot distribution are more controllable, compared with other clustered dot halftone algorithms such as R. Levien's algorithm (see IS&T 8th Int. Congress on Advances in Non-Impact Printing Technologies, p.280-2, 1992). This method also effectively reduces structured worm-like artifacts in midtones that occur in Levien's algorithm. The dot distribution is further improved by using different error diffusion weights for different input gray levels.

We address the problem of optimal periodic clustered-dot color-screen design. In traditional clustered-dot color screening, the screen for each colorant is rotated to a different angle relative to the others. If the angles are not carefully chosen, visible moire and rosette artifacts may appear These artifacts primarily result from the interaction of the periodic structures associated with the halftone screens of different colorants. Registration errors can also introduce unwanted artifacts in the screened images. Using lattice theory and a model for the perceived rendered halftone, we present a systematic method for designing moire- and rosette-free clustered-dot color screens for discrete-raster color systems. We also investigate strategies for choosing the periodicities so that the resulting screen is robust to registration errors.

A novel halftoning approach that has embedded in it a model for the electrophotographic process is presented. Models for the laser beam, exposure of the organic photoconductor, and the resulting absorptance on the paper are embedded into the direct binary search (DBS) halftoning algorithm. The algorithm is applicable to any arbitrary pixel modulation scheme and is also highly portable between different electrophotographic print engines. Computational issues are addressed to make the approach viable. Results show good exploitation of pixel modulation and improvement over DBS with no printer model throughout most of the tone scale.

A model for the human visual system (HVS) is an important component of many halftoning algorithms. Using the iterative direct binary search (DBS) algorithm, we compare the halftone texture quality provided by four different HVS models that have been reported in the literature. Choosing one HVS model as the best for DBS, we then develop an approximation to that model which significantly improves computational performance while minimally increasing the complexity of the code. By varying the parameters of this model, we find that it is possible to tune it to the gray level being rendered, and to thus yield superior halftone quality across the tone scale. We then develop a dual-metric DBS algorithm that effectively provides a tone-dependent HVS model without a large increase in computational complexity.

We introduce a new approach to optimal image scaling called resolution synthesis (RS). In RS, the pixel being interpolated is first classified in the context of a window of neighboring pixels; and then the corresponding high-resolution pixels are obtained by filtering with coefficients that depend upon the classification. RS is based on a stochastic model explicitly reflecting the fact that pixels falls into different classes such as edges of different orientation and smooth textures. We present a simple derivation to show that RS generates the minimum mean-squared error (MMSE) estimate of the high-resolution image, given the low-resolution image. The parameters that specify the stochastic model must be estimated beforehand in a training procedure that we have formulated as an instance of the well-known expectation-maximization (EM) algorithm. We demonstrate that the model parameters generated during the training may be used to obtain superior results even for input images that were not used during the training.

A spread spectrum watermarking algorithm for hardcopy imaging applications using a human visual system model based halftoning technique called direct binary search (DBS) is presented. DBS is used to design a halftone that jointly optimizes a human visual system based error metric and correlation watermark detector output. The original image is required for the correlation watermark detector.

Screening is a low complexity halftoning algorithm that has been widely used in many applications. However, screen design requires that the stacking property be obeyed. This constraint limits the texture quality at each gray level. We present a look-up-table based halftoning algorithm for which the stacking constraint is not necessarily satisfied; but the binary patterns for individual levels are still correlated. The binary patterns are designed level by level using the direct binary search method. The algorithm improves halftone image quality compared with screening.

The direct binary search (DBS) algorithm employs a search heuristic to minimize the mean-squared perceptually filtered error between the halftone and continuous-tone original images. Based on an efficient method for evaluating the effect on the mean squared error of trial changes to the halftone image, we show that DBS also minimizes in a pointwise sense the absolute error under the same visual model, but at twice the viewing distance associated with the mean-squared error metric. This dual interpretation sheds light on the convergence properties of the algorithm, and clearly explains the tone bias that has long been observed with halftoning algorithms of this type. It also demonstrates how tone bias and texture quality are linked via the scale parameter, the product of printer resolution and viewing distance. Finally, we show how the tone bias can be eliminated by tone-correcting the continuous-tone image prior to halftoning it.

We incorporate a higher order measurement-based model for printer dot interactions within the iterative direct binary search (DBS) halftoning algorithm. We also present an efficient strategy for evaluating the change in computational cost as the search progresses. Experimental results are shown which demonstrate the efficacy of the approach.

A wavelet decomposition based surface compression technique to represent the forward transformation for a color printer is presented. The method creates a multi-scale grid structure for a given maximum pointwise error threshold. The error performance, the memory requirement, and the computational cost of this technique are compared with those of minimax sequential linear interpolation (SLI) and minimum mean-squared (MMSE) SLI.

Screen design requires that the stacking property be obeyed. This constraint limits the texture quality at each gray level. To improve the texture quality, we introduce a look-up-table based halftoning algorithm for which the stacking constraint is not necessarily satisfied but the binary patterns for individual levels are still correlated. The algorithm improves the halftone image quality compared with screening.

The direct binary search (DBS) algorithm is an iterative method designed to minimize one metric of error between the grayscale original and halftone image. This metric incorporates a model for the human visual system. To achieve the optimization, an initial halftone is adjusted until a local minimum of the metric is achieved at each pixel. However this local minimum can be shown to simultaneously minimize another metric of error using another visual model; this alternative metric and visual model combination are referred to as the dual interpretation of DBS. An analysis of this dual reveals the origin of tone reproduction bias in DBS. Many model based halftoning algorithms are effected by this problem. The dual can also be used to quantify the perceived tone reproduction capabilities of DBS and provides insight on why DBS yield very high quality halftones. Despite the complexity of the DBS algorithm it can be implemented with surprising efficiency. We demonstrate how the algorithm simultaneously exploits two different visual models and error metrics to efficiently yield very high quality halftones.

We derive a closed form solution to the problem of finding an optimal grid structure for sequential linear interpolation (SLI) of a 3-D function. The criterion of optimality is to minimize the maximum absolute interpolation error. The approach we take is to equate the interpolation error within each interpolated cell. We compare the performance of our method with that of a previously developed iterative method.

Many halftoning algorithms tend to render highlight regions with objectionable dot distributions. To alleviate this artifact, we introduce a halftone postprocessing algorithm called the Springs algorithm. The objective of Springs is to rearrange dots in affected regions for a smoother more attractive rendition. We describe the Springs algorithm, and we show results which demonstrate its effectiveness. The heart of this algorithm is a simple dot-rearrangement heuristic which results in a more isotropic dot distribution. The approach is to treat any well-isolated dot as if it were connected to neighboring dots by springs, and to move it to a location where the energy in the springs is a minimum. Applied to the whole image, this could degrade the halftone appearance. However, Springs only moves dots in selected regions of the image. To select these regions, Springs employs a segmentation scheme which is based on finding light regions which do not exhibit strong edge structures.

We exploit the techniques of tree structured vector quantization (TSVQ), branch and bound search, and the triangle inequality to speed the search of large image databases. Our method can reduce search computation required to locate images which best match a query image provided by a user. While exact search is possible, a free parameter allows search accuracy to be reduced, thereby providing a substantially better speed-up versus accuracy tradeoff.

The direct binary search (DBS) algorithm is an iterative method which minimizes a metric of error between the grayscale original and halftone image. This is accomplished by adjusting an initial halftone until a local minimum of the metric is achieved at each pixel. The metric incorporates a model for the human visual system (HVS). In general, the DBS time complexity and halftone quality depend on three factors: the HVS model parameters, the choice of initial halftone, and the search strategy used to update the halftone. Despite the complexity of the DBS algorithm, it can be implemented with surprising efficiency. We demonstrate how the algorithm exploits the model for the HVS to efficiently yield very high quality halftones.

We introduce a new approach that we call sequential linear interpolation (SLI) for approximating multidimensional nonlinear functions. The SLI is a partially separable grid structure that allows us to allocate more grid points to the regions where the function to be interpolated is more nonlinear. This approach reduces the mean squared error (MSE) between the original and approximated function while retaining much of the computational advantage of the conventional uniform grid interpolation. To obtain the optimal grid point placement for the SLI structure, we appeal to an asymptotic analysis similar to the asymptotic vector quantization (VQ) theory. In the asymptotic analysis, we assume that the number of interpolation grid points is large and the function to be interpolated is smooth. Closed form expressions for the MSE of the interpolation are obtained from the asymptotic analysis. These expressions are used to guide us in designing the optimal SLI structure. For cases where the assumptions underlying the asymptotic theory are not satisfied, we develop a postprocessing technique to improve the MSE performance of the SLI structure. The SLI technique is applied to the problem of color printer characterization where a highly nonlinear multidimensional function must be efficiently approximated. Our experimental results show that the appropriately designed SLI structure can greatly improve the MSE performance over the conventional uniform grid.

We are developing a new graduate program at Purdue in Video and Image Systems Engineering (VISE). The project is comprised of three parts: a new curriculum centered around a degree option in VISE to be earned as part of the Masters or Ph.D. degrees; a state-of-the-art lecture/laboratory facility for instruction, laboratory experiments, and project and homework activities in VISE courses; and enhancement of existing courses and development of new courses in the VISE area.

We describe an algorithm to design a frequency modulated screen using the direct binary search algorithm. Compared with the direct binary search algorithm itself, we show that we can maintain halftone image quality while significantly reducing the required computation.

We present a new method for digitally interpolating images to higher resolution. It consists of two phases: rendering and correction. The rendering phase is edge-directed. From the low resolution image data, we generate a high resolution edge map by first filtering with a rectangular center-on-surround-off filter and then performing piecewise linear interpolation between the zero crossings in the filter output. The rendering phase is based on bilinear interpolation modified to prevent interpolation across edges, as determined from the estimated high resolution edge map. During the correction phase, we modify the mesh values on which the rendering is based to account for the disparity between the true low resolution data, and that predicted by a sensor model operating on the high resolution output of the rendering phase. The overall process is repeated iteratively. We show experimental results which demonstrate the efficacy of our interpolation method.

This paper describes the design of color filters for a surface color measurement device. The function of the device is to return the XYZ tristimulus vector characterizing the color of the surface. The device is designed to measure emissive as well as reflective surfaces. It uses an internal set of LEDs to illuminate reflective surfaces while characterizing their color under assumed standard illuminants. In the design of the filters, we formulate a nonlinear optimization problem with the goal of minimizing error in the uniform color space CIE L*a*b*. Our optimization criteria employs a technique to retain a linear structure while approximating the true L*a*b* error. In addition, our solution is regularized to account for system noise, filter roughness, and filter implementation errors. Experimental results indicate average and worst-case device accuracy of 0.27 L*a*b* DeltaE units and 1.56 L*a*b* DeltaE units for a "system tolerance" of 0.0005.

Searching an image for the occurrence of a pattern or a template is an essential step in a number of image processing applications. We propose a new multiresolution matching criterion based on the generalized log likelihood ratio. We also developed a multiscale search technique which facilitates finding the best solution by searching a small subset of the entire set of possible template locations. The search technique is designed to keep the amount of computation at each resolution approximately the same. The results obtained on our example images demonstrate the robustness and accuracy of the matching criterion along with a speed-up of over two orders of magnitude by the search technique.

This paper describes the design of color filters for a surface color measurement device. The function of the device is to return the XYZ tristimulus vector characterizing the color of the surface. The device is designed to measure emissive as well as reflective surfaces. It uses an internal set of LEDs to illuminate reflective surfaces while characterizing their color under assumed standard illuminants. In the design of the filters, we formulate a nonlinear optimization problem with the goal of minimizing error in the uniform color space CIE L*a*b*. Our optimization criteria employs a technique to retain a linear structure while approximating the true L*a*b* error. In addition, our solution is regularized to account for system noise, filter roughness and filter implementation errors. Experimental results indicate average and worst case device accuracy of 0.27 L*a*b* /spl Delta/E units and 1.56 L*a*b* /spl Delta/E units for a "system tolerance" of 0.0005.

Proposes an efficient vector quantization (VQ) technique called sequential scalar quantization (SSQ). The scalar components of the vector are individually quantized in a sequence, with the quantization of each component utilizing conditional information from the quantization of previous components. Unlike conventional independent scalar quantization (ISQ), SSQ has the ability to exploit intercomponent correlation. At the same time, since quantization is performed on scalar rather than vector variables, SSQ offers a significant computational advantage over conventional VQ techniques and is easily amenable to a hardware implementation. In order to analyze the performance of SSQ, the authors appeal to asymptotic quantization theory, where the codebook size is assumed to be large. Closed-form expressions are derived for the quantizer mean squared error (MSE). These expressions are used to compare the asymptotic performance of SSQ with other VQ techniques. The authors also demonstrate the use of asymptotic theory in designing SSQ for a practical application (color image quantization), where the codebook size is typically small. Theoretical and experimental results show that SSQ far outperforms ISQ with respect to MSE while offering a considerable reduction in computation over conventional VQ at the expense of a moderate increase in MSE.

Determining the parameters of motion within a time-varying scene is an important problem in such fields as computer vision, motion compensated video coding, and tracking. Most motion estimation algorithms operate on image data that has been sampled in both space and time. However, very little work has been done to investigate the impact of the underlying sampling strategy on the motion estimation problem. The authors investigate motion estimation with time-sequentially sampled image data. They consider both centroid-displacement-based and Fourier-based approaches to motion estimation with this type of data. For comparision, they also examine the performance of these estimators with conventional, frame-instantaneously sampled data. The motion estimators are developed and evaluated in the context of the tracking problem. In particular, they present extensive numerical results showing the performance of the motion estimators in a simulated tracking environment within which the assumptions underlying the development of the estimators are violated. These results suggest empirical rules for choosing parameter values for the estimators.

The new approach to undergraduate DSP education at Purdue is based on a simple idea: emphasize applications. Students are assumed to have a significant exposure to sampling and discrete-time signals, systems, and transforms at the junior level. In the senior course, the traditional DSP topics of digital filter design, the DFT, radix-2 FFT's, and quantization are covered in the first five weeks of the semester. Coverage of these topics is augmented by treatment in the laboratory component of the course using diverse software tools and by Matlab based homework assignments. The remainder of the course is devoted to treating the topics of speech processing and image processing in substantial depth and involves a design project. The course has been very successful in terms of increasing enrolment and outstanding student evaluations.>

Introduces a new approach which we call sequential linear interpolation (SLI) for interpolating multidimensional nonlinear functions. SLI grid points can be nonuniformly placed. By applying asymptotic analysis, we obtain optimal conditions for placing the interpolation grid points in the SLI grid structure to minimize the interpolation error. Thus, we use the grid points more efficiently. We apply this technique to the color printer calibration problem where highly nonlinear functions must be efficiently implemented.>

Two iterative methods are used to reconstruct a continuous-tone image from its halftone: successive approximation and projection onto convex sets. The halftone image is assumed to be generated by thresholding with an unknown screen function which is estimated from the halftone image in the first stage of the reconstruction process. The screen function is an integral part of both iterative reconstruction methods. The other key element is a constraint on the bandwidth of the reconstructed image. Relaxing this constraint during the course of the iterations not only reduces the RMS error after each iteration but also achieves a relatively sharp full-band reconstruction at the end.>

A maximum likelihood (ML) estimate of the magnitude of a complex-valued image from measurements of its Fourier transform in a limited region offset from the origin is derived under the assumption of independent, uniformly distributed image phase samples. The expectation-maximization (EM) algorithm is employed to solve the resulting nonlinear maximum likelihood equation, and it yields a computationally efficient iterative estimator. Reconstructions from this algorithm contain significantly more energy than conventional reconstructions, but with only slightly improved reconstruction quality.>

Two new algorithms are presented for estimating optical flow fields. These algorithms are based on the spatiotemporal properties of the data obtained time-sequentially from the field of view (FOV). The first is a centroid-based algorithm which estimates the velocity from the distance traveled by the centroid of the time-sequential data during the observation period. This algorithm performs well considering its simplicity. Next, a Fourier-based algorithm is presented. When motion within a time-varying scene is constant speed translation, the spectral energy is confined to a plane in the frequency domain. It is shown how to estimate a set of points on this plane using the 1-D FFT of the weighted time-sequential data. An estimate of the velocity vector is obtained by finding the orientation of the plane which gives a least squares fit to this set.>

The concept of perfect reconstruction in filter banks is examined using the Smith form of the polyphase matrix. A class of M-channel FIR (finite-impulse response) perfect-reconstruction filter banks is introduced that includes the recently developed lossless filter banks. With the proposed filter banks, the synthesis filters are of the same complexity as the analysis filters. In addition, the synthesis filter bank is easily obtained from the analysis filter bank by inverting a set of M*M constant coefficient matrices. A statistical method for the design of such filter banks is presented. These can be used as building blocks in large tree-structured filter banks where the overall number of channels is any composite integer. An analysis of the computational and storage complexity of such tree structures is given. The results of the analysis are simple formulas for required storage and computation, which helps in the selection of efficient tree structures. >

The design of multirate bandpass filters requires a search for the optimum multistage structure. The size of the search space grows exponentially as the filter bandwidth decreases, and excessive computer time is required to search it exhaustively. A fast branch-and-bound algorithm for searching this space is presented. In our tests, this algorithm decreases search time by an average factor which increases from 7.7 to 23.0 as the filter passband width decreases from 8 × 10-4to 3.2 × 10-6. It is also shown that when coefficient storage is the cost criterion, the actual filter structure resembles an unrealizable optimum filter structure for small bandwidth filters.

Extraction of 3D information from 2D views is an important problem in the inspection of manufactured parts. Inspection for defects and correctness of shape and size requires accurate 3D information. In this paper we present a method for 3D reconstruction by backprojection of 2D views. This method uses readily available passive imaging sensors and a computer-controlled positioner to directly produce a 3D reconstruction of the part. Consistency based processing is introduced as a method of increasing reconstruction accuracy by exploiting redundant data in the multiple images. Maximum effciency and accuracy in reconstruction is achieved using optimal view patterns and reliable silhouette extraction techniques. Although this method is applicable to the inspection of general machine parts, our immediate application was in the inspection of laser-drilled rivet holes and laser-drilled cooling holes in jet engine turbine blades.

A number of different methods for reconstruction of bandlimited 1-D and 2-D signals from nonuniformly spaced samples are compared experimentally. The six 1-D algorithms are solution for Fourier series coefficients, two methods due to Yen, Papoulis' coordinate transformation, cubic splines, and an iterative method similar to those of Wiley and Gerchberg. Three of the 2-D algorithms are iterative. They differ in the mode of space domain error correction. The fourth 2-D algorithm is the thin-plate splines method. The methods are evaluated on the basis of mean signal-to-error energy ratios calculated over an ensemble consisting of reconstructions of several functions, each sampled with many different random point sets. Sample plots of reconstructions are also included.

With spatiotemporal signals sampled on a non-time-sequential lattice, continuous- and discrete-parameter convolution are equivalent. In many real systems, however, signals are actually sampled on a time-sequential lattice. The implementation of continuous-parameter 3D convolution with such sampled data generally requires interpolation to a non-time-sequential lattice. The exception occurs with data sampled time-sequentially according to a congruential ordering. With this data, a simple 1D convolution may be performed; and the frequency response of a desired spatiotemporal filter may be mapped directly to the frequency response of a 1D digital filter.