We present a new framework for creating lane-level detailed HD-maps at scale for autonomous vehicles (AVs). In order to overcome scaling challenges of ground survey-based HD-map creation, we propose a number of innovations by leveraging three data sources: high-resolution aerial imagery, aggregated vehicle telemetry, and a navigation map. We first divide map creation problem into several categories based on lane configurations. The road category is predicted in a supervised setting using aerial imagery, which is pre-processed by using aggregated vehicle telemetry without supervision. We utilized the navigation map to guide the process along the road network and used aerial imagery and aggregated vehicle telemetry to extract lane level features at each step. We propose a multi-task convolutional neural network (CNN) to predict road polygons, road-way centerline, number of lanes, lane and road edges using both the aerial imagery and the corresponding aggregated vehicle telemetry. The predicted road features for each image are then stitched along a road segment to construct the road and lane edge polylines, which are then used to predict lane marking and road edge types in a sliding window fashion along the road segment. The extracted features are finally utilized to calculate higher level features for each point in the HD-map. Our experimental results show that the proposed framework works well, offering a flexible solution for creating HD-maps for AVs at scale.

Automated license plate recognition (ALPR) is essential in several roadway imaging applications. For ALPR systems deployed in the United States, variation between jurisdictions on character width, spacing, and the existence of noise sources (e.g., heavy shadows, non-uniform illumination, various optical geometries, poor contrast, and so on) present in LP images makes it challenging for the recognition accuracy and scalability of ALPR systems. Font and plate-layout variation across jurisdictions further adds to the difficulty of proper character segmentation and increases the level of manual annotation required for training classifiers for each state, which can result in excessive operational overhead and cost. In this paper, we propose a new ALPR workflow that includes novel methods for segmentation- and annotation-free ALPR, as well as improved plate localization and automation for failure identification. Our proposed workflow begins with localizing the LP region in the captured image using a two-stage approach that first extracts a set of candidate regions using a weak sparse network of winnows classifier and then filters them using a strong convolutional neural network (CNN) classifier in the second stage. Images that fail a primary confidence test for plate localization are further classified to identify localization failures, such as LP not present, LP too bright, LP too dark, or no vehicle found. In the localized plate region, we perform segmentation and optical character recognition (OCR) jointly by using a probabilistic inference method based on hidden Markov models (HMMs) where the most likely code sequence is determined by applying the Viterbi algorithm. In order to reduce manual annotation required for training classifiers for OCR, we propose the use of either artificially generated synthetic LP images or character samples acquired by trained ALPR systems already operating in other sites. The performance gap due to differences between training and target domain distributions is minimized using an unsupervised domain adaptation. We evaluated the performance of our proposed methods on LP images captured in several US jurisdictions under realistic conditions.

Skin prick test is a common diagnosis method for allergic diseases. The test causes a controlled allergic response and provokes a wheal on the skin, whose size and shape change depending on the sensitivity of a patient against a specific allergen. The physical size of the wheal can be estimated from images acquired by digital cameras, which requires segmentation of the wheal from the captured SPT images. In this paper, we propose a new method for wheal detection from skin prick test images. The proposed method operates on localized images using a set of calibration marks and a perspective transformation. An initial segmentation of the localized SPT image is performed using normalized cuts to identify coherent regions in the image. The segmented region containing the wheal is selected by formulating an optimization problem to identify the segment with distinctive color, texture, and shape attributes. Our experiments conducted on images acquired from 50 different patients show the performance of the proposed wheal detection algorithm.

High Occupancy Vehicle (HOV) and High Occupancy Tolling (HOT) lanes have been commonly practiced in several jurisdictions to reduce traffic congestion and promote car pooling. Camera-based methods have been recently proposed for a cost-efficient, safe and effective HOV/HOT lane enforcement with the prevalence of video cameras in transportation imaging applications. An important step in automated lane enforcement systems is classification of localized window/windshield images to distinguish passenger from no-passenger vehicles to identify violators. This can be performed using deep convolutional neural networks (CNNs), which are shown to significantly outperform hand-crafted features in several classification tasks. Training/fine-tuning CNNs require a set of passenger/no-passenger images manually labeled by an operator, which requires substantial time and effort that can result in excessive operational cost and overhead. In this paper, we study adaptability of three popular CNNs (i.e., AlexNet, VGG-M, and GoogLeNet) across different domains in classifying passenger/no-passenger images as part of an automated HOV/HOT lane enforcement system. Our experiments over 40, 000 side-view vehicle images show many interesting insights for domain adaptability of these deep learning architectures.

Due to the high volume of traffic on modern roadways, transportation agencies have proposed high occupancy vehicle (HOV) and high occupancy tolling (HOT) lanes to promote carpooling. Enforcement of the rules of these lanes is currently performed by roadside enforcement officers using visual observation. Officer-based enforcement is, however, known to be inefficient, costly, potentially dangerous, and ultimately ineffective. Violation rates up to 50%-80% have been reported, whereas manual enforcement rates of less than 10% are typical. Near-infrared (NIR) camera systems have been recently proposed to monitor HOV/HOT lanes and enforce the regulations. These camera systems bring an opportunity to automatically determine vehicle occupancy from captured HOV/HOT NIR images. Due to their ability to see through windshields of vehicles, these cameras also enable enforcement of other passenger compartment violations such as seatbelt violation and driver cell phone usage, in addition to determining vehicle occupancy. In this paper, we propose computer vision methods for detecting vehicle occupancy, seatbelt violation, and driver cell phone usage from NIR images captured from HOV/HOT lanes. Our methods consist of two stages. First, we localize the vehicle's front windshield and side window from captured HOV/HOT images using the deformable part model (DPM). Next, we define a region of interest in the localized images for each violation type and perform image classification using one of the local aggregation-based image features, i.e., bag-of-visual-words (BOW), vector of locally aggregated descriptors (VLAD), and Fisher vectors (FV), and compare their performances for each case. We also compare the performance of DPM-based detection with the image classification methods for vehicle occupancy and seatbelt violation detection. A data set over 4000 images including front/side view vehicle images with seatbelt and cell phone violations was collected on a public roadway and is used to perform the experiments.

We propose a color barcode framework for mobile phone applications by exploiting the spectral diversity afforded by the cyan (C), magenta (M), and yellow (Y) print colorant channels commonly used for color printing and the complementary red (R), green (G), and blue (B) channels, respectively, used for capturing color images. Specifically, we exploit this spectral diversity to realize a three-fold increase in the data rate by encoding independent data in the C, M, and Y print colorant channels and decoding the data from the complementary R, G, and B channels captured via a mobile phone camera. To mitigate the effect of cross-channel interference among the print-colorant and capture color channels, we develop an algorithm for interference cancellation based on a physically-motivated mathematical model for the print and capture processes. To estimate the model parameters required for cross-channel interference cancellation, we propose two alternative methodologies: a pilot block approach that uses suitable selections of colors for the synchronization blocks and an expectation maximization approach that estimates the parameters from regions encoding the data itself. We evaluate the performance of the proposed framework using specific implementations of the framework for two of the most commonly used barcodes in mobile applications, QR and Aztec codes. Experimental results show that the proposed framework successfully overcomes the impact of the color interference, providing a low bit error rate and a high decoding rate for each of the colorant channels when used with a corresponding error correction scheme.

The Cincinnati Waterfront Daguerreotype Panorama, owned by the Public Library of Cincinnati and Hamilton County, was brought to George Eastman House in 2007 for conservation treatment. At George Eastman House the plates were imaged under a microscope to capture details at nearly their full resolution and composited mosaic images comprised of 756 images per plate were created. Although the mosaic yields an otherwise unattainable contiguous high resolution and detailed view of the Cincinnati waterfront at the time of capture, non-uniformity of the capture illumination introduced a patterning that impairs the visual and analytic value of the mosaic. In this paper, we develop a mathematical model for the mosaic image capture to analyze the degradation caused by the illumination non-uniformity. Motivated by the analysis, we develop a simple signal processing solution to correct for the lighting variation by utilizing suitably tuned frequency domain filtering. Using the proposed method, we demonstrate that the patterning due to non-uniformity of illumination is corrected, resulting in a significantly improved mosaic.

Encoding data independently in cyan, magenta, and yellow (CMY) print colorant channels with detection in complementary Red, green, and blue (RGB) image capture channels offers an attractive framework for extending monochrome barcodes to color with increased data rates. The undesired absorption of colorants in regions of spectral sensitivity of the noncomplementary capture channels, however, gives rise to cross-channel color interference that significantly deteriorates the performance of the color barcode system. In this paper, we propose an Expectation Maximization (EM) style algorithm to estimate and cancel this color interference and improve the overall performance of the barcode system. Our method utilizes a physical model for print-capture process where the model parameters vary depending on printer, capture device, and illumination. We estimate the model parameters using an iterative EM-style approach and obtain an estimate of CMY colorant channels from the scanned RGB barcode by using the estimated model parameters. Our experimental results show that the proposed method mitigates the effect of color interference and significantly reduces the bit error rates for the recovered data.

Halftone dot orientation modulation has recently been proposed as a method for data hiding in printed images. Extraction of data embedded with halftone orientation modulation is accomplished by computing, from the scanned hardcopy image, detection statistics that uniquely identify the embedded orientation. From a communications perspective, this data hiding setup forms an interesting class of channels with dot orientation as input and a vector of statistics as the output. This paper derives capacity expressions for these channels that allow for numerical evaluation of the capacity. Results provide significant insight for orientation modulation based print-scan resilient data hiding: the capacity varies significantly as a function of the image graylevel and experimentally observed error free data rates closely mirror the variation in capacity.

We present a new high capacity color barcode. The barcode we propose uses the cyan, magenta, and yellow (C,M,Y) colorant separations available in color printers and enables high capacity by independently encoding data in each of these separations. In each colorant channel, payload data is conveyed by using a periodic array of elliptically shaped dots whose individual orientations are modulated to encode the data. The orientation based data encoding provides beneficial robustness against printer and scanner tone variations. The overall color barcode is obtained when these color separations are printed in overlay as is common in color printing. A reader recovers the barcode data from a conventional color scan of the barcode, using red, green, and blue (R,G,B) channels complementary, respectively, to the print C, M, and Y channels. For each channel, first the periodic arrangement of dots is exploited at the reader to enable synchronization by compensating for both global rotation/scaling in scanning and local distortion in printing. To overcome the color interference resulting from colorant absorptions in noncomplementary scanner channels, we propose a novel interference minimizing data encoding approach and a statistical channel model (at the reader) that captures the characteristics of the interference, enabling more accurate data recovery. We also employ an error correction methodology that effectively utilizes the channel model. The experimental results show that the proposed method works well, offering (error-free) operational rates that are comparable to or better than the highest capacity barcodes known in the literature.

We present a new framework for data hiding in images printed with clustered dot halftones. Our application scenario, like other hardcopy embedding methods, encounters fundamental challenges due to extreme bilevel quantization inherent in halftoning, the stringent requirements of image fidelity, and other unavoidable printing and scanning distortions. To overcome these challenges, while still allowing for automated extraction of the embedded data and a high embedding capacity, we propose a number of innovations. First, we perform the embedding jointly with the halftoning by employing an analytical halftone threshold function that allows steering of the halftone spot orientation within each halftone cell based upon embedded data. In this process, image fidelity is emphasized and, if necessary, the capability to recover individual data values is sacrificed resulting in unavoidable erasures and errors. To overcome these and other sources of errors, we propose a suitable data detection and error control methodology based upon a statistical representation for the print-scan channel that effectively models the channel dependence upon the cover image gray-level. To combat the geometric distortion inherent in the print-scan process, we exploit the periodic halftone structure to recover from global scaling and rotation and propose a novel decision directed synchronization technique that counters locally varying printing distortion. Experimental results demonstrate the power of the proposed framework: we achieve high operational rates while preserving halftone image quality.

We present a forensic technique for analyzing a printed image in order to trace the originating printer. Our method, which is applicable for commonly used electrophotographic (EP) printers, operates by exploiting the geometric distortion that these devices inevitably introduce in the printing process. In the proposed method, first a geometric distortion signature is estimated for an EP printer. This estimate is obtained using only the images printed on the printer and without access to the internal printer controls. Once a database of printer signatures is available, the printer utilized to print a test image is identified by computing the geometric distortion signature from test image and correlating the computes signatures against the printer signatures in the database. Experiments conducted over a corpus of EP printers demonstrate that the geometric distortion signatures of test documents exhibit high correlation with the corresponding printer signatures and a low correlation with other printer signatures. The method is therefore quite promising for forensic printer identification applications. We highlight several of the capabilities and challenges for the method.

By formulating the problem of ordering the outputs observed from a device over time, we pose a new problem in forensics and propose a framework for addressing this problem of device temporal forensics. Our proposed framework is based on a two-stage approach wherein time-dependent device parameters are first estimated from observed outputs and the resulting estimates are then temporally ordered by employing a Markov model for the temporal evolution of device parameters and exploiting the data processing inequality in information theory. We demonstrate and evaluate a simple realization of the framework for digital camera forensics based on photo-response non-uniformity. Results obtained over a database of online images indicate that the method provides accurate temporal ordering.

Clustered-dot halftones are extensively utilized in hardcopy printing. Modulation of the dot orientation in these halftones offers an avenue for data embedding which has been exploited in a number of different methods. We consider the capacity of these channels, modeling them as binary orientation input channels with vector valued output detection statistics. We derive upper bounds on the capacity for three channel conditional distributions corresponding to sub-Gaussian, Gaussian and super-Gaussian distributions. Using experimentally estimated channel parameters our bounds reveal that channel capacity has noticeable variations as a function of gray level. Highlights, shadows and mid-tones offer negligible capacity, on the contrary the regions between highlights and mid-tones or shadows and mid-tones offer high capacity for data embedding.

Halftone image watermarking techniques that allow automated extraction of the embedded watermark data are useful in a variety of document security and workflow applications. The print-scan process inherent in these applications introduces distortions whose characteristics exhibit a strong spatial dependence on the cover image in which the data is embedded. In this paper, we demonstrate that a characterization of this channel dependence and the use of error correction coding that exploits this dependence via image adaptive decoding offers a significant performance gain. We show this advantage in the specific context of a high rate data embedding method that utilizes orientation modulation for data embedding in clustered dot halftones and moment based detection at the receiver. Channel coding for this scenario utilizing convolutional codes and Repeat Accumulate (RA) codes highlights the advantage of the proposed adaptive decoding methodology. The performance for the RA codes with adaptive decoding also reveals that the resulting system has significantly higher operational rates than prior schemes.