Sungnyun Kim, Haofu Liao, Srikar Appalaraju, Peng Tang, Zhuowen Tu, Ravi Kumar Satzoda, R. Manmatha, Vijay Mahadevan, Stefano Soatto. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.

In this work, instead of directly predicting the pixel-level segmentation masks, the problem of referring image seg-mentation is formulated as sequential polygon generation, and the predicted polygons can be later converted into segmentation masks. This is enabled by a new sequence-to-sequence framework, Polygon Transformer (PolyFormer), which takes a sequence of image patches and text query to-kens as input, and outputs a sequence of polygon vertices autoregressively. For more accurate geometric localization, we propose a regression-based decoder, which predicts the precise floating-point coordinates directly, without any co-ordinate quantization error. In the experiments, PolyFormer outperforms the prior art by a clear margin, e.g., 5.40% and 4.52% absolute improvements on the challenging Re-fCOCO+ and RefCOCOg datasets. It also shows strong generalization ability when evaluated on the referring video segmentation task without fine-tuning, e.g., achieving competitive 61.5% J&F on the Ref-DAVIS17 dataset.

We present a new formulation for structured information extraction (SIE) from visually rich documents. We address the limitations of existing IOB tagging and graph-based formulations, which are either overly reliant on the correct ordering of input text or struggle with decoding a complex graph. Instead, motivated by anchor-based object detectors in computer vision, we represent an entity as an anchor word and a bounding box, and represent entity linking as the association between anchor words. This is more robust to text ordering, and maintains a compact graph for entity linking. The formulation motivates us to introduce 1) a Document Transformer (DocTr) that aims at detecting and associating entity bounding boxes in visually rich documents, and 2) a simple pre-training strategy that helps learn entity detection in the context of language. Evaluations on three SIE benchmarks show the effectiveness of the proposed formulation, and the overall approach outperforms existing solutions.

Vision-based detection of hazards in the path of ego-vehicle is a challenging task because of the variability in the type of hazards. In this paper, we present a detailed review of vision-based hazard detection methods followed by a set of new architectures and methods include semantic segmentation, instance segmentation, object detection, monocular vision with depth fusion based methods and ensembles. Additionally, we propose a set of new (and some old) benchmarking metrics that accurately capture the effectiveness of hazard detection algorithms, in terms of both algorithmic accuracy and deployability in vehicles. Detailed performance evaluations show that the proposed methods using Mask-RCNN, ensembles and monocular-stereo fusion surpass current state-of-the-art techniques in terms of accuracy and computational speed. Additionally, our fusion based object detection architectures provide a good tradeoff between accuracy (e.g. Average Precision) and computation requirements, with operating speeds that are 15 times faster than existing techniques.

Existing nighttime vehicle detection methods use color as the primary cue for detecting vehicles. However, complex road and ambient lighting conditions, and camera configurations can influence the effectiveness of such explicit rule and threshold-based methods. In this paper, there are three main contributions. First, we present a novel method to detect vehicles during nighttime involving both learned classifiers and explicit rules, which can operate in the presence of varying ambient lighting conditions. The proposed method that is titled as Vehicle Detection using Active-learning during Nighttime (VeDANt) employs a modified form of active learning for training Adaboost classifiers with Haar-like features using gray-level input images. The hypothesis windows are then further verified using the proposed techniques involving perspective geometries and color information of the taillights. Second, VeDANt is extended to analyze the dynamics of the vehicles during nighttime by detecting three taillight activities-braking, turning left, and turning right. Third, we release three new and fully annotated Laboratory for Intelligent and Safe Automobiles-Night data sets with over 5000 frames for evaluation and benchmarking, which capture a variety of complex traffic and lighting conditions. Such comprehensively annotated and complex public data sets are a first in the area of nighttime vehicle detection. We show that VeDANt is able to detect vehicles during nighttime with over 98% accuracy and less than 1% false detections.

Naturalistic driving studies (NDSs) capture drive data from multiple sensor modalities over long periods of time and under varying road conditions. NDS data reduction dictionaries list a range of events that are directly related to the conflicts and threat posed by the dynamics of the surrounding vehicles on the ego-vehicle. Manual reduction of such large scale data for events/conflicts related to dynamics of multiple vehicles is inefficient and prone to errors. In this paper, we present drive analysis techniques for automated NDS data reduction that can be deployed to identify, quantify and visualize threats posed to the ego-vehicle. In this regard, we propose safe maneuver zones (SMZs) that are derived based on the dynamics of surrounding vehicles with respect to the ego-vehicle. A set of metrics are formulated using the SMZs to quantify the threat posed by surrounding vehicles on the ego-vehicle. A detailed drive analysis of naturalistic driving data comprising more than 500,000 frames of data from over 5 hours of highway driving is presented. The resulting drive analysis reports characterize 7 different drives using the different metrics from the SMZs.

On-road vehicle detection is a critical operation in automotive active safety systems such as collision avoidance, merge assist, lane change assistance, etc. In this paper, we present VeDAS-Vehicle Detection using Active learning and Symmetry. VeDAS is a multipart-based vehicle detection algorithm that employs Haar-like features and Adaboost classifiers for the detection of fully and partially visible rear views of vehicles. In order to train the classifiers, a modified active learning framework is proposed that selects positive and negative samples of multiple parts in an automated manner. Furthermore, the detected parts from the classifiers are associated by using a novel iterative window search algorithm and a symmetry-based regression model to extract fully visible vehicles. The proposed method is evaluated on seven different datasets that capture varying road, traffic, and weather conditions. Detailed evaluations show that the proposed method gives high true positive rates of over 95% and performs better than existing state-of-the-art rear-view-based vehicle detection methods. Additionally, VeDAS also detects partially visible rear views of vehicles using the residues left behind after detecting the fully visible vehicles. VeDAS is able to detect partial rear views with a detection rate of 87% on a new partially visible rear-view vehicle dataset that we release as part of this paper.

Many types of epileptic seizures are accompanied by characteristic eye movement, along with the movement of other parts of the body. Vision-based detection of eye movement in the context of seizures is less explored. The contribution of this paper is three-fold: (1) a compute-efficient algorithm for eyeball detection is proposed and evaluated on a standard database and yields an accuracy of 95%. (2) A new video dataset for absence seizure has been created. (3) The proposed algorithm for eyeball detection is applied on this dataset in order to detect the occurrence of absence seizure.

In order to understand dangerous situations in the driving environment, naturalistic driving studies (NDS) are conducted by collecting and analyzing data from sensors looking inside and outside of the car. Manually processing the overwhelming amounts of data that are generated in such studies is very comprehensive. We propose a method for automatic data reduction for NDS based on stereo vision vehicle detection and tracking during day- and nighttime. The developed system can automatically register five NDS events, mainly related to intersections, from an existing NDS dictionary. We propose a new drive event which takes advantage of the extra dimension provided by stereo vision. In total, six drive events are selected on the basis of them being problematic to detect automatically using conventional monocular computer vision approaches. The proposed system is evaluated on day-and nighttime data, resulting in drive analysis report. The proposed system reach an overall precision of 0.78 and an overall recall of 0.72.

Analysis of naturalistic driving data provides a rich set of semantics which can be used to determine the driving characteristics that could lead to crashes and near-crashes. In this paper, we introduce “drive quality” analysis as part of the drive analysis process of naturalistic driving studies (NDSs) that we have previously introduced in [1]. In this first work on drive quality analysis for NDS data reduction, lane change maneuvers that are reduced from naturalistic driving data are further analyzed in a detailed manner in order to characterize them. Visual data from multiple perspectives and the data from in-vehicle sensors are used to characterize lane changes based on both the ego-vehicle kinematics and ego-vehicle surround dynamics. According to available literature on NDS and data reduction, this is the first work that presents an analysis of visual data from multiple perspectives to characterize and extract semantics related to ego-vehicle maneuvers in NDSs such as SHRP2.

In the recent years, mobile computing platforms are becoming increasingly cheaper and yet more powerful in terms of computational resources. Automobiles provide a suitable environment to deploy such mobile platforms in order to provide low cost driver assistance systems. In this paper, we propose Snap-DAS which is a vision-based driver assistance system that is implemented on a SnapdragonTMembedded platform. A forward facing camera combined with the SnapdragonTMplatform constitute Snap-DAS. The compute efficient implementation of the LASeR lane estimation algorithm in [1] is exploited to implement a set of lane related functions on Snap-DAS, which include lane drift warning and lane change event detection. A detailed evaluation is performed on live data and Snap-DAS is also field tested on freeways. Furthermore, we explore the possibility of using Snap-DAS for analyzing drives for online naturalistic driving studies.

Vision-based lane detection is a critical component of modern automotive active safety systems. Although a number of robust and accurate lane estimation (LE) algorithms have been proposed, computationally efficient systems that can be realized on embedded platforms have been less explored and addressed. This paper presents a framework that incorporates contextual cues for LE to further enhance the performance in terms of both computational efficiency and accuracy. The proposed context-aware LE framework considers the state of the ego vehicle, its surroundings, and the system-level requirements to adapt and scale the LE process resulting in substantial computational savings. This is accomplished by synergistically fusing data from multiple sensors along with the visual data to define the context around the ego vehicle. The context is then incorporated as an input to the LE process to scale it depending on the contextual requirements. A detailed evaluation of the proposed framework on real-world driving conditions shows that the dynamic and static configuration of the lane detection process results in computation savings as high as 90%, without compromising on the accuracy of LE.

Naturalistic driving studies (NDSs) capture large volumes of drive data from multiple sensor modalities, which are analyzed for critical information about driver behavior and driving characteristics that lead to crashes and near crashes. One of the key steps in such studies is data reduction, which is defined as a process by which “trained employees” review segments of driving video and record a taxonomy of variables that provides information regarding the sequence of events leading to crashes. Given the volume of sensor data in NDSs, such manual analysis of the drive data can be time-consuming. In this paper, we introduce “drive analysis” as one of the first steps toward automating the process of extracting midlevel semantic information from raw sensor data. Techniques are proposed to analyze the sensor data from multiple modalities and to extract a set of 23 semantics about lane positions, vehicle localization within lanes, vehicle speed, traffic density, and road curvature. The proposed techniques are demonstrated using real-world test drives comprising over 150 000 frames of visual data, which are also accompanied by vehicle dynamics that are captured from an in-vehicle controller-area-network bus, an inertial motion unit, and a Global Positioning System.

Accurate and efficient lane estimation is a critical component of active safety systems in automobiles such as lane departure warning systems. A substantial number of lane estimation methods have been proposed and evaluated in literature. However, a common set of evaluation metrics that assess different components of lane estimation process has not been addressed. This paper proposes a set of performance evaluation metrics for lane estimation process, that can be deployed to evaluate different kinds of lane estimation algorithms. Evaluation by applying the proposed metrics is demonstrated using a recent lane estimation method.

Naturalistic driving studies (NDS) provide critical information about driving behaviors and characteristics that could lead to crashes and near-crashes. Such studies involve analysis of large volumes of data from multiple sensors and detection and extraction of critical events is an important step in NDS. This paper introduces techniques that analyze the visual data complemented with other sensors in the vehicle to determine critical events related to lane drifts, road departures and road delineations. To the best knowledge of the authors, this is the first work that detects and extract events listed in visual reference dictionary of NDS studies like Strategic Highway Research Program 2 (SHRP2). Detailed evaluations with real-world NDS data is presented.

In this paper, we propose a novel method to robustly identify and classify arrow markings in road images. In the proposed method, simple and unique signatures are first derived for the various arrow types, based on signed edge maps and decomposing the arrows into smaller parts. The signed edge maps are processed using Hough Transform (HT), and the resulting Hough spaces are analyzed systematically, using a set of simple rules. The signatures are rotation-invariant and scale-invariant, thereby making the approach robust to variations in the appearance of the arrow markings. It is shown that the method yields a high detection and classification accuracy, of as high as 97% in the test images considered.

In this paper, we propose novel block-based techniques for robust extraction of lane marking edges in complex scenarios, such as in the presence of shadows, vehicles, other road markings etc. The techniques are based on the properties of lane markings and involve a two-stage processing: (1) generation of customized edge maps using histograms of gradient angles, and (2) directional signed edges in combination with Hough Transform to identify lane markings. It is shown that the proposed techniques show a detection accuracy of as high as 98% on test data collected on real road scenarios, representing the various complex cases.

This paper proposes a novel adaptive pseudo-carry compensation truncation (PCT) scheme, which is derived for the multiplexer based array multiplier. The proposed method yields low average error among existing truncation methods. The new PCT based truncated array multiplier outperforms other existing truncated array multipliers by as much as 25% in terms of silicon area and delay, and consumes about 40% less dynamic power than the full-width multiplier for 32-bit operation. The proposed truncation scheme is applied to an image compression algorithm. Due to its low truncation error, the mean square errors (MSE) of various reconstructed images are found to be comparable to those obtained with full-precision multiplication.

Non-collinear edge pixels are equivalent to noise for the linear Hough transform (LHT). Existing methods that reduce the number of points for Hough voting are based on random and/or probabilistic selection. Such methods select both collinear and noisy pixels, thereby incurring unwanted computational costs. In this paper, we propose a novel gradient angle histogram based technique to generate modified straight line edge map (SLEM), which largely retains the straight line edges and eliminates noisy edge pixels. A block-based SLEM generation is proposed to increase the robustness of straight line extraction and validated on test images. Further, effect of varying block sizes on accuracy of straight line detection is studied and appropriate block settings are derived. The proposed gradient angle histogram based method reduces the number of edge pixels by as much as 85%.

Real-time image rotation is a necessary task in many vision based applications. Most available image rotation architectures are serial in nature, while the existing parallel techniques suffer from computational bottlenecks, limiting their performance. In this paper, we propose techniques that not only speed up the rotation process but also reduce area, yielding an area-time efficient architecture. They exploit properties of symmetry in image coordinates for parallel image rotation. We show that the number of computations, the total computation time and area cost of the proposed architecture are lesser by as much as 80%, 55% and 67% respectively when compared to the most efficient parallel architecture in recent literature.

Accelerating Hough transform in hardware has been of interest due its popularity in real-time capable image processing applications. In most existing linear Hough transform architectures, an m xm edge map is serially read for processing, resulting in a total computation time of at least m(2) cycles. In this paper, we propose a novel parallel Hough transform computation method called the Additive Hough transform (AHT), wherein the image is divided using a k x k grid to reduce the total computation time by a factor of k(2). We have also proposed an efficient implementation of the AHT consisting of a look-up table (LUT) and two-operand adder arrays for every angle. Techniques to condense the LUT size have also been proposed to further reduce area utilization by as much as 50%. Our investigations based on employing an 8 x 8 grid shows a 1000 x speedup compared to existing architectures for a range of image sizes. Area-time trade-off analysis has been presented to demonstrate that the area-time product of the proposed AHT-based implementation is at least 43% lower than other implementations reported in the literature. We have also included and characterized a hierarchical addition step in order to generate a global accumulation space equivalent to that of the conventional HT. It is shown that the proposed implementation with the hierarchical addition step remains superior to other methods in terms of both performance and area-time product metrics. Finally, we show that the proposed solution is equally efficient when applied on rectangular images.

In this paper, we propose programmable serialin parallel-out LSB-first and MSB-first modular multipliers for elliptic curve cryptosystems (ECCs). The proposed multipliers can operate in any arbitrary field GF(2m) such that m is less than a maximum field order M. A linear array of processing elements is designed with a parallel switching circuitry to incorporate programmability in the fixed order multipliers. The proposed architectures are qualitatively compared against existing programmable multipliers in terms of gate count, delay and latency. The application specific integrated circuit (ASIC) implementation of the proposed multipliers using TSMC 0.18μm standard cell library is also analyzed.

The Hough transform, a widely used operation in image processing, suffers from being computationally intensive. In this letter, we propose a method called additive Hough transform (AHT) to accelerate Hough transform computation. It is based on parallel processing of k2points, obtained by dividing the edge map into uniform blocks using a k times k grid. It has been shown that AHT on verification gives the same Hough profile as the conventional Hough transform. Further, AHT reduces the total computation time by at least k2times as compared to existing Hough transform architectures.

Image preprocessing is an important step in most real-time neural-network based pattern recognition systems. Thumbnail generation is a commonly used preprocessing method to reduce the hardware complexity of the input layer of a neural network. A direct implementation of the averaging method to create thumbnails incurs high hardware cost. In this paper we propose an optimized averaging method that outperforms standard averaging method in terms of device utilization, maximum operating frequency and latency, when implemented using FPGA design methodology. It eliminates the use of accumulators and dividers and offers hardware savings of as much as 96% over standard averaging method. Also, the proposed averaging is performed in a single cycle making it more suitable for real-time implementations

Modular multiplication in Galois Fields - GF(p) and GF(2/sup m/) is an ineluctable and time stumbling block in public key cryptosystems. Montgomery modular multiplication has emerged as a VLSI efficient implementation of this operation. In this paper, a new scalable and pipelined Montgomery multiplier architecture that unifies the two important finite fields, GF(p) and GF(2/sup m/), is presented. The proposed architecture has successfully reduced the slack of the Montgomery multiplication in GF(2/sup m/) without jeopardizing the timing of its operation in GF(p). Acceleration of multiplication in GF(2/sup m/) for all ranges of modulus and in GF(p) for higher precision modulus is made possible through a new dual field adder and processing unit which can be pipelined in a kernel. The proposed dual field adder has been optimized to operate in an existing architecture that has been retimed to overcome the conflicts for speeding up the pipelined architecture. The latency has been analytically formulated in terms of the input wordlength, modulus precision and number of pipeline stages to evaluate its total computation time. The processing unit has been implemented on FPGA and the experimental results show evidence of throughput rate and latency improvement over existing dual field processing unit.