Zhenhai Zhu, Radu Soricut. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Pretraining from unlabelled web videos has quickly become the de-facto means of achieving high performance on many video understanding tasks.Features are learned via prediction of grounded relationships between visual content and automatic speech recognition (ASR) tokens.However, prior pretraining work has been limited to only instructional videos; a priori, we expect this domain to be relatively "easy:" speakers in instructional videos will often reference the literal objects/actions being depicted.We ask: can similar models be trained on more diverse video corpora?And, if so, what types of videos are "grounded" and what types are not?We fit a representative pretraining model to the diverse YouTube8M dataset, and study its success and failure cases.We find that visualtextual grounding is indeed possible across previously unexplored video categories, and that pretraining on a more diverse set results in representations that generalize to both noninstructional and instructional domains.

Instructional videos get high-traffic on video sharing platforms, and prior work suggests that providing time-stamped, subtask annotations (e.g., "heat the oil in the pan") improves user experiences.However, current automatic annotation methods based on visual features alone perform only slightly better than constant prediction.Taking cues from prior work, we show that we can improve performance significantly by considering automatic speech recognition (ASR) tokens as input.Furthermore, jointly modeling ASR tokens and visual features results in higher performance compared to training individually on either modality.We find that unstated background information is better explained by visual features, whereas fine-grained distinctions (e.g., "add oil" vs. "add olive oil") are disambiguated more easily via ASR tokens.

Current image captioning methods are usually trained via maximum likelihood estimation. However, the log-likelihood score of a caption does not correlate well with human assessments of quality. Standard syntactic evaluation metrics, such as BLEU, METEOR and ROUGE, are also not well correlated. The newer SPICE and CIDEr metrics are better correlated, but have traditionally been hard to optimize for. In this paper, we show how to use a policy gradient (PG) method to directly optimize a linear combination of SPICE and CIDEr (a combination we call SPIDEr): the SPICE score ensures our captions are semantically faithful to the image, while CIDEr score ensures our captions are syntactically fluent. The PG method we propose improves on the prior MIXER approach, by using Monte Carlo rollouts instead of mixing MLE training with PG. We show empirically that our algorithm leads to easier optimization and improved results compared to MIXER. Finally, we show that using our PG method we can optimize any of the metrics, including the proposed SPIDEr metric which results in image captions that are strongly preferred by human raters compared to captions generated by the same model but trained to optimize MLE or the COCO metrics.

In this paper, a new approach for detecting and analyzing sea surface slicks caused by hydrocarbon seepage of offshore petroleum accumulations has been developed. This approach uses remote sensing radar technology and geophysical exploration techniques and has been developed based on hydrocarbon seepage theory. In this study, Synthetic Aperture Radar (SAR) data were used as the main data source. These data were integrated with gravity data inversed from satellite altimeter data, geophysical abnormal data from airborne magnetic data, and geological data of oil-and gas-bearing basins. Using the geographical information system, the oil and gas accumulating areas were outlined by the prospect models. This approach for the exploration and evaluation for offshore petroleum accumulations has been applied to two study areas in offshore petroleum basins in China: the Bohai Sea and Pearl River Mouth basins. By comparing the drilling outcomes and relative materials, our results show that the application of this integrated method is very effective.

We formulate the mask modeling as a parametric model order reduction problem based on the finite element discretization of the Helmholtz equation. By using a new parametric mesh and a machine learning technique called Kernel Method, we convert the nonlinearly parameterized FEM matrices into affine forms. This allows the application of a well-understood parametric reduction technique to generate compact mask model. Since this model is based on the first principle, it naturally includes diffraction and couplings, important effects that are poorly handled by the existing heuristic mask models. Further more, the new mask model offers the capability to make a smooth trade-off between accuracy and speed.

In this paper, we propose methods to accelerate the solution of multiple related linear systems of equations. Such systems arise, for example, in building pattern libraries for interconnect parasitic extraction, parasitic extraction under process variation, and parameterized interconnect characterization. Our techniques include methods based on a generalized form of ldquorecycledrdquo Krylov subspace methods that use the sharing of information between related systems of equations to accelerate the iterative solution and methods to reuse computational effort during system matrix setup. Experimental results on electrostatic problems demonstrate significant improvement over existing methods that are based on solving each system individually. The proposed methods are generic, fully treat nonlinear perturbations without approximation and can potentially be applied to a wide variety of application domains outside electrostatic analysis.

In this paper we propose methods for fast iterative solution of multiple related linear systems of equations. Such systems arise, for example, in building pattern libraries for interconnect parasitic extraction, parasitic extraction under process variation, and parameterized interconnect characterization. Our techniques are based on a generalized form of "recycled" Krylov subspace methods that use sharing of information between related systems of equations to accelerate the iterative solution. Experimental results on electromagnetics problems demonstrate that the proposed method can achieve a speed-up of 5X~30X compared to direct GMRES applied sequentially to the individual systems. These methods are generic, fully treat nonlinear perturbations without approximation, and can be applied in a wide variety of application domains outside electromagnetics.

This paper is concerned with model order reduction of large scale dynamic systems that have sparse matrix representations, particularly systems with large numbers of input/output ldquoports.rdquo We present an algorithm that combines the advantages of widely-used approaches such as PRIMA and TICER but avoids many of the drawbacks of both. The resulting algorithm is capable of high-order rational approximation, exploits network sparsity, preserves passivity, can be extended to general non-symmetric systems, and can be applied to networks with hundreds or thousands of ports. We develop a common mathematical framework that can encompass all three algorithms, show mathematical relations between them, and point out certain special cases where they are equivalent. We show examples from analysis of industrial on-chip RC/RLC networks that demonstrate performance advantages of more than three orders of magnitude.

Variability in digital integrated circuits makes timing verification an increasingly challenging task. Statistical static timing analysis has been proposed as a solution to this problem, but most of the work has concentrated in the development of timing engines for computing delay propagation. Such tools rely on the availability of delay formulas accounting for both cell and interconnect delay. In this paper, we concentrate on the impact of interconnect on delay and propose an extension to the standard modeling strategies that is variation-aware and compatible with such statistical engines. Our approach, based on a specific type of perturbation analysis, allows for the analytical computation of the quantities needed for statistical delay propagation. We also show how perturbation analysis can be performed when only the standard delay table lookup models are available for the standard cells. Results from applying our proposed modeling strategy to computing delays and slews in several instances accurately match similar results obtained using electrical level simulation

In this paper we describe several novel sparsification techniques used in a fast stochastic integral equation solver to compute the mean value and the variance of capacitance of 3D interconnects with random surface roughness. With the combination of these numerical techniques, the computational cost has been reduced from O(N/sup 4/) to O(Nlog/sup 2/(N)), where N is the number of panels used for the discretization of nominal smooth surfaces. Numerical experiments show that the proposed numerical techniques are accurate and efficient.

In this paper, we describe the algorithms used in FastImp, a program for accurate analysis of wide-band electromagnetic effects in very complicated geometries of conductors. The program is based on a recently developed surface integral formulation and a precorrected fast Fourier transform (FFT) accelerated iterative method, but includes a new piecewise quadrature panel integration scheme, a new scaling and preconditioning technique as well as a generalized grid interpolation and projection strategy. Computational results are given on a variety of integrated circuit interconnect structures to demonstrate that FastImp is robust and can accurately analyze very complicated geometries of conductors.

In This work we describe a stochastic integral equation method for computing the mean value and the variance of capacitance of interconnects with random surface roughness. An ensemble average Green's function is combined with a matrix Neumann expansion to compute nominal capacitance and its variance. This method avoids the time-consuming Monte Carlo simulations and the discretization of rough surfaces. Numerical experiments show that the results of the new method agree very well with Monte Carlo simulation results.

In this paper we describe the algorithms used in FastImp, a program for accurate analysis of wide-band electromagnetic effects in very complicated geometries of conductors. The program is based on a recently developed surface integral formulation and a precorrected-FFT accelerated iterative method, but includes a new scaling and preconditioning technique as well as a generalized grid interpolation and projection strategy. Computational results are given on a variety of integrated circuit interconnect structures to demonstrate that FastImp is robust and can accurately analyze very complicated geometries of conductors.

Detailed electromagnetic analysis of three-dimensional structures in multi-layered dielectric media is critical for automatic generation of equivalent circuit models for the interconnects and packages in RF or mixed signal integrated circuits. In this paper we present a new wideband surface integral formulation that can be readily combined with the well-established layered Green's function techniques. In discretizing the formulation, we have used the well-known RWG linear basis function to reduce the number of unknowns. Using a so-called loop-star basis transformation and the frequency normalization, the accuracy at low frequencies has been improved substantially. Several numerical examples are used to validate the accuracy and robustness of this new formulation.

In order for parasitic extraction of high-speed integrated circuit interconnect to be sufficiently efficient, and fit with model-order reduction techniques, a robust wideband surface integral formulation is essential. One recently developed surface integral formulation has shown promise, but was plagued with numerical difficulties of poorly understood origin. We show that one of that formulation's difficulties was related to the inaccuracy in the approach to evaluate integrals over discretization panels, and we present an accurate approach based on an adapted piecewise quadrature scheme. We also show that the condition number of the original system of integral equations can be reduced by differentiating one of the integral equations. Computational results on a ring and a spiral inductor are used to show that the new quadrature scheme and the differentiated integral formulation improve accuracy and accelerate the convergence of iterative solution methods.

In this paper, we present a new capacitance extraction method named Dimension Reduction Technique (DRT) for 3D VLSI interconnects. The DRT converts a complex 3D problem into a series of cascading simple 2D problems. Each 3D problem is solved separately, so we can choose the most efficient method according to the arrangement of conductors. More importantly, it is very easy to obtain the analytical solutions of 2D problem in many layers such as the pure dielectric layers and the layers with parallel signal lines. Therefore, the domain that has to be analyzed numerically is minimized. This leads to the drastic reduction of the computing time and memory needs. We have used the DRT to extract the capacitances of multilayered and multiconductor cross-overs, bends, via with signal lines and open-end. The results are in good agreement with those of Ansoft's SPICELINK and MIT's FastCap, but the computing time and memory size used by the DRT are several even tens times less than those used by SPICELINK and FastCap.