Long context fed into Large Language Models (LLMs) becomes one of the main limiting factors for their successful deployment in industry, which is related to the high deployment costs and dedicated hardware with large memory onboard. The computational overhead related to the long context stems directly from the attention layer computation. In this work, we aim to accelerate the attention layer computation, while minimizing performance degradation by exploiting theoretical framework of attention sparsity in LLMs. Based on its findings, we propose a customized loss function which is designed to enforce sparsity of the attention layer by restricting the energy of a predefined number of top elements in the attention matrix. We perform experiments with GPT-2 language model to show the efficacy of our sparsification approach. The attention matrices of the models trained with the proposed loss reveal predefined level of sparsity, while, at the same time, capture relevant input dependencies and have minimal impact on the final accuracy. Future work aims to demonstrate the value of our approach on larger models and various model architectures.

This paper presents a technique for trajectory classification with applications to dynamic free-air hand gesture recognition. Such gestures are unencumbered and drawn in free air. Our approach is an extension to the longest common subsequence (LCS) classification algorithm. A learning preprocessing stage is performed to create a probabilistic 2-D template for each gesture, which allows taking into account different trajectory distortions with different probabilities. The modified LCS, termed the most probable LCS (MPLCS), is developed to measure the similarity between the probabilistic template and the hand gesture sample. The final decision is based on the length and probability of the extracted subsequence. Validation tests using a cohort of gesture digits from video-based capture show that the approach is promising with a recognition rate of more than 98 % for video stream preisolated digits. The MPLCS algorithm can be integrated into a gesture recognition interface to facilitate gesture character input. This can greatly enhance the usability of such interfaces.

This paper explores how shape, motion, and lighting interact in the case of a two-frame motion sequence. We consider a rigid object with Lambertian reflectance properties undergoing small motion with respect to both a camera and a stationary point light source. Assuming orthographic projection, we derive a single, first order quasilinear partial differential equation that relates shape, motion, and lighting, while eliminating out the albedo. We show how this equation can be solved, when the motion and lighting parameters are known, to produce a 3D reconstruction of the object. A solution is obtained using the method of characteristics and can be refined by adding regularization. We further show that both smooth bounding contours as well as surface markings can be used to derive Dirichlet boundary conditions. Experimental results demonstrate the quality of this reconstruction.

We present a method for shape reconstruction from several images of a moving object. The reconstruction is dense (up to image resolution). The method assumes that the motion is known, e.g., by tracking a small number of feature points on the object. The object is assumed Lambertian (completely matte), light sources should not be very close to the object but otherwise arbitrary, and no knowledge of lighting conditions is required. An object changes its appearance significantly when it changes its orientation relative to light sources, causing violation of the common brightness constancy assumption. While a lot of effort is devoted to deal with this violation, we demonstrate how to exploit it to recover 3D structure from 2D images. We propose a new correspondence measure that enables point matching across views of a moving object. The method has been tested both on computer simulated examples and on a real object.