As the space of causal sequence modeling architectures continues to grow, the need to develop a general framework for their analysis becomes increasingly important. With this aim, we draw insights from classical signal processing and control theory, to develop a quantitative measure of memory utilization: the internal mechanisms through which a model stores past information to produce future outputs. This metric, which we call effective state-size (ESS), is tailored to the fundamental class of systems with input-invariant and input-varying linear operators, encompassing a variety of computational units such as variants of attention, convolutions, and recurrences. Unlike prior work on memory utilization, which either relies on raw operator visualizations (e.g. attention maps), or simply the total memory capacity (i.e. cache size) of a model, our metrics provide highly interpretable and actionable measurements. In particular, we show how ESS can be leveraged to improve initialization strategies, inform novel regularizers and advance the performance-efficiency frontier through model distillation. Furthermore, we demonstrate that the effect of context delimiters (such as end-of-speech tokens) on ESS highlights cross-architectural differences in how large language models utilize their available memory to recall information. Overall, we find that ESS provides valuable insights into the dynamics that dictate memory utilization, enabling the design of more efficient and effective sequence models.

We approach designing a state-space model for deep learning applications through its dual representation, the transfer function, and uncover a highly efficient sequence parallel inference algorithm that is state-free: unlike other proposed algorithms, state-free inference does not incur any significant memory or computational cost with an increase in state size. We achieve this using properties of the proposed frequency domain transfer function parametrization, which enables direct computation of its corresponding convolutional kernel’s spectrum via a single Fast Fourier Transform. Our experimental results across multiple sequence lengths and state sizes illustrates, on average, a 35% training speed improvement over S4 layers – parametrized in time-domain – on the Long Range Arena benchmark, while delivering state-of-the-art downstream performances over other attention-free approaches. Moreover, we report improved perplexity in language modeling over a long convolutional Hyena baseline, by simply introducing our transfer function parametrization. Our code is available at https://github.com/ruke1ire/RTF.

Spherical cameras, which can acquire all-round information, are effective to estimate rotation for robotic applications. Recently, Convolutional Neural Networks have shown great robustness in solving such regression problems. However they are designed for planar images and cannot deal with the non-uniform distortion present in spherical images, when expressed in the planar equirectangular projection. This can lower the accuracy of motion estimation. In this research, we propose an Equirectangular-Convolutional Neural Network (E-CNN) to solve this issue. This novel network regresses 3D spherical camera rotation by uniformizing distorted optical flow patterns in the equirectangular projection. We experimentally show that this results in consistently lower error as opposed to learning from the distorted optical flow.

Comparing to perspective cameras, the all-round 3D measurement of the environment can be done by spherical cameras in a more efficient way. However, the measurement using binocular spherical stereo has two singularity points at the epipoles of each spherical camera, where the measurement result gets extremely sensitive to the error when getting close to the epipoles and along the epipolar directions. This affects the accuracy of 3D reconstruction along with the epipolar directions. A three-way measurement method using three spherical cameras with trinocular spherical stereo setup is proposed in this paper to achieve accurate all-round 3D measurement. The improved accuracy of 3D measurement by the implementation of weighted reprojection error optimization was verified in experiments.

Conventional techniques for frame-to-frame camera motion estimation rely on tracking a set of sparse feature points. However, images taken from spherical cameras have high distortion which can induce mistakes in feature point tracking, offsetting the advantage of their large fields-of-view. Hence, in this research, we attempt a novel approach of using dense optical flow for distortion-robust spherical camera motion estimation. Dense optical flow incorporates smoothing terms and is free of local outliers. It encodes the camera motion as well as dense 3D information. Our approach decomposes dense optical flow into epipolar geometry and the dense disparity map, and reprojects this disparity map to estimate 6 DoF camera motion. The approach handles spherical image distortion in a natural way. We experimentally demonstrate its accuracy and robustness.

In camera-equipped teleoperated robots, it is often tedious for the operator to manage both the viewpoint and the shaky/unstable navigation, leading to disorientation. Our proposal is to create a virtual, freely rotatable camera that is decoupled from the robot's rotation. It is implemented using a complete spherical camera and removing its rotation in-image with a novel algorithm based on aligning the dense spherical optical flow field along the epipolar direction. Finally, any area on the rotation-less image sequence can be undistorted, resulting in the desired decoupled camera. We illustrate the concept by showing the effect on some videos taken from a spherical camera under different robot motions.

This paper deals with the background maintenance problem and proposes a novel pixel-wise solution. The proposed background maintenance algorithm is histogram-based. The algorithm has the following main features: fast background initialization, high accuracy in describing the real background and fast reaction to sudden changes. The basic idea of our algorithm is that the pixels are updated only if a statistic measure on the intensity variations of each pixels is greater to an adaptive threshold, thus reducing the I/O channel occupation. Experimental results on dynamic scenes taken from a fixed camera show that the proposed algorithm produces background images with an improved quality with respect to classical pixel-wise algorithms.

This paper proposes a human detection method that combines range image segmentation and human detection based on image local features. The method uses a stereo vision system called Subtraction Stereo, which extracts a range image of foreground regions. An extracted range image is segmented for each object by Mean Shift Clustering. Human detection based on local features is applied to each segment of foreground regions to detect humans. In this process, regions to scan a detection window for extracting local features are restricted. In addition, the size of the detection window is obtained using the distance information of a range image and camera parameters. Therefore, processing time and false detection can be reduced. Joint HOG features are used as the image local features. When applying the Joint HOG based human detection, occlusion of multiple humans is considered in construction of a classifier and in integration of detection windows, which improves the detection performance for the occluded humans. The proposed method is evaluated by experiments comparing with the method using Joint HOG features only. 11fps fast human detection is achieved.

A fast human detection system using a stereo camera is constructed. “Subtraction stereo”, that can measure distance information of foreground regions, is used to restrict regions for human detection and to adapt the detection window size. Two methods are introduced for human detection. One is a method based on template matching using gradient images, and the other is a method using approximated Shape Context (aSC) descriptors focusing on human upper bodies. High human detection performance better than the standard HOG-based method with low calculation cost is achieved by the combination of the two methods. The effectiveness of the proposed system is verified experimentally.

In this paper, we propose a fast and stable human detection based on "subtraction stereo" which can measure distance information of foreground regions. Scanning an input image by detection windows is controlled in their window sizes and number using the distance information obtained from subtraction stereo. This control can skip a large number of detection windows and leads to reduce the computational time and false detection for fast and stable human detection. Additionally, we propose two-step boosting as a new training way of classifier with whole and upper human body models. Experimental results show that the proposal is faster and less false detection than the method described in the reference [1].

In this paper we present a method for the detection of moving objects for unknown and generic environments under cast shadow and periodic movements of non relevant objects (like waving leaves), using a combination of non-parametric thresholding algorithms and local cast shadow analysis with stereo camera information. Good detection rates were achieved in several environments under different lighting conditions, and objects could be detected independently of scene illumination, shadow, and periodic changes.

We propose a method for multi-object segmentation in a projection plane. Our algorithm requires a stereo camera system called Subtraction Stereo, which extracts foreground information with a fixed stereo camera. The main contribution of this paper is how the image sequences that include partial occlusion of the foreground objects can be accurately segmented using mean shift clustering in real-time processing. The proposed method is suitable for inside a medium-sized environment, such as a room. Finally, we try to segment the sequences that include occlusion and show the accuracy of the proposed method.

In this paper, we present a novel approach for accurate modeling of computer workloads. According to this approach, the sequences of features generated by a program during its execution are considered as time series and are processed with signal processing techniques both for feature extraction and statistical pattern matching. In the feature extraction phase we used spectral analysis for describing the sequence and to retain the important information. In the pattern matching phase we used a simplified form of bidimensional Hidden Markov Model, called pseudo2D-HMM, as Statistical Machine Learning Algorithm. Several processes of the same workload are necessary to obtain a 2D-HMM model of the workload. In this way, the models are obtained in an initial training phase; we developed techniques for on-line workload classification of a running process and for synthetic traces generation. The proposed algorithms is evaluated via trace-driven simulations using the SPEC 2000 workloads. We show that pseudo2D-HMMs accurately describe memory references sequences; the classification accuracy is about 92% with six different workloads.