We propose a method for learning the posture and structure of agents from unlabelled behavioral videos. Starting from the observation that behaving agents are generally the main sources of movement in behavioral videos, our method, Behavioral Keypoint Discovery (B-KinD), uses an encoder-decoder architecture with a geometric bottleneck to reconstruct the spatiotemporal difference between video frames. By focusing only on regions of movement, our approach works directly on input videos without requiring manual annotations. Experiments on a variety of agent types (mouse, fly, human, jellyfish, and trees) demonstrate the generality of our approach and reveal that our discovered keypoints represent semantically meaningful body parts, which achieve state-of-the-art performance on keypoint regression among self-supervised methods. Additionally, B-KinD achieve comparable performance to supervised keypoints on downstream tasks, such as behavior classification, suggesting that our method can dramatically reduce model training costs vis-a-vis supervised methods.

Specialized domain knowledge is often necessary to accurately annotate training sets for in-depth analysis, but can be burdensome and time-consuming to acquire from domain experts. This issue arises prominently in automated behavior analysis, in which agent movements or actions of interest are detected from video tracking data. To reduce annotation effort, we present TREBA: a method to learn annotation-sample efficient trajectory embedding for behavior analysis, based on multi-task self-supervised learning. The tasks in our method can be efficiently engineered by domain experts through a process we call "task programming", which uses programs to explicitly encode structured knowledge from domain experts. Total domain expert effort can be reduced by exchanging data annotation time for the construction of a small number of programmed tasks. We evaluate this trade-off using data from behavioral neuroscience, in which specialized domain knowledge is used to identify behaviors. We present experimental results in three datasets across two domains: mice and fruit flies. Using embeddings from TREBA, we reduce annotation burden by up to a factor of 10 without compromising accuracy compared to state-of-the-art features. Our results thus suggest that task programming and self-supervision can be an effective way to reduce annotation effort for domain experts.

We present a novel method for analyzing social behavior. Continuous videos are segmented into action `bouts' by building a temporal context model that combines features from spatio-temporal energy and agent trajectories. The method is tested on an unprecedented dataset of videos of interacting pairs of mice, which was collected as part of a state-of-the-art neurophysiological study of behavior. The dataset comprises over 88 hours (8 million frames) of annotated videos. We find that our novel trajectory features, used in a discriminative framework, are more informative than widely used spatio-temporal features; furthermore, temporal context plays an important role for action recognition in continuous videos. Our approach may be seen as a baseline method on this dataset, reaching a mean recognition rate of 61.2% compared to the expert's agreement rate of about 70%.

We investigate a new approach for the problem of source separation/classification of non-orthogonal, non-stationary noisy signals impinging on an array of sensors. We propose a solution to the problem when the contaminating noise is temporally and spatially correlated. The observations are projected onto a nested set of multiresolution spaces prior to classical eigendecomposition. An inherent invariance property of the signal subspace is observed in a subset of the multiresolution spaces that depends on the level of approximation expressed by the orthogonal basis. This feature, among others revealed by the algorithm, is eventually used to separate the correlated signal sources in the context of 'best basis' selection. The technique shows robustness to source non-stationarity as well as anisotropic properties of the channel characteristics under no constraints on the array design. We illustrate the high performance of the technique on simulated and experimental multichannel neurophysiological data measurements

This paper describes the development of a high-density electronic interface to the central nervous system. Silicon micromachined electrode arrays now permit the long-term monitoring of neural activity in vivo as well as the insertion of electronic signals into neural networks at the cellular level. Efforts to understand and engineer the biology of the implant/tissue interface are also underway. These electrode arrays are facilitating significant advances in our understanding of the nervous system, and merged with on-chip circuitry, signal processing, microfluidics, and wireless interfaces, they are forming the basis for a family of neural prostheses for the possible treatment of disorders such as blindness, deafness, paralysis, severe epilepsy, and Parkinson's disease.

The problem of detecting transient signals of unknown waveforms has been widely studied in recent years due to the numerous applications associated with it. In particular, the detection of neural spikes using a recording array of closely spaced sensors in the brain is one such application. In this paper, we propose a new approach for solving this problem when no apriori knowledge is given about the signal and/or the noise processes. By extending our previous multiresolution analysis framework for noise suppression and source identification [1,2], we show that it is feasible to achieve reasonable detection performance in low SNR scenarios. Comparison to traditional detection schemes is presented and the overall performance is evaluated.

Technology for recording action potentials from the nervous system has changed in recent years from single or at most two or three neurons to tens of neurons recorded from geometrically precise arrays of recording sites. To fully exploit these research opportunities, signal delivery and processing are key factors. We desire not only to sort the array input into neural channels but to locate neurons with respect to the array so they can be later identified in histology and serve to further decode the activity of each neuron. The distribution of a neuron's signal across the sensor array is predictive of the cell location at least in projection onto the array but also in the dimension above the array. To achieve 3-dimensional locating power, careful estimation of signal strength at each site must be achieved taking into account the distance and the field distortion for all anticipated source positions. Achieving this independent of source strength appears to be feasible.

We investigated a new framework for the problem of blind source identification in multichannel signal processing. Inspired by a neurophysiological data environment, where an array of closely spaced recording electrodes is surrounded by multiple neural cell sources, significant spatial correlation of source signals motivated the need for an efficient technique for reliable multichannel blind source identification. In a previous work Oweiss and Anderson (see Proceedings of the 34th. Asilomar Conference on Signals, Systems and Computers, Pacific Grove, 2000) adopted a new approach for noise suppression based on thresholding an array discrete wavelet transform (ADWT) representation of the multichannel data. We extend the work of Oweiss and Anderson to identify sources from the observation mixtures. The technique relies on separating sources with highest spatial energy distribution in each frequency subband spanned by the corresponding wavelet basis. Accordingly, the best basis selection criterion we propose benefits from the additional degree of freedom offered by the space domain. The amplitude and shift invariance properties revealed by this technique make it very efficient to track spatial source variations sometimes encountered in multichannel neural recordings. Results from multichannel multiunit neural data are presented and the overall performance is evaluated.

The problem of estimating the properties of smooth, continuous contours from discrete, noisy samples is used as vehicle to demonstrate the robustness of cross-validated regularization applied to a vision problem. A method for estimation of contour properties based on smoothing spline approximations is presented. Generalized cross-validation is to devise an automatic algorithm for finding the optimal value of the smoothing (regularization) parameter from the data. The cross-validated smoothing splines are then used to obtain optimal estimates of the derivatives of quantized contours. Experimental results are presented which demonstrate the robustness of the method applied to the estimation of curvature of quantized contours under variable scale, rotation, and partial occlusion. These results suggest the application of generalized cross-validation to other computer-vision algorithms involving regularization.>

The performance of the cross-correlation technique for time-delay estimation in presence of jitter is discussed in this paper. It is shown that the effect of a sampling jitter in the signal to be correlated with a reference template is the convolution of the jitter density function with the original cross-correlation function in the absence of jitter. A simulation of the approach showing how the error caused by the jitter can be removed using a simple deconvolution process is presented. A practical application of the approach in a rotation measurement algorithm is given.

This paper addresses the problem of the nonuniform intersample distance of digital curves obtained as a result of quantization of continuous contours on a square lattice. A resampling algorithm based on variable-factor interpolation and decimation is presented, and its performance is evaluated analytically and by computer simulations assuming the grid-intersect quantization. It is shown that the output of the resampling algorithm outperforms the original digital curve in terms of the average and maximum error in the measurement of length by 50 and 73 percent, respectively. The resampling algorithm can be used as a preprocessing stage in shape analysis systems to enhance their performance by increasing the accuracy and consistency of both local and global features such as curvature and the Fourier shape descriptors.