Quantifying motion in 3D is important for studying the behavior of humans and other animals, but manual pose annotations are expensive and time-consuming to obtain. Self-supervised keypoint discovery is a promising strategy for estimating 3D poses without annotations. However, current keypoint discovery approaches commonly process single 2D views and do not operate in the 3D space. We propose a new method to perform self-supervised keypoint discovery in 3D from multi-view videos of behaving agents, without any keypoint or bounding box supervision in 2D or 3D. Our method, BKinD-3D, uses an encoder-decoder architecture with a 3D volumetric heatmap, trained to reconstruct spatiotemporal differences across multiple views, in addition to joint length constraints on a learned 3D skeleton of the subject. In this way, we discover keypoints without requiring manual supervision in videos of humans and rats, demonstrating the potential of 3D keypoint discovery for studying behavior.

We introduce MABe22, a large-scale, multi-agent video and trajectory benchmark to assess the quality of learned behavior representations. This dataset is collected from a variety of biology experiments, and includes triplets of interacting mice (4.7 million frames video+pose tracking data, 10 million frames pose only), symbiotic beetle-ant interactions (10 million frames video data), and groups of interacting flies (4.4 million frames of pose tracking data). Accompanying these data, we introduce a panel of real-life downstream analysis tasks to assess the quality of learned representations by evaluating how well they preserve information about the experimental conditions (e.g. strain, time of day, optogenetic stimulation) and animal behavior. We test multiple state-of-the-art self-supervised video and trajectory representation learning methods to demonstrate the use of our benchmark, revealing that methods developed using human action datasets do not fully translate to animal datasets. We hope that our benchmark and dataset encourage a broader exploration of behavior representation learning methods across species and settings.

The mathematical study of real-world dynamical systems relies on models composed of differential equations. Numerical methods for solving and analyzing differential equation systems are essential when complex biological problems have to be studied, such as the spreading of a virus, the evolution of competing species in an ecosystem, or the dynamics of neurons in the brain. Here we present PyRates, a Python-based software for modeling and analyzing differential equation systems via numerical methods. PyRates is specifically designed to account for the inherent complexity of biological systems. It provides a new language for defining models that mirrors the modular organization of real-world dynamical systems and thus simplifies the implementation of complex networks of interacting dynamic entities. Furthermore, PyRates provides extensive support for the various forms of interaction delays that can be observed in biological systems. The core of PyRates is a versatile code-generation system that translates user-defined models into "backend" implementations in various languages, including Python, Fortran, Matlab, and Julia. This allows users to apply a wide range of analysis methods for dynamical systems, eliminating the need for manual translation between code bases. PyRates may also be used as a model definition interface for the creation of custom dynamical systems tools. To demonstrate this, we developed two extensions of PyRates for common analyses of dynamic models of biological systems: PyCoBi for bifurcation analysis and RectiPy for parameter fitting. We demonstrate in a series of example models how PyRates can be used in combination with PyCoBi and RectiPy for model analysis and fitting. Together, these tools offer a versatile framework for applying computational modeling and numerical analysis methods to dynamical systems in biology and beyond.

We propose a method for learning the posture and structure of agents from unlabelled behavioral videos. Starting from the observation that behaving agents are generally the main sources of movement in behavioral videos, our method, Behavioral Keypoint Discovery (B-KinD), uses an encoder-decoder architecture with a geometric bottleneck to reconstruct the spatiotemporal difference between video frames. By focusing only on regions of movement, our approach works directly on input videos without requiring manual annotations. Experiments on a variety of agent types (mouse, fly, human, jellyfish, and trees) demonstrate the generality of our approach and reveal that our discovered keypoints represent semantically meaningful body parts, which achieve state-of-the-art performance on keypoint regression among self-supervised methods. Additionally, B-KinD achieve comparable performance to supervised keypoints on downstream tasks, such as behavior classification, suggesting that our method can dramatically reduce model training costs vis-a-vis supervised methods.

Specialized domain knowledge is often necessary to accurately annotate training sets for in-depth analysis, but can be burdensome and time-consuming to acquire from domain experts. This issue arises prominently in automated behavior analysis, in which agent movements or actions of interest are detected from video tracking data. To reduce annotation effort, we present TREBA: a method to learn annotation-sample efficient trajectory embedding for behavior analysis, based on multi-task self-supervised learning. The tasks in our method can be efficiently engineered by domain experts through a process we call "task programming", which uses programs to explicitly encode structured knowledge from domain experts. Total domain expert effort can be reduced by exchanging data annotation time for the construction of a small number of programmed tasks. We evaluate this trade-off using data from behavioral neuroscience, in which specialized domain knowledge is used to identify behaviors. We present experimental results in three datasets across two domains: mice and fruit flies. Using embeddings from TREBA, we reduce annotation burden by up to a factor of 10 without compromising accuracy compared to state-of-the-art features. Our results thus suggest that task programming and self-supervision can be an effective way to reduce annotation effort for domain experts.

Competency-based education (CBE) has been the focus of much attention lately - from institutions, employers, and policymakers alike. Institutions - from ivy leagues to community colleges to K-12 - are incorporating some form of CBE. In computer science, CBE is an excellent modality to measure mastery and thus prepare students for the next phase whether it is further education or a job. This workshop introduces participants to CBE with an emphasis on CBE course design and competency development. This workshop will be useful for faculty and course designers at all levels - from K-12 through graduate school.