Abstract Markerless methods for animal posture tracking have been rapidly developing recently, but frameworks and benchmarks for tracking large animal groups in 3D are still lacking. To overcome this gap in the literature, we present 3D-MuPPET, a framework to estimate and track 3D poses of up to 10 pigeons at interactive speed using multiple camera views. We train a pose estimator to infer 2D keypoints and bounding boxes of multiple pigeons, then triangulate the keypoints to 3D. For identity matching of individuals in all views, we first dynamically match 2D detections to global identities in the first frame, then use a 2D tracker to maintain IDs across views in subsequent frames. We achieve comparable accuracy to a state of the art 3D pose estimator in terms of median error and Percentage of Correct Keypoints. Additionally, we benchmark the inference speed of 3D-MuPPET, with up to 9.45 fps in 2D and 1.89 fps in 3D, and perform quantitative tracking evaluation, which yields encouraging results. Finally, we showcase two novel applications for 3D-MuPPET. First, we train a model with data of single pigeons and achieve comparable results in 2D and 3D posture estimation for up to 5 pigeons. Second, we show that 3D-MuPPET also works in outdoors without additional annotations from natural environments. Both use cases simplify the domain shift to new species and environments, largely reducing annotation effort needed for 3D posture tracking. To the best of our knowledge we are the first to present a framework for 2D/3D animal posture and trajectory tracking that works in both indoor and outdoor environments for up to 10 individuals. We hope that the framework can open up new opportunities in studying animal collective behaviour and encourages further developments in 3D multi-animal posture tracking.

Recent advances in machine learning and computer vision are revolutionizing the field of animal behavior by enabling researchers to track the poses and locations of freely moving animals without any marker attachment. However, large datasets of annotated images of animals for markerless pose tracking, especially high-resolution images taken from multiple angles with accurate 3D annotations, are still scant. Here, we propose a method that uses a motion capture (mo-cap) system to obtain a large amount of annotated data on animal movement and posture (2D and 3D) in a semi-automatic manner. Our method is novel in that it extracts the 3D positions of morphological keypoints (e.g eyes, beak, tail) in reference to the positions of markers attached to the animals. Using this method, we obtained, and offer here, a new dataset - 3D-POP with approximately 300k annotated frames (4 million instances) in the form of videos having groups of one to ten freely moving birds from 4 different camera views in a 3.6m x 4.2m area. 3D-POP is the first dataset of flocking birds with accurate keypoint annotations in 2D and 3D along with bounding box and individual identities and will facilitate the development of solutions for problems of 2D to 3D markerless pose, trajectory tracking, and identification in birds.

The core idea in an XR (VR/MR/AR) application is to digitally stimulate one or more sensory systems (e.g. visual, auditory, olfactory) of the human user in an interactive way to achieve an immersive experience. Since the early 2000s biologists have been using Virtual Environments (VE) to investigate the mechanisms of behavior in non-human animals including insects, fish, and mammals. VEs have become reliable tools for studying vision, cognition, and sensory-motor control in animals. In turn, the knowledge gained from studying such behaviors can be harnessed by researchers designing biologically inspired robots, smart sensors, and rnulti-agent artificial intelligence. VE for animals is becoming a widely used application of XR technology but such applications have not previously been reported in the technical literature related to XR. Biologists and computer scientists can benefit greatly from deepening interdisciplinary research in this emerging field and together we can develop new methods for conducting fundamental research in behavioral sciences and engineering. To support our argument we present this review which provides an overview of animal behavior experiments conducted in virtual environments.

The evolution of costly cooperation, where cooperators pay a personal cost to benefit others, requires that cooperators interact more frequently with other cooperators. This condition, called positive assortment, is known to occur in spatially-structured viscous populations, where individuals typically have low mobility and limited dispersal. However many social organisms across taxa, from cells and bacteria, to birds, fish and ungulates, are mobile, and live in populations with considerable inter-group mixing. In the absence of information regarding others' traits or conditional strategies, such mixing may inhibit assortment and limit the potential for cooperation to evolve. Here we employ spatially-explicit individual-based evolutionary simulations to incorporate costs and benefits of two coevolving costly traits: cooperative and local cohesive tendencies. We demonstrate that, despite possessing no information about others' traits or payoffs, mobility (via self-propulsion or environmental forcing) facilitates assortment of cooperators via a dynamically evolving difference in the cohesive tendencies of cooperators and defectors. We show analytically that this assortment can also be viewed in a multilevel selection framework, where selection for cooperation among emergent groups can overcome selection against cooperators within the groups. As a result of these dynamics, we find an oscillatory pattern of cooperation and defection that maintains cooperation even in the absence of well known mechanisms such as kin interactions, reciprocity, local dispersal or conditional strategies that require information on others' strategies or payoffs. Our results offer insights into differential adhesion based mechanisms for positive assortment and reveal the possibility of cooperative aggregations in dynamic fission-fusion populations.

Learning has been studied extensively in the context of isolated individuals. However, many organisms are social and consequently make decisions both individually and as part of a collective. Reaching consensus necessarily means that a single option is chosen by the group, even when there are dissenting opinions. This decision-making process decouples the otherwise direct relationship between animals' preferences and their experiences (the outcomes of decisions). Instead, because an individual's learned preferences influence what others experience, and therefore learn about, collective decisions couple the learning processes between social organisms. This introduces a new, and previously unexplored, dynamical relationship between preference, action, experience and learning. Here we model collective learning within animal groups that make consensus decisions. We reveal how learning as part of a collective results in behavior that is fundamentally different from that learned in isolation, allowing grouping organisms to spontaneously (and indirectly) detect correlations between group members' observations of environmental cues, adjust strategy as a function of changing group size (even if that group size is not known to the individual), and achieve a decision accuracy that is very close to that which is provably optimal, regardless of environmental contingencies. Because these properties make minimal cognitive demands on individuals, collective learning, and the capabilities it affords, may be widespread among group-living organisms. Our work emphasizes the importance and need for theoretical and experimental work that considers the mechanism and consequences of learning in a social context.

Robustness and adaptability are central to the functioning of biological systems, from gene networks to animal societies. Yet the mechanisms by which living organisms achieve both stability to perturbations and sensitivity to input are poorly understood. Here, we present an integrated study of a living architecture in which army ants interconnect their bodies to span gaps. We demonstrate that these self-assembled bridges are a highly effective means of maintaining traffic flow over unpredictable terrain. The individual-level rules responsible depend only on locally-estimated traffic intensity and the number of neighbours to which ants are attached within the structure. We employ a parameterized computational model to reveal that bridges are tuned to be maximally stable in the face of regular, periodic fluctuations in traffic. However analysis of the model also suggests that interactions among ants give rise to feedback processes that result in bridges being highly responsive to sudden interruptions in traffic. Subsequent field experiments confirm this prediction and thus the dual nature of stability and flexibility in living bridges. Our study demonstrates the importance of robust and adaptive modular architecture to efficient traffic organisation and reveals general principles regarding the regulation of form in biological self-assemblies.

The spontaneous emergence of pattern formation is ubiquitous in nature, often arising as a collective phenomenon from interactions among a large number of individual constituents or sub-systems. Understanding, and controlling, collective behavior is dependent on determining the low-level dynamical principles from which spatial and temporal patterns emerge; a key question is whether different group-level patterns result from all components of a system responding to the same external factor, individual components changing behavior but in a distributed self-organized way, or whether multiple collective states co-exist for the same individual behaviors. Using schooling fish (golden shiners, in groups of 30 to 300 fish) as a model system, we demonstrate that collective motion can be effectively mapped onto a set of order parameters describing the macroscopic group structure, revealing the existence of at least three dynamically-stable collective states; swarm, milling and polarized groups. Swarms are characterized by slow individual motion and a relatively dense, disordered structure. Increasing swim speed is associated with a transition to one of two locally-ordered states, milling or highly-mobile polarized groups. The stability of the discrete collective behaviors exhibited by a group depends on the number of group members. Transitions between states are influenced by both external (boundary-driven) and internal (changing motion of group members) factors. Whereas transitions between locally-disordered and locally-ordered group states are speed dependent, analysis of local and global properties of groups suggests that, congruent with theory, milling and polarized states co-exist in a bistable regime with transitions largely driven by perturbations. Our study allows us to relate theoretical and empirical understanding of animal group behavior and emphasizes dynamic changes in the structure of such groups.

Integrating robotic agents into animal groups creates significant opportunities for advancing experimental investigations of collective animal behavior. In the case of fish schooling, new insights into processes such as collective decision making and leadership have been made in recent experiments in which live fish were interacting with robotic fish driven along preplanned paths. We introduce a new cyber–physical implementation that enables robotic fish to use real-time feedback to control their motion in response to live fish and other environmental features. Each robotic fish is magnetically connected to, and thus moved by, a wheeled robot underneath the tank. Real-time image processing of a video stream from an overhead camera provides measurements of both the robotic fish and the live fish moving together in the tank. Feedback responses computed from these measurements are communicated to the robotic fish using Bluetooth. We show results of demonstrations and discuss possibilities that our implementation affords for new kinds of behavioral experiments with fish schools.

Animals can exhibit complex movement patterns that may be the result of interactions with their environment or may be directly the mechanism by which their behaviour is governed. In order to understand the drivers of these patterns we examine the movement behaviour of individual desert locusts in a homogenous experimental arena with minimal external cues. Locust motion is intermittent and we reveal that as pauses become longer, the probability that a locust changes direction from its previous direction of travel increases. Long pauses (of greater than 100 s) can be considered reorientation bouts, while shorter pauses (of less than 6 s) appear to act as periods of resting between displacements. We observe power-law behaviour in the distribution of move and pause lengths of over 1.5 orders of magnitude. While Lévy features do exist, locusts' movement patterns are more fully described by considering moves, pauses and turns in combination. Further analysis reveals that these combinations give rise to two behavioural modes that are organized in time: local search behaviour (long exploratory pauses with short moves) and relocation behaviour (long displacement moves with shorter resting pauses). These findings offer a new perspective on how complex animal movement patterns emerge in nature.

Understanding cooperation in animal social groups remains a significant challenge for evolutionary theory. Observed behaviours that benefit others but incur some cost appear incompatible with classical notions of natural selection; however, these behaviours may be explained by concepts such as inclusive fitness, reciprocity, intra-specific mutualism or manipulation. In this work, we examine a seemingly altruistic behaviour, the active recruitment of conspecifics to a food resource through signalling. Here collective, cooperative behaviour may provide highly nonlinear benefits to individuals, since group functionality has the potential to be far greater than the sum of the component parts, for example by enabling the effective tracking of a dynamic resource. We show that due to this effect, signalling to others is an evolutionarily stable strategy under certain environmental conditions, even when there is a cost associated to this behaviour. While exploitation is possible, in the limiting case of a sparse, ephemeral but locally abundant nutrient source, a given environmental profile will support a fixed number of signalling individuals. Through a quantitative analysis, this effective carrying capacity for cooperation is related to the characteristic length and time scales of the resource field.

Collective decision making and especially leadership in groups are among the most studied topics in natural, social, and political sciences. Previous studies have shown that some individuals are more likely to be leaders because of their social power or the pertinent information they possess. One challenge for all group members, however, is to satisfy their needs. In many situations, we do not yet know how individuals within groups distribute leadership decisions between themselves in order to satisfy time-varying individual requirements. To gain insight into this problem, we build a dynamic model where group members have to satisfy different needs but are not aware of each other's needs. Data about needs of animals come from real data observed in macaques. Several studies showed that a collective movement may be initiated by a single individual. This individual may be the dominant one, the oldest one, but also the one having the highest physiological needs. In our model, the individual with the lowest reserve initiates movements and decides for all its conspecifics. This simple rule leads to a viable decision-making system where all individuals may lead the group at one moment and thus suit their requirements. However, a single individual becomes the leader in 38% to 95% of cases and the leadership is unequally (according to an exponential law) distributed according to the heterogeneity of needs in the group. The results showed that this non-linearity emerges when one group member reaches physiological requirements, mainly the nutrient ones - protein, energy and water depending on weight - superior to those of its conspecifics. This amplification may explain why some leaders could appear in animal groups without any despotism, complex signalling, or developed cognitive ability.