MOTIVATION: Many 'automated gating' algorithms now exist to cluster cytometry and single cell sequencing data into discrete populations. Comparative algorithm evaluations on benchmark datasets rely either on a single performance metric, or a few metrics considered independently of one another. However, single metrics emphasise different aspects of clustering performance and do not rank clustering solutions in the same order. This underlies the lack of consensus between comparative studies regarding optimal clustering algorithms and undermines the translatability of results onto other non-benchmark datasets. RESULTS: We propose the Pareto fronts framework as an integrative evaluation protocol, wherein individual metrics are instead leveraged as complementary perspectives. Judged superior are algorithms that provide the best trade-off between the multiple metrics considered simultaneously. This yields a more comprehensive and complete view of clustering performance. Moreover, by broadly and systematically sampling algorithm parameter values using the Latin Hypercube sampling method, our evaluation protocol minimises (un)fortunate parameter value selections as confounding factors. Furthermore, it reveals how meticulously each algorithm must be tuned in order to obtain good results, vital knowledge for users with novel data. We exemplify the protocol by conducting a comparative study between three clustering algorithms (ChronoClust, FlowSOM and Phenograph) using four common performance metrics applied across four cytometry benchmark datasets. To our knowledge, this is the first time Pareto fronts have been used to evaluate the performance of clustering algorithms in any application domain. AVAILABILITY: Implementation of our Pareto front methodology and all scripts to reproduce this article are available at https://github.com/ghar1821/ParetoBench.

Computational and mathematical modelling has become a valuable tool for investigating biological systems. Modelling enables prediction of how biological components interact to deliver system-level properties and extrapolation of biological system performance to contexts and experimental conditions where this is unknown. A model's value hinges on knowing that it faithfully represents the biology under the contexts of use, or clearly ascertaining otherwise and thus motivating further model refinement. These qualities are evaluated through calibration, typically formulated as identifying model parameter values that align model and biological behaviours as measured through a metric applied to both. Calibration is critical to modelling but is often underappreciated. A failure to appropriately calibrate risks unrepresentative models that generate erroneous insights. Here, we review a suite of strategies to more rigorously challenge a model's representation of a biological system. All are motivated by features of biological systems, and illustrative examples are drawn from the modelling literature. We examine the calibration of a model against distributions of biological behaviours or outcomes, not only average values. We argue for calibration even where model parameter values are experimentally ascertained. We explore how single metrics can be non-distinguishing for complex systems, with multiple-component dynamic and interaction configurations giving rise to the same metric output. Under these conditions, calibration is insufficiently constraining and the model non-identifiable: multiple solutions to the calibration problem exist. We draw an analogy to curve fitting and argue that calibrating a biological model against a single experiment or context is akin to curve fitting against a single data point. Though useful for communicating model results, we explore how metrics that quantify heavily emergent properties may not be suitable for use in calibration. Lastly, we consider the role of sensitivity and uncertainty analysis in calibration and the interpretation of model results. Our goal in this manuscript is to encourage a deeper consideration of calibration, and how to increase its capacity to either deliver faithful models or demonstrate them otherwise.

The advent of two-photon microscopy now reveals unprecedented, detailed spatio-temporal data on cellular motility and interactions in vivo. Understanding cellular motility patterns is key to gaining insight into the development and possible manipulation of the immune response. Computational simulation has become an established technique for understanding immune processes and evaluating hypotheses in the context of experimental data, and there is clear scope to integrate microscopy-informed motility dynamics. However, determining which motility model best reflects in vivo motility is non-trivial: 3D motility is an intricate process requiring several metrics to characterize. This complicates model selection and parameterization, which must be performed against several metrics simultaneously. Here we evaluate Brownian motion, Lévy walk and several correlated random walks (CRWs) against the motility dynamics of neutrophils and lymph node T cells under inflammatory conditions by simultaneously considering cellular translational and turn speeds, and meandering indices. Heterogeneous cells exhibiting a continuum of inherent translational speeds and directionalities comprise both datasets, a feature significantly improving capture of in vivo motility when simulated as a CRW. Furthermore, translational and turn speeds are inversely correlated, and the corresponding CRW simulation again improves capture of our in vivo data, albeit to a lesser extent. In contrast, Brownian motion poorly reflects our data. Lévy walk is competitive in capturing some aspects of neutrophil motility, but T cell directional persistence only, therein highlighting the importance of evaluating models against several motility metrics simultaneously. This we achieve through novel application of multi-objective optimization, wherein each model is independently implemented and then parameterized to identify optimal trade-offs in performance against each metric. The resultant Pareto fronts of optimal solutions are directly contrasted to identify models best capturing in vivo dynamics, a technique that can aid model selection more generally. Our technique robustly determines our cell populations' motility strategies, and paves the way for simulations that incorporate accurate immune cell motility dynamics.

In this paper we present the Relay Chain: a new algorithm central to a novel strategy for exploring underwater environments using a swarm of autonomous underwater vehicles (AUVs). The Relay Chain provides a mobile, scalable and dynamic communication link between the water’s surface and a shoal of AUVs exploring the sea bed. The chain tracks the shoal as it explores, recruiting and returning AUVs from and to the shoal to modulate the length of the chain as required. The chain can instruct the shoal to reverse course when it travels too far from the starting point and there are no further AUVs to recruit. Given the challenging underwater environment, chain breakages are however inevitable, and as such we consider a number of recovery strategies to address chain breakages and minimise the time for which messages destined to the surface from the shoal are delayed. A simple ‘turn and search’ strategy is contrasted with strategies that depend on absolute positioning systems of varying accuracy. Implementing underwater absolute positioning systems is challenging, and our results highlight how accurate such a system must be to outperform more naive strategies, and hence be considered a worthwhile investment. We find the accuracy must be within 20cm, being 40% of AUV sensor range.

BACKGROUND: Experimental autoimmune encephalomyelitis has been used extensively as an animal model of T cell mediated autoimmunity. A down-regulatory pathway through which encephalitogenic CD4Th1 cells are killed by CD8 regulatory T cells (Treg) has recently been proposed. With the CD8Treg cells being primed by dendritic cells, regulation of recovery may be occuring around these antigen presenting cells. CD4Treg cells provide critical help within this process, by licensing dendritic cells to prime CD8Treg cells, however the spatial and temporal aspects of this help in the CTL response is currently unclear. RESULTS: We have previously developed a simulator of experimental autoimmune encephalomyelitis (ARTIMMUS). We use ARTIMMUS to perform novel in silico experimentation regarding the priming of CD8Treg cells by dendritic cells, and the resulting CD8Treg mediated killing of encephalitogenic CD4Th1 cells. Simulations using dendritic cells that present antigenic peptides in a mutually exclusive manner (either MBP or TCR-derived, but not both) suggest that there is no significant reliance on dendritic cells that can prime both encephalitogenic CD4Th1 and Treg cells. Further, in silico experimentation suggests that dynamics of CD8Treg priming are significantly influenced through their spatial competition with CD4Treg cells and through the timing of Qa-1 expression by dendritic cells. CONCLUSION: There is no requirement for the encephalitogenic CD4Th1 cells and cytotoxic CD8Treg cells to be primed by the same dendritic cells. We conjecture that no significant portion of CD4Th1 regulation by Qa-1 restricted CD8Treg cells occurs around individual dendritic cells, and as such, that CD8Treg mediated killing of CD4Th1 cells occurring around dendritic cells is not critical for recovery from the murine autoimmune disease. Furthermore, the timing of the CD4Treg licensing of dendritic cells and the spatial competition between CD4Treg and CD8Treg cells around the dendritic cell is critical for the size of the cytotoxic T lymphocyte response, because dendritic cells have a limited lifespan. If treatments can be found to either speed up the licensing process, or increase the spatial competitiveness of CD8Treg cells, the magnitude of the cytotoxic T lymphocyte response can be increased.

Integrating computer simulation with conventional wet-lab research has proven to have much potential in furthering the understanding of biological systems. Success requires the relationship between simulation and the real-world system to be established: substantial aspects of the biological system are typically unknown, and the abstract nature of simulation can complicate interpretation of in silico results in terms of the biology. Here we present spartan (Simulation Parameter Analysis RToolkit ApplicatioN), a package of statistical techniques specifically designed to help researchers understand this relationship and provide novel biological insight. The tools comprising spartan help identify which simulation results can be attributed to the dynamics of the modelled biological system, rather than artefacts of biological uncertainty or parametrisation, or simulation stochasticity. Statistical analyses reveal the influence that pathways and components have on simulation behaviour, offering valuable biological insight into aspects of the system under study. We demonstrate the power of spartan in providing critical insight into aspects of lymphoid tissue development in the small intestine through simulation. Spartan is released under a GPLv2 license, implemented within the open source R statistical environment, and freely available from both the Comprehensive R Archive Network (CRAN) and http://www.cs.york.ac.uk/spartan. The techniques within the package can be applied to traditional ordinary or partial differential equation simulations as well as agent-based implementations. Manuals, comprehensive tutorials, and example simulation data upon which spartan can be applied are available from the website.

In studying complex systems, agent-based simulations offer the possibility of directly modelling components in an environment. However, the scientific value of agent-based simulations has been limited by inadequate scientific rigour. The paper focuses on agent-based simulations that are used in biological and bio-medical research. Starting from a review of best practice in simulation engineering, the paper identifies some of the key activities in developing complex systems simulations that support scientific research, and how these contribute to the essential development of mutual trust among developers and scientists. Examples from the authors' own experience illustrate how a range of studies have manifested these key activities, and identifies some successes and problems encountered.