We describe how we leveraged best practices in big data processing pipeline design and visual analytics to prototype the Maritime Patterns-of-Life Information Service (MPoLIS), an information product currently under development at the NATO Centre for Maritime Research and Experimentation (CMRE). MPoLIS supports the maritime industry, governments, and international organizations with visual analytics on vessel traffic in seaports. It addresses three main requirements: a) storing and processing large amounts of data; b) on-demand availability of statistical summaries of vessel traffic in ports; c) intuitive and interactive interface for subject matter experts (SMEs) in the maritime domain. MPoLIS has contributed to building a data-driven, self-service analytics culture within NATO and has been sanctioned for use in support of maritime situational awareness (MSA) in ongoing NATO operations.

Seaports play a vital role in the global economy, as they operate as the connection corridors to all other modes of transport and as engines of growth for the wider region. But ports today are faced with numerous unique challenges and for them to remain competitive, significant investments are required. In support of greater transparency in policy making, decisions regarding investment need to be supported by data-driven intelligence. It is often an overlooked fact that seaports do not remain static over time; such spatial units often evolve according to environmental patterns both in size but also connectivity and operational capacity. As such any valid decision making regarding port investment and policy making, essentially needs to take into account port evolution over time and space. In this work, we leverage the huge amounts of vessel data that are progressively becoming available through the Automatic Identification System (AIS) and distributed machine learning to define a seaport's extended area of operation. Specifically, we present our adaptation of the well-known KDE algorithm to the map-reduce paradigm, and report results on the port of Shanghai.

Computational Maritime Situational Awareness (MSA) supports the maritime industry, governments, and international organizations with machine learning and big data techniques for analyzing vessel traffic data available through the Automatic Identification System (AIS). A critical challenge of scaling computational MSA to big data regimes is integrating the core learning algorithms with big data storage modes and data models. To address this challenge, we report results from our experimentation with MongoDB, a NoSQL document-based database which we test as a supporting platform for computational MSA. We experiment with a document model that avoids database joins when linking position and voyage AIS vessel information and allows tuning the database index and document sizes in response to the AIS data rate. We report results for the AIS data ingested and analyzed daily at the NATO Centre for Maritime Research and Experimentation (CMRE).

We describe a framework for automatically identifying and visualizing the most differentiating attributes of each cluster in a clustered data set. A dissimilarity function measures the cluster-conditional distinguishing saliency of each attribute with respect to a reference realization of the same attribute. For each cluster, the N attributes that are most dissimilar are presented first to the human expert, along with the overall dissimilarity of the cluster. We discuss the computational benefits of the proposed framework, how it can be implemented with map-reduce, its application to the behavioral analysis of mobile phone users, and it broad applicability to diverse problem domains.

We investigate parameter-based and distribution-based approaches to regularizing the generative, similarity-based classifier called local similarity discriminant analysis classifier (local SDA). We argue that regularizing distributions rather than parameters can both increase the model flexibility and decrease estimation variance while retaining the conceptual underpinnings of the local SDA classifier. Experiments with four benchmark similarity-based classification datasets show that the proposed regularization significantly improves classification performance compared to the local SDA classifier, and the distribution-based approach improves performance more consistently than the parameter-based approaches. Also, regularized local SDA can perform significantly better than similarity-based SVM classifiers, particularly on sparse and highly nonmetric similarities.

We propose a local, generative model for similarity-based classification. The method is applicable to the case that only pairwise similarities between samples are available. The classifier models the local class-conditional distribution using a maximum entropy estimate and empirical moment constraints. The resulting exponential class conditional-distributions are combined with class prior probabilities and misclassification costs to form the local similarity discriminant analysis (local SDA) classifier. We compare the performance of local SDA to a non-local version, to the local nearest centroid classifier, the nearest centroid classifier, k-NN, and to the recently-developed potential support vector machine (PSVM). Results show that local SDA is competitive with k-NN and the computationally-demanding PSVM while offering the advantages of a generative classifier.

A generative model for similarity-based classification is proposed using maximum entropy estimation. First, a descriptive set of similarity statistics is assumed to be sufficient for classification. Then the class conditional distributions of these descriptive statistics are estimated as the maximum entropy distributions subject to empirical moment constraints. The resulting exponential class conditional distributions are used in a maximum a posteriori decision rule, forming the similarity discriminant analysis (SDA) classifier. The relationship between SDA and the quadratic discriminant analysis classifier is discussed. An example SDA classifier is given that uses the class centroids as the descriptive statistics. Compared to the nearest-centroid classifier, which is also based only on the class centroids, simulation and experimental results show SDA consistently improves performance.

We introduce a definition of similarity based on Tversky's set-theoretic linear contrast model and on information-theoretic principles. The similarity measures the residual entropy with respect to a random object. This residual entropy similarity strongly captures context, which we conjecture is important for similarity-based statistical learning. Properties of the similarity definition are established and examples illustrate its characteristics. We show that a previously-defined information-theoretic similarity is also set-theoretic, and compare it to the residual entropy similarity. The similarity between random objects is also treated