Inferring social relationships from mobility data is crucial for many applications because it reflects real-world connections among people. However, large-scale trajectory datasets with ground-truth social ties are exceedingly scarce, making it difficult to train deep models for relationship inference. To address this gap, we propose a transferable social relationship inference framework that can be trained on one high-quality, labeled dataset and then generalized to new datasets, even from different cities. Our framework rests on the key insight that social bonds depend largely on the frequency of individual meetings and the popularity of those meeting locations, both of which can be inferred statistically from raw trajectory data, irrespective of the underlying geographic semantics. It comprises two main modules: 1) Universal Social Relationship Classifier (USRC): A model trained to infer social relationships from trajectory data, and 2) Spatial Embedding Transfer (SET): A location embedding alignment technique that adapts new datasets to the pre-trained USRC model. By aligning location embeddings, SET module enables the pre-trained USRC to interpret previously unseen datasets without extra supervision. Experiments on five public datasets demonstrate that our method achieves state-of-the-art performance in zero-shot social relationship inference, surpassing other unsupervised, and in some cases, even supervised, approaches. Additionally, the SET module significantly improves location embedding alignment, outperforming existing baseline methods. The source code and data are available at https://github.com/chuchen2017/SET.

Given the temporal GPS coordinates from a large set of human agents, how can we model their mobility behavior toward effective anomaly (e.g., bad-actor or malicious behavior) detection without any labeled data? Human mobility and trajectory modeling have been extensively studied, showcasing varying abilities to manage complex inputs and balance performance-efficiency trade-offs. In this work, we formulate anomaly detection in complex human behavior by modeling raw GPS data as a sequence of stay-point events, each characterized by spatio-temporal features, along with trips (i.e., commutes) between the stay-points. Our problem formulation allows us to leverage modern sequence models for unsupervised training and anomaly detection. Notably, we equip our proposed model USTAD (for Uncertainty-aware Spatio-Temporal Anomaly Detection) with aleatoric (i.e., data) uncertainty estimation to account for inherent stochasticity in certain individuals' behavior, as well as epistemic (i.e., model) uncertainty to handle data sparsity under a large variety of human behaviors. Together, aleatoric and epistemic uncertainties unlock a robust loss function as well as uncertainty-aware decision-making in anomaly scoring. Extensive experiments show that USTAD significantly outperforms baselines in industry-scale data. We open-source all code at https://github.com/wenhaomin/USTAD.

Generating realistic human mobility data is essential for various application domains, including transportation, urban planning, and epidemic control, as real data is often inaccessible to researchers due to high costs and privacy concerns. Existing deep generative models learn from real trajectories to generate synthetic ones. Despite the progress, most of them suffer from training stability issues and scale poorly with increasing data size. More importantly, they often lack control mechanisms to guide the generated trajectories under constraints such as enforcing specific visits. To address these limitations, we formally define the controlled trajectory generation problem for effectively handling multiple spatiotemporal constraints. We introduce Geo-Llama, a novel LLM finetuning framework that can enforce multiple explicit visit constraints while maintaining contextual coherence of the generated trajectories. In this approach, pre-trained LLMs are fine-tuned on trajectory data with a visit-wise permutation strategy where each visit corresponds to a specific time and location. This strategy enables the model to capture spatiotemporal patterns regardless of visit orders while maintaining flexible and in-context constraint integration through prompts during generation. Extensive experiments on real-world and synthetic datasets validate the effectiveness of Geo-Llama, demonstrating its versatility and robustness in handling a broad range of constraints to generate more realistic trajectories compared to existing methods.

Human mobility modeling from GPS-trajectories and synthetic trajectory generation are crucial for various applications, such as urban planning, disaster management and epidemiology. Both of these tasks often require filling gaps in a partially specified sequence of visits, - a new problem that we call "controlled" synthetic trajectory generation. Existing methods for next-location prediction or synthetic trajectory generation cannot solve this problem as they lack the mechanisms needed to constrain the generated sequences of visits. Moreover, existing approaches (1) frequently treat space and time as independent factors, an assumption that fails to hold true in real-world scenarios, and (2) suffer from challenges in accuracy of temporal prediction as they fail to deal with mixed distributions and the inter-relationships of different modes with latent variables (e.g., day-of-the-week). These limitations become even more pronounced when the task involves filling gaps within sequences instead of solely predicting the next visit.

Modeling human mobility under uncertain conditions and individual preferences remains a difficult and unsolved problem. Data-driven deep learning approaches require extensive trajectory data for training, while more traditional methods often assume deterministic conditions or simple minimum-cost paths. We propose an implicit neural representation (INR) to learn continuous, latent fields of stochastic traffic properties over space and time. We successfully impute speeds on a road network with hundreds of thousands of edges from only a few hundred vehicles, then illustrate the quality of these representations on a trajectory generation task. A near-shortest-path algorithm weighted by the INR’s predictions produces plausible real-world routing choices, showing potential for applications in route planning and anomaly detection.

Human mobility research has significantly benefited from recent advances in machine learning, as have numerous other industries. Aided by the ever-increasing availability of geospatial and mobility data, machine learning models have enabled large-scale systems for simulating city-wide macro and micro mobility behaviors, urban planning, transportation management, and disaster relief optimization. However, while many fields have invested significant effort in solving the model transferability and generalization problem, the inability of machine learning-based human mobility models to generalize to new locations has come to be implicitly accepted in most geospatial research. In this vision paper, we focus on this geospatial generalization problem, its root causes, and how it is restricting the applications of otherwise-promising research. Most importantly, we argue for several data- and modeling-driven innovations which could help remedy this problem, spanning mega-scale simulations, large foundation models, and multi-task, transfer, and meta-learning. We also spotlight a handful of promising ideas which have recently emerged from the community. We hope that these proposals take root and help develop more capable, flexible, and generalizable models in research and industry.

The recent proliferation of real-world human mobility datasets has catalyzed geospatial and transportation research in trajectory prediction, demand forecasting, travel time estimation, and anomaly detection. However, these datasets also enable, more broadly, a descriptive analysis of intricate systems of human mobility. We formally define patterns of life analysis as a natural, explainable extension of online unsupervised anomaly detection, where we not only monitor a data stream for anomalies but also explicitly extract normal patterns over time. To learn patterns of life, we adapt Grow When Required (GWR) episodic memory from research in computational biology and neurorobotics to a new domain of geospatial analysis. This biologically-inspired neural network, related to self-organizing maps (SOM), constructs a set of "memories" or prototype traffic patterns incrementally as it iterates over the GPS stream. It then compares each new observation to its prior experiences, inducing an online, unsupervised clustering and anomaly detection on the data. We mine patterns-of-interest from the Porto taxi dataset, including both major public holidays and newly-discovered transportation anomalies, such as festivals and concerts which, to our knowledge, have not been previously acknowledged or reported in prior work. We anticipate that the capability to incrementally learn normal and abnormal road transportation behavior will be useful in many domains, including smart cities, autonomous vehicles, and urban planning and management.

A need to understand and predict vehicles' behavior underlies both public and private goals in the transportation domain, including urban planning and management, ride-sharing services, and intelligent transportation systems. Individuals' preferences and intended destinations vary throughout the day, week, and year: for example, bars are most popular in the evenings, and beaches are most popular in the summer. Despite this principle, we note that recent studies on a popular benchmark dataset from Porto, Portugal have found, at best, only marginal improvements in predictive performance from incorporating temporal information. We propose an approach based on hypernetworks, a variant of meta-learning ("learning to learn") in which a neural network learns to change its own weights in response to an input. In our case, the weights responsible for destination prediction vary with the metadata, in particular the time, of the input trajectory. The time-conditioned weights notably improve the model's error relative to ablation studies and comparable prior work, and we confirm our hypothesis that knowledge of time should improve prediction of a vehicle's intended destination.

This work introduces DIVINIA, a feature extractor and novel training objective for content-based image retrieval. DIVINIA combines a semantic matching objective with a ranking objective to produce a feature extractor that is able to retrieve semantically relevant regions from a large search corpus. It further ranks them appropriately according to visual similarity. Furthermore, DIVINIA provides a mechanism for performing one-shot and even zero-shot object localization without the need to fine-tune the feature extraction model or re-index the corpus of search features. We demonstrate the capabilities of the DIVINIA system in the context of object localization in satellite imagery. We present quantitative and qualitative results that show robust domain transfer between satellite image optics and sensor modalities. We show good precision and search relevance ordering when returning areas of interest to specific object classes.

It is common today for even consumer-grade cameras to tag images and videos with the location of the imagery on the earth's surface. Some imagery, however, does not have a geo-location tag and it thus becomes necessary to ascertain the location of the camera, image, or objects in the scene. For such imagery, users must work hard to deduce geo-locations using reference data. Geo-tagging of such image/video is an extremely time-consuming and labor-intensive activity that often meets with limited success. In this paper, we propose a system to estimate the geo-location and viewing direction of a query photo using geometric configuration of objects in the query. Our experiment using a set of ground-truth data within our proposed system shows promising results.

We propose a novel method to model and learn the scene activity, observed by a static camera. The proposed model is very general and can be applied for solution of a variety of problems. The motion patterns of objects in the scene are modeled in the form of a multivariate nonparametric probability density function of spatiotemporal variables (object locations and transition times between them). Kernel Density Estimation is used to learn this model in a completely unsupervised fashion. Learning is accomplished by observing the trajectories of objects by a static camera over extended periods of time. It encodes the probabilistic nature of the behavior of moving objects in the scene and is useful for activity analysis applications, such as persistent tracking and anomalous motion detection. In addition, the model also captures salient scene features, such as the areas of occlusion and most likely paths. Once the model is learned, we use a unified Markov Chain Monte Carlo (MCMC)-based framework for generating the most likely paths in the scene, improving foreground detection, persistent labeling of objects during tracking, and deciding whether a given trajectory represents an anomaly to the observed motion patterns. Experiments with real-world videos are reported which validate the proposed approach.

This paper presents a multi-frame data association algorithm for tracking multiple targets in video sequences. Multi-frame data association involves finding the most probable correspondences between target tracks and measurements (collected over multiple time instances) as well as handling the common tracking problems such as, track initiations and terminations, occlusions, and noisy detections. The problem is known to be NP-Hard for more than two frames. A rank constrained continuous formulation of the problem is presented that can be efficiently solved using nonlinear optimization methods. It is shown that the global and local extrema of the continuous problem respectively coincide with the maximum and the maximal solutions of the discrete counterpart. A scanning window based tracking algorithm is developed using the formulation that performs well under noisy conditions with frequent occlusions and multiple track initiations and terminations. The above claims are supported by experiments and quantitative evaluations using both synthetic and real data under different operating conditions.

This paper presents an unsupervised data driven scheme to automatically estimate the relative topology of overlapping cameras in a large visual sensor network. The proposed method learns the camera topology by employing the statistics of co-occurring observations (of moving targets) in each sensor. Since target observation data is typically very noisy in realistic scenarios, an efficient two step method is used for robust estimation of the planar homography between camera views. In the first step, modes in the co-occurrence data are learned using meanshift. In the second step, a RANSAC based procedure is used to estimate the homography from weighted co-occurrence modes. Note that the first step not only lessens the effects of noise but also reduces the search space for efficient calculation. Unlike most existing algorithms for overlapping camera calibration, the proposed method uses an update mechanism to adapt online to the changes in network topology. The method does not assume prior knowledge about the scene, target, or network properties. It is also robust to noise, traffic intensity, and the amount of overlap between the fields of view. Experiments and quantitative evaluation using both synthetic and real data are presented to support the above claims.

When viewed from a system of multiple cameras with non-overlapping fields of view, the appearance of an object in one camera view is usually very different from its appearance in another camera view due to the differences in illumination, pose and camera parameters. In order to handle the change in observed colors of an object as it moves from one camera to another, we show that all brightness transfer functions from a given camera to another camera lie in a low dimensional subspace and demonstrate that this subspace can be used to compute appearance similarity. In the proposed approach, the system learns the subspace of inter-camera brightness transfer functions in a training phase during which object correspondences are assumed to be known. Once the training is complete, correspondences are assigned using the maximum a posteriori (MAP) estimation framework using both location and appearance cues. We evaluate the proposed method under several real world scenarios obtaining encouraging results.

This paper presents a novel object-based video coding framework for videos obtained from a static camera. As opposed to most existing methods, the proposed method does not require explicit 2D or 3D models of objects and hence is general enough to cater for varying types of objects in the scene. The proposed system detects and tracks objects in the scene and learns the appearance model of each object online using incremental principal component analysis (IPCA). Each object is then coded using the coefficients of the most significant principal components of its learned appearance space. Due to smooth transitions between limited number of poses of an object, usually a limited number of significant principal components contribute to most of the variance in the object's appearance space and therefore only a small number of coefficients are required to code the object. The rigid component of the object's motion is coded in terms of its affine parameters. The framework is applied to compressing videos in surveillance and video phone domains. The proposed method is evaluated on videos containing a variety of scenarios such as multiple objects undergoing occlusion, splitting, merging, entering and exiting, as well as a changing background. Results on standard MPEG-7 videos are also presented. For all the videos, the proposed method displays higher Peak Signal to Noise Ratio (PSNR) compared to MPEG-2 and MPEG-4 methods, and provides comparable or better compression.

This paper presents a framework for finding point correspondences in monocular image sequences over multiple frames. The general problem of multiframe point correspondence is NP-hard for three or more frames. A polynomial time algorithm for a restriction of this problem is presented and is used as the basis of the proposed greedy algorithm for the general problem. The greedy nature of the proposed algorithm allows it to be used in real-time systems for tracking and surveillance, etc. In addition, the proposed algorithm deals with the problems of occlusion, missed detections, and false positives by using a single noniterative greedy optimization scheme and, hence, reduces the complexity of the overall algorithm as compared to most existing approaches where multiple heuristics are used for the same purpose. While most greedy algorithms for point tracking do not allow for entry and exit of the points from the scene, this is not a limitation for the proposed algorithm. Experiments with real and synthetic data over a wide range of scenarios and system parameters are presented to validate the claims about the performance of the proposed algorithm.

The mapping that relates the image irradiance to the image brightness (intensity) is known as the Radiometric Response Function or Camera Response Function. This usually unknown mapping is nonlinear and varies from one color channel to another. In this paper, we present a method to estimate the radiometric response functions (of R, G and B channels) of a color camera directly from the images of an arbitrary scene taken under different illumination conditions (The illumination conditions are not assumed to be known). The response function of a channel is modeled as a gamma curve and is recovered by using a constrained nonlinear minimization approach by exploiting the fact that the material properties of the scene remain constant in all the images. The performance of the proposed method is demonstrated experimentally.

Conventional tracking approaches assume proximity in space, time and appearance of objects in successive observations. However, observations of objects are often widely separated in time and space when viewed from multiple non-overlapping cameras. To address this problem, we present a novel approach for establishing object correspondence across non-overlapping cameras. Our multicamera tracking algorithm exploits the redundance in paths that people and cars tend to follow, e.g. roads, walk-ways or corridors, by using motion trends and appearance of objects, to establish correspondence. Our system does not require any inter-camera calibration, instead the system learns the camera topology and path probabilities of objects using Parzen windows, during a training phase. Once the training is complete, correspondences are assigned using the maximum a posteriori (MAP) estimation framework. The learned parameters are updated with changing trajectory patterns. Experiments with real world videos are reported, which validate the proposed approach.

We present a framework for finding point correspondences in monocular image sequences over multiple frames. The general problem of multiframe point correspondence is NP hard for three or more frames. A polynomial time algorithm for a restriction of this problem is presented, and is used as the basis of proposed greedy algorithm for the general problem. The greedy nature of the proposed algorithm allows it to be used in real time systems for tracking and surveillance etc. In addition, the proposed algorithm deals with the problems of occlusion, missed detections, and false positives, by using a single noniterative greedy optimization scheme, and hence, reduces the complexity of the overall algorithm as compared to most existing approaches, where multiple heuristics are used for the same purpose. While most greedy algorithms for point tracking do not allow for entry and exit of points from the scene, this is not a limitation for the proposed algorithm. Experiments with real and synthetic data show that the proposed algorithm outperforms the existing techniques and is applicable in more general settings.