Despite LiDAR (Light Detection and Ranging) being an effective privacy-preserving alternative to RGB cameras to perceive human activities, it remains largely underexplored in the context of multi-modal contrastive pre-training for human activity understanding (e.g., human activity recognition (HAR), retrieval, or person re-identification (RE-ID)). To close this gap, our work explores learning the correspondence between LiDAR point clouds, human skeleton poses, IMU data, and text in a joint embedding space. More specifically, we present DeSPITE, a Deep Skeleton-Pointcloud-IMU-Text Embedding model, which effectively learns a joint embedding space across these four modalities. At the heart of our empirical exploration, we have combined the existing LIPD and Babel datasets, which enabled us to synchronize data of all four modalities, allowing us to explore the learning of a new joint embedding space. Our experiments demonstrate novel human activity understanding tasks for point cloud sequences enabled through DeSPITE, including Skeleton<->Pointcloud<->IMU matching, retrieval, and temporal moment retrieval. Furthermore, we show that DeSPITE is an effective pre-training strategy for point cloud HAR through experiments in MSR-Action3D and HMPEAR.

Realistic crowd simulations are essential for immersive virtual environments, relying on both individual behaviors (microscopic dynamics) and overall crowd patterns (macroscopic characteristics). While recent data-driven methods like deep reinforcement learning improve microscopic realism, they often overlook critical macroscopic features such as crowd density and flow, which are governed by spatio-temporal spawn dynamics, namely, when and where agents enter a scene. Traditional methods, like random spawn rates, stochastic processes, or fixed schedules, are not guaranteed to capture the underlying complexity or lack diversity and realism. To address this issue, we propose a novel approach called nTPP-GMM that models spatio-temporal spawn dynamics using Neural Temporal Point Processes (nTPPs) that are coupled with a spawn-conditional Gaussian Mixture Model (GMM) for agent spawn and goal positions. We evaluate our approach by orchestrating crowd simulations of three diverse real-world datasets with nTPP-GMM. Our experiments demonstrate the orchestration with nTPP-GMM leads to realistic simulations that reflect real-world crowd scenarios and allow crowd analysis.

Recent advancements in deep learning have significantly enhanced the development of efficient models for multi-modal path prediction within urban environments, offering approaches to navigate complex environments accurately. Despite their performance, models grounded in deep learning techniques frequently encounter challenges related to interpretability. This limitation not only hampers their practical application but also complicates the process of diagnosing and rectifying errors within these systems, which is a critical factor for ensuring reliability and safety in realworld deployments. In this paper we propose NeSyMoF, a Neuro-Symbolic model for Motion Forecasting, to address this critical gap by combining the predictive power of deep neural networks with the interpretable logic inherent in symbolic reasoning. Data processing in NeSyMoF involves extracting pertinent features from the agent’s environment and channeling them into a neuro-symbolic reasoning module. The neurosymbolic reasoning module generates first-order logic rules that describe and condition the path prediction process, thereby providing clear explanations and intentions behind the forecasts of the model. We evaluate our model with the Argoverse benchmark for path forecasting, as it includes challenging driving situations, necessary to extensively evaluate our model. The results of our evaluation show that NeSyMoF outperforms state-of-the-art interpretable models for single-mode predictions while providing logic-based explanations for its forecasts, that articulate the reasoning behind predictions, making NeSyMoF more adapted for human-centric applications.

In this paper, we propose LiOn-XA, an unsupervised domain adaptation (UDA) approach that combines LiDAR-Only Cross-Modal (X) learning with Adversarial training for 3D LiDAR point cloud semantic segmentation to bridge the domain gap arising from environmental and sensor setup changes. Unlike existing works that exploit multiple data modalities like point clouds and RGB image data, we address UDA in scenarios where RGB images might not be available and show that two distinct LiDAR data representations can learn from each other for UDA. More specifically, we leverage 3D voxelized point clouds to preserve important geometric structure in combination with 2D projection-based range images that provide information such as object orientations or surfaces. To further align the feature space between both domains, we apply adversarial training using both features and predictions of both 2D and 3D neural networks. Our experiments on 3 real-to-real adaptation scenarios demonstrate the effectiveness of our approach, achieving new state-of-the-art performance when compared to previous uni- and multi-model UDA methods. Our source code is publicly available at https://github.com/JensLe97/lion-xa.

The annotation of 3D point cloud datasets is an expensive and tedious task. To optimize the annotation process, recent works have proposed the use of environments with higher levels of immersion in combination with different types of visual aids. However, two problems remain unresolved. First, the proposed environments limit the user to a unique level of immersion and a fixed hardware setup. Second, their design overlooks the interaction effects between the level of immersion and the visual aids on the quality of the annotation process. To address these issues, we propose PointCloudLab, an environment for 3D point cloud annotation that allows the use of different levels of immersion that work in combination with visual aids. Using PointCloudLab, we conducted a controlled experiment (N=20) to investigate the effects of levels of immersion and visual aids on the annotation process. Our findings reveal that higher levels of immersion combined with object-based visual aids lead to a faster and more accurate annotation. Furthermore, we found significant interaction effects between the levels of immersion and the visual aids on the accuracy of the annotation.

The recent advances in deep learning have paved the way for autonomous vehicles (AVs) to take charge of more complex tasks in the navigation process. However, predictions of deep learning models are subject to different types of uncertainty that may put the user and the surrounding environment in danger. In this paper, we investigate the effects that AR-based visualizations of 3 types of uncertainties in deep learning modules for path planning in AVs may have on drivers. The uncertainty types of the deep learning models that we consider are: the waypoint uncertainty, the situation uncertainty, and the path uncertainty. We propose 3 concepts to visualize the 3 uncertainty types on a Windshield display. We evaluate our AR-based concepts with a user study $(\mathrm{N}=20)$ using a VR-based immersive environment, to ensure the security of the participants. The results of our evaluation reveal that the absence of uncertainty visualization leads to lower driver engagement. More importantly, the combination of situation uncertainty and path uncertainty visualizations leads to higher driver engagement, and higher trust in the automated vehicle, while inducing an acceptable mental load for the drive.

In this paper, we propose a continual learning (CL) approach that adapts to the vehicle CAN-data flexibly and continuously. Our approach is capable of learning from vehicle CAN-bus data in multiple driving scenarios, adapting to the various drifts within each driving scenario. The basis for our approach corresponds to a common solver model and a series of supervisor models. Our solver model extends the memory-aware synapses approach with the use of weight cloning and weighted experience replay. Our supervisor model selects the output of the solver model that corresponds to the driving scenario present at the input. We evaluate our approach using a Tesla Model 3 CAN-data and 8 different driving scenarios. Our evaluation results show that our approach effectively learns multiple driving scenarios sequentially without forgetting the previous knowledge.

In this work, we address the problem of unsupervised moving object segmentation (MOS) in 4D LiDAR data recorded from a stationary sensor, where no ground truth annotations are involved. Deep learning-based state-of-the-art methods for LiDAR MOS strongly depend on annotated ground truth data, which is expensive to obtain and scarce in existence. To close this gap in the stationary setting, we propose a novel 4D LiDAR representation based on multivariate time series that relaxes the problem of unsupervised MOS to a time series clustering problem. More specifically, we propose modeling the change in occupancy of a voxel by a multivariate occupancy time series (MOTS), which captures spatio-temporal occupancy changes on the voxel level and its surrounding neighborhood. To perform unsupervised MOS, we train a neural network in a self-supervised manner to encode MOTS into voxel-level feature representations, which can be partitioned by a clustering algorithm into moving or stationary. Experiments on stationary scenes from the Raw KITTI dataset show that our fully unsupervised approach achieves performance that is comparable to that of supervised state-of-the-art approaches.

In this paper, we present an approach to discover periodic-frequent multi-step human routines in event data from smart devices and sensors deployed at home 1. Based on the discovered routines, our approach is able to suggest rules to automate the control of different aspects of the home environment. We evaluate our approach through an in the lab study, a study based on synthetic data, and an in-the-wild study. Our results show that our approach exhibits a high recall-precision performance, with a recovery rate of around 90% for most of the cases under investigation.

In this paper, we propose a new approach capable of performing continuous identification of users in home and office environments based on hand and arm motion patterns obtained from a wrist-worn inertial measurement unit (IMU). Different from state-of-the-art methods, our approach is not constrained to particular types of movements, gestures, or activities, thus allowing users to perform freely and unconstrained their daily routines while the identification takes place. We evaluate our approach by conducting an in the lab study and two in-situ studies, one in home environment and one in office environment. Our studies involved a total of 29 different participants and the data collected corresponds to approximately 256 hours. The results obtained in the studies indicate that our approach is able to perform continuous user identification with an accuracy of 0.88 for office environments and 0.71 for the average size of a household.

Context: Onboarding is a process that helps newcomers become integrated members of their organisation. Successful onboarding programs can result in increased performance in conventional organisations, but there is little guidance on how to onboard new developers in Open Source Software (OSS) projects. Goal: In this study, we examine how mentoring and project characteristics influence the effectiveness and efficiency of the onboarding process. We study a collaboration program involving a total of nine Open Source Software projects and more than 120 students from different universities around the world as part of Facebook's Education Modernization Program. Method: We use quantitative measurements of source code repositories, issue tracking systems, and discussion fora to examine how newcomers become contributing members of their OSS projects. Results: We found that developers receiving deliberate onboarding support through mentoring were more active at an earlier stage than developers entering projects through conventional means. Also, we found that project size and lifetime influenced onboarding. Conclusion: Empirical decision support can contribute to a more effective onboarding process in OSS projects. Mentor support in critical stages can accelerate the process, but project maturity is also a significant factor that increases the effect of onboarding.