Abstract Trajectory data combines the complexities of time series, spatial data, and (sometimes irrational) movement behavior. As data availability and computing power have increased, so has the popularity of deep learning from trajectory data. This review paper provides the first comprehensive overview of deep learning approaches for trajectory data. We have identified eight specific mobility use cases which we analyze with regards to the deep learning models and the training data used. Besides a comprehensive quantitative review of the literature since 2018, the main contribution of our work is the data-centric analysis of recent work in this field, placing it along the mobility data continuum which ranges from detailed dense trajectories of individual movers (quasi-continuous tracking data), to sparse trajectories (such as check-in data), and aggregated trajectories (crowd information).

In order to perform cutting-edge research like AI model training, a large amount of data needs to be accessed. However, data providers are often reluctant to share their data with researchers as these might contain personal data and thereby sharing may introduce serious risks with significant personal, institutional or societal impacts. Apart from the need to control these risks, data providers must also comply with regulations like GDPR, which creates an additional overhead that makes data sharing even less appealing to data providers. Technologies like anonymization can play a critical role when sharing data that may contain personal information by offering privacy preservation measures like face or license plate anonymization. Therefore, we propose a framework to support data sharing of personal data for research by integrating anonymization, risk assessment and automatic licence agreement generation. The framework offers a practical and efficient solution for organisations seeking to enhance data-sharing practices without compromising information security.

This paper introduces M3fed, a novel solution for federated learning of movement anomaly detection models. This innovation has the potential to improve data privacy and reduce communication costs in machine learning for movement anomaly detection. We present the novel federated learning (FL) strategies employed to train M3fed, perform an example experiment with maritime AIS data, and evaluate the results with respect to communication costs and FL model quality by comparing classic centralized M3and the new federated M3fed.

In this paper, we present a deep learning based multimodal system for classifying daily life videos. To train the system, we propose a two-phase training strategy. In the first training phase (Phase I), we extract the audio and visual (image) data from the original video. We then train the audio data and the visual data with independent deep learning based models. After the training processes, we obtain audio embeddings and visual embeddings by extracting feature maps from the pre-trained deep learning models. In the second training phase (Phase II), we train a fusion layer to combine the audio/visual embeddings and a dense layer to classify the combined embedding into target daily scenes. Our extensive experiments, which were conducted on the benchmark dataset of DCASE (IEEE AASP Challenge on Detection and Classification of Acoustic Scenes and Events) 2021 Task 1B Development, achieved the best classification accuracy of 80.5%, 91.8%, and 95.3% with only audio data, with only visual data, both audio and visual data, respectively. The highest classification accuracy of 95.3% presents an improvement of 17.9% compared with DCASE baseline and shows very competitive to the state-of-the-art systems.

Artificial Intelligence (AI) is increasingly explored in various domains and industries. Many companies experiment with AI, but too often those experiments are one-off analyses based on outdated data and the resulting models never make it into production. This paper proposes a framework for building and operating AI models at the industrial edge. The center of this framework is the model artifact, a model-generating entity. We analyze three AI model use-cases and user roles involved in industrial AI applications to illustrate the challenges in deploying and operating AI applications in industrial edge scenarios. We propose to structure the AI models into predefined artifacts that enable deployments with only a few clicks. The edge device links sensor data with the model input and returns the model output as feedback back into the industrial process. Model training, deployment, and management can be carried out in a scalable manner even by a non-expert. Several models can be managed in parallel, and data can be linked to the respective sensor or machine.

The digital transformation provides industries with unparalleled opportunities for value creation. AI and Machine learning (AI/ML)-driven approaches for data analysis applied to the massive amounts of data steaming from industrial processes can lead to enhanced operation, costs reduction, and powerful decision-making strategies. In this paper we address the problem of Quality Assurance (QA) in industrial manufacturing. We propose Smart Active Sampling (SAS), a new QA sampling strategy for quality inspection outside the production line. Based on the principles of active learning, an AI/ML model trained for quality prediction decides which produced pieces or samples are sent to quality inspection, to further improve its own prediction accuracy. SAS reduces the production of scrap parts due to earlier detection of quality violations. By inspecting a much lower number of samples as compared to traditional random sampling approaches, SAS improves QA efficiency and cuts down quality inspection costs, resulting in an overall smoother operation. We elaborate on some of the challenges faced in smart sampling strategies for quality inspection, describe the main concepts behind SAS, and showcase its application in a real-world manufacturing QA use case, training an AI/ML model for product defect prediction. Compared to a standard random sampling strategy, widely applied today in industrial QA applications, SAS improves model prediction accuracy requiring a significantly lower number of inspected samples, up to five time less samples in the analyzed dataset.

Traveler flows and crossings at the external borders of the EU are increasing and are expected to increase even more in the future; trends which encompass great challenges for travelers, border guards and the border infrastructure. In this paper, we present a new handheld device that enables border control authorities to check European, visaholding and frequent third country travelers in a comfortable, fast and secure way. The mobile solution incorporates new multimodal biometric capturing and matching units for face and 4-finger authentication. Thereby, the focus is on the capturing unit and fingerprint verification, which is evaluated in detail. The use in border control requires high security measurements and trustworthy use of credentials, which are also presented. Tests of the handheld device at a land border indicate great acceptance by travelers and border guards.

Facial animation has been combined with text-to-speech synthesis to create innovative multimodal interfaces, such as Web stores and Web-based customer services. This paper presents an image-based facial animation system using active appearance models (AAM) for precisely detecting feature points in the human face, which are required for selecting mouth images from the database of the face model. In order to minimize the impact of human error when creating the training data, a new optimization method for building an AAM is produced. Optimized training set reduces the average feature point location error from 1.15 pixels to 0.17 pixel. The feature points are suitable for automatic morphing between mouth images with large visual differences. Subjective tests show that morphing improves the visual quality of the animation from "fair" to "good".

This paper describes parameterization of mouth images for an image-based facial animation system. The analysis part of the facial animation system produces a face model, which is composed of a personalized mask as well as a large database of mouth images and their related phonetic and visual information. Then, a photo-realistic talking head is synthesized by rendering a personalized mask textured with a mouth image, which is selected from the database. The selection is driven by a unit selection algorithm, which finds the appropriate mouth images from the database such that they match the words spoken by the talking head. The selection of mouth images is based on parameters describing the mouth images. Therefore, the parameterization of mouth images is the key part for creating a photo-realistic facial animation. Hereby the visual parameterization of mouth images by LLE (locally linear embedding) is investigated comprehensively and compared with PCA (principal component analysis). Experimental results show that the parameterization of mouth images by LLE performs better for an image-based facial animation system than by PCA

In this paper we present a system to robustly estimate the 3D position of a human head. Before the face model is positioned in the initial frame, it is adapted to the 3D scan of the tracked human head. Head tracking is achieved by minimizing a robust cost function with a stochastic optimization algorithm called differential evolution. This approach enables the estimation of large motions between consecutive frames. Furthermore, the algorithm can even handle a large number of outliers e.g. caused by occlusion and still estimate the precise position

This paper describes an image-based facial animation system, which consists of the audiovisual analysis of a human subject and the synthesis of a photo-realistic facial animation. The unit selection algorithm selects for a given audio output the best mouth samples from the database by assigning two costs, the phonetic context and the visual distance between two consecutive samples. Here a novel approach to adapt the unit selection algorithm to an individual human subject is presented, such that a photo-realistic facial animation can be generated