Sequence embeddings are essential for tasks like time series analysis and natural language processing, yet pooling techniques to create compact sequence representations remain underexplored. Poor pooling methods can lead to significant information loss, diminishing the effectiveness of strong feature extractors. In this early results paper, we propose an autoencoder-based sequence pooling approach that leverages autoencoders' ability to compress information and supports pretraining during self-supervised learning. Evaluated on time series data with a transformer-based encoder, our method outperforms traditional pooling techniques, such as mathematical functions and learnable weighted sums.

We present a semi-supervised fault detection method that combines contrastive and active learning techniques to efficiently analyze sensor data in industrial systems. It uses a transformer-based encoder with rotary positional embeddings for self-supervised pre-training, which allows it to build structured representations that can be used for density-based clustering using DBSCAN. This method significantly reduces the need for manual labeling, as it only requires 20% of the data to achieve high accuracy in clustering. It outperforms unsupervised alternatives and is scalable and adaptable to evolving fault patterns. Its reduced manual intervention makes it a valuable tool for real-world industrial health monitoring.