Production high-performance computing (HPC) systems are adopting and integrating GPUs into their design to accommodate artificial intelligence (AI), machine learning, and data visualization workloads. To aid with the design and operations of new and existing GPU-based large-scale systems, we provide a detailed characterization of system operations, job characteristics, user behavior, and trends on a contemporary GPU-accelerated production HPC system. Our insights indicate that the pre-mature phases in modern AI workflow take up significant GPU hours while underutilizing GPUs, which opens up the opportunity for a multi-tier system. Finally, we provide various potential recommendations and areas for future investment for system architects, operators, and users.

Filtering is the process of recovering a signal, x(t), from noisy measurements z(t). One common filter is the Kalman Filter, which is proven to be the conditional minimum variance estimator of x(t) when the measurements are a Gaussian random processes. However, in practice (a) the measurements are not necessarily Gaussian and (b) an estimation of the measurement covariance is problematic and often tuned using cross-validation and domain knowledge. In order to address the Kalman Filter's suboptimal performance in situations where non-Gaussian noise is present, we train a deep autoencoder to learn a mapping of the noisy measurements to a "learned" noise process covariance R(t) and measurements z(t), which are passed to a Kalman Filter. The Kalman Filter's output is then mapped to the original measurement space via the decoder portion of the autoencoder, where this output and original measurements are compared. Training this autoencoder-Kalman Filter (AEKF) via domain randomization on simulated noisy sensor responses, we show the AEKF's estimate of x(t), in the majority of cases, achieves a lower MSE than both a standard Kalman Filter and a Long Short-Term Memory (LSTM) recurrent deep neural network. Most significantly, the AEKF outperforms all methods in the case of simulated Cauchy noise.