Predictive models are essential in big data platforms to tackle the needs of several real-world applications. However, static models are known to be prone to obsolescence in dynamic environments. While concept drift detection represents a viable way to deal with this problem, it is scarcely explored in the context of big data streams. In this paper, we propose a distributed drift detection workflow based on the DDM algorithm paired with a predictive model. Our workflow updates the predictive model as soon as drifts are detected, adjusting to the most recent data characteristics. To enable the analysis of large-scale datasets, we leverage Pandas UDFs and Apache Spark, efficiently distributing this workload across multiple worker node instances. Our experiments on two real-world drift detection datasets show the positive results obtained in terms of Speedup, Scaleup, and a limited impact in detection delay in comparison to a single worker node instance.

Prolonged air leak is a complication arising from a collapsed lung which can lead to serious illness such as pneumonia and empyema, and patient suffering from indwelling chest tubes. Drainage of air and liquid from chest drains can be monitored and recorded using novel digital chest drainage devices. The collected data can be analyzed by predictive models, which can provide decision support in chest tube management. Despite the promising adoption of predictive models in this context, existing approaches are still in their infancy and are mostly based on autoregressive and conventional machine learning models. In this paper, we present a LSTM-based model architecture for air leak forecasting that is able to deal with non-linear dependencies among different features and contiguous time points. We devise a post-processing procedure that leverages predictions to suggest whether the patient could have their chest tube safely removed in the upcoming hours, and evaluate the results according to a medical protocol. Experimental results show that our model is able to outperform currently adopted models, in terms of both forecasting and classification performance, suggesting the feasibility of our approach for chest tube management.

Clouds at different altitudes play different roles in Earth’s climate. Comprehensive understanding of overlapping clouds is important for climate and weather prediction. The East Pacific region is where El Niño and La Niña originate and where multi-layer clouds frequently occur. The overlap of clouds at different altitudes in this region increases the classification complexity for cloud-based climatological studies. Unlike prior work in cloud layer classification that assumes single layer or two-layer of clouds, in this work, we consider multi-layer cloud classification with 8 cloud-level classes (clear-sky, high, middle, low, high+middle, high+low, middle+low, high+middle+low). We develop and analyze machine learning models on features extracted from satellite images from the East Pacific regions collected by GOES Advanced Baseline Imager (ABI). These are used to classify CloudSat/CALIPSO observed multi-layer clouds. Due to the imbalanced nature of the data, we investigate the adoption of conventional resampling methods, as well as deep learning methods with data augmentation. In our experiments, we utilize the random forest classifier and Multilayer perceptron classifier with data augmentation methods to reduce the class imbalance during training. With these approaches, we achieve a classification accuracy of 83.6% without exploiting any ancillary information.

Proton therapy is a unique form of radiotherapy that utilizes protons to treat cancer by irradiating cancerous tumors while avoiding unnecessary radiation exposure to surrounding healthy tissues. Real-time imaging of prompt gamma rays can be used as a tool to make this form of therapy more effective. The use of Compton cameras is one proposed method for the real-time imaging of prompt gamma rays that are emitted by the proton beams as they travel through a patient’s body. The non-zero time resolution of the Compton camera, during which all interactions are recorded as occurring simultaneously, causes the reconstructed images to be noisy and insufficiently detailed to evaluate the proton delivery for the patient. Deep Learning has been a promising method used to remove and correct the different problems existing within the Compton Camera’s data. Previous papers have demonstrated the effectiveness of using deep fully connected networks to correct improperly ordered gamma interactions within the data. We do a moderately large hyperparameter grid search to find a promising set which yields competitive performance but contains fewer neurons making it compact. The studies which have many neurons, many layers, and a non-zero dropout rate have the best testing accuracy. These many neuron and many layer networks still have significantly fewer total neurons than the current neural network implementation. If given considerably more training time these compact networks could yield equal, if not superior, testing accuracy when compared to larger networks. More improvements are still needed for clinical use and we are currently experimenting with recurrent neural networks to test the viability of this type of architecture for this application.