Manning’s roughness coefficients (Manning’s n) are numerical values which represent the resistance to flows in river channels and floodplains. Field surveys and land use land cover (LULC) maps are commonly used to infer these Manning’s n values. Field surveys can be time and labour intensive while the LULC maps derived from Landsat and RadarSat-2 are often too coarse and may produce inaccurate representations of true surface roughness, leading to errors in the outputs from hydraulic models. In this work we tested two approaches to generate high resolution surfaces depicting Manning’s n. The first is using unclassified LiDAR point clouds and a voxel/connection (VC) based approach in combination with a rule-based table to set the roughness values. In the second method, we created training data for several land use categories and trained an Image Segmentation (IS) U-Net model to predict the category of each pixel and then applied a Manning's n value to each category. Our results indicate the LiDAR based approach is able to label all pixels within the study region, but the reclass table needs more attention. The IS method has the capability to cover a larger area more efficiently, but many pixels did not fit the existing training labels, leaving the output data incomplete. Future work is needed before either of these techniques can be reliably implemented.

Deep neural networks are vulnerable to adversarial attacks. Due to their black-box nature, it is rather challenging to interpret and properly repair these incorrect behaviors. This paper focuses on interpreting and repairing the incorrect behaviors of Recurrent Neural Networks (RNNs). We propose a lightweight model-based approach (RNNRepair) to help understand and repair incorrect behaviors of an RNN. Specifically, we build an influence model to characterize the stateful and statistical behaviors of an RNN over all the training data and to perform the influence analysis for the errors. Compared with the existing techniques on influence function, our method can efficiently estimate the influence of existing or newly added training samples for a given prediction at both sample level and segmentation level. Our empirical evaluation shows that the proposed influence model is able to extract accurate and understandable features. Based on the influence model, our proposed technique could effectively infer the influential instances from not only an entire testing sequence but also a segment within that sequence. Moreover, with the sample-level and segment-level influence relations, RNNRepair could further remediate two types of incorrect predictions at the sample level and segment level.

The state-of-the-art deep neural network (DNN) achieves impressive performance on the input that is similar to training data. However, it fails to make reasonable decisions on the input that is quite different from training data, i.e., out-of-distribution (OOD) examples. Although many techniques have been proposed to detect OOD examples in recent years, it is still a lack of a systematic study about the effectiveness and robustness of different techniques as well as the performance of OOD-aware DNN models. In this paper, we conduct a comprehensive study to unveil the mystery of current OOD detection techniques, and investigate the differences between OOD-unaware/-aware DNNs in model performance, robustness, and uncertainty. We first compare the effectiveness of existing detection techniques and identify the best one. Then, evasion attacks are performed to evaluate the robustness of techniques. Furthermore, we compare the accuracy and robustness between OOD-unaware/-aware DNNs. At last, we study the uncertainty of different models on various kinds of data. Empirical results show OOD-aware detection modules have better performance and are more robust against random noises and evasion attacks. OOD-awareness seldom degrades the accuracy of DNN models in training/test datasets. In contrast, it makes the DNN model more robust against adversarial attacks and noisy inputs. Our study calls for attention to the development of OOD-aware DNN models and the necessity to take data distribution into account when robust and reliable DNN models are desired.

As Deep Learning (DL) is continuously adopted in many industrial applications, its quality and reliability start to raise concerns. Similar to the traditional software development process, testing the DL software to uncover its defects at an early stage is an effective way to reduce risks after deployment. According to the fundamental assumption of deep learning, the DL software does not provide statistical guarantee and has limited capability in handling data that falls outside of its learned distribution, i.e., out-of-distribution (OOD) data. Although recent progress has been made in designing novel testing techniques for DL software, which can detect thousands of errors, the current state-of-the-art DL testing techniques usually do not take the distribution of generated test data into consideration. It is therefore hard to judge whether the "identified errors" are indeed meaningful errors to the DL application (i.e., due to quality issues of the model) or outliers that cannot be handled by the current model (i.e., due to the lack of training data). Tofill this gap, we take the first step and conduct a large scale empirical study, with a total of 451 experiment configurations, 42 deep neural networks (DNNs) and 1.2 million test data instances, to investigate and characterize the impact of OOD-awareness on DL testing. We further analyze the consequences when DL systems go into production by evaluating the effectiveness of adversarial retraining with distribution-aware errors. The results confirm that introducing data distribution awareness in both testing and enhancement phases outperforms distribution unaware retraining by up to 21.5%.