The absence of comprehensive situational awareness information poses a significant challenge for humanitarian organizations during their response efforts. We present Flood Insights, an end-to-end system that ingests data from multiple non-traditional data sources such as remote sensing, social sensing, and geospatial data. We employ state-of-the-art natural language processing and computer vision models to identify flood exposure, ground-level damage and flood reports, and most importantly, urgent needs of affected people. We deploy and test the system during a recent real-world catastrophe, the 2022 Pakistan floods, to surface critical situational and damage information at the district level. We validated the system's effectiveness through geographic regression analysis using official ground-truth data, showcasing its strong performance and explanatory power. Moreover, the system was commended by the United Nations Development Programme stationed in Pakistan, as well as local authorities, for pinpointing hard-hit districts and enhancing disaster response.

The devastating 2022 floods in Pakistan resulted in a catastrophe impacting millions of people and destroying thousands of homes. While disaster management efforts were taken, crisis responders struggled to understand the country-wide flood extent, population exposure, urgent needs of affected people, and various types of damage. To tackle this challenge, we leverage remote and social sensing with geospatial data using state-of-the-art machine learning techniques for text and image processing. Our satellite-based analysis over a one-month period (25 Aug–25 Sep) revealed that 11.48% of Pakistan was inundated. When combined with geospatial data, this meant 18.9 million people were at risk across 160 districts in Pakistan, with adults constituting 50% of the exposed population. Our social sensing data analysis surfaced 106.7k reports pertaining to deaths, injuries, and concerns of the affected people. To understand the urgent needs of the affected population, we analyzed tweet texts and found that South Karachi, Chitral and North Waziristan required the most basic necessities like food and shelter. Further analysis of tweet images revealed that Lasbela, Rajanpur, and Jhal Magsi had the highest damage reports normalized by their population. These damage reports were found to correlate strongly with affected people reports and need reports, achieving an R-Square of 0.96 and 0.94, respectively. Our extensive study shows that combining remote sensing, social sensing, and geospatial data can provide accurate and timely information during a disaster event, which is crucial in prioritizing areas for immediate and gradual response.

Although speech driven facial animation has been studied extensively in the literature, works focusing on the affective content of the speech are limited. This is mostly due to the scarcity of affective audio-visual data. In this article, we improve the affective facial animation using domain adaptation by partially reducing the data scarcity. We first define a domain adaptation to map affective and neutral speech representations to a common latent space in which cross-domain bias is smaller. Then the domain adaptation is used to augment affective representations for each emotion category, including angry, disgust, fear, happy, sad, surprise, and neutral, so that we can better train emotion-dependent deep audio-to-visual (A2V) mapping models. Based on the emotion-dependent deep A2V models, the proposed affective facial synthesis system is realized in two stages: first, speech emotion recognition extracts soft emotion category likelihoods for the utterances; then a soft fusion of the emotion-dependent A2V mapping outputs form the affective facial synthesis. Experimental evaluations are performed on the SAVEE audio-visual dataset. The proposed models are assessed with objective and subjective evaluations. The proposed affective A2V system achieves significant MSE loss improvements in comparison to the recent literature. Furthermore, the resulting facial animations of the proposed system are preferred over the baseline animations in the subjective evaluations.

In human-to-computer interaction, facial animation in synchrony with affective speech can deliver more naturalistic conversational agents. In this paper, we present a two-stage deep learning approach for affective speech driven facial shape animation. In the first stage, we classify affective speech into seven emotion categories. In the second stage, we train separate deep estimators within each emotion category to synthesize facial shape from the affective speech. Objective and subjective evaluations are performed over the SAVEE dataset. The proposed emotion dependent facial shape model performs better in terms of the Mean Squared Error (MSE) loss and in generating the landmark animations, as compared to training a universal model regardless of the emotion.

Lips deliver visually active clues for speech articulation. Affective states define how humans articulate speech; hence, they also change articulation of lip motion. In this paper, we investigate effect of phonetic classes for affect recognition from lip articulations. The affect recognition problem is formalized in discrete activation, valence and dominance attributes. We use the symmetric KullbackLeibler divergence (KLD) to rate phonetic classes with larger discrimination across different affective states. We perform experimental evaluations using the IEMOCAP database. Our results demonstrate that lip articulations over a set of discriminative phonetic classes improves the affect recognition performance, and attains 3-class recognition rates for the activation, valence and dominance (AVD) attributes as 72.16%, 46.44% and 64.92%, respectively.