Natural language understanding (NLU) is one of the most critical components in goal-oriented dialog systems and enables innovative Big Data applications such as intelligent voice assistants (IVA) and chatbots. While recent advances in deep learning-based NLU models have achieved significant improvements in terms of accuracy, most existing works are monolingual or bilingual. In this work, we propose and experiment with techniques to develop multilingual NLU models. In particular, we first propose a purely language-agnostic multilingual NLU framework using a multilingual BERT (mBERT) encoder, a joint decoder design for intent classification and s lot filling tasks, and a novel co-appearance regularization technique. Then three distinct language-aware multilingual NLU approaches are proposed including using language code as explicit input; using language-specific parameters during decoding; and using implicit language identification as an auxiliary task. We show results for a large-scale, commercial IVA system trained on a various set of intents with huge vocabulary sizes, as well as on a public multilingual NLU dataset. We performed experiments in explicit consideration of code-mixing and language dissimilarities which are practical concerns in large-scale real-world IVA systems. We have found that language-aware designs can improve NLU performance when language dissimilarity and code-mixing exist. The empirical results together with our proposed architectures provide important insights towards designing multilingual NLU systems.

With the ubiquitous network connectivity and the proliferation of mobile devices, people are increasingly consuming digital contents from social media driven music sharing platforms (e.g., YouTube, Soundcloud). In this paper, we study a novel problem of connotation-aware music retrieval that focuses on the connotation which expresses the implicit feeling or emotion beyond the explicit content in artworks. Our goal is to automatically retrieve relevant music on social media based on the connotation of visual inputs (e.g., images, photos) provided by the users. The problem is challenging as it requires the accurate identification of the implicit connotation from both images and music pieces, and the precise matching of the identified connotation across different data modalities. We develop a connotation-aware music retrieval (CaMR) framework to address the above challenges. Evaluation results from a real-world social media dataset demonstrate that the CaMR framework can retrieve music that is highly relevant to the connotation of the input image.

Fauxtography is a category of multi-modal posts that spreads misleading information on various online social platforms (e.g., Facebook, Twitter, Reddit). A fauxtography post usually consists of an image, a text description and comments from its readers. In this paper, we focus on an explainable fauxtography detection problem where the goal is to explain which a specific component of a post leads to the fauxtography decision. This problem is motivated by the limitations of current fauxtography detection solutions that only focus on the detection but ignore the important explanation aspect of their results. Two critical challenges exist in solving our problem: i) it is difficult to accurately identify the "guilty" component of a fauxtography post given the fact that different components of the post and their associations could all lead to the fauxtography; ii) it is expensive and time-consuming to obtain a good training set with fine-grained labels of fauxtography posts in terms of explainability, making the corresponding solutions weakly supervised in nature. To address the above challenges, we develop ExFaux, an end-to-end graph-based fauxtography explanation framework, to effectively explain which part of the post contributes to its fauxtography. We evaluate the ExFaux by creating a real-world dataset from online social media (Twitter and Reddit). The results show that ExFaux not only detects the fauxtography posts more accurately than the state-of-the-arts but also provides well-justified explanations to its results.

The recent advance of the federated learning (FL) has brought new opportunities for privacy-aware distributed machine learning (ML) applications to train a powerful ML model without accessing the private training data of the participants. In this paper, we focus on addressing a novel fair classification problem in FL where the model trained by FL displays discriminatory bias towards particular demographic groups. Addressing the fairness issue in a FL framework posts three critical challenges: fairness and performance trade-offs, restricted information, and constrained coordination. To address these challenges, we develop FairFL, a fair federated learning framework dedicated to reducing the bias in privacy-sensitive ML applications. It consists of a principled deep multi-agent reinforcement learning framework and a secure information aggregation protocol that optimizes both the accuracy and the fairness of the learned model while respecting the strict privacy constraints of the clients. Evaluation results on real-world applications showed that FairFL can achieve significant performance gains in both fairness and accuracy of the learned model compared to state-of-the-art baselines.

Social sensing has emerged as a new sensing paradigm to observe the physical world by exploring the "wisdom of crowd" on social media. This paper focuses on the abnormal traffic event localization problem using social media sensing. Two critical challenges exist in the state-of-the-arts: i) "content-only inference": the limited and unstructured content of a social media post provides little clue to accurately infer the locations of the reported traffic events; ii) "informal and scarce data": the language of the social media post (e.g., tweet) is informal and the number of the posts that report the abnormal traffic events is often quite small. To address the above challenges, we develop SyntaxLoc, a syntax-based probabilistic learning framework to accurately identify the location entities by exploring the syntax of social media content. We perform extensive experiments to evaluate the SyntaxLoc framework through real world case studies in both New York City and Los Angeles. Evaluation results demonstrate significant performance gains of the SyntaxLoc framework over state-of-the-art baselines in terms of accurately identifying the location entities that can be directly used to locate the abnormal traffic events.

With the increasing popularity of portable devices with cameras (e.g., smartphones and tablets) and ubiquitous Internet connectivity, travelers can share their instant experience during the travel by posting photos they took to social media platforms. In this paper, we present a new image-driven poetry recommender system that takes a traveler's photo as input and recommends classical poems that can enrich the photo with aesthetically pleasing quotes from the poems. Three critical challenges exist to solve this new problem: i) how to extract the implicit artistic conception embedded in both poems and images? ii) How to identify the salient objects in the image without knowing the creator's intent? iii) How to accommodate the diverse user perceptions of the image and make a diversified poetry recommendation? The proposed iPoemRec system jointly addresses the above challenges by developing heterogeneous information network and neural embedding techniques. Evaluation results from real-world datasets and a user study demonstrate that our system can recommend highly relevant classical poems for a given photo and receive significantly higher user ratings compared to the state-of-the-art baselines.

Road traffic accidents are a major challenge in urban transportation systems. An effective countermeasure to address this problem is to accurately forecast the traffic risks in a city before accidents actually happen. Current traffic accident prediction solutions largely rely on accurate data collected from infrastructure-based sensors, which is not always available due to various resource constraints or privacy and legal concerns. In this paper, we address this limitation by exploring social sensing, a new sensing paradigm that uses humans as sensors to report the states of the physical world. In particular, we consider two types of publicly available social sensing data sources: social media data (e.g., traffic posts on Twitter) and open city data (e.g., traffic data from the city web portal). In this paper, we develop the RiskCast, an inductive multi-view learning approach to accurately forecast the traffic risk by exploiting the social sensing data under a principled co-regularization framework. The evaluation results on a real world dataset from New York City show that RiskCast significantly outperforms the state-of-the-art baselines in forecasting the traffic risks in a city.

With the increasing popularity of online video platforms (e.g., YouTube, Vimeo), the spread of hateful videos and the lack of rigorous hateful content control have become a critical issue. This paper focuses on the problem of identifying online hatred-vulnerable videos where the videos themselves do not contain any hateful content but unexpectedly trigger hateful comments from the audience. It is suboptimal to simply treat the hatred-vulnerable videos as hateful ones and remove them from the sharing platforms. This will discourage the uploaders of such videos from sharing valid and informative videos in the future. However, treating these hatred-vulnerable videos as hatred-free ones will provide undesirable opportunities for hateful users to spread their toxic comments and extreme ideology. In this paper, we develop VulnerCheck, an end-to-end supervised learning approach to effectively classify hatred-vulnerable videos from hateful and hatred-free ones by exploring the structure and semantics features of audience's comment networks. VulnerCheck is content-agnostic in the sense that it does not analyze the content of the video and is therefore robust against sophisticated content creators who craft hateful videos to bypass the current content censorship. We evaluate VulnerCheck on a real-world dataset collected from YouTube. Results demonstrate that our scheme is both effective and efficient in identifying hatred-vulnerable videos and significantly outperforms the state-of-the-art baselines.

Urban land usage classification is a critical task in big data based smart city applications that aim to understand the social-economic land functions and physical land attributes in urban environments. This paper focuses on a migratable urban land usage classification problem using remote sensing data (i.e., satellite images). Our goal is to accurately classify the land usage of locations in a target city where the ground truth land usage data is not available by leveraging a classification model from a source city where such data is available. This problem is motivated by the limitation of current solutions that primarily rely on a rich set of ground-truth data for accurate model training, which encounters high annotation costs. Two important challenges exist in solving our problem: i) the target and source cities often have different urban characteristics that prevent the direct application of a model learned from the source city to the target city; ii) the complex visual features in satellite images make it non-trivial to “translate” the images from the target city to the source city for an accurate classification. To address the above challenges, we develop TransLand, an adversarial transfer learning framework to translate the satellite images from the target city to the source city for accurate land usage classification. We evaluate our scheme on the real-world satellite imagery and land usage datasets collected from live different cities in Europe. The results show that TransLand significantly outperforms the state-of-the-art land usage classification baselines in classifying the land usage of locations in a city.

Combating copyright infringing multimedia content has arisen as a critical undertaking in online video sharing platforms, such as YouTube and Twitch. In contrast to the traditional copyright detection problem that studies the static content (e.g., music, films, digital documents), the proposed system focuses on a much more challenging problem: detecting copyright infringements in live video streams. This is motivated by the observation that a large amount of copyright-infringing videos bypass the detector while many legal videos are taken down by mistake. In this paper, we present an end-to-end system that is dedicated to combating the copyright infringements in live video streams. The system to be demonstrated consists of 1) a web front-end for user interaction and customized video query, 2) a scalable and real-time video crawling system that can collect video metadata, live chat messages, and visual content of the live video streams on video sharing platforms, and 3) a novel supervised copyright detection engine that leverages the live chat messages of the audience to detect the copyright infringement of live videos.

Social media sensing has emerged as a new application paradigm to collect observations from online social media users about the physical environment. A fundamental problem in social media sensing applications lies in estimating the evolving truth of the measured variables and the reliability of data sources without knowing either of them a priori. This problem is referred to as dynamic truth discovery. Two major limitations exist in current truth discovery solutions: i) existing solutions cannot effectively address the missing truth problem where the measured variables do not have any reported measurements from the data sources; ii) the latent correlations among the measured variables were not fully captured and utilized in current solutions. In this paper, we proposed a Reliable Missing Truth Finder (RMTF) to address the above limitations in social media sensing applications. In particular, we develop a novel data-driven technique to identify the lagged and latent correlations among measured variables, and incorporate such correlation information into a holistic spatiotemporal inference model to infer the missing truth. We evaluated the RMTF using the real-world Twitter data feeds. The results show that the RMTF scheme significantly outperforms the state-of-the-art truth discovery solutions by correctly inferring the missing truth of the measured variables.

With the increasing popularity of online video sharing platforms (such as YouTube and Twitch), the detection of content that infringes copyright has emerged as a new critical problem in online social media. In contrast to the traditional copyright detection problem that studies the static content (e.g., music, films, digital documents), this paper focuses on a much more challenging problem: one in which the content of interest is from live videos. We found that the state-of-the-art commercial copyright infringement detection systems, such as the ContentID from YouTube, did not solve this problem well: large amounts of copyright-infringing videos bypass the detector while many legal videos are taken down by mistake. In addressing the copyright infringement detection problem for live videos, we identify several critical challenges: i) live streams are generated in real-time and the original copyright content from the owner may not be accessible; ii) streamers are getting more and more sophisticated in bypassing the copyright detection system (e.g., by modifying the title, tweaking the presentation of the video); iii) similar video descriptions and visual contents make it difficult to distinguish between legal streams and copyright-infringing ones. In this paper, we develop a crowdsourcing-based copyright infringement detection (CCID) scheme to address the above challenges by exploring a rich set of valuable clues from live chat messages. We evaluate CCID on two real world live video datasets collected from YouTube. The results show our scheme is significantly more effective and efficient than ContentID in detecting copyright-infringing live videos on YouTube.

With the ever-increasing number of road traffic accidents worldwide, the road traffic safety has become a critical problem in intelligent transportation systems. A key step towards improving the road traffic safety is to identify the locations where severe traffic accidents happen with a high probability so the precautions can be applied effectively. We refer to this problem as risky traffic location identification. While previous efforts have been made to address similar problems, two important limitations exist: i) data availability: many cities (especially in developing countries) do not maintain a publicly accessible database for the traffic accident records in a city, which makes it difficult to accurately estimate the accidents in the city; ii) location accuracy: many self-reported traffic accidents (e.g., social media posts from common citizens) are not associated with the exact GPS locations due to the privacy concerns. To address these limitations, this paper develops the RiskSens, a multi-view learning approach to identifying the risky traffic locations in a city by jointly exploring the social and remote sensing data. We evaluate RiskSens using a real world dataset from New York. The evaluation results show that RiskSens significantly outperforms the state-of-the- art baselines in identifying risky traffic locations in a city.

With the increasing popularity of online social media (e.g., Facebook, Twitter, Reddit), the detection of misleading content on social media has become a critical undertaking. This paper focuses on an important but largely unsolved problem: detecting fauxtography (i.e., social media posts with misleading images). We found that the existing literature falls short in solving this problem. In particular, current solutions either focus on the detection of fake images or misinformed texts of a social media post. However, they cannot solve our problem because the detection of fauxtography depends not only on the truthfulness of the images and the texts but also on the information they deliver together on the posts. In this paper, we develop the FauxBuster, an end-to-end supervised learning scheme that can effectively track down fauxtography by exploring the valuable clues from user's comments of a post on social media. The FauxBuster is content-free in that it does not rely on the analysis of the actual content of the images, and hence is robust against malicious uploaders who can intentionally modify the presentation and description of the images. We evaluate FauxBuster on real-world data collected from two mainstream social media platforms - Reddit and Twitter. Results show that our scheme is both effective and efficient in addressing the fauxtography problem.

Copyright infringement detection is a critical problem in large-scale online video sharing systems: the copyright-infringing videos must be correctly identified and removed from the system to protect the copyright of the content owners. This paper focuses on a challenging problem of detecting copyright infringement in live video streams. The problem is particularly difficult because i) streamers can be sophisticated and modify the title or tweak the presentation of the video to bypass the detection system; ii) legal videos and copyright-infringing ones may have very similar visual content and descriptions. We found current commercial copyright detection systems did not address this problem well: a large amount of copyrighted content bypasses the detection system while legal streams are taken down by mistake. In this paper, we develop the StreamGuard, an unsupervised Bayesian network based copyright infringement detection system that addresses the above challenges by leveraging live chat messages from the audience. We evaluate StreamGuard on real-world live video streams collected from YouTube. The results show that StreamGuard is effective and efficient in identifying the copyright-infringing videos.

Social media sensing has emerged as a new big data application paradigm to collect observations and claims about the measured variables in physical environment from common citizens. A fundamental problem in social media sensing applications lies in estimating the evolving truth of claims and the reliability of data sources without knowing either of them a priori, which is referred to as dynamic truth discovery. We identified two critical challenges that are not fully addressed by solutions from current literature. The first challenge is “physical constraint-awareness” where the transition of truth is constrained by some physical rules that must be followed to ensure correct estimation of the evolving truth. The second one is “noisy and incomplete data” where the social media sensing data is sparse in nature and contains a lot of rumors and misinformation, making it difficult to capture the constantly evolving truth of measured variables. In this paper, we developed a new Constraint-Aware Dynamic Truth Discovery (CA-DTD) scheme to address the above challenges. To address the physical constraint-awareness challenge, CA-DTD develops a new constraint-aware Hidden Markov Model to effectively infer the evolving truth of measured variables by incorporating physical constraints. To address the noisy and incomplete data challenge, CA-DTD fuses sensing observations from online social media with information from traditional news media using a principled approach. We evaluate CA-DTD scheme using two real-world social media sensing data traces and the results show that CA-DTD significantly outperforms the state-of-the-art baselines.

Point-of-Interest (POI) recommendation is an important application in Location-based Social Networks (LBSN). The category prediction problem is to predict the next POI category that users may visit. The predicted category information is critical in large-scale POI recommendation because it can significantly reduce the prediction space and improve the recommendation accuracy. While efforts have been made to address the POI category prediction problem, several important challenges still exist. First, existing solutions did not fully explore the temporal dependency (e.g., “long range dependency”) of users' check-in traces. Second, the hidden contextual information associated with each check-in point has been underutilized. In this work, we propose a Context-Aware POI Category Prediction (CAP-CP) scheme using Natural Language Processing (NLP) models. In particular, to address temporal dependency challenge, we develop a novel Temporal Adaptive Ngram (TA-Ngram) model to capture the dynamic dependency between check-in points. To address the challenge of hidden context incorporation, CAP-CP leverages the Probabilistic Latent Semantic Analysis (PLSA) model to infer the semantic implications of the context variables in the prediction model. Empirical results on a real world dataset show that our scheme can effectively improve the performance of the state-of-the-art POI recommendation solutions.

Users' home location is important information for many advanced information services in big data applications (e.g., localized recommendation, target ads of local business and urban planning). In this paper, we study the problem of accurately inferring the home locations of people from the noisy and sparse data they voluntarily share on online social media. Previous studies have developed supervised learning approaches to predict a person's home location in a city. However, the accuracy of these techniques largely depends on a high quality training dataset, which is difficult and expensive to obtain in practice. In this study, we propose a new analytical framework, Unsupervised Home Location Inference (UHLI), to accurately infer the home locations of people using a set of principle approaches. In particular, the UHLI scheme addresses the critical challenges of using sparse and noisy online social media data and derives an optimal solution to the home location inference problem. We evaluated the performance of our scheme and compared it to the state-of-the-art baselines using three real world data traces collected from Foursquare. The results showed that our scheme can accurately infer the home location of people and significantly outperform the state-of-the-art baselines.

In the big data era, it's important to identify trustworthy information from an influx of noisy data contributed by unvetted sources from online social media (e.g., Twitter, Instagram, Flickr). This task is referred to as truth discovery which aims at identifying the reliability of the sources and the truthfulness of claims they make without knowing either of them a priori. There are two important challenges that have not been well addressed in current truth discovery solutions. The first one is “misinformation spread” where a majority of sources are contributing to false claims, making the identification of truthful claims difficult. The second challenge is “data sparsity” where sources contribute a small number of claims, providing insufficient evidence to accomplish the truth discovery task. In this paper, we developed a Robust Truth Discovery (RTD) scheme to address the above two challenges. In particular, the RTD scheme explicitly quantifies different degrees of attitude that a source may express on a claim and incorporates the historical contributions of a source using a principled approach. The evaluation results on two real world datasetsshow that the RTD scheme significantly outperforms the state-of-the-art truth discovery methods.