With the new developments of natural language processing, increasing attention has been given to the task of Named Entity Recognition (NER). However, the vast majority of work focus on a small number of large-scale annotated datasets with a limited number of entities such as person, location and organization. While other datasets have been introduced with domain-specific entities, the smaller size of these largely limits the applicability of state-of-the-art deep models. Even if there are promising new approaches for performing zero-shot learning (ZSL), they are not designed for a cross-domain settings. We propose Cross Domain Zero Shot Named Entity Recognition with Knowledge Graph (DOZEN), which learns the relations between entities across different domains from an existing ontology of external knowledge and a set of analogies linking entities and domains. Experiments performed on both large scale and domain-specific datasets indicate that DOZEN is the most suitable option to extracts unseen entities in a target dataset from a different domain.

Our world is witnessing the on-going substantial increase in the number of multimodal physical and social sensors that are ubiquitously distributed and observing or reporting what is happening in their surroundings. These sensors provide massive amounts of spatio-temporal digital footprints which can be analyzed for various tasks such as event detection or situation awareness. However, inherent noise due to the nature of these sensors result in imprecise data and hence imprecise analysis. Also, the heterogeneous data from different modalities, formats and sources make interpreting different levels of information a big challenge. To overcome these limitations, we propose a novel unified matrix factorization-based model to fuse physical and social sensor signals for spatio-temporal analysis. Readings of physical sensor signals are represented by a spatio-temporal situation matrix, which then incorporates social content that can provide explanations for the signal strengths. We test our framework on large-scale real-world data including PSI stations data, traffic CCTV camera images, and tweets for situation prediction as well as for filtering noise to detect events of diverse situations. The experimental results suggest that the proposed matrix factorization approach can utilize the sources correlation, resulting in better performances in various situational understanding tasks.

Online Social Network Sites have become a primary platform for brands and organizations to engage their audience by sharing image and video posts on their timelines. Different from traditional advertising, these posts are not restricted to the products or logo but include visual elements that express more in general the values and attributes of the brand, called brand associations. Since marketers are increasingly spending time in discovering and re-posting user generated posts that reflect the brand attributes, there is an increasing demand for such discovery systems. The goal of these systems is to assist brand experts in filtering through online collections of new user media to discover actionable posts, which match the brand value and have the potential to engage the consumers. Driven by this real-life application, we define and formulate a new task of content discovery for brands and propose a framework that learns to rank posts for brands from their historical timeline. We design a Personalized Content Discovery (PCD) framework to address the three challenges of high inter-brand similarity, sparsity of brand--post interactions, and diversification of timeline. To learn fine-grained brand representation and to generate explanations for the ranking, we automatically learn visual elements of posts from the timeline of brands and from a set of brand attributes in the domain of marketing. To test our framework we use two large-scale Instagram datasets that contain a total of more than 1.5 million image and video posts from the historical timeline of hundreds of brands from multiple verticals such as food and fashion. Extensive experiments indicate that our model can effectively learn fine-grained brand representations and outperform the closest state-of-the-art solutions.

Recent years have seen unprecedented research on using artificial intelligence to understand the subjective attributes of images and videos. These attributes are not objective properties of the content but are highly dependent on the perception of the viewers. Subjective attributes are extremely valuable in many applications where images are tailored to the needs of a large group, which consists of many individuals with inherently different ideas and preferences. For instance, marketing experts choose images to establish specific associations in the consumers' minds, while psychologists look for pictures with adequate emotions for therapy. Unfortunately, most of the existing frameworks either focus on objective attributes or rely on large scale datasets of annotated images, making them costly and unable to clearly measure multiple interpretations of a single input. Meanwhile, we can see that users or organizations often interact with images in a multitude of real-life applications, such as the sharing of photographs by brands on social media or the re-posting of image microblogs by users. We argue that these aggregated interactions can serve as auxiliary information to infer image interpretations. To this end, we propose a probabilistic learning framework capable of transferring such subjective information to the image-level labels based on a known aggregated distribution. We use our framework to rank images by subjective attributes from the domain knowledge of social media marketing and personality psychology. Extensive studies and visualizations show that using auxiliary information is a viable line of research for the multimedia community to perform subjective attributes prediction.

Brands and organizations are using social networks such as Instagram to share image or video posts regularly, in order to engage and maximize their presence to the users. Differently from the traditional advertising paradigm, these posts feature not only specific products, but also the value and philosophy of the brand, known as brand associations in marketing literature. In fact, marketers are spending considerable resources to generate their content in-house, and increasingly often, to discover and repost the content generated by users. However, to choose the right posts for a brand in social media remains an open problem. Driven by this real-life application, we define the new task of content discovery for brands, which aims to discover posts that match the marketing value and brand associations of a target brand. We identify two main challenges in this new task: high inter-brand similarity and brand-post sparsity; and propose a tailored content-based learning-to-rank system to discover content for a target brand. Specifically, our method learns fine-grained brand representation via explicit modeling of brand associations, which can be interpreted as visual words shared among brands. We collected a new large-scale Instagram dataset, consisting of more than 1.1 million image and video posts from the history of 927 brands of fourteen verticals such as food and fashion. Extensive experiments indicate that our model can effectively learn fine-grained brand representations and outperform the closest state-of-the-art solutions.

Messages like "If You Drink Don't Drive", "Each water drop count" or "Smoking causes cancer" are often paired with visual content in order to persuade an audience to perform specific actions, such as clicking a link, retweeting a post or purchasing a product. Despite its usefulness, the current way of discovering actionable images is entirely manual and typically requires marketing experts to filter over thousands of candidate images. To help understand the audience, marketers and social scientists have been investigating for years the role of personality in personalized services by leveraging AI technologies and social network data. In this work, we analyze how personality affects user actions on images in a social network website, and which visual stimuli contained in image content influence actions from users with certain Big Five traits. In order to achieve this goal, we ground this research on psychological studies which investigate the interplay between personality and emotions. Given a public Twitter dataset containing 1.6 million user-image timeline retweet actions, we carried out two extensive statistical analysis, which show significant correlation between personality traits and affective visual concepts in image content. We then proposed a novel model that combines user personality traits and image visual concepts for the task of predicting user actions in advance. This work is the first attempt to integrate personality traits and multimedia features, and moves an important step towards building personalized systems for automatically discovering actionable multimedia content.

Images in social networks share different destinies: some are going to become popular while others are going to be completely unnoticed. In this paper we propose to use visual sentiment features together with three novel context features to predict a concise popularity score of social images. Experiments on large scale datasets show the benefits of proposed features on the performance of image popularity prediction. Exploiting state-of-the-art sentiment features, we report a qualitative analysis of which sentiments seem to be related to good or poor popularity. To the best of our knowledge, this is the first work understanding specific visual sentiments that positively or negatively influence the eventual popularity of images.

Two important challenges for social networks are the creation of targeted and personalized content for their users, selecting the most interesting material from the huge amount of user-generated content, and keeping user engagement , e.g. through creation and curation of users' profiles. In this demo we show a system for video commenting, sharing and interest discovery that combines recommendation algorithms, clustering techniques, tools for video tagging and evaluation of semantic resources relatedness. Combining these tools and techniques it becomes possible to provide personalized multimedia services and to improve and propagate interests and inter-personal connections through the network.