Named Entity Recognition and Disambiguation (NERD) systems are foundational for information retrieval, question answering, event detection, and other natural language processing (NLP) applications. We introduce TweetNERD, a dataset of 340K+ Tweets across 2010-2021, for benchmarking NERD systems on Tweets. This is the largest and most temporally diverse open sourced dataset benchmark for NERD on Tweets and can be used to facilitate research in this area. We describe evaluation setup with TweetNERD for three NERD tasks: Named Entity Recognition (NER), Entity Linking with True Spans (EL), and End to End Entity Linking (End2End); and provide performance of existing publicly available methods on specific TweetNERD splits. TweetNERD is available at: https://doi.org/10.5281/zenodo.6617192 under Creative Commons Attribution 4.0 International (CC BY 4.0) license. Check out more details at https://github.com/twitter-research/TweetNERD.

Automated affective computing in the wild setting is a challenging problem in computer vision. Existing annotated databases of facial expressions in the wild are small and mostly cover discrete emotions (aka the categorical model). There are very limited annotated facial databases for affective computing in the continuous dimensional model (e.g., valence and arousal). To meet this need, we collected, annotated, and prepared for public distribution a new database of facial emotions in the wild (called AffectNet). AffectNet contains more than 1,000,000 facial images from the Internet by querying three major search engines using 1,250 emotion related keywords in six different languages. About half of the retrieved images were manually annotated for the presence of seven discrete facial expressions and the intensity of valence and arousal. AffectNet is by far the largest database of facial expression, valence, and arousal in the wild enabling research in automated facial expression recognition in two different emotion models. Two baseline deep neural networks are used to classify images in the categorical model and predict the intensity of valence and arousal. Various evaluation metrics show that our deep neural network baselines can perform better than conventional machine learning methods and off-the-shelf facial expression recognition systems.

Social robots are becoming an integrated part of our daily lives with the goal of understanding humans' social intentions and feelings, a capability which is often referred to as empathy. Despite significant progress towards the development of empathic social agents, current social robots have yet to reach the full emotional and social capabilities. This paper presents our recent effort on incorporating an automated Facial Expression Recognition (FER) system based on deep neural networks into the spoken dialog of a social robot (Ryan) to extend and enrich its capabilities beyond spoken dialog and integrate the user's affect state into the robot's responses. In order to evaluate whether this incorporation can improve social capabilities of Ryan, we conducted a series of Human-Robot-Interaction (HRI) experiments. In these experiments the subjects watched some videos and Ryan engaged them in a conversation driven by user's facial expressions perceived by the robot. We measured the accuracy of the automated FER system on the robot when interacting with different human subjects as well as three social/interactive aspects, namely task engagement, empathy, and likability of the robot. The results of our HRI study indicate that the subjects rated empathy and likability of the affect-aware Ryan significantly higher than non-empathic (the control condition) Ryan. Interestingly, we found that the accuracy of the FER system is not a limiting factor, as subjects rated the affect-aware agent equipped with a low accuracy FER system as empathic and likable as when facial expression was recognized by a human observer.

Automated Facial Expression Recognition (FER) has remained a challenging and interesting problem in computer vision. Despite efforts made in developing various methods for FER, existing approaches lack generalizability when applied to unseen images or those that are captured in wild setting (i.e. the results are not significant). Most of the existing approaches are based on engineered features (e.g. HOG, LBPH, and Gabor) where the classifier's hyper-parameters are tuned to give best recognition accuracies across a single database, or a small collection of similar databases. This paper proposes a deep neural network architecture to address the FER problem across multiple well-known standard face datasets. Specifically, our network consists of two convolutional layers each followed by max pooling and then four Inception layers. The network is a single component architecture that takes registered facial images as the input and classifies them into either of the six basic or the neutral expressions. We conducted comprehensive experiments on seven publicly available facial expression databases, viz. MultiPIE, MMI, CK+, DISFA, FERA, SFEW, and FER2013. The results of our proposed architecture are comparable to or better than the state-of-the-art methods and better than traditional convolutional neural networks in both accuracy and training time.

This paper presents an anthropomorphic robotic bear for the exploration of human-robot interaction including verbal and non-verbal communications. This robot is implemented with a hybrid face composed of a mechanical faceplate with 10 DOFs and an LCD-display-equipped mouth. The facial emotions of the bear are designed based on the description of the Facial Action Coding System as well as some animal-like gestures described by Darwin. The mouth movements are realized by synthesizing emotions with speech. User acceptance investigations have been conducted to evaluate the likability of these facial behaviors exhibited by the eBear. Multiple Kernel Learning is proposed to fuse different features for recognizing user's facial expressions. Our experimental results show that the developed Bear-Like robot can perceive basic facial expressions and provide emotive conveyance towards human beings.