Emotion detection from face is an important problem and has received attention from industry and academia. Although emotion recognition from videos has a very high performance, emotion recognition from a single image stays a challenging task. In this paper, we try to use information from videos to do emotion recognition on a single image. More specifically, we leverage contrastive loss for pretraining the network on the videos and experiment with different sampling methods to select consistently hard triplets for continual learning of the network. Once the embeddings have been trained, we test them on a standard emotion classification task. Our method significantly improves the performance of the models and shows the efficacy of self-supervision in emotion recognition.