Visual sentiment analysis aims to predict human emotional responses to visual stimuli. It has attracted considerable attention owing to the increasing popularity of online image sharing. Most researchers have focused on improving emotion recognition using holistic and local information derived from given images. Relatively less attention has been paid to the semantic information of objects in images, which influences human emotional responses to the images. Therefore, we propose a novel object semantic attention network (OSANet) that attempts to unravel the semantic information of objects in images that contribute to emotion detection. The OSANet combines both global representation and semantic information of objects to predict the emotion elicited by a given image. First, the holistic features that represent the entire image are extracted using convolutional blocks. Subsequently, the object-level semantic information is obtained from pre-trained word embedding and then weighted according to the relative importance of the object using the attention mechanism. Notably, a new loss function to address the subjectivity of sentiment analysis is introduced, which improves the performance of the emotion detection task. Extensive experiments on three image emotion datasets demonstrated the superiority and interpretability of the OSANet. The results show that the OSANet outperforms extant image emotion detection frameworks.