Semantic scene understanding is important for autonomous robots that aim to navigate dynamic environments, manipulate objects, or interact with humans in a natural way. In this paper, we address the problem of jointly performing semantic segmentation as well as instance segmentation in an online fashion, so that autonomous robots can use this information on-the-go and without sacrificing accuracy. We achieve this by exploiting a local connectivity prior of objects in the real world and a multi-task convolutional neural network architecture. The network identifies the individual object instances and their classes without region proposals or pre-segmentation of the images into individual classes. We implemented and thoroughly evaluated our approach, and our experiments suggest that our method can be used to accurately segment instance masks of objects and identify their class in an online fashion.