Co-saliency detection aims to identify the common and salient objects from a group of relevant images. The main challenge for co-saliency detection is how to mine and exploit the saliency cues of both intra-image and inter-image. In this paper, we present a novel unified hierarchical neural network (UHGNN). We first construct the graph model by segmenting the images into super-pixels and extracting the intra-image hierarchical saliency cues. Then, the inter-image hierarchical saliency representation is mined to form the unified two-dimensional hierarchical feature setup. We further propose the geometric attention module to make the most of the intra-image and inter-image cues. Our UHGNN model competes or outperforms the state-of-the-art methods on two co-saliency detection benchmark datasets (MSRC, iCoSeg).