Image memorability prediction aims to estimate the degree to which an image will be remembered by observers. Generally, the core problem in image memorability prediction is how to obtain effective representations to characterize the visual content of an image. In contrast to existing methods, which focus more on exploring the factors that make images memorable, in this paper, we first propose a general framework for learning joint low-rank and sparse principal feature representations, called the LSPFR framework, to obtain the lowest-rank intrinsic representation for image memorability prediction. By considering the joint optimization of the nuclear and$\ell _{1}$-norms, the global low-rank structure and the local patterns embedded in data can be exploited to make the learned features more robust and informative. To improve our framework based on the exploitation of sample relationship structure information, we present an extended version of LSPFR, named E-LSPFR, in which the underlying relationship structure matrix is inferred through a negative log-likelihood term with a sparsity constraint. The results of experiments conducted on four publicly available datasets confirm the superior performance of our proposed approaches.