The point of interest (POI) recommendation algorithm in location based social network (LBSN) can assist people to find more appealing locations and satisfy their specific demands. However, it is challengeable to infer user’s preference due to the sparsity of the user’s check-in data. To address the problem and improve recommendation performance, this paper proposes an improved context-aware weighted matrix factorization algorithm for POI recommendation (ICWMF). It takes advantage of time factor, geographical information, and social relationship to obtain user’s preference for locations. Firstly, the Ebbinghaus forgetting curve is employed to model the influence of time attenuation, so as to reflect that user preferences change over time. In order to assign dynamic weights to unvisited POI and infer user preference, we build the implicit feedback term by modeling the geographical influence from user perspective and the social relationship. In addition, the Gaussian model is employed to construct proximity location relationship to represent the probability of locations being discovered by users. Then, it is taken as the regularization term to avoid overfitting. Finally, the objective function of weighted matrix factorization is reconstructed with the implicit feedback term and the regularization term we designed. ICWMF naturally learns two potential feature matrices during weighted matrix decomposition based on new designed objective function to achieve better recommendation results. The results of simulation experiments on Brightkite and Gowalla dataset indicate that ICWMF outperforms other four comparison methods in terms of precision and recall.