The timely and accurate prediction of winter wheat yields is of importance in maintaining food security. However, existing deep-learning methods used for crop yield prediction are limited. While most methods utilize recurrent neural networks (RNNs) to interpret crop time series data, they struggle to learn geographical spatial information from input data and often prove challenging to interpret with prior knowledge. In this study, hyperspectral images were used as the input to a BS-Nets network for band selection, and a graph-based RNN framework GT-long-short-term memory (LSTM) [two channels based LSTM-graph neural network (GNN)], was proposed for predicting the winter wheat yield at the county level. Using the BS-Nets for hyperspectral bands selection at the field scale, we obtained the top 4 bands in selection results for all six stages, the results were band 46 (791 nm), 50 (825 nm), 66 (954 nm), and 161 (2484 nm). Based on the results of hyperspectral bands selection, at the county scale, the similar wavelength bands of Sentinel-2 red-edge 3 (783 nm), NIR (834 nm), red-edge 4 (865 nm), SWIR2 (2190 nm) were chosen as inputs for the GT-LSTM county-level estimates of winter wheat yield. When only remote sensing data were used, the highest prediction accuracy ($R^{2}= 0.688$, RMSE = 0.54 t/ha) was obtained for DOY135 (30 days before harvest). The incorporation of the meteorological data improved the accuracy by 7% ($R^{2}= 0.714$, RMSE = 0.50 t/ha), and the optimal time for predicting wheat yield was at DOY115 (50 days before harvest). Further addition of GNN layers to the model improved the accuracy of the results by an additional 14% ($R^{2}= 0.757$, RMSE = 0.43 t/ha), and the best prediction results were then obtained at DOY105 (60 days before harvest).