Accurate trajectory prediction of surrounding vehicles is important for automated vehicles. To solve several existing problems of maneuver-based trajectory prediction, we propose four targeted solutions and establish a trajectory prediction model that integrates semi-supervised And-or Graph (AOG) and Spatio-temporal LSTM (ST-LSTM). To reduce the dependence on the well-labeled dataset, we introduce the concept of sub-maneuvers to improve the classifications of vehicle movements based on the given rough maneuver labels. AOG is used as the backbone of the probabilistic motion inference considering sub-maneuvers. We only define the basic units and inference logics of AOG and design a semi-supervised approach to directly learn the sub-maneuvers and the inference model structure from the training data, without manually specifying the structure (layers and nodes) of the inference model. This approach helps to avoid excessive artificial design or biases. The learned hierarchical motion inference model improves the interpretability of the overall trajectory prediction process. To utilize vehicle interaction information and further yield more accurate prediction, we adopt two different methods to consider vehicle interaction in the two sub-models (maneuver recognition and trajectory prediction). The experiment on NGSIM I-80 dataset shows that the maneuver-based model proposed in this paper (AOG-ST and refined AOG-ST-TB) performs more accurate trajectory prediction results. Although the AOG-ST seems clumsy and slow, we show that it is a flexible and quick model for trajectory prediction for various driving scenarios through the discussion and experiment.