For successful peg-in-hole assembly, predefined sub-tasks should be executed sequentially according to the current contact state. Therefore, recognizing contact state transitions is essential in order to determine whether to continue the current task or proceed to the next. In that context, learning-based solutions have shown outstanding results. However, these methods heavily rely on balanced datasets, which are challenging to obtain due to the short duration of certain contact states and rare failure cases. To address this issue, this paper proposes a framework for estimating contact state transitions using anomaly detection through input data reconstruction. The proposed framework operates in a semi-supervised manner, eliminating the need for balanced datasets during training. For input data reconstruction, a convolutional neural network is combined with a variational autoencoder to process various sensor measurements as a multivariate time series. Unlike traditional binary anomaly detection, the proposed anomaly detector scores reconstruction errors and leverages domain knowledge to identify various contact state transitions in the peg-in-hole assembly. The effectiveness of the proposed framework is validated through experiments using a torque-controlled dual manipulator system.