Controller area network (CAN) protocol is an efficient standard enabling communication among electronic control units (ECUs). However, the CAN bus is vulnerable to malicious attacks because of a lack of defense features. In this article, a novel vehicle intrusion detection system (IDS) is developed. The challenge is that existing techniques of IDSs rarely consider attacks with small-batch, which are characterized by their small attack scale and concealed attack patterns, posing a significant threat to driving safety. To solve this problem, we developed an algorithm model that merges multidimensional long short-term memory (MD-LSTM) and self-attention mechanism (SAM), shortly named MULSAM. The MULSAM model was compared with other baseline models, including stacked long short-term memory (LSTM), MD-LSTM, etc. Experiments show that our approach has the best-detection accuracy (98.98%) and training stability. Further, to speed up the inference of MULSAM on edge, the hardware accelerator is implemented on FPGA devices using technologies, such as parallelization, modular, pipeline, and fixed-point quantization. Experiments show that our FPGA-based acceleration scheme has a better-energy efficiency than the CPU platform. Even with a certain degree of quantification, the acceleration model for MULSAM still displays a high-detection accuracy of 98.81% and a low latency of 1.88 ms.