Autonomous Underwater Vehicles (AUVs) encounter significant challenges in confined spaces like ports and testing tanks, where vehicle-environment interactions, such as wave reflections and unsteady flows, introduce complex, time-varying disturbances. Model-based state estimation methods can struggle to handle these dynamics, leading to localization errors. To address this, we propose a data-driven velocity estimation approach using Inertial Measurement Units (IMUs) and a Gated Recurrent Unit (GRU) neural network, capturing temporal dependencies and rejecting external disturbances. This velocity estimator is then integrated into a sensor fusion framework using an asynchronous Kalman filter to improve localization by fusing on-board and off-board sensor information. Experimental validation on miniature AUVs demonstrates the effectiveness of the proposed method in enhancing accuracy for velocity and position estimation in environments with significant disturbances due to interactions between the vehicle and the environment.