In distributed manufacturing environments, Auto-mated Guided Vehicles (AGVs) equied with visual camera play a crucial role in automating material handling and optimizing production efficiency. Detecting anomalies during AGV operation is crucial to prevent potential malfunctions that could disrupt industrial processes. However, anomaly detection is challenging due to privacy concerns and the heterogeneity of data collected by AGVs across different factories. While sharing data across factories can improve the generalization capabilities of models, this can lead to overgeneralization in reconstruction-based anomaly detection, where the model reconstructs both normal and anomalous data too well, reducing its ability to detect anomalies. To address this problem, we propose FedCali, a federated learning framework that balances generalization and specialization across AGVs monitoring different manufacturing processes. Our proposed Gradient Guiding Mechanism (GGM) selectively aligns local model gradients with global knowledge only when necessary. This allows local models to retain their unique characteristics while benefiting from shared insights. Experiments with the MVTec dataset show that FedCali improves both reconstruction quality and anomaly detection accuracy, achieving higher AUROC scores and lower losses compared to baseline methods. This shows that FedCali is able to effectively process various manufacturing data collected by AGVs while maintaining data privacy.