The rapid development of distributed control technologies has made Fieldbus networks widely used in industrial control systems (ICSs). Meanwhile, the weak security protection of Fieldbus networks exposes potential attack paths for attackers. Attackers can tap covert and unauthorized external devices (i.e., intrusion devices) into the network to launch attacks. As the intrusion device can remain silent when eavesdropping, there is no detectable abnormal traffic in the network to detect the intrusion device. In this paper, we analytically prove that the observed signals sent from any benign device will inevitably change when the intrusion device is tapped into the Fieldbus network. With this knowledge, we construct the channel-state group fingerprint from the communication signals of each benign device and propose a collaborative intrusion detection mechanism for physical access, PhyCID, to passively detect the covert intrusion device. Detection results on a real power distribution cabinet, an RS485 bus testbed, and a controller area network (CAN) bus testbed indicate that PhyCID is purely passive, environmentally adaptive, and protocol-independent in most Fieldbus networks, including RS485 and CAN. Furthermore, extensive experiments under different scenarios demonstrate the effectiveness and robustness of PhyCID.