This paper presents a system integrating physical human-robot interaction in a dual-arm object manipulation task using multi-modal robot skin. The skin covers the end-effectors, enabling distance and active force control for aligning and grasping unknown objects, and extends over the entire arms, allowing multi-contact interaction by a user to guide the manipulation motion. Additionally, the skin patches on the non-contact area of the end-effectors function as tactile buttons, with gestures like short taps or long presses designed to switch control objectives. The system integrates two main controllers: a direct wrench controller regulating the grasp force and a Cartesian impedance controller governing the end-effector motion. Interaction wrenches on the arm patches are projected into the end-effector frame or object frame for pre-grasp adjustments or generating manipulation motion. Our method is validated on a dual-arm mobile manipulator equipped with robotic skin, demonstrating smooth motion in response to tactile interaction while consistently maintaining a stable grasp. The system's robustness enables it to handle objects of various geometries without prior knowledge. The system's capability is further highlighted through a practical application where, with human guidance, the robot successfully hangs a toolbox on two nails.