In the literature on object grasping, the robot often determines the grasp point and posture from visual information. They predict the grasping point uniquely from the object's shape characteristics. However, as a practical matter, there are cases where there are constraints on grasp point due to the object states, the limitation of the robot's hardware and the surrounding environment. In this study, we propose a neural network that can easily constrain the input. It determines the grasp pose from visual information and outputs the grasp success probability. The grasp pose is modified using backpropagation to increase the success rate of the grasp. As for the target object, we deal with some dirty tableware scattered on the table. We have developed a system that autonomously collects supervised data so that the robot can learn by itself whether it has succeeded in a grasp attempt. Finally, the robot can grasp an object which avoids dirty parts and find the suboptimal grasp pose.