A combined method of graph embedding (GE) and reinforcement learning (RL) is developed for discrete cross-section optimization of planar steel frames, in which the section size of each member is selected from a prescribed list of standard sections. The RL agent aims to minimize the total structural volume under various practical constraints. GE is a method for extracting features from data with irregular connectivity. While most of the existing GE methods aim at extracting node features, an improved GE formulation is developed for extracting features of edges associated with members in this study. Owing to the proposed GE operations, the agent is capable of grasping the structural property of columns and beams considering their connectivity in a frame with an arbitrary size as feature vectors of the same size. Using the feature vectors, the agent is trained to estimate the accurate return associated with each action and to take proper actions on which members to reduce or increase their size using an RL algorithm. The applicability of the proposed method is versatile because various frames different in the numbers of nodes and members can be used for both training and application phases. In the numerical examples, the trained agents outperform a particle swarm optimization method as a benchmark in terms of both computational cost and design quality for cross-sectional design changes; the agents successfully assign reasonable cross-sections considering the geometry, connectivity, and support and load conditions of the frames.