Abstract Community structure is an important topological feature of complex networks. Detecting community structure is a highly challenging problem in analyzing complex networks and has great importance in understanding the function and organization of networks. Up until now, numerous algorithms have been proposed for detecting community structure in complex networks. A wide range of these algorithms use the maximization of a quality function called modularity . In this article, three different algorithms, namely, MEM‐net, OMA‐net, and GAOMA‐net, have been proposed for detecting community structure in complex networks. In GAOMA‐net algorithm, which is the main proposed algorithm of this article, the combination of genetic algorithm (GA) and object migrating automata (OMA) has been used. In GAOMA‐net algorithm, the MEM‐net algorithm has been used as a heuristic to generate a portion of the initial population. The experiments on both real‐world and synthetic benchmark networks indicate that GAOMA‐net algorithm is efficient for detecting community structure in complex networks.