Abstract Glaucoma is a leading cause of permanent vision loss. Early detection and treatment of this infection is critical for recovery and slowing the progression of vision loss. An efficient novel system focused on customized particle swarm optimization (CPSO) and four state‐of‐the‐art machine‐learning classifiers is proposed to boost prediction performance. This interconnected architecture detects glaucoma through five main phases: (1) pre‐processing, (2) segmentation, (3) feature extraction, (4) finding the best scored features, and (5) classification using the proposed CPSO‐machine learning dependent classifier. The subject images belong to the publically available benchmark Digital Retinal Images for Optic Nerve Segmentation retinal fundus data set. Rather than focusing on the initial 20 extracted features of the retinal fundus, half of the critical features are chosen to form a feature vector based on scores provided by the univariate method and the feature importance method separately. These features are fed into this system for training, testing, and multiple sets of results are created as a result of multiple combinations of CPSO and supervised machine‐learning classifiers. These result sets are evaluated using six efficiency metrics. According to the simulation results, the best output is recorded when a univariate selected feature vector is fed into the CPSO—K‐nearest neighbour dependent hybrid method. This model outperformed other models with a maximum accuracy of 0.99, a specificity of 0.96, a sensitivity of 0.97, a precision of 0.97, an F1‐score of 0.97, and a Kappa of 0.94. A fivefold cross‐validation method is used to derive the values. This research would help to achieve good levels of glaucoma care since the proposed system is excellent at distinguishing between stable and glaucomatous eyes. For ophthalmologists, this new technique can be used as second opinion for improving diagnostic accuracy for glaucoma.