Deep learning excels in advanced inference tasks using electronic neural networks (ENN), but faces energy consumption and limited computation speed challenges. To mitigate this, optical neural networks (ONNs) were developed, utilizing light for computations. However, their high manufacturing costs limited accessibility. In this work, we first introduce the binary optical neural network (BONN) - a streamlined ONN variant with binarized weights, which significantly reduces fabrication complexities and costs. Specifically, we address (i) the development of a binarization weight function aligned with backward-error propagation, and (ii) a simulation-based training for extra-large neural networks housing millions of neurons. We prototype six BONNs, each comprising four 0.8 × 0.8mm2 layers with one million 800 nm diameter neurons. Costs are cut to 0.13 USD per layer, marking a substantial decrease of 769× from previous ONNs. Experimental results reveal BONNs consume 2, 405× less power than leading ENNs while maintaining an average recognition accuracy of 74% across six datasets.