The problem of reconstructing a continuous-tone image given its ordered dithered halftone or its error-diffused halftone image is considered. We develop a modular class of nonlinear filters that can reconstruct the continuous-tone information preserving image details and edges that provide important visual cues. The proposed nonlinear reconstruction algorithms, denoted as binary permutation filters, are based on the space and rank orderings of the halftone samples provided by the multiset permutation of the "on" pixels in a halftone observation window. For a given window size, we obtain a wide range of filters by varying the amount of space-rank ordering information utilized in the estimate. For image reconstructions from ordered dithered halftones, we develop periodically space-varying filters that can account for the periodical nature of the underlying screening process. A class of suboptimal but simpler space-invariant reconstruction filters are also proposed and tested. Constrained LMS type algorithms are employed for the design of reconstruction filters that minimize the reconstruction mean squared error. We present simulations showing that binary permutation filters are modular, robust to image source characteristics, and that they produce high visual quality image reconstruction.