A common technique for the partial encryption of compressed images and videos encrypts only the sign bits of some syntax elements such as the quantized transform coefficients or the motion vector differences. The sign bit can be interpreted as the most significant bit (MSB) in the binary representation of the syntax element. Our work is motivated by the key observation that the binary code used for this representation has an impact on the quality of the reconstruction at the eavesdropper and on the size of the stream to be encrypted. Therefore, we address the problem of optimal binary code design for partial encryption. Ideally, the goal is to simultaneously maximize the eavesdropper's distortion and minimize the length of the compressed MSB stream. Since these two objectives are conflicting in general, we formulate the problem as the maximization of a weighted sum of the eavesdropper's distortion and of the probability of the MSB being 0. We cast the problem as a binary integer linear program equivalent to a maximum weight matching problem, which has a polynomial-time solution algorithm. We show that when the source to be quantized and the quantizer are symmetric, the problem can be converted to a linear program of a smaller size, for a family of distortion metrics. Extensive experiments assess the performance of the optimized binary code in comparison with existing approaches. The results reveal that certain existing partial encryption schemes could benefit from the proposed design.