Cryptographic libraries are a vital security component of software systems, yet their misuse has caused several incidents. Prior work has established that misuse of cryptographic libraries is common, and developers struggle to use their APIs correctly. However, it is currently unknown how the design and implementation decisions that shape cryptographic library APIs are made. To investigate these decisions and associated challenges in the design and implementation process of cryptographic library APIs, we conducted 21 semi-structured interviews with experienced developers of cryptographic libraries and used thematic analysis to identify overarching topics and challenges they encountered. We find that design decisions span a spectrum of abstraction levels and are heavily influenced by cryptographic standards, other libraries, legacy code, and developers' intuitions. Developers are challenged by the optimal level of abstraction for cryptographic APIs to balance security, usability, and flexibility. They lack systematic knowledge on defining usability and achieving such balance. Consequently, developers rely on usability self-tests, personal experiences, and opinions. Based on our findings, we make detailed recommendations to tailor future research toward better empirically validated support of cryptographic library API design and implementation decisions. Further, we advocate for integrating research-based usability guidance into cryptographic standardization to foster community discussion early on and better support secure, usable, and flexible cryptographic library APIs.