The use of social media data in research is common, spanning fields from computer science to social science, from human-computer interaction to law and criminology. However, social media data often contains personal and sensitive information. While prior work discusses the ethics of research using social media data, focusing on ethics broadly can be insufficient to unravel granular privacy risks and possible mitigations. Focusing on research papers that use social media data to study security-related topics, we systematically analyze 601 papers across 16 years, covering a wide array of academic disciplines. Our findings highlight a lack of transparency in reporting - only 35% of papers mention any considerations of data anonymization, availability, and storage. Applying Solove's taxonomy to classify the identified privacy risks in the social media setting, we observe that Solove's taxonomy was prescient in capturing aggregation risk, but the volume, timeliness, and micro details of data, combined with modern data science, yield risks beyond what was considered 20 years ago. We present the implications of our findings for various stakeholders: researchers, ethics boards, and publishing venues. While there are already signs of improvement, we posit that some small behavioral changes from the academic community may make a big difference in user privacy.