User studies to investigate which human traits affect a user's response to cyber security related situations are typically conducted via self-reported surveys. However, it has been observed that factors such as peer perception, socially desirable responding, and responder bias etc. frequently impact the results, which then do not necessarily reflect the actual behavior of the user when subjected to real world security incidents. To mitigate such biases, we developed PsychoRithm - a software system that presents different real-world security scenarios for the subjects and records their real-time reactions to these scenarios. This paper describes the architecture of PsychoRithm, the design choices we had to make, and the challenges we faced in the design process.