Information security awareness (ISA) is a practice focused on the set of skills which help a user successfully mitigate social engineering (SE) attacks. Evaluating the ISA of users is crucial, since early identification of users who are more vulnerable to SE attacks improves system security. Previous studies for evaluating the ISA of smartphone users rely on subjective data sources (questionnaires) and do not address the differences between classes of SE attacks. This paper presents a framework for evaluating the ISA of smartphone users for specific attack classes. In addition to questionnaires, we utilize objective data sources: a mobile agent, a network traffic monitor, and cybersecurity challenges. We evaluated the framework by conducting a long-term user study involving 162 users. The results show that: the self-reported behavior of users differs significantly from their actual behavior and the ISA level derived from the actual behavior of users is highly correlated with their ability to mitigate SE attacks.