While previous studies of Conversational Agents (e.g. Siri, Google Assistant, Alexa and Cortana) have focused on evaluating usability and exploring capabilities of these systems, little work has examined users' affective experiences. In this paper we present a survey study with 171 participants to examine CA users' affective experiences. Specifically, we present four major usage scenarios, users' affective responses in these scenarios, and the factors which influenced the affective responses. We found that users' overall experience was positive with interest being the most salient positive emotion. Affective responses differed depending on the scenarios. Both pragmatic and hedonic qualities influenced affect. The factors underlying pragmatic quality are: helpfulness, proactivity, fluidity, seamlessness and responsiveness. The factors underlying hedonic quality are: comfort in human-machine conversation, pride of using cutting-edge technology, fun during use, perception of having a human-like assistant, concern about privacy and fear of causing distraction.