The emergence of large language models (LLMs) represented by ChatGPT has profoundly influenced the conventional software development process. Nevertheless, there is little research on how developers interact with LLMs. To investigate the interaction between developers and LLMs during software development, we conducted a user study with 56 participants, who were randomly assigned into two groups to perform different types of software development tasks. The first task was solving two simple coding puzzles, and the second was to fix two real-world bugs from a small-scale open source projects. We captured the full screen histories of all participants to construct a Markov activity transition model, depicting transitions among prevalent development activities such as coding, writing prompt and debugging. By characterizing developer behavior of the two groups of participants in the task, our study contributes empirical insights into developer behavior while interacting with LLMs. Such knowledge could guide the development of LLM -specific capabilities in supporting software development tasks, and offer valuable insights for developers aiming to utilize LLMs for more efficient problem-solving in their development practices.