Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming

Code-recommendation systems, such as Copilot and CodeWhisperer, have thepotential to improve programmer productivity by suggesting and auto-completingcode. However, to fully realize their potential, we must understand howprogrammers interact with these systems and identify ways to improve thatinteraction. To seek insights about human-AI collaboration with coderecommendations systems, we studied GitHub Copilot, a code-recommendationsystem used by millions of programmers daily. We developed CUPS, a taxonomy ofcommon programmer activities when interacting with Copilot. Our study of 21programmers, who completed coding tasks and retrospectively labeled theirsessions with CUPS, showed that CUPS can help us understand how programmersinteract with code-recommendation systems, revealing inefficiencies and timecosts. Our insights reveal how programmers interact with Copilot and motivatenew interface designs and metrics.