It has been more than a year since GitHub launched the GitHub Copilot technology preview last year to the official launch this year. How helpful is this tool, which calls itself an “AI programming assistant”, to developers? The GitHub Next team investigated the issue using surveys and experiments, and said it had both expected and unexpected results.

The GitHub Next team said that because AI-assisted development is a relatively new field, its researchers have no existing cases to refer to. So when they want to assess the impact of GitHub Copilot on developers, they don’t know where to start. After early observations and interviews with users, they surveyed more than 2,000 developers to understand their experience with GitHub Copilot at scale. The researchers considered three points when designing their research methods:

• Look at productivity as a whole.At GitHub, we like to think broadly and sustainably about developer productivity and the many factors that influence it. We used the SPACE Productivity Framework to select aspects to investigate.
• Refer to the developer’s first point of view.The researchers set up rounds of studies, including qualitative (perceived) and quantitative (observed) data, hoping to get a glimpse of the big picture through this combination. They want to verify: (a) Does the developer’s actual experience confirm what they infer from the telemetry data? (b) Does their qualitative feedback apply to their large user base?
• Evaluate how GitHub Copilot works in everyday development scenarios.When setting up the study, professional developers were specifically recruited, and tests were designed around typical tasks a developer might accomplish in a day.

Based on this, the researchers made the following findings.

1. Developers care more about productivity

Through these large-scale surveys, the research team hopes to see what developers using GitHub Copilot gain in addition to increased development speed. Here are the notable results.

• Improved developer satisfaction (happiness index).Between 60% and 75% of users say they are more satisfied with their work, feel less frustrated while coding, and are able to focus on more satisfying work while using GitHub Copilot. It’s a win for developers who feel good about what they’re doing.
• Effectively preserve mental energy.Developers report that GitHub Copilot helps them stay in a state of “flow” (73%) and keep mental effort during repetitive tasks (87%). This is where developer happiness lies, knowing from previous research that context switches and interruptions can ruin a developer’s day, and that certain types of work are draining energy.

2. Development speed is equally important

In the survey, researchers saw developers report that they completed tasks faster when using GitHub Copilot, especially repetitive tasks. This is an expected finding (after all, GitHub Copilot writes faster than humans), but the consistency of over 90% is still a pleasant surprise. The vast majority of developers feel that GitHub Copilot is helping them get things done faster.

We recruited 95 professional developers, randomly divided them into two groups, and calculated how long it would take them to write an HTTP server in JavaScript. One group used GitHub Copilot to complete the task, the other group did not. We try to control as many factors as possible – all developers are already familiar with JavaScript.

In the experiments, we measured, on average, how successful each group was at completing the task and how long it took for each group to complete the task.

• Groups using GitHub CopilotHigher rate of task completion(78% versus 70% for the group without Copilot).
• The notable difference is thatdevelopers who use GitHub Copilot complete tasks significantly faster than developers who don’t use GitHub Copilot by 55%. Specifically, developers using GitHub Copilot took an average of 1 hour and 11 minutes to complete the task, while developers who did not use GitHub Copilot took an average of 2 hours and 41 minutes. These results are statistically significant ( P=.0017 ), the 95% confidence interval for the percentage of speed gain is [21%, 89%].