Software engineer workflows have been transformed in recent years with the influx of AI coding tools such as Cursor and Github Copilot. This promises to increase productivity by automatically writing lines of code, creating bug fixes, and testing changes automatically. The tool is powered by AI models from Openai, Google Deepmind, Anthropic and Xai, and has rapidly improved performance in recent software engineering tests.
However, a new study published Thursday by the non-profit AI research group Metr raises questions about the extent to which today’s AI coding tools increase productivity for experienced developers.
METR recruited 16 experienced open source developers and conducted a randomized controlled trial for this study by completing 246 real tasks in a large code repository that regularly contributes. Researchers randomly assigned about half of these tasks as “AI-Allowed,” giving developers permission to use state-of-the-art AI coding tools such as Cursor Pro, and banning the use of AI tools for the other half of the tasks.
Before completing the assigned task, developers predicted that using AI coding tools would reduce the completion time by 24%. That wasn’t the case.
“Amazingly, we found that allowing AI actually increases completion time by 19%. Developers use AI tools to slow them down,” the researchers said.
In particular, only 56% of the developers of this study had experience using Cursor, the main AI tool offered in this study. Almost all developers (94%) had experience using web-based LLM in their coding workflows, but this study was the first time they had used cursors in particular. Researchers note that developers were trained in using cursors in preparation for their research.
Nevertheless, Metr’s findings raise questions about the universal productivity gains promised by AI coding tools in 2025. Based on research, developers should not assume that they will become known as AI coding tools, particularly “atmosphere coders.”
METR researchers point out some potential reasons why developers slow down developers rather than AI speed up developers. Developers are prompting AI when using Vibe Coders and waiting for it to respond, rather than actually coding. AI also tends to struggle with the large, complex codebase used by this test.
The authors of this study are careful not to draw strong conclusions from these findings, and explicitly point out that they do not believe that AI systems cannot speed up many software developers or most software developers at the moment. Other large studies have shown that AI coding tools speed up the workflow of software engineers.
The author also points out that AI progress has been substantial in recent years and does not expect the same results even in three months. Metr has also discovered that AI coding tools have significantly improved their ability to complete complex, long-term tasks in recent years.
However, this study offers another reason to be skeptical of the promised benefits of AI coding tools. Other research shows that today’s AI coding tools can introduce mistakes and, in some cases, security vulnerabilities.
Source link
