The number of merged PRs increased. But what does that actually tell us?
There is no comparison of the complexity, or depth of those PRs with work previously done without agents. The authors themselves acknowledge that PR count rewards small, frequent PRs and that whether the additional output justifies its cost remains unanswered. We do not know whether end-to-end feature delivery became faster, whether the additional output justified the cost, or what happened to defects, maintainability, review effort, and rework.
The conclusion seems to be little more than: Microsoft engineers are using AI tools, Microsoft is spending more on AI usage, and AI vendors are earning more subscription and token revenue.
More PRs are not necessarily more value. Without quality, delivery-time, and ROI metrics, the claimed productivity gain is largely meaningless.
Especially in conditions where AI assisted PRs are tracked for performance.
I had some extra usage on claude yesterday and put several ultra workflows to look for code cleanup opportunities. It made -40 tasks and PRs just like that. This is 10% of all PRs/commits in the project history, but the before and after is arguably meaningless.
Exactly. Once AI-assisted PR count becomes a performance metric, the metric is no longer merely incomplete—it actively creates an incentive to generate more tasks and PRs
There is no comparison of the complexity, or depth of those PRs with work previously done without agents. The authors themselves acknowledge that PR count rewards small, frequent PRs and that whether the additional output justifies its cost remains unanswered. We do not know whether end-to-end feature delivery became faster, whether the additional output justified the cost, or what happened to defects, maintainability, review effort, and rework.
The conclusion seems to be little more than: Microsoft engineers are using AI tools, Microsoft is spending more on AI usage, and AI vendors are earning more subscription and token revenue.
More PRs are not necessarily more value. Without quality, delivery-time, and ROI metrics, the claimed productivity gain is largely meaningless.