Skip to content
Contact Us
Back to blog

AI & Machine Learning

Attention Is All You Need (But It's a Scarce Resource)

Agustin Manganelli

· 10 min read

Scaling pull requests in the era of agentic development

In 2017, eight researchers at Google published a paper called “Attention Is All You Need”, and the transformer it introduced set off the wave of AI we are living through. Few industries have felt it as directly as software development. Up until mid-2025, we (and the whole industry) worked in an AI-assisted mode. From mid-2025 to early 2026 the change became clear and inevitable, with the consolidation of development tools such as Claude Code and Codex.

This has led to an incredible leap in the speed of software development and, almost without noticing, we started facing another key problem: how do we review all the code being generated, staying efficient while still looking out for quality?

Every major tech company is facing this same problem, and their answers are starting to converge. This post covers what we found in our data, what the industry is doing, and what we are changing.

Facing the issue

As we adopted this new way of working, we focused on improving our development techniques and left the way we review the generated code mostly untouched. The industry sees it too: Meta, Anthropic and Stripe all report massive increases in diffs per developer, open PRs and shipped code.

The open-source community is also facing this problem. Daniel Stenberg, founder of cURL, ended the project's bug bounty program at the start of 2026, when the maintainers could no longer keep up with the number of AI-generated reports coming in.

At OrangeLoops we began to see the exact same pattern: loads of lines of code being generated, lots of open PRs, and no real process to review them safely and efficiently. In numbers: on one of our recent projects, built from scratch, we reached a peak of 40 active PRs authored by a single engineer. Combined with the work of the other 2 or 3 developers on the team, they piled up really fast. The diffs were also an entire order of magnitude larger than in our pre-AI development experience.

Looking into the data

PR sizes

We reviewed the patterns that emerged from our PRs, and a lot of insights came out. One particular thing about agent-driven development is the speed at which engineers can ship features or fix issues, and that speed led to bigger PRs.

Once we moved to agent-driven work, our median PR grew nearly 3×, and very large PRs became 2.5× more common. In projects built AI-first from day one, they were 4–5× more common than before. So we went back and reviewed the PRs shipped this way, tracing every bug fix to the PR that introduced it. Large PRs introduced a later-fixed bug about 4× as often as small ones. A second, independent analysis on a different set of projects found the same gap.

How we measured: 381 merged PRs across five projects, July 2025 to October 2026. A “large” PR is one with 400 or more reviewable lines: production code only, excluding tests, docs, generated code and lock files. A PR counts as introducing a bug when a later fix rewrote its lines (traced with git blame), and review comments count only reviewers other than the author.

Not every PR deserves the same review

If attention is the scarce resource, the next question is where to spend it. Here we borrowed from what the largest engineering organizations are doing. Meta's RADAR (Risk Aware Diff Auto Review) scores every diff for risk and lets low-risk ones land with lighter review. They report that the diffs RADAR reviews show a third of the revert rate and a fiftieth of the incident rate of the rest (on diffs pre-selected as low-risk, so the comparison is observational).

Anthropic triages the same way, by blast radius: the less a change can break, the less human review it gets (their Claude Code team published the details on October 15). The common recommendation is a deterministic risk tier: authentication, secrets, payments, data migrations and deployment pipelines always get a human, while docs, tests and generated code can move fast.

Lighter review does not mean no one is responsible. Even where AI does most of the reviewing, a human still owns the merge. At Anthropic, low-risk changes can merge on automated checks alone, but an agent's approval never counts as a person's, and whoever prompted the change is on the hook for it. Stripe's agents open more than a thousand PRs a week, and all of them are human-reviewed.

Google, where 75% of new code is now AI-generated, describes its engineers as supervising the agents. Atlassian keeps the final accept or decline decision on every AI review suggestion with a human, and Microsoft's .NET team says it has full ownership of everything it ships. The Linux kernel made it a rule: an AI agent cannot sign off a patch; the human who submits it carries full responsibility. As Simon Willison, co-creator of Django, put it: “A computer can never be held accountable. That's your job as the human in the loop.”

We built a lightweight version of that tier classification and replayed our history through it. Out of the box, generic rules already worked: PRs marked high-risk introduced bugs about 2× as often as the rest, and PRs touching less critical functionality introduced fewer.

But generic rules don't know your business. In a point-of-sale app, for instance, they flagged almost nothing, because the riskiest code there is fiscal billing, and no generic rule knows that. Once each project declared its core, the tier sharpened to 2–7×. Declaring it too broadly broke it again: when a whole source folder was marked as core, half of all PRs landed in high risk, and the tier stopped telling them apart.

The PRs that need attention the most get the least

A risk tier tells you where attention should go. We measured how much review each PR actually received, as review comments per line of reviewable code (excluding autogenerated code and other files that add noise to a PR), and compared it across sizes and risk levels. The result runs exactly backwards.

The bigger the PR, the thinner the review. From our smallest PRs to our largest, comment density fell 15×. A second, independent analysis on a different set of projects found a 21× drop.

Two bar charts by PR size. Share of PRs that introduced a later-fixed bug: XS 3%, S 12%, M 4%, L 16%, XL 25%. Review comments per 100 lines: XS 4.1, S 1.2, M 1.0, L 0.6, XL 0.2.

PR sizes in reviewable lines: XS under 50, S 50–199, M 200–399, L 400–999, XL 1,000 or more.

Risk made it worse, not better. Among our largest PRs, the high-risk ones received less than half the attention of the medium-risk ones. And in 2026, with agents writing most of the code, attention on large high-risk PRs halved again. The PRs touching auth, infrastructure and the business core, and carrying the most lines, were the ones reviewed most lightly.

This is also why having a review did not protect us. PRs that later turned out to introduce a bug had been reviewed just as often as the rest (90% vs 91%). In one case, a multi-PR stack covering a critical feature, several thousand lines in total, was approved in just over an hour with three comments. An approval shows that someone clicked. It does not show that someone read the code.

Bar chart: 90% of the PRs that later introduced a bug had been reviewed before merging, against 91% of all other PRs.

None of this is a failure of individual reviewers. Attention is fixed, and agents made code cheap to produce but no cheaper to read. When a PR is too big to read, people skim it, and skimming is hardest to notice on exactly the changes where it matters most.

What we are doing about it

Looking at our own data next to what the largest engineering organizations have published, the conclusions line up. Nobody serious is answering the flood of agent PRs by reviewing harder, or by dropping review. The answer is to change what reaches the reviewer, and to tell the reviewer where to look. That gives us three steps, in this order.

1. Strengthen the harness first

An agent is only as good as its ability to check its own work. Microsoft's .NET team raised the success rate of its agent PRs from 38% to 69% without changing the model, just by improving build scripts, setup and repository instructions. OpenAI reports the same loop: when unattended agents failed, the team added end-to-end tests, browser tooling and QA smoke tests instead of fixing each output by hand. Stripe puts it simply: “if it's good for humans, it's good for LLMs, too.”

For us, that means every repo needs tests and end-to-end checks the agent can run before it opens a PR, so it can verify what it built rather than claim it works. One warning from Microsoft matters here: an agent can write tests that lock in incorrect behavior as the expected result. Tests written by the same agent that wrote the code are not an independent check; they can encode the same mistake. The checks that guard the core must come from somewhere else.

We also looked into cross-provider reviewing. Current research suggests it only helps when the reviewer model is at least as strong as the one that did the work. With a weaker reviewer, there was no improvement, and in one study the results got worse.

2. Keep PRs small and scoped

Large PRs introduced bugs about 4× as often as small ones in our data, and they received a fraction of the review. Microsoft found that task scope matters even more than raw size: well-scoped, mechanical tasks were where agents performed best. Small, focused PRs help on both sides. The model generating the code works better with a narrow task, the reviewer can actually read the result, and neither needs to hold the whole domain in their head to judge it. In practice: one concern per PR, stacked PRs instead of one giant one, and a size budget that warns early and stops the PRs nobody can review.

3. Add a boring, deterministic risk classifier to CI

Finally, every PR gets classified automatically, as part of the CI run, before anyone looks at it. The classifier is deliberately boring: path rules, file types, destructive migrations, new permissions, secrets, plus a short list of each project's core. No model decides the risk. A rule-based classifier can't be talked out of it by a persuasive PR description, and its output is the same every time. It labels the PR with its size and risk tier and leaves a short comment explaining why. That tells reviewers where they can step on the gas, and where they need to slow down and read every line.

None of this removes the human. Someone still owns every merge. What changes is that their attention, the one resource that did not scale with the agents, gets spent where it matters.

References

Industry practice

Open source and accountability

AI review research

[Get In Touch]

What are you building?

Tell us about your project. We're curious about what you're building and where we might fit.
Or reach us at contact@orangeloops.com