In coverage that drew 1,306 points on Hacker News, Sean Goedecke advanced a deceptively simple thesis: working with an LLM amplifies the skilled and exposes the unskilled. Expertise, he argued, is rewarded rather than replaced (Goedecke, “LLMs reward expertise”). The post resonated because it flatly contradicts both panic narratives — that AI will erase knowledge workers — and triumphalist ones — that anyone can now produce expert output by typing a sentence.

The argument deserves scrutiny because it makes a falsifiable empirical claim, not a philosophical one. Does the data support the idea that expertise is the variable that determines how much value a person extracts from an LLM? Or does the evidence point somewhere more nuanced? Goedecke himself flagged the risk in his own comment thread: some readers, he noted, are “rightly suspicious of a view that’s reassuring them about how they’re still valuable.” That honesty is the right starting point. We should test the claim against studies, not vibes.

What Goedecke actually argues

Goedecke’s core mechanic is straightforward. Before LLMs, a technical gap — say, not knowing CSS — forced you to either recruit a skilled colleague or hope a matching answer already existed online. Today the same person can delegate that gap to a model and produce “sort-of-okay” output. Everyone becomes a generalist (Goedecke).

From this, a tempting conclusion follows: if everyone talks to the same model, prompting skill is irrelevant and expertise no longer matters. Goedecke rejects that. His central claim is that the most important skill in prompting is expertise in the domain you are prompting about. A novice and an expert may get similar first drafts, but only the expert can steer, prune, and verify the result. He extends this to codebases specifically: if you hold a strong “theory of your codebase,” you can push the LLM far harder than someone with no familiarity, because you have a prior sense of what a good solution looks like.

The mechanism he proposes is information retrieval, not magic. The answer is “in the model” already; the scarce resource is the human ability to pull the right answer out. That reframes expertise as a compression and filtering skill: knowing which of the model’s many plausible lines to keep, and which to throw away before they calcify into a confident mistake.

The Terence Tao conversation, and what it really shows

His flagship exhibit is Terence Tao’s public conversation with ChatGPT about a recently discovered counterexample to the Jacobian Conjecture (Tao’s shared chat). Tao does not merely prompt; he makes leaps, proposes reformulations, and pushes back when outputs “look weird.” Goedecke notes the model shifts into “talking-to-mathematicians” mode for Tao, producing terser, denser replies than a layperson receives.

The lesson is not that Tao has mastered a secret prompt syntax. It is that domain knowledge lets you pull the right idea out of a multi-paragraph response and discard the rest. As we explored in our own analysis of Tao’s method, the genius sees the shape of the problem before the model finishes speaking, and he almost never takes the model’s advice about where to go next (TheAIprism: what a genius sees that we don’t). The model is a sparring partner, not an authority.

This maps onto a broader pattern Goedecke observes in his own engineering work: familiarity with concrete specifics beats generic principles. He can ask sharp questions about the systems he owns at GitHub that he could never ask about abstract mathematics. Expertise, in other words, is local — and LLMs reward the locality. The same person can be expert and novice in the same afternoon, depending on the domain, which is why the claim “expertise is rewarded” is true only relative to a specific task.

What the controlled studies actually say

Goedecke’s claim is anecdotal. The cleanest counter-evidence comes from randomized experiments. In a 2023 study, Shakked Noy and Whitney Zhang gave 453 professionals incentivized writing tasks, randomly assigning ChatGPT access (Noy & Zhang, Science). Output quality rose and completion time fell — but the gains were concentrated among lower-ability workers. The productivity distribution compressed rather than spread.

A large field study of customer-service agents reached the same pattern at scale. Brynjolfsson, Li, and Raymond studied thousands of agents before and after AI deployment and found an average 15% productivity lift, but a 34% lift for novice and low-skilled workers, with minimal effect on the best performers (Brynjolfsson, Li & Raymond, “Generative AI at Work”). Their interpretation: AI transmits the best practices of top performers downward, lifting the floor for everyone below them.

A 2024 age-classification experiment by Caplin et al. compounds the point: AI raised performance across ability levels but reduced dispersion most when users were well calibrated about their own skill (Law & Economics Center review). The recurring result across writing, support, and classification tasks is skill compression, not elite-only reward. If the only evidence were these three papers, Goedecke’s thesis would look wrong — which is exactly why the next study matters.

The mechanism behind compression is best-practice transmission. A junior who has never seen a strong example of the task suddenly has one on tap, every time. The senior, who already embodied those practices, gains little from re-encountering them. That is why the floor moves and the ceiling barely does — at least on the tasks these studies measured, which tend to be bounded and verifiable.

The jagged technological frontier

The most important caveat comes from the BCG–Harvard study of 758 consultants and 18 realistic tasks (Dell’Acqua et al., “Navigating the Jagged Technological Frontier”). Within the model’s capabilities, GPT-4 users completed 12.2% more tasks, 25.1% faster, and 40% produced higher-quality work. But on tasks just outside that “jagged frontier,” AI users were 19% less likely to reach a correct answer than people with no AI at all.

This is the crux. AI does not fail uniformly; it fails on a ragged boundary the user cannot see. The researchers distinguish “centaurs” (clean human/AI task splits) from “cyborgs” (constant interaction) — both work, but both depend on the human knowing where the frontier sits. Lakhani’s blunt warning captures it: “This is not Google.” Treating the model as a search box is exactly how users fell 19% behind on the hard tasks.

The frontier finding does something subtle to Goedecke’s thesis. It suggests the expert’s advantage is not merely producing better drafts — it is knowing which tasks to hand the model at all. Outside the frontier, more delegation is worse. That is a meta-judgment the novice lacks, and it is invisible in aggregate productivity numbers that average over easy and hard tasks alike. The expert’s reward, in other words, shows up as avoidance of catastrophe rather than headline speed gains.

Two effects, not one: compression and amplification

Set the studies side by side and a cleaner picture emerges. There are two distinct effects running in opposite directions:

Compression at the floor. AI lifts weaker workers most. Noy & Zhang, Brynjolfsson et al., and Caplin et al. all find performance dispersion shrinks, especially when users are well calibrated about their own skill. The floor rises fast.

Amplification at the ceiling. Experts extract more at the top. Goedecke’s Tao example and the BCG finding — that outside-frontier failure depends on user judgment — both imply the expert’s edge grows precisely where tasks are hard and the model is silent or wrong.

Goedecke is right that expertise is rewarded. He understates how much AI compresses the gap below. The honest synthesis is asymmetric: the floor rises faster than the ceiling. A junior with a model can now mimic a competent senior on routine work, but no amount of model access converts a novice into Tao on the frontier.

Consider a concrete split. On a bounded writing task — summarize this memo, draft this email — the novice-plus-model and the expert-plus-model land close, because the frontier encloses the task and the model supplies the missing structure. On an open mathematical proof or a subtle production incident, the novice gets fluent nonsense the expert immediately flags. The same tool, two regimes: compression where the frontier is generous, amplification where it is thin.

One caveat tempers the whole comparison: the frontier is not fixed. As models improve, tasks that were once outside it migrate inside, and the compression effect expands with them. If the boundary keeps moving outward, the era in which expertise is decisively rewarded at the ceiling may shrink to a thinner and thinner sliver of remaining-hard problems — unless expertise itself is what defines where the frontier currently lies.

Why calibration is the new bottleneck

If AI both lifts the floor and rewards expertise at the ceiling, what exactly does the skilled person contribute? The BCG data points to one scarce skill: calibration — knowing when to trust the model and when to ignore it. Outside the frontier, over-trust was actively harmful (−19% correctness). The human who suspects “this looks more complex than I hoped” and reroutes is the human who stays accurate.

Goedecke’s own phrasing fits: “the human is the bottleneck, not the model,” because the hard part is communicating exactly what solution you want (Goedecke). I would sharpen that: the bottleneck is judgment about the model’s limits, a meta-skill that sits above raw domain expertise. Domain expertise helps you recognize a wrong answer; calibration tells you whether to ask at all. Both are human, neither is automatable yet.

This also explains the Hacker News skeptic’s objection — that anyone can now feel rewarded. Feeling rewarded and being right are different. The model happily confirms the incompetent, which is precisely why calibration, not confidence, separates the expert from the amateur. Calibration is buildable: it grows from repeated, consequential feedback where wrong answers carry a cost the model cannot absorb for you. That is another reason expertise, earned through consequences, stays relevant.

Implications for knowledge work and hiring

For organizations, the data argues against two instincts. First, do not assume AI erases the need for senior talent; you need experts precisely to set direction and catch errors outside the frontier. Second, do not assume juniors are now interchangeable with seniors — AI narrows the gap but does not close it, and someone must still validate the output. The realistic play is mixed teams where experts handle the frontier and juniors, augmented, handle the floor.

This reframes the jobs debate away from “will AI replace us” toward “who can steer it” — a theme we examine in our broader review of what is actually happening to jobs (TheAIprism: separating AI hype from reality). The scarce role is the editor of the machine, not its operator. Hiring should weight demonstrated calibration — can this person tell good model output from fluent nonsense? — above raw output volume, because volume is now nearly free and discernment is not.

The same logic reshapes internal metrics. If AI narrows the spread between your best and worst contributors on routine work, average throughput becomes a worse signal of talent. Managers should monitor the tail — the hard, frontier cases where only calibration prevents regressions — rather than aggregate speed, or they will reward the person who delegates most and ships the most plausible errors.

Implications for education and evaluating AI output

The synthesis also reshapes how we should teach and assess. If AI compresses the floor, drilling rote execution matters less; teaching calibration matters more. Students need to learn not “how to write the essay” but “how to tell whether the essay the model wrote is correct.” That is a higher-order skill, and it is exactly the one experts already possess. Education that skips the fundamentals in favor of pure prompt reliance risks producing adults who cannot catch the model when it is wrong.

For evaluation, the lesson is uncomfortable: we can no longer grade the artifact without grading the process. A flawless draft may be expert-steered or expert-blind. The differentiator is whether the author can defend every line — a capacity AI cannot fake and expertise alone supplies. Assessment must move toward oral defense, source tracing, and revision history rather than the final product alone.

There is a Carnegie-style lesson here too. Just as cognitive tools historically offloaded routine computation, LLMs offload routine composition — and in both cases the expert’s value migrated to the parts the tool could not do. The frontier, not the floor, is where expertise lives, and curricula that teach only floor-level execution are teaching the part the machine now owns. The goal of training shifts from producing flawless executors to producing reliable judges.

The verdict, and an open question

Goedecke’s thesis survives contact with the data, but in a revised form. Expertise is rewarded — yet so is the absence of it, because AI raises the floor for everyone. The net effect is not replacement but reorganization: execution cheapens, judgment appreciates. The people who thrive are those who treat the model as a brilliant, unreliable junior colleague rather than an oracle, and who invest in the calibration the studies show is decisive.

The practical takeaway for knowledge workers is unglamorous. Spend less energy on prompt incantations and more on deepening the domain sense the model cannot fake; build feedback loops where your mistakes are visible; and reserve the model for the tasks inside its frontier while you guard the boundary yourself. Expertise is not obsolete. It is redistributed toward the places the model cannot reach.

That leaves the question the studies have not settled: if AI keeps lifting the floor while the expert’s edge persists mainly at the frontier, will deep expertise become a smaller share of total value — or the only part that still commands a premium?

References

1. Goedecke, S. (2026). LLMs reward expertise. seangoedecke.com. https://www.seangoedecke.com/llms-reward-expertise/
2. Dell’Acqua, F., Lakhani, K. R., McFowland, E., et al. (2023). Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality. Harvard Business School / BCG. Coverage
3. Noy, S., & Zhang, W. (2023). Experimental evidence on the productivity effects of generative artificial intelligence. Science, 381(6654), 187–192. https://www.science.org/doi/10.1126/science.adh2586
4. Brynjolfsson, E., Li, D., & Raymond, L. (2023). Generative AI at Work. NBER Working Paper 31161. https://www.nber.org/papers/w31161
5. Law & Economics Center (2024). AI, Productivity, and Labor Markets: A Review of the Empirical Evidence. George Mason University. https://laweconcenter.org/resources/ai-productivity-and-labor-markets-a-review-of-the-empirical-evidence/

The post ‘LLMs Reward Expertise’: What the Data Actually Shows appeared first on The AI Prism.

Walk into almost any knowledge-work office in 2026 and you will find assistants embedded in the email client, the code editor, the CRM, the support queue and the slide deck. Ask the same organisation what any of it did to output per hour, and the room goes quiet.

That silence is the paradox. Adoption is close to universal at the individual level, spending is enormous, and yet the macro statistics that are supposed to register a productivity boom look stubbornly ordinary. Something has to give — either the tools do less than the demos suggest, or our instruments are pointed in the wrong direction.

The honest answer, laid out well in Bjorn Roche’s essay on the AI productivity gap and in the long Hacker News discussion it kicked off, is that both are partly true — and that neither is a reason to panic. This is what technology adoption has always looked like from the inside.

The Paradox, Stated Plainly

The shape of the problem is old enough to have a name. In 1987 Robert Solow observed that you could see the computer age everywhere but in the productivity statistics — a line the NBER literature on the productivity paradox has been unpacking ever since.

The 2026 version is tighter and faster. US Census Bureau survey work found firm-level AI use climbing from low single digits in 2023 to roughly 9% of firms by late 2024, and the Business Trends and Outlook Survey has kept tracking the curve upward since.

Meanwhile US labour productivity growth has run in the low single digits — respectable, not transformational, and well inside the range you would expect from a normal cyclical recovery, per Bureau of Labor Statistics series.

So the question is not whether there is a gap. It is which of several unglamorous mechanisms is producing it.

Measurement Lag: We Count the Wrong Things

Productivity is output divided by hours. Both halves of that fraction are hard to observe in knowledge work, and AI makes them harder.

If a support agent resolves the same number of tickets but each reply is clearer, output measurement records nothing. If a developer ships the same number of features with fewer defects, the defects that never happened do not appear in any denominator. National accounts are built to count widgets and billable hours, not avoided rework.

There’s a subtler problem: much of what AI produces is free. Generated images, drafts and summaries that would previously have been purchased or skipped entirely show up as consumer surplus, and GDP-based statistics famously undercount consumer surplus, as the Brookings work on measurement and intangibles has argued.

Roche’s essay makes a version of this point from inside a company rather than inside an econometrics paper: most firms deploying assistants never established a baseline, so they have no way to know what changed.

Diffusion Takes Years, Not Quarters

The second explanation is the one economists find most boring and most convincing. General-purpose technologies pay out slowly because the technology is the cheap part and the reorganisation around it is the expensive part.

Paul David’s canonical study of electrification showed factories took roughly 40 years to reap the full productivity benefit of electric motors — not because motors were bad, but because the payoff required abandoning the central-shaft factory layout and rebuilding the floor plan entirely.

Erik Brynjolfsson and colleagues formalised this as the “J-curve” of general purpose technologies: measured productivity falls first, because firms are pouring resources into intangible complementary investment — training, process redesign, data plumbing — that the statistics treat as cost rather than capital formation.

If that model holds, we are currently in the dip. The trough is not evidence of failure; it is the price of the reorganisation.

Task Automation Is Not Workflow Automation

Here is the mechanism that most directly explains why individual enthusiasm doesn’t aggregate into organisational output.

Assistants are extremely good at bounded tasks: draft this, summarise that, refactor this function, name these variables. Almost every credible study measures exactly that kind of task. The Brynjolfsson, Li and Raymond study of a customer support deployment found roughly a 14% average increase in issues resolved per hour, concentrated among less experienced workers.

But a workflow is a chain of tasks with handoffs, approvals and queues between them. Speeding up one link in a chain moves the bottleneck; it does not necessarily move the throughput. Anyone who has drafted a document in ninety seconds and then waited nine days for legal review knows this in their bones.

Amdahl’s law is a useful mental model here. If AI touches 30% of the work and makes that portion twice as fast, the end-to-end gain is about 18% at best — and that’s before the coordination costs of the new step.

The gains show up in the statistics only when the queue between the links is redesigned, and queue redesign is a management problem, not a model problem.

The Quality Offset: Busier, Not Better

The fourth mechanism is the least comfortable one. Some of the output AI enables is work that nobody needed.

When drafting becomes nearly free, the marginal cost of producing a document collapses — so more documents get produced. More documents means more reading, more review, more meetings about the documents. The volume of artefacts rises while the volume of decisions stays flat.

Open-source maintainers have been unusually blunt about this. Several major projects reported a surge in low-quality, AI-assisted submissions that consumed more reviewer time than they saved, with the curl project’s experience with AI-generated security reports becoming the canonical example of generated volume imposing a review tax.

This is a genuine negative externality: the producer captures the speed gain and the reviewer absorbs the cost. In an aggregate measure the two cancel, and the statistics record nothing.

Who’s Actually Faster — and Who Only Feels Faster

The most instructive recent finding is one that cuts against the tools’ own users.

A 2025 randomised controlled trial by METR on experienced open-source developers working in their own large repositories found that participants were roughly 19% slower when using AI assistance — while believing they had been about 20% faster. The METR study writeup is careful about its limits, but that perception gap is the single most important number in this debate.

The pattern across studies is consistent rather than contradictory. Novices and people working outside their expertise gain the most. Experts in familiar, complex codebases gain the least and sometimes lose, because verifying a plausible-looking suggestion costs more than writing the line yourself.

Self-reported productivity is therefore close to useless as evidence. Everyone feels faster, because the friction of the blank page disappears and the friction of review is diffuse and unmemorable.

The policy implication is uncomfortable for vendors. The people most likely to renew a licence are the ones who gained the least, because the speed they felt was real to them even when the throughput was not. The people most likely to churn are the experts who quietly lost time and noticed. Adoption metrics and value metrics point in different directions, and most dashboards only show the first.

If your organisation’s AI ROI case rests on a survey asking employees how much time they saved, you do not have evidence. You have a mood.

What Would Actually Close the Gap

Strip out the extremes and a moderate picture emerges from the credible studies. The strongest gains cluster in tasks that are text-heavy, low-stakes and easily verified. The weakest cluster in tasks that are context-heavy, high-stakes and expensive to verify — which describes most of what senior people are paid for.

Adoption itself is lopsided. Anthropic’s usage analysis found delegation-style use growing relative to collaborative use, and the Anthropic Economic Index shows adoption concentrating heavily in software and technical writing rather than spreading evenly across the economy.

The widely cited finding that a large majority of enterprise generative AI pilots produced no measurable P&L impact should therefore be read as a claim about pilots, not about the technology. Pilots without process change rarely move a P&L, whatever the technology. For a broader read on how this pattern plays out in employment rather than output, see The AI Prism’s look at what’s actually happening to jobs.

The pattern repeats across every general-purpose technology before this one. Electricity, the computer, the internet, each spent years as a disappointment in the aggregate statistics before the reorganisation caught up. The mistake was never to expect gains; it was to expect them on the deployment timeline rather than the absorption timeline. AI is behaving exactly as its predecessors did, which is the most reassuring and most ignored fact in the whole debate.

None of this means the tools are useless. It means the gains are conditional, and the conditions are organisational rather than algorithmic. That is good news, because organisations can be changed faster than physics.

None of the fixes are technical, which is precisely why they are slow.

Measure before you deploy. A two-week baseline of cycle time, defect rate and rework volume on one workflow is worth more than a year of adoption dashboards. Seat counts and token spend measure input, not output.

Pick one end-to-end workflow rather than sprinkling assistants across ten. Gains are only visible when the whole chain — including the approval steps and the handoffs — is redesigned around the new capability.

Then account for the review tax explicitly. If generated volume increases, reviewer capacity has to increase or throughput standards have to tighten; otherwise the saving quietly migrates from producer to reviewer and disappears.

Finally, be patient with the macro data. Electrification took decades and computing took roughly twenty years to register clearly. Expecting a general-purpose technology deployed at scale from 2023 to show up in national statistics by 2026 was never a reasonable timeline.

The Management Question Nobody Asks

Underneath all the measurement debate sits a simpler, more awkward possibility: the gap persists because most organisations never treated AI adoption as a change-management project in the first place.

Rolling out assistants is treated as a software rollout — buy licences, send a training email, watch the dashboard. Reorganising work around a new capability is a different activity entirely, and it is the one the productivity literature says actually moves the number. The tool arrives; the workflow does not.

This reframes the paradox in a useful way. The lag is not only a statistical artefact or a diffusion delay. Some of it is simply the ordinary cost of deploying a general-purpose technology without the complementary investment the theory predicts. The good news is that this is the one mechanism on the list a manager can actually do something about on a Tuesday.

The Bottom Line

The productivity gap is not proof that AI doesn’t work, and it isn’t proof that the statistics are broken. It is what the middle of an adoption curve looks like when the tooling has outrun the organisational plumbing around it — task-level speed with workflow-level friction, real gains for novices, unproven gains for experts, and a measurement apparatus that was never designed to see any of it.

The interesting question isn’t when the numbers will finally arrive. It’s whether the organisations currently counting seats and licences will have built the baselines they need to recognise the gains when they do — or whether they’ll still be measuring adoption and calling it impact?

References

1. Bjorn Roche — The AI Productivity Gap
2. Hacker News discussion — The AI Productivity Gap
3. Brynjolfsson, Rock & Syverson — Artificial Intelligence and the Modern Productivity Paradox (NBER w24001)
4. Brynjolfsson, Li & Raymond — Generative AI at Work (NBER w31161)
5. METR — Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
6. US Census Bureau — Business Trends and Outlook Survey
7. US Bureau of Labor Statistics — Productivity Data
8. Brookings — Productivity in the Age of Artificial Intelligence
9. Anthropic Economic Index
10. Brynjolfsson & Hitt — Computing Productivity (NBER w7833)
11. curl project — on AI-generated security reports
12. The AI Prism — What Is Actually Happening to Jobs

The post The AI Productivity Paradox: Why the Gains Haven’t Shown Up Yet appeared first on The AI Prism.