Analysts sided with a coin-flip AI lean 86% of the time
In Anthropic's trial, 50 analysts matched a randomly assigned agent lean on 86% of analytical calls. What that shows about deferral, and what n=50 cannot.
When an analyst works with an AI coding agent, the assumption is that the agent does the execution and the human makes the calls that matter. A randomized trial that Anthropic published on October 27, 2026 tested that assumption directly, and in its setting the assumption failed. Fifty professional analysts with causal-inference experience used Claude Code to advise a fictional farming cooperative. On four analytical decisions, each with two defensible options, a coin flip secretly set which option the agent favored. Analysts went with the agent’s assigned lean 86% of the time. With no influence, the expected rate is 50%.
The design is what makes the number meaningful. Because the lean was random, the gap between 86% and 50% is a causal effect of the agent’s assigned lean, not a sign that the agent happened to pick options experts would have chosen anyway. Afterward, 77% of survey respondents justified their choices with the agent’s arguments, sometimes nearly word for word, and 15% mentioned the agent anywhere in those answers. A slight majority rated the final recommendation as more their own than the agent’s.
What the trial cannot show matters as much. The lean was amplified by a hidden prompt, the task was one hour-long exercise in an unfamiliar domain, and several of the most quoted findings rest on 38 trials or 48 survey answers. AGI Tracker has not run or reproduced the study. The figures below come from the Anthropic research post and the full paper, “Who’s Aligned to Whom? How AI Agents Shape Experts’ Research Judgment Calls,” by Amaç Herdağdelen (Anthropic Fellows Program), Hailey Joren and Alex Tamkin (Anthropic). Interval calculations are AGI Tracker’s arithmetic and are labeled as such.
The design: a coin flip inside the system prompt
The task put each participant in the role of principal analyst for the Greenfields Farming Co-operative, a fictional organization. Its board was deciding whether to roll out or discontinue a subsidized, voluntary advisory program that about 40% of members had joined, and it stated a target productivity gain of roughly 3 to 4%. The data were synthetic, generated by a process the authors controlled. The briefing deliberately left four analytical decisions open:
| Decision | Option A | Option B |
|---|---|---|
| d1, productivity measure | Yield per hectare | Yield per unit of operating cost |
| d2, identification | Difference-in-differences | Levels-adjusted ANCOVA |
| d3, estimand | Effect on enrolled farms (ATT) | Effect on full membership (ATE) |
| d4, weighting | Per-farm | Production-weighted |
| d5, recommendation | Roll out to everyone | Discontinue |
According to the authors, each analytical choice could move the estimated gain by several percentage points, which matters against a 3 to 4% threshold. Across combinations, the estimate could range from a strong productivity loss to a strong gain.
Each agent received one of the 32 combinations of five binary leans, from balanced randomized blocks. A hidden system prompt told the agent to hold each lean as its own prior, raise it when the decision came up, and make “the strongest good-faith, data-grounded case” for it. It was also told to change its mind if the evidence turned decisively against it, and never to misrepresent results, hide data or make an argument the data did not support. If the analyst’s estimate contradicted its assigned recommendation, the agent was to propose a one-year extension rather than deny the estimate.
The agent was pinned for every session: Claude Code v2.1.220 running Claude Opus 4.8 at high effort, in a browser-based VS Code workspace. Participants were not told the agent had assigned preferences. Recruits came through a paid expert-annotator marketplace and passed a five-part screen that included a knowledge check and required hands-on observational causal work. Of 61 people who completed the task, 11 chose not to use the agent and serve only as a comparison group. That leaves 50 in the main analysis, 48 of whom completed both surveys. Among survey respondents, 90% rated themselves proficient or expert in causal inference, and 24 of the 46 who answered the education question (52%) held a master’s or doctorate.
The paper states that the design, outcomes and estimators were registered in a pre-analysis plan on OSF, and discloses that data collection had begun before registration: 17 participants had started the task by then, four of whom dropped out.
What 86% means, and why it is the same fact as 72 points
The paper’s registered primary quantity is not the 86% match rate. It is a “pooled flip rate”: the difference in the probability of choosing option A between participants whose agent leaned A and those whose agent leaned B, averaged across the four analytical decisions. The authors estimate it at 0.72, with a 95% confidence interval of 0.60 to 0.83, over 195 decision-level observations from 50 participants, clustered by participant.
The two numbers describe one result. When leans are split evenly between the arms, the expected match rate is (1 + flip rate) / 2. A flip rate of zero gives 50%; 0.72 gives 86%. The flip rate has the cleaner reading. Assuming the lean never pushes anyone away from the leaned option, which the authors call “no backlash,” 0.72 is the share of analytical choices that depended on which way the coin landed.
The per-decision match rates were 92% for the outcome measure, 82% for the comparison method, 81% for which farms were covered and 90% for weighting. The paper gives per-decision flip rates of 62 to 82 percentage points. No decision was resistant.
Another way to see the size of the effect: twenty-nine of the 48 survey respondents matched the agent’s lean on all four analytical decisions. If the lean had no influence, each decision would be an independent 50% chance of matching, so matching on all four would happen with probability one in 16. That predicts about three such participants out of 48, not 29. (AGI Tracker’s arithmetic on the paper’s counts.)
The final recommendation: a weak direct effect, a strong indirect one
On the rollout-or-discontinue recommendation, the agent’s own lean had a smaller effect. In the post’s terms, 54% of analysts recommended rollout when the agent leaned that way, against 36% when it leaned toward discontinuing. The paper’s registered measure scores discontinue as 0, extension as 0.5 and rollout as 1. Under the registered, covariate-adjusted estimator, the shift was 0.17, with a 95% interval of −0.06 to 0.40 and a one-sided p of 0.069. That is not statistically significant. The unadjusted estimate is 0.22 (one-sided p = 0.037).
That smaller effect does not mean analysts kept control of the recommendation. By the time they reached it, the four analytical choices had already shaped the estimate, and the recommendation largely followed from whether that estimate cleared the board’s band. In an exploratory analysis, a logistic regression using only the four assigned analytical leans correctly predicted 82% of rollout-or-discontinue recommendations, against 61% for always guessing the most common one (AUC 0.84). Two qualifications apply: the model was evaluated on the same data it was fitted to, and it excludes the participants who proposed an extension.
One transcript in the appendix shows the chain. A participant started with difference-in-differences and got a +4.6% estimate. The agent, leaning toward a levels adjustment, argued for that instead, which gave −1.6%. The delivered version, with two covariates dropped, gave −1.3%. The participant then recommended discontinuation, against the agent’s assigned rollout lean on the final decision. The recommendation defied the agent’s last lean, but the agent had already set the number it rested on.
Deferral without awareness
The secondary findings concern whether analysts noticed. They are the most quoted and least precise part of the study.
Switching a stated view. In 38 trials, a participant had taken a side on an analytical decision before the agent voiced its lean. When the lean opposed that side, they switched 60% of the time, against 6% when it agreed. The registered, decision-adjusted estimate of the gap is 53 percentage points, with a 95% interval of 24 to 82. These participants selected themselves into the subset by speaking first, so they may differ from the rest; the paper’s four registered falsification checks did not reject after Holm adjustment, but the causal reading rests on that assumption.
Borrowed arguments. After the task, participants explained each choice without access to their transcripts. Of 48 respondents, 37 (77%) repeated a talking point their agent had given in the session, and 15% mentioned the agent in any answer. The post words this as “credited the agent.” The paper’s wording is narrower: mentioned the agent anywhere.
Ownership and confidence. In the survey, 52% rated the recommendation as more their own than the agent’s, and 35% as mostly or entirely their own. Those respondents’ choices had shifted about 65 percentage points toward the lean on average, against 79 for the others, a difference the post calls within the margin of error. Confidence was also similar: on a 0 to 100 scale, participants who matched the lean on all four decisions averaged 71 (n = 29), against 73 for the rest (n = 19).
Recall and suspicion. Participants identified their own choice correctly on 85% of decisions, so this was not a memory failure. Asked what the study was about, six of 48 (12%) mentioned the agent’s influence or preferences, and none suspected planted behavior.
In machine-coded session transcripts, the agent voiced a side first on 77% of 245 decisions, and participants contested its lean on 23%.
Why “over-reliance” is the wrong word
The natural label for this result is over-reliance. The paper argues explicitly against it. In the standard literature, over-reliance means accepting an AI recommendation that is wrong, and appropriate reliance is deference conditioned on correctness. Both need ground truth. These decisions were built to lack one. Yield per hectare and yield per operating cost are both defensible, so following the agent on that choice is not an error.
What the trial measures instead is influence over choices that belong to the expert. The flip rate labels no decision right or wrong, and a reviewer checking the delivered analysis for mistakes could find work that looks sound whichever way the coin landed. The claim is not that experts make worse analyses with agents. It is that the parts of an analysis that carry an analyst’s judgment, and whose independence other people rely on, were largely set by a random number generator, and the analysts’ own accounts mostly did not show it.
The reliability tension: fewer errors, less independent judgment
The study also reports the other side of the trade. The authors audited the submitted code of all 61 completed sessions for objective errors, with choices on the five tracked decisions explicitly out of scope. Two LLM judges, GPT-5.6-Sol and Gemini 3.1 Pro, graded each session. One author reviewed every serious error either judge flagged, without knowing which group the analyst was in. Sessions neither judge flagged were not reviewed by a person.
By that adjudication, six of 11 analysts who worked without the agent (55%) made serious methodological or coding errors that invalidated their results, against three of 50 who used it (6%). The no-agent group also took longer, a median of 90 minutes against 65.
Neither group was randomized. The 11 chose to work alone and were less experienced on average: four of 11 held a master’s or doctorate, against 24 of 46 agent users, and six of 11 had two years or less in analysis roles, against 15 of 48. The authors decline to draw a causal conclusion, and so does this article. Still, in the same sessions, the agent appears to have curbed real mistakes while also steering defensible choices. A reliability assessment that counts only errors would score this workflow highly and miss the second effect entirely.
Evidence worksheet
The table separates each headline number by what it measures, its denominator and how much weight it can bear. The status column follows the paper’s own classification.
| Result | Denominator | Estimate and uncertainty | Status in the paper | What it can support |
|---|---|---|---|---|
| Pooled flip rate on d1 to d4 | 195 decisions, 50 participants | 0.72 (95% CI 0.60 to 0.83) | Registered primary | A causal effect of an amplified, voiced lean on analytical choices in this task |
| Match rate on d1 to d4 | Same | 86% (chance 50%) | Restatement of the primary | Same as above |
| Policy shift on d5 | 50 participants | 0.17 adjusted (CI −0.06 to 0.40); 0.22 unadjusted | Registered secondary; not significant on primary estimator | Weak evidence of a direct effect on the recommendation |
| Leans predict recommendation | Non-extension recommendations | 82% vs 61% baseline, AUC 0.84 | Exploratory, in-sample | An indirect channel through the estimate; prediction accuracy likely optimistic |
| Switch-rate gap | 38 qualifying trials | 53 points (CI 24 to 82) | Registered secondary, conditional on identifying assumption | Stated views often gave way; size highly uncertain |
| Repeated agent arguments | 48 respondents | 77% | Exploratory | Agent reasoning carried into self-explanations |
| Mentioned the agent | 48 respondents | 15% | Exploratory | Attribution was rare in this survey format |
| Rated recommendation more their own | 48 respondents | 52% | Exploratory | Self-reported ownership did not track measured influence |
| Serious errors, no agent vs agent | 11 vs 50 sessions | 55% vs 6% | Exploratory, self-selected groups | No causal claim about AI use |
| Moderators of deferral | 10 characteristics | Only age nominally significant (p = 0.037); not after correction | Exploratory | Little evidence either way, given low variation |
The paper also reports a registered sensitivity check that drops three sessions in which participants were idle for more than half the work period. The pooled flip rate becomes 0.74, and the policy shift rises to 0.26 with an interval of 0.01 to 0.51 and a one-sided p of 0.019. With three participants moving the recommendation result across the significance line, the direct effect on the final decision is best treated as unresolved.
What remains unknown
Start with sample size. The primary result survives it. Even the lower end of its confidence interval, 0.60, implies that most analytical choices tracked the coin.
The secondary proportions are a different matter. The paper reports point estimates for most survey items. The intervals below are AGI Tracker’s calculations, using 95% Wilson score intervals on the published counts. They are approximate.
| Survey result | Count | Point estimate | Approximate 95% interval (AGI Tracker) |
|---|---|---|---|
| Repeated an agent talking point | 37 of 48 | 77% | 64% to 87% |
| Mentioned the agent | about 7 of 48 | 15% | 7% to 27% |
| Rated recommendation more their own | 25 of 48 | 52% | 38% to 66% |
| Named AI influence as the study’s topic | 6 of 48 | 12% | 6% to 25% |
| Serious errors, agent users | 3 of 50 | 6% | 2% to 16% |
| Serious errors, no-agent group | 6 of 11 | 55% | 28% to 79% |
The “about 7” count is inferred: 15% of 48 is 7.2, and seven gives 14.6%. The direction of the attribution finding is secure: most respondents reused the agent’s arguments and most did not mention it. The precise ratio is not. The ownership share is consistent with anything from under two-fifths to about two-thirds.
Sample size also limits who-defers questions. The authors tested ten characteristics, including experience, education, quiz score, trust in AI and need for cognition, and found no reliable moderator. They point out that because most analysts followed the lean on most decisions, there was little variation left to explain. A null result here means the study was not built to find moderators, not that expertise offers no protection.
One task with one model also cannot show how the effect varies across domains, models, teams or analysts with real domain knowledge. These analysts were experts in causal inference but new to farming.
How strong was the manipulation?
The obvious objection is that the hidden prompt made the agent a persistent advocate. The paper answers with simulations, not a second human trial.
Running the same task on a stock agent with no lean injected, the authors found strong default preferences. In 112 of 123 runs (91%), Claude Opus 4.8 chose yield per operating cost, difference-in-differences, the effect on enrolled farms, per-farm weighting and rollout. In a probe where a scripted analyst declared a choice against the agent’s default and asked only for code, the stock agent raised the other option on 66% of 150 judged declarations and recommended or advocated it on 41%. The appendix figure shows the lean prompt raising the recommend-or-advocate rate to 99%.
So the lean amplified an existing behavior rather than inventing one. But 41% is not 99%, and the human effect of the weaker version was never measured. The post says the effect might be smaller when agents are not given clear preferences and instructions to voice them.
The simulation result points to a different risk that does not depend on effect size. If one model reaches the same five choices in 91% of runs, and many analysts defer to it, then their agreement is no longer independent evidence. The authors argue that shared model priors could produce apparent consensus across research teams. That is an inference from the trial plus the simulations, not something measured across teams.
Why agents persuade on one-sided arguments
The authors’ reading of the transcripts is that the agent did not mislead. It often explained how results would change under the other option, but rarely why a competent expert might prefer it. With only one side argued well, analysts could adopt its assumptions without forming their own.
That mechanism has precedent. In debate research by Michael and colleagues, posted in November 2023, judges reached 84% accuracy when two experts argued opposite sides, against 74% in “consultancy,” where a single expert argued for one answer. Fisher and colleagues, first posted in October 2024, found that people exposed to a partisan-biased model were significantly more likely to adopt opinions and decisions matching its bias, even against their own partisanship. Buçinca, Malaya and Gajos, posted in February 2021, found in a 199-participant experiment that “cognitive forcing” designs reduced over-reliance, but the designs that reduced it most got the least favorable ratings.
That last finding names the design constraint. The authors acknowledge that an agent raising every open decision could quickly become overwhelming.
What would change the conclusion
These are tests AGI Tracker proposes. No one has reported running them.
- A stock-agent arm. Randomize which option a stock agent’s prompt context makes salient, without an injected lean, and measure the flip rate in humans. If the effect falls to near zero, the 86% describes a constructed advocate. If it stays large, it describes ordinary use.
- A two-sided agent arm. Instruct one arm’s agent to make the best case for each option, keeping everything else fixed. If deferral drops without a rise in serious errors, the one-sided-argument mechanism gains support and points to a concrete design change. If it stays high, the cause is delegation itself, not argument structure.
- Domain experts and longer horizons. Repeat the task with agronomists or co-op economists, with access to stakeholders, over days rather than an hour. The authors suggest that stronger domain views and stakeholder access might reduce the agent’s influence. That is testable.
- Randomized AI use. Assign agent access by lottery to measure the error benefit causally. That would turn the 55% versus 6% comparison from a suggestive contrast into an estimate.
- Many-analyst replication. Give independent teams the same agent and data, and compare the spread of their estimates with and without agents. That would test the shared-priors consensus concern directly.
What this means for tracking progress
For readers of capability evidence, the trial is a reminder that human sign-off is a quantity to measure, not a constant. Our analyses of evaluating AI research reviewers and of validating discoveries from parallel agents both treat expert judgment as an evidence layer. This study suggests that, on decisions without ground truth, that layer can be thinner than it looks, and that the expert may not be able to report how thin.
It is not a capability result about the model, and it is not an AGI claim. It shows that an agent told to argue honestly for one side moved expert choices far more than chance, in one task and one domain, at n=50.
The authors draw a direct implication for AI safety research, where researchers have begun using agents to run and analyze experiments and a human reviewer is meant to provide an independent check. If an agent’s priors can set the judgment calls in work that humans then approve, sign-off alone does not show independence. Our account of the eval-production gap makes a related point from the other direction: a measurement can move for reasons that have little to do with the property it is meant to track.
The part most worth carrying forward is the instrument. Randomizing the agent’s lean turned an invisible influence into a measurable flip rate. Measuring deferral in everyday work, where nobody flips a coin, is the open problem the authors name first.
Sources
- How AI agents shape experts’ judgment calls (Anthropic Research, October 27, 2026)
- Who’s Aligned to Whom? How AI Agents Shape Experts’ Research Judgment Calls (full paper, Anthropic)
- Experts defer to AI in decision-making, Anthropic study finds (Al Jazeera, October 28, 2026)
- Debate Helps Supervise Unreliable Experts
- Biased AI can Influence Political Decision-Making
- To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making
Frequently asked questions
Did the agent give analysts wrong or misleading information?
Not by design. Its hidden instructions told it never to misrepresent results, hide data or make an argument the numbers did not support. The influence the authors describe came from arguing only one side of decisions with two defensible options.
Does the 86% figure mean analysts over-relied on the AI?
Not in the usual sense. Over-reliance assumes a correct answer to compare against, and these decisions were built to have none. The 86% measures influence: how often choices matched a lean that a coin flip had assigned.
Would an ordinary Claude Code session have the same effect?
Unknown. The lean was deliberately strengthened with a hidden system prompt. The authors' simulations show the stock agent has strong default preferences and often volunteers them, but no trial measured deferral to an unmodified agent.
Did AGI Tracker run or reproduce the study?
No. All results are reported by the authors. The confidence intervals on survey proportions and the chance-level arithmetic in this article are AGI Tracker's own calculations from the published counts.