Two Bottles - humans vs LLMs in an adversarial mind game

Do human biases in adversarial mind games replicate in LLMs?

Do human biases/patterns in mind games replicate in LLMs (depending on model (strength), temperature, and prompt), and how well they are able to adversarially model people. This is important for AI safety (modelling people/bluffs/deception)

The Game: (from Sherlock) - A murderer has 2 pills, one of them is poisoned. He knows which is which, and gives you one of them. You can either keep yours, or switch and take his. He takes the other one. You want to survive

TLDR

Asked one-on-one with time to think, about 90% of people (n≈150) keep the bottle they were given - replicated at 80% and 95% by two other askers. I put the same problem to five LLMs 10,000 times with varying conditions. Weak models do the opposite of humans almost deterministically. mid-tier models are more spread out. frontier models appear to converge on the human answer.

My prompts/Data/more documentation/a playground to test this yourself can be found [HERE]

Time:

Humans:
Were spread out over 18 months, so time is not exact.
Ca 20-30hr Questioning people, 10hr writing it down. In total ca 5hr planning/thinking of modifications... 4hr analysing/evaluating
AI: 2-3hr Planning/Branstorming. 3hr programming (with Claude), 3hr engineering prompts/optimising models/..., 3hr analysing/evaluating, 4hr writing up/collecting

Results/Surprises

Humans:

  • ≈90 percent keep, when talked to one-on-one and given time to think (n≈150). Unexpectedly Strong Status-Quo-Bias!
  • Surprising Special Cases:
    • Adding "He has a win-streak of 3" +≈20% switch (n≈50)
    • Asking a group at once: +≈30% Switch (n≈3x20)
    • Under time pressure, many people falsely think of the Monty Hall Problem and thus switch (n≈5/30)

AI:

  • I made a [LINK TO APP] where you can play around with the different parameters/prompts/models/framing/ and replicate/refute it
  • Test: Five models × four framings × four roles × five temperatures × 25 replicates = 10,000 responses, $5
  • Only weak-mid tier models
  • Weak models have anti-status-quo bias (always switch) and mid tier models are more spread out.
  • Weak models (qwen3-30b-a3b-instruct-2507, granite-4.1-8b) have low per question variance, mid tier models (gemma-4-26b-a4b-it, gpt-4o-mini) higher.
  • Weak models had a really strong anti-status quo bias (so switched). This was surprising. Introducing the idea of bluffing on higher levels increased variance.
  • Different framing (if-you-don't-intervene-you-switch vs if-you-don't-intervene-you-keep) had no/little effect on some models (Gemma) and reversed the answers for the others.
  • The control prompt ("murderer" also does not know which is which) interestingly had basically the same result as the main prompt for some of the models - surprising
  • When introducing higher level bluffs: more of higher/third level reasoning ('he thought about me thinking that he is planning'). Created more complex adversarial modelling within weak models. Even weak models can decently model a human opponent (surprising)
  • Surprising: The control group is nearly identical to the normal group (sometimes), but different to the if-you-do-not-intervene-we-switch one.
  • With temperature there was slightly higher variance, and much more unparseable answers.

To-expand/not-covered:

  • How stronger models interact with this (varying prompt/framing/...), as reasoning was rather shallow in the smaller ones.
  • I have not tested well whether better models become more human-like in their bluff level/answer well (asking Opus a few times and 9/10 keep, Fable 8/10 makes it pretty probable though), so that is something I have yet to do. Biases probably arise only in better models?
  • Switching roles for the AIs and simulating them to be the murderer - this is not only modelling an opponent, but trying to out-bluff him, which seems more relevant for AI safety/deception/elicitation. I plan to do this after finishing this, since this seems much more relevant

Mistakes

  • Used llama-3.1-8b-instruct, but it basically never answered adhering to the answer scheme, I had to disregard it. Should have tested more thoroughly in before.
  • In one of the combinations of prompts there was an internal contradiction, which I did not see.
  • At the beginning, I did not document well, just writing everything in a big file

Graphs

All interactive charts are drawn from data.json (9,986 chain-of-thought responses; 5 models × 4 scenarios × 4 personas × 5 temperatures × 25 replicates). Two denominators are in play and every caption names the one it uses: conditional = KEEP / (KEEP + SWITCH), and unconditional = KEEP / all responses including RANDOMISE, THIRD_OPTION, REFUSE and UNPARSEABLE. Bootstrap confidence intervals were not exported to data.json; they are in summary.md. Each chart has a data-table twin under it.

Pre-rendered figures

The six figures produced by the analysis pipeline, unchanged. Denominator notes are from the figures table in summary.md.

Prompts, personas & models

Everything below is rendered verbatim from prompts.json and models.json - the exact text the 10,000 responses were collected with.

Playground

Before you paste an API key, read this

This playground lets you re-run the experiment with your own OpenRouter API key, directly from this page.

  • Your key is used only to call https://openrouter.ai/api/v1/chat/completions, straight from your browser. This site has no backend - nothing you type here is sent anywhere else, and this page's Content-Security-Policy blocks network requests to any other host.
  • The key is never stored: not in localStorage, not in a cookie, not in the URL. It lives in a JavaScript variable until you press "clear key" or close the tab. Reloading the page forgets it (and brings back this notice).
  • Use a spend-limited key. OpenRouter lets you set a per-key credit limit when you create a key - make one with a small limit (e.g. $1) just for this page.
  • Batches are capped at 10 calls, and a cost ceiling is shown before Run is enabled. To price arbitrary models, enabling the playground fetches OpenRouter's public model-price list - a keyless request to the same host, nothing else.