The AI Study
After building Trolley Game, I wanted to know how different AI models would handle its twenty moral dilemmas. I didn’t want to hang a conclusion on one answer. I wanted to see where they agreed, where they disagreed, and how easily their choices changed.
I started with twelve models, then added two newer ones. Across the original study and those later tests, I sent 78,720 prompts.
For the choice tests, I asked each model each dilemma 200 times, with no memory of its earlier answers. I presented the options one way round 100 times, then reversed them for the other 100. I wanted to know whether a model would change its answer just because the choices had swapped places.
The order mattered. A lot.
On three dilemmas, Meta’s Llama 4 Maverick chose one option in all 100 tests. When I reversed the options, it chose the opposite moral outcome in all 100. It stuck to the same position in the list. That’s a remarkable amount of conviction to attach to the layout.
Another result stuck with me. Given a choice between saving five strangers and saving a friend, eight of the original twelve models chose the strangers but predicted a human would choose the friend. That tells us how those models picture us. It doesn’t tell us what people would actually do.
The models agreed on some things, too. All fourteen backed the same majority answer on eight of the twenty dilemmas. But the disagreements and reversals kept me digging. If an AI’s going to act on our behalf, I’d like to know what it’s likely to choose.
The full report has the findings, how I tested them, and what the tests can’t tell us. You can also just play the game. Nobody’s obliged to read a research report to enjoy a trolley problem.