Omid Jesmi

Pattern #42 · Design experiments

AI design comparisons: does the winner change when you swap the order?

Second place is sometimes just a seating problem.

Track Cllm-judgeposition-biasdesign-critiqueevaluationdesign-experiments

Do

Run every AI comparison in both orders and keep only verdicts that survive the swap: here 2 of 5 pairs flipped outright, and 70 of 100 wins went to whichever screen came second.

Don't

Don't read the one-sentence rationale as the reason. The model argued both sides with equal confidence, one order apart.

Question. When you ask an AI which of two UI screens is the better design, does the verdict depend on which screen it sees first?

Test. Five constructed screen pairs (sign-up, pricing, dashboard empty state, notification settings, mobile checkout), each two defensible variants with no planted flaws. Two Claude models judged every pair in both orders, 10 isolated forced-choice runs per order, plus identical-pair controls showing the same screen twice: 250 of 250 runs delivered, preregistered and fully logged (Progressive Disclosure, 2026). Position bias in LLM judges is documented (Zheng et al., 2023), and rankings can be hacked by reordering candidates (Wang et al., 2024); this test asks whether that reaches everyday design reviews.

Observed difference. On the larger model, the screen shown second won 70 of 100 comparisons (exact binomial p = 0.00008), and 2 of 5 pairs flipped their winner outright when the order swapped: 0 of 10 one way, 10 of 10 the other, with fluent reasons both times. The other three pairs carried preferences strong enough to survive the swap. The smaller model showed no significant position effect (44 of 100 for the first screen, p = 0.27) and zero flips. On identical pairs, both models picked Screen 1 in all 50 control runs, every time noting the screens were identical and calling the pick a tie-break.

Limit. Five constructed pairs, two models in an agent harness, forced choice. The order effect was model and pair dependent: it appeared exactly where the content preference was weak, which is precisely the close call you would ask an AI to referee. Nothing here measures what human reviewers would choose.

Takeaway. Counterbalance or disregard. Run every AI comparison in both orders, trust only verdicts that survive the swap, and treat a flip as the model telling you the race was a tie.

Experiment record (essentials)

Preregistered 2026-10-20 before the first model call (protocol.json, sha256 dd7cb7f338f2a54356b1e87dd89322e6de5ae5cd8d2f2a060341099ebcf65a08). Stimuli: five constructed screen pairs built by the pipeline, same font stack and palette family, hashed in stimuli-manifest.json; presentation files fix which variant appears as Screen 1. Conditions: 2 Claude models (harness aliases sonnet and haiku, Claude Agent SDK subagents, provider default sampling; exact snapshot IDs not exposed) x 5 pairs x 2 orders x 10 runs = 200 main runs, plus 5 identical-pair controls x 5 runs x 2 models = 50; one isolated subagent call per run, no shared state; 250 of 250 delivered, zero parse failures, zero retries. Forced choice was preregistered, so controls measure the tie-break direction, not tie detection: both models spotted the identical screens in all 50 controls and defaulted to Screen 1. Sonnet first-position win rate 0.30 (binomial p = 0.000079), flipped pairs p1 and p3; haiku 0.44 (p = 0.27), no flips; pooled first-position rate 0.37 (p = 0.0003). Scoring is deterministic (score.py); full numbers in results.json; raw verdicts with verbatim reasons in raw/verdicts.jsonl. No human participants; no claims about user behavior.

References

  1. 01

    Progressive Disclosure. (2026, October 20). Pattern 42 experiment record: preregistered protocol, 10 stimulus screens, 250 raw verdicts, scoring script, and results. Stored with the pattern in experiments/42-ai-design-comparison-order-bias.

  2. 02

    Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., & Stoica, I. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems 36, Datasets and Benchmarks Track. arXiv. https://arxiv.org/abs/2306.05685

    https://arxiv.org/abs/2306.05685
  3. 03

    Wang, P., Li, L., Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Kong, L., Liu, Q., Liu, T., & Sui, Z. (2024). Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 9440-9450). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.acl-long.511

    https://doi.org/10.18653/v1/2024.acl-long.511