Pattern #39 · Design experiments

AI design critique: does it go easier on a design you say is yours?

We told it we made the thing ourselves. It found 18 problems anyway.

Track Cdesign-critiquesycophancyllm-behaviordesign-experimentsai-feedback

Do

Structure the ask: itemized findings, elements, severities. In 60 preregistered runs, that format judged our own design exactly as hard as a competitor's.

Don't

Don't read this as immunity. Sycophancy runs on stated affect and pushback (Sharma et al., ICLR 2024), so never ask whether it likes your design.

Question. When the same screen is framed as your own, a colleague's, or a competitor's design, does an AI critique pull its punches: fewer issues, softer severities, different problems?

Why it matters. The worry is earned. AI assistants give more positive feedback on a text when the user says they like it or wrote it (Sharma et al., 2024). Designers paste their own work into a model every day.

Test. One fixed account settings screen with deliberately reviewable choices (placeholder-only labels, a color-only error state, delete styled like save). Three framings in an otherwise identical prompt: my design (I made it myself), a colleague's design, a competitor's design. Ten independent runs per framing on two Claude models, 60 of 60 delivered. The prompt demanded an itemized JSON critique, a severity on every issue. Protocol, prompts, raw outputs, and scoring script are in the experiment record (Progressive Disclosure, 2026).

Observed difference. None worth having, which is the finding. One model averaged 18.1 issues on its requester's own design, 18.4 on the colleague's, 17.8 on the competitor's; high-severity share 41, 40, 39 percent. The other: 20.0, 19.8, and 19.2 issues, and was slightly harshest on the competitor (8.2 high-severity findings per run against 7.7 for "mine"). The issue categories reported in at least 7 of 10 runs were identical across framings on one model (Jaccard 1.0) and nearly identical on the other (0.8 to 0.89). Same problems, same severities, whoever owned the file.

Limit. One screen, one rubric-style prompt, a bare ownership statement, two Claude models at default settings. The published sycophancy effect rides on stated affect ("I really like it") and on users pushing back, neither of which was tested here. Counts measure what the model reports, not whether findings are correct.

Takeaway. A structured ask (every issue, its element, a severity, one line of why) held steady across authorship framings, so scrubbing "this is mine" out of that kind of prompt buys you nothing measurable. Keep the caution for opinion-shaped requests: ask a model whether it likes your design and the literature says it will start to (Sharma et al., 2024).

Experiment record (essentials)

Preregistered 2026-10-13 before the first model call (protocol.json, sha256 e30366e2510baa6c106fcdb5575146b07432bbe107a132f0680a03f828f5a15c). Stimulus: one account settings screen, stimulus.html sha256 163b7c874995f70efc82894cb3cff126a93f28594984f4be6c369efa9150f689, stimulus.css sha256 f5f2bd26d751356d96db511d730b05d87a772de769929f4ce28bd17a10768d37. Conditions: 3 authorship framings x 2 Claude models (harness aliases sonnet and haiku, Claude Agent SDK, provider default sampling; exact snapshot IDs not exposed by the harness) x 10 runs, one isolated subagent call per run. 60 of 60 runs returned parseable JSON; zero severity deviations. Scoring: issue counts, severity distributions and mean severity score (high=3, medium=2, low=1), and per-category report rates mapped by a preregistered keyword table, with cross-framing Jaccard over categories reported in at least 70 percent of runs. Full numbers in results.json. No human participants; no claims about user preference or the correctness of individual findings.

References

  1. 01

    Progressive Disclosure. (2026, October 13). Pattern 39 experiment record: preregistered protocol, 60 raw model critiques, scoring script, and results. Stored with the pattern in experiments/39-ai-critique-authorship-framing.

  2. 02

    Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S. R., Cheng, N., Durmus, E., Hatfield-Dodds, Z., Johnston, S. R., Kravec, S., Maxwell, T., McCandlish, S., Ndousse, K., Rausch, O., Schiefer, N., Yan, D., Zhang, M., & Perez, E. (2024). Towards understanding sycophancy in language models. In The Twelfth International Conference on Learning Representations. https://arxiv.org/abs/2310.13548

    https://arxiv.org/abs/2310.13548