Pattern #38 · Design experiments

Vague design feedback: what an AI actually changes when told to make it pop

Ask ten times, get the same indigo. To a model, make it pop is a recipe.

Track Cdesign-feedbackdesign-critiquellm-behaviordesign-experimentsstakeholder-communication

Do

Use the model as a decoder: generate the default reading of a vague phrase, show it to your stakeholder, and ask if that is the pop they meant.

Don't

Don't treat the output as their intent. The model never heard your stakeholder; it heard the internet say make it pop, twenty thousand times.

Question. When the only feedback on a fixed UI card is a classic vague phrase, does a model interpret it differently every time, or does each phrase carry one stable meaning?

Why it matters. Feedback research keeps finding that effective critique is "specific, actionable, and justified" (Ngoon et al., 2018). Stakeholders keep saying "make it pop" anyway.

Test. One plain pricing card: six selectors, 27 CSS declarations. Four phrases: make it pop, make it cleaner, make it more modern, make it feel more premium. Ten independent runs per phrase on two Claude models with an identical prompt; 79 of 80 runs delivered. Protocol, prompts, raw outputs, and scoring script are in the experiment record (Progressive Disclosure, 2026).

Observed difference. Each phrase behaved like a preset, not a coin flip. Runs of the same phrase changed largely the same declarations: mean pairwise overlap (Jaccard) of 0.80 to 0.95 in seven of eight cells, 0.56 in the eighth. The style signatures were nearly unanimous. Pop: a saturated indigo or blue accent, colored border, and glow shadow in 19 of 19 runs. Cleaner: all 20 runs stripped the list bullets and none added a shadow. Modern: a system-font stack and larger corner radius in 20 of 20. Premium: wide uppercase letterspacing in 20 of 20, a Georgia serif price in 18, a gold accent in 17; three went black and gold. Across phrases the models turned the same knobs (background, border, radius, shadow, type); the phrase decided the direction.

Limit. Two Claude models in one harness with default sampling. The consistency measures the models' conventions, not your stakeholder's intent, and nothing here says the indigo version is better.

Takeaway. Vague feedback is not ambiguous to a model; it is a recipe, served identically every time. So use it as a decoder: generate the default interpretation, show it to the person who said "pop," and ask if that is what they meant. A model that answers everyone the same way is also a known homogenizer (Padmakumar & He, 2024), which is why it cannot stand in for the one human in your meeting.

Experiment record (essentials)

Preregistered 2026-10-12 before the first model call (protocol.json, sha256 06a5d1bc34c07d75f1525f10a167ea4780998d6391ff15534d2336081e579a2c). Stimulus: one pricing card, stimulus.html sha256 9e174b66e749dafaeac5481547367e2079406db09823ccf1d5c4f08688cc6f9d, stimulus.css sha256 262629def1fc4006b74fd06d5967745d035d16a5a36a3c2e5fa9b27e5b0934b5. Conditions: 4 phrases x 2 Claude models (harness aliases sonnet and haiku, Claude Agent SDK, provider default sampling; exact snapshot IDs not exposed by the harness) x 10 runs, one isolated subagent call per run. 79 of 80 runs returned a parseable CSS block; one haiku run failed to deliver output and is logged as a delivery failure. Scoring: changed (selector, property) sets vs baseline over the six fixed selectors; measures are per-cell mean pairwise Jaccard, per-property agreement rates, and cross-phrase overlap of high-agreement sets. Full numbers in results.json; value signature counts in value-signatures.json. No human participants; no claims about preference or usability.

References

  1. 01

    Progressive Disclosure. (2026, October 12). Pattern 38 experiment record: preregistered protocol, 79 raw model outputs, scoring script, and results. Stored with the pattern in experiments/38-make-it-pop.

  2. 02

    Ngoon, T. J., Fraser, C. A., Weingarten, A. S., Dontcheva, M., & Klemmer, S. (2018). Interactive guidance techniques for improving creative feedback. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM. https://doi.org/10.1145/3173574.3173629

    https://doi.org/10.1145/3173574.3173629
  3. 03

    Padmakumar, V., & He, H. (2024). Does writing with language models reduce content diversity? In The Twelfth International Conference on Learning Representations. https://arxiv.org/abs/2309.05196

    https://arxiv.org/abs/2309.05196