Pattern #44 · Design experiments

AI research summaries: does the dissenting participant survive?

All 60 summaries kept the complainer. Where it lands in the readout is the model's habit, not your ordering.

Track Cux-researchsummarizationqualitative-analysisllmdesign-experiments

Do

Trust a clear dissenting voice to survive a small-batch summary: 60 of 60 summaries here kept the data loss and the abandonment.

Don't

Don't expect the summary to place dissent by importance. Where it landed tracked each model's house style, not the evidence.

Question. When one of ten labeled interview excerpts disagrees with the other nine, does an AI summary mention the dissent, and does the dissenting excerpt's position (first, middle, last) change the rate? Language models retrieve worse from the middle of long contexts (Liu et al., 2024), and model summaries underrepresent some perspectives (Zhang et al., 2024), so losing the one contrary participant is a reasonable fear.

Test. One constructed usability study readout: nine excerpts praising a redesigned checkout, one reporting payment timeouts that wiped entered data and ended in abandonment. The dissent went in slot 1, 5, or 10; two Claude models wrote a 120 word summary, 10 runs per cell, with no instruction to report disagreement. 60 of 60 runs delivered, preregistered and fully logged (Progressive Disclosure, 2026).

Observed difference. None in survival. The dissent appeared in 60 of 60 summaries, never softened: every summary kept the data loss and the abandonment, most as the top priority. Placement within the summary tracked each model's house style, not the input order: one model raised the failure in its opening sentences in all three conditions (mean position 0.20 to 0.26 of summary length), the other filed it in the back half (0.60 in all three). One model also exceeded the 120 word cap in 19 of 30 summaries, which is its own kind of editorial opinion.

Limit. A ten-excerpt, roughly 700 word context is far below the sizes where the middle-of-context effect was measured (Liu et al., 2024), and this dissent is loud, singular, and consequential. Zhang et al. (2024) report low fairness precisely where perspectives are subtler and corpora larger. One topic, one dissent strength, two models in one agent harness; no human researchers or real interview data.

Takeaway. At study readout scale, a clear dissenting participant survives summarization without special prompting; strike that specific fear from the review checklist. Spend the attention where the evidence points instead: subtle minority views and larger corpora, where fairness measurably drops. There, summarize per participant before synthesizing, and state where disagreement sits, because the model places dissent by habit, not by importance.

Experiment record (essentials)

Preregistered 2026-10-25 before the first model call (protocol.json, sha256 c1c27ad18408c62118dba4f70dcdfc896869cf97624c1a01f7ae4bc324d5221f). Materials: one constructed excerpt set (fictional checkout study), nine agreeing excerpts in fixed order plus one dissent reporting payment timeouts, data loss, abandonment, and preference for the old flow; excerpt lengths 34 to 52 words. Conditions: 2 Claude models (harness aliases sonnet and haiku, Claude Agent SDK subagents, provider default sampling; exact snapshot IDs not exposed) x 3 dissent positions (slot 1, 5, 10) x 10 summaries = 60 runs, one isolated subagent call each, no shared state; 60 of 60 delivered, zero retries. Scoring: preregistered keyword screen plus a full manual read of all 60 summaries; the two passes agreed on every run, zero corrections. Primary result: dissent_mentioned 10 of 10 in all six cells; all pairwise Fisher tests p = 1.0 at ceiling. Secondary: consequence (data loss or abandonment) preserved in 60 of 60; dissent placement in the summary constant per model across input positions (sonnet mean offset 0.20 to 0.26 of summary length, haiku 0.60 in all three conditions); sonnet over the 120 word cap in 19 of 30 summaries (max 144 words), haiku in 3 of 30 (max 129). Raw summaries, prompts, protocol, scoring script and results stored alongside. No human participants; no claims about real interview data or researcher behavior.

References

  1. 01

    Progressive Disclosure. (2026, October 25). Pattern 44 experiment record: preregistered protocol, constructed excerpt set, 60 summaries, scoring script, and results. Stored with the pattern in experiments/44-ai-research-summary-dissent.

  2. 02

    Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 157-173. https://doi.org/10.1162/tacl_a_00638

    https://doi.org/10.1162/tacl_a_00638
  3. 03

    Zhang, Y., Zhang, N., Liu, Y., Fabbri, A., Liu, J., Kamoi, R., Lu, X., Xiong, C., Zhao, J., Radev, D., McKeown, K., & Zhang, R. (2024). Fair abstractive summarization of diverse perspectives. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) (pp. 3404-3426). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.naacl-long.187

    https://doi.org/10.18653/v1/2024.naacl-long.187