Omid Jesmi

Pattern #13 · Explainability

Don't present chain-of-thought as an explanation

A reasoning trace is testimony, not evidence.

Track Aexplainabilitychain-of-thoughttransparency

Do

Explain with artifacts users can check: sources named beside the claim, the data retrieved, the actions taken.

Don't

Don't pipe the model's inner monologue into the UI and call it transparency. It can omit the input that decided the answer.

The rule. When an AI product justifies an answer, show evidence the user can check: sources, retrieved data, actions taken. The reasoning trace is one more generated output, not a record of how the answer was produced.

Why. Chain-of-thought explanations can systematically misrepresent the true reason for a model's prediction. When Turpin and colleagues reordered few-shot answer options so the correct answer was always A, models followed the bias, never mentioned it, and accuracy dropped by as much as 36% on a suite of 13 tasks (Turpin et al., 2023). Dedicated reasoning models behave the same way: when Anthropic planted hints its test models demonstrably used, Claude 3.7 Sonnet mentioned the hint in its chain of thought 25% of the time on average, DeepSeek R1 39% (Chen et al., 2025). The trace reads like a reason; it is not a reliable record of the one.

Seen in the wild. OpenAI shows a model-generated summary of the chain of thought for its o-series models rather than the raw trace, which stays internal as a monitoring surface (OpenAI, 2024).

References

  1. 01

    Turpin, M., Michael, J., Perez, E., & Bowman, S. R. (2023). Language models don't always say what they think: Unfaithful explanations in chain-of-thought prompting. Advances in Neural Information Processing Systems, 36. https://doi.org/10.48550/arXiv.2305.04388

    https://doi.org/10.48550/arXiv.2305.04388
  2. 02

    Chen, Y., Benton, J., Radhakrishnan, A., Uesato, J., Denison, C., Schulman, J., Somani, A., Hase, P., Wagner, M., Roger, F., Mikulik, V., Bowman, S. R., Leike, J., Kaplan, J., & Perez, E. (2025). Reasoning models don't always say what they think. arXiv. https://doi.org/10.48550/arXiv.2505.05410

    https://doi.org/10.48550/arXiv.2505.05410
  3. 03

    OpenAI. (2024, September 12). Learning to reason with LLMs. https://openai.com/index/learning-to-reason-with-llms/

    https://openai.com/index/learning-to-reason-with-llms/