The rule. Aim user trust at the system's actual competence: the design target is reliance calibrated to what the AI does well and badly, not the highest trust you can induce.
Why. The foundational review of trust in automation names the goal in its title, appropriate reliance: people should neither over-depend on imperfect systems nor dismiss capable ones, and design should help users match their confidence to actual system performance (Lee & See, 2004). Both failure directions were catalogued decades before chat interfaces: misuse, relying on automation more than its reliability warrants, and disuse, neglecting capable automation, often after false alarms (Parasuraman & Riley, 1997). In AI-assisted decision-making the calibration problem survives explanation: people "accept an AI's suggestion even when that suggestion is wrong," while cognitive forcing functions such as having people decide before seeing the AI's recommendation "significantly reduced overreliance compared to the simple explainable AI approaches" (Buçinca et al., 2021).
Seen in the wild. GitHub's Copilot documentation tells developers the product "is still a tool capable of making mistakes" and that they "should always validate the code it suggests": a vendor designing its own users away from maximum trust (GitHub, n.d.).