The Psychology Behind AI Recommendations

When a system recommends something, the person has to decide how much weight to give it. That decision is not primarily about the system's accuracy, because in most cases the person has no way to assess accuracy. It is about a set of psychological factors that determine reliance, and those factors behave in ways that are consistent, measurable and often counterproductive.

Initial reliance is high

Logg, Minson and Moore (2019) found that people frequently weighted algorithmic advice more heavily than advice from other people — algorithm appreciation, not aversion. The effect held across estimation tasks and was reduced, though not eliminated, when the participant was choosing between the algorithm's judgement and their own.

This runs against the folk assumption that people distrust machines by default. In unfamiliar quantitative tasks, an algorithm carries an implicit claim to objectivity that human advisers do not, and that claim is largely granted without evidence.

Reliance collapses asymmetrically after error

Dietvorst, Simmons and Massey (2015) demonstrated the opposite pattern under a specific condition: after seeing an algorithm make a mistake, participants abandoned it far more readily than they abandoned a human forecaster who made the same mistake — even when the algorithm was demonstrably more accurate overall.

The asymmetry is the finding. Human error is treated as an instance; machine error is treated as evidence of a defect in the system. One implication is that transparency about model performance is safer before deployment than after: a person told in advance that a model is right eighty percent of the time interprets the first error differently from a person who discovers fallibility by encountering it.

Dietvorst, Simmons and Massey (2018) found a partial remedy. Allowing people to adjust an imperfect algorithm's output, even slightly, substantially increased their willingness to use it. Modest control restored reliance more effectively than accuracy claims did.

Explanation changes acceptance, not always accuracy

Explanations increase perceived understanding and stated trust. They do not reliably increase the person's ability to tell when the system is wrong, which is the capability that actually matters. An explanation that reads plausibly for a correct output usually also reads plausibly for an incorrect one.

Lee and See (2004) framed the objective correctly for automation generally: the goal is appropriate reliance, calibrated to actual reliability, rather than maximum trust. A system that is trusted more than it deserves is a failure mode, not a success.

Recommendations shape the preferences they claim to serve

A recommender trained on behavior and then influencing that behavior creates a feedback loop. What the person is shown constrains what they can engage with, which becomes the training signal for what they are shown next. Preferences inferred from that loop are partly the system's own artefact.

This is not a reason to abandon recommendation, but it is a reason to be careful about the claim that recommenders reveal preferences. They reveal behavior within an option set the recommender determined.

What this means in practice

Set expectations about fallibility before first use rather than after first failure. Give people a way to adjust output, since modest control buys disproportionate acceptance. And evaluate the system on whether reliance is calibrated — people overriding it when it is wrong and following it when it is right — rather than on whether reliance is high. The wider evidence on machine advice is examined in algorithm aversion and appreciation.

References

1. Logg, J. M., Minson, J. A., & Moore, D. A. (2019). Algorithm appreciation: People prefer algorithmic to human judgment. Organizational Behavior and Human Decision Processes, 151, 90–103.
2. Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015). Algorithm aversion: People erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General, 144(1), 114–126.
3. Dietvorst, B. J., Simmons, J. P., & Massey, C. (2018). Overcoming algorithm aversion: People will use imperfect algorithms if they can (even slightly) modify them. Management Science, 64(3), 1155–1170.
4. Lee, J. D., & See, K. A. (2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50–80.
5. Parasuraman, R., & Riley, V. (1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39(2), 230–253.