Looking ahead to the next decade, we see plenty of reason for optimism. Rapid advancements in big data analytics, AI and machine learning are set to benefit humankind dramatically in the next ten years, making us safer (i.e. less prone to destructive human error) and happier (due to less time spent on tedious tasks such as driving and shopping). Businesses, too, are poised to profit hugely – AI-based additions to global GDP growth could reach USD22 trillion by 2030.

The consumer platforms powered by this new technology have been massively popular. Netflix’s 158-million strong global subscriber base and Amazon’s USD233 billion in net sales for 2018 are but two of many examples that attest to that this popularity.

Yet algorithmic systems remain controversial. Worldwide media have chronicled their rise with a constant undertone of suspicion, punctuated by eruptions of scandal that may well prefigure a full-blown backlash. The most familiar objection against AI centres on privacy – the widespread perception that tech companies know more about us than we have assented to, and doubts about whether these companies can be trusted to use our data responsibly.

In a recent article for Sloan Management Review, we discuss another important risk to consumer acceptance of AI. The very ease and predictive accuracy of algorithmic platforms could backfire, if they lead users to feel their autonomy, or free will, has been compromised. Tech companies should not confuse the popularity of their offerings with carte blanche to automate and optimise the entire customer journey. As much as people appreciate algorithms making seemingly unnecessary choices for them, they will react negatively when their compliance is considered a fait accompli.

Evidence of autonomy threat

This phenomenon, which we call autonomy threat, shows up in a variety of research studies, many of which are discussed in the 2017 review paper on which our Sloan piece is based. For instance, in a recent study two of us found that when customers believed their future choices could be predicted based on past patterns, they gravitated away from their most preferred option and chose differently. In other words, consumers violated their own preferences in order to re-establish their sense of autonomy.

We also suspect that autonomy threat plays a hidden role in the ongoing AI backlash. For example, the Cambridge Analytica scandal of 2018, in which that firm targeted Facebook users precisely for deceptive political ads, was manifestly about privacy. Yet the stakes involved – the 2016 United States presidential election – raised questions about ads affecting the outcome of the election, therefore threatening the voter autonomy on which democracy depends. Concerns about privacy and autonomy dovetailed to produce an especially ferocious market response – wiping nearly USD40 billion from Facebook’s market value in a single day.

For tech companies, buffering against autonomy threat requires understanding three different points:

• Uniqueness and autonomy
• Predictability and autonomy
• Overlapping privacy and autonomy

Encroaching too far in any one of these areas could trigger consumer backlash. There are also measures firms can take to bolster users’ sense of autonomy and therefore reduce the likelihood of an adverse reaction.

Uniqueness and autonomy

First, people want to be seen as unique individuals and not interchangeable with others. This becomes an issue when, for example, customer service chat bots give advice in lieu of human agents. Even if the bot dispenses first-rate advice, users may resent being treated as though their problems were not special enough to command a person’s individual attention.

Academic research finds that this holds true even in high-stakes situations, such as a medical diagnosis. Patients will choose to see a human doctor even when they are told a robot would provide a more statistically accurate diagnosis, out of a deep-seated belief that the robot will not be able to account for their unique situation.

In addition to performing simple tweaks such as bots addressing customers by name, companies can respond to such autonomy threats by ensuring that consumers always have a voice in curating their experience. Though this will at times run counter to convenience, it’s important to remember the “IKEA effect”, in which the sweat equity of personally assembling their own flat-packed furniture helps buyers feel more bonded to their purchases. Studies have found that consumers will create more mental work for themselves when an important decision (e.g. between two products) seems too easy. A “perfect” choice-making process that eliminates all mental labour may thus provoke consumer rebellion.

Predictability and autonomy

Second, people want to be able to change their minds. If data-driven platforms leave them feeling locked into their past choices, autonomy threat may be triggered and shoppers may make sub-optimal choices, as in the above-mentioned “predictability” study.

Interestingly, however, in that same study consumers felt less threatened when the word “consistent” rather than “predictable” was used to describe their choices. Through shifting their messaging, companies can promote the continuance of desired consumer behaviour – for example, by framing marketing messages as invitations to deepen expertise or complete a task.

Moreover, enabling consumers to change what they choose creates opportunities that few AI-driven companies have exploited. Helping people improve their lives – for instance, by eating better, quitting smoking, etc. – is quite feasible for these companies, as algorithms can be developed to identify tell-tale signs of customers who may be ready to choose differently. Such consumers may, as one of us has shown, buy smaller packages of unhealthy snacks rather than the more cost-effective jumbo sizes to make sure that they cannot give in to the temptation to overeat, an example of a self-control strategy called “precommitment”.

Companies can then offer pre-committing consumers options to help them realise large-scale lifestyle changes – a possible win-win for business and public health. That is, as long as companies respect consumers’ autonomy by offering them precommitment options, which they can accept, reject or delay as they see fit, rather than choosing for the consumer.

Where privacy and autonomy overlap

Third, we must remember that privacy and autonomy are partly overlapping concepts. A critical part of free will is deciding which of our behaviours and preferences we would like to share with others, and which we want to keep private. When privacy is taken away, part of our autonomy goes with it.

The precariousness of internet privacy is becoming difficult to ignore. Big Tech’s laxity on the issue has implications not only for adult consumers but the next generation as well: The US Federal Trade Commission recently fined YouTube USD170 million for collecting data on children under 13 years old.

Clearly, many companies believe they have no choice but to play fast and loose with privacy so that they can harvest the consumer data they need to train their algorithms. But research suggests that more rigid and transparent privacy policies may produce better results for users and firms alike, perhaps because reinforcing autonomy helps soothe sensitivities regarding privacy. A 2015 paper in Science argued, “Control [over personal information] can reduce privacy concern, which in turn can have unintended effects.” The paper goes on to cite a finding that people were surprisingly willing to have their private information posted online when they felt the decision to do so was in their hands. Paradoxically, when explicitly given the option not to divulge, participants opted to reveal more instead of erring on the side of caution.

Autonomy and privacy interact in other ways too. Research by one of us suggests that companies create less autonomy threat when they track consumers’ existing preferences rather than those that are still being formed. We surmise that this distinction would apply to e-commerce – purchase histories would be fair game for algorithmic tracking, while preparatory activity for a purchase (search history, price comparisons, etc.) should be off-limits.

The spectre of regulation

Our Sloan article was written out of serious concern that the profoundly beneficial impact of AI solutions may be blunted by backlash, if Big Tech isn’t careful.

As the new decade begins, media reports warn of a “rising tide of AI regulation”, possibly including an outright ban on data collection by bots or even transparency requirements for algorithms. If changes aren’t made to appease consumers’ pervasive sense of autonomy and privacy violations, the likely result will be mounting public outrage, probably followed by regulatory crackdown.

We believe regulatory action represents a far greater threat to innovation than our rather modest autonomy-related recommendations. Let’s hope the tech industry takes heed so we don’t have to find out.

Ziv Carmon is the Dean of Research and The Alfred H. Heineken Chaired Professor of Marketing at INSEAD.

Klaus Wertenbroch is the Novartis Chaired Professor of Management and the Environment and a Professor of Marketing at INSEAD. He directs the Strategic Marketing Programme, one of INSEAD’s Executive Education programmes.

Rom Schrift is an Associate Professor of Marketing at the Indiana University Kelley School of Business.

Haiyang Yang is an Assistant Professor of Marketing at the Johns Hopkins University Carey Business School. He was awarded his PhD at INSEAD.

Don’t miss our latest content. Download the free INSEAD Knowledge app today.

Follow INSEAD Knowledge on Twitter and Facebook.