Would you let a machine score your morals?

Attitudes to artificial intelligence (AI) can swing quickly from excitement to dystopian dread.

Researchers at TSE Digital Center aim to provide a more measured response, helping regulators and other decision-makers to consider the social acceptability of digital technologies and ethical concerns about privacy, surveillance, and discrimination. An empirical investigation by IAST psychologist, Zoe Purcell, and leader of the Center’s ‘Artificial Intelligence and Society’ program, Jean-François Bonnefon, suggests that applications such as AI moral scoring will run into persistent psychological barriers. In particular, people tend to feel that their own moral profile is too special to be accurately evaluated by machines.

Why might our moral judgments benefit from the input of machines?

Successful cooperative societies depend on the ability of their members to build good reputations which are based on shared information about moral traits such as loyalty and generosity. In small-scale societies, personal experience and gossip can be effective ways to obtain this information. But these traditional channels do not scale up well in modern societies where we can interact with hundreds of thousands of strangers.

One radical solution is to delegate the acquisition and collation of information about human behavior to intelligent machines which transform their behavioral observations into moral scores. Based on the recommendation of its expert group on AI, the European Commission is considering a ban on large-scale scoring of moral personality. However, there may be a place for less extreme applications. For example, people may be willing to let private companies use AI to issue certified moral scores which could be used to enhance CVs or dating profiles.

Although digital traces and machine learning have been used to successfully predict demographics like age, gender and ethnicity, the assessment of moral traits and personal values is more difficult. As these technologies improve, it will be important to understand the psychological drivers of acceptance since these will determine public support for AI moral scoring policies and consumption of AI moral scoring products.

Peer ratings on online platforms have been embraced by many users. Why might AI ratings provoke a frostier reception?

Platforms such as Uber, Airbnb and eBay encourage humans to rate their experience with another human, then aggregate these ratings for other humans to predict what their experience will be. But even when these mechanisms are taken to the extreme – as in the Black Mirror episode ‘Nosedive’, depicting a dystopian world in which people rate every social encounter – the scoring is done by humans, not AI.

In contrast, our paper is inspired by the growing use of AI to construct moral scores. Building on algorithms that extract personality profiles from social media and other online activity, there are now AI tools that automatically rate the aggressivity or sexism of individual users. AI technology also underpins the Chinese social credit system in which every citizen is given a single score based on both ‘desirable’ and ‘undesirable’ behaviors and low scorers face social and legal penalties.

Unchecked, AI moral scoring thus poses major ethical challenges and could become a sinister tool of Orwellian surveillance. Its use in large-scale social engineering projects such as Chinese system has raised concerns about authoritarian social control, mischaracterization of individuals, and disproportionate sanctions.

What does your research suggest about the public’s appetite for AI moral scoring?

In a series of experiments using representative UK and USA samples, we find that people believe machines will struggle to accurately assess their moral profile. This suggests they may support strong regulations against AI moral scoring and that the commercial potential of this tool may be limited.

Our participants were more likely to accept AI moral scoring if they perceived it as more accurate. They found it less acceptable for AI to score negative moral traits like racism, but this may be a simple framing effect. For example, people may find it more acceptable for AI to score ‘antiracism’ than ‘racism’. Rapid improvements in the accuracy of AI moral scoring seem likely, but this does not guarantee that our powerful psychological biases will be overcome. Notably, we show that people have a strong tendency to overestimate their moral peculiarity. Very few participants believed that their moral profile was among the most common, and 88% underestimated its prevalence. Together with their belief that AI struggles to score unusual moral profiles, this led our participants to doubt AI’s ability to score their morals accurately.

Perceptions of medical AI offer an intriguing parallel. Although people have long been exposed to the idea of the quantified physical self, they still believe AI is unable to grasp the uniqueness of their medical profile, and thus resist the introduction of AI diagnosis tools. Familiarity with the idea of a quantified moral self may similarly fail to reduce resistance to AI moral scoring.

We also found that left-leaning participants overestimated their moral peculiarity to a larger degree which aligns with previous findings that liberals feel a greater need for psychological uniqueness. This may intensify political polarization about the acceptability of AI moral scoring in parallel with political disagreements about AI’s social impact.