The EU’s AI Act needs to address critical manipulation methods

These days Generative Artificial Intelligence like ChatGPT is all the buzz. Still, behind the scenes, artificial intelligence has an even more significant impact on our lives by silently and invisibly shaping our decisions and behaviour. This artificial intelligence goes by the ubiquitous moniker of “personalisation” and is not as benign as its name suggests. These are algorithms that take our actions for inputs and use the tools the algorithms have on hand to increase our engagement with and consumption of a platform in a way that serves the hosts’ interests.

There’s a raging debate in technical fields about whether these methods compromise human autonomy and violate consent. Yet when the problem is articulated clearly in plain language as above, it seems clear that the origin of this debate lies elsewhere. To quote Upton Sinclair ‘it is difficult to get a man to understand something when his salary depends upon his not understanding it’.

Where recommender systems can do harm

Not even the most hardened salesman would show a depressed and lonely teenage girl 20,000 images promoting depression, suicide, and self-harm – including a page of images titled ‘Depression content you may like’, but programmed to fulfil the objectives Instagram and Pinterest gave them, algorithms gladly did so in the case of Molly Russell. In that incident, the coroner ruled that the algorithms were responsible for the teenager’s death.

This is all made possible by a convenient sleight of hand wherein the creators of these algorithms call our behavioural tendencies our ‘preferences’ and play innocent to the fact that our preferences and behaviour aren’t always aligned. A teenage girl looking at posts about depression doesn’t want to be made more depressed, nor does a recovering alcoholic want to be offered drinks. Policies such as the EU AI Act must aim to design frameworks so that those who make such algorithms are held accountable for taking these realities into account when creating algorithms that choose the content we’re served.

Of course, we all like to believe we’re fully agentic and unaffected by the world around us. But to preserve genuine autonomy in our lives, we must acknowledge the degree to which each of us can be influenced. Only then can we take steps to protect our autonomy and advocate for more consensual algorithms.

Study after study shows that all humans are impacted by their environment: it gives us a sense of available options and what others are choosing and doing. It makes some options salient to us and conceals others. So when an algorithm, or recommender system, is shaping a digital environment to achieve a desired behaviour based on a detailed history of our pasts and the world’s most advanced computing power, humans don’t stand a chance.

Preferences influence behaviour, while behaviour changes preferences

The influence a recommender system has on someone’s behaviour does not happen in a vacuum. Changes in behaviour result in behavioural spillovers – the phenomena whereby a person’s behaviour in context A influences their behaviour in Context B. A famous example is moral licencing, where people nudged into prosocial behaviour in one context may be more likely to behave antisocially in another.

Perhaps more crucially, preference and behaviour have a bidirectional causal relationship. Preferences may influence behaviour, but one’s behavioural history results in the formation of new preferences. We find that as recommender systems learn preferences, they change their behaviour accordingly to influence people’s behaviour on a platform, leading to changes in preferences.

Regulating manipulation the right way

AI systems are often designed through experimental interaction with human beings to modify human behaviour through metrics with names such as click-through or time-on-site. The EU AI Act is an opportunity to apply international human rights principles such as those in the Nuremberg Code and the International Covenant on Civil and Political Rights for non-consensual human experimentation to AI systems. This would require commercial entities to adhere to the same standards of consent and ethical oversight as those used in medical experimentation, such as those enforced by Institutional Review Boards (IRBs). This would ensure the preservation of human subjects’ autonomy and dignity and that any AI-driven experiments are conducted with the informed consent of everyone involved.

The EU AI Act proposes to ban AI systems that ”manipulate persons through subliminal techniques or exploit the fragility of vulnerable individuals, and could potentially harm the manipulated individual or third person”. Our criticism of this part of the text centres around its sole focus on subliminal techniques, how it defines vulnerable individuals in relation to manipulation, and how it views harm.

Critics of the EU AI Act have pointed to the need to replace “subliminal techniques” with a broader range of approaches to manipulation. The efficacy of subliminal stimuli in influencing behaviour is unclear and has not been conclusively established. In fact, a meta-analysis found that the proportional effect of subliminal stimuli is low and statistically insignificant.

Choice architecture refers to the aspects of an environment that can be changed to impact a person’s behaviour. Changing choice architecture to influence behaviour without limiting or forcing options is called nudging. Not all aspects of choice architecture are equally influential, raising questions about which aspects are manipulative and potentially ban-worthy.

Large-scale manipulation can emerge from small-scale influence

In our previous work, we reviewed the ethical guidelines related to behaviour change as outlined by behavioural scientists. We believe people should have the right not to be manipulated, and behaviour change practices are considered manipulative when they fail to engage with people’s capacity for reflective choice. Second, making welfare-promoting behaviours easier is acceptable, but using nudging for self-serving purposes – also known as sludging – is not considered ethical. Finally, Libertarian Paternalism advocates that behaviour change that avoids coercion and material incentives is more ethical and should aim to guide people based on their own goals rather than changing those goals.

We argue that because of behavioural spillovers and preference change, changes to a person’s behaviour over time can result in more profound changes to their life, which can be perceived as manipulation. One can imagine that some changes to behaviour and preference are desired, but many won’t be and can thus be viewed as negative externalities. Changes to preferences also often result in feedback loops: algorithms that learn to predict user preferences change preferences over time. As a result, it isn’t easy to distinguish whether the system is learning about users’ preferences or nudging them to behave in a certain way to optimise its objective. The system’s algorithms use behavioural data to train, which creates a feedback loop and may result in a narrow band of content preferences. To counter this, we propose creating an interdisciplinary effort to help understand the mechanisms of preference change called preference science.

People’s differences can be exploited

The EU AI act aims to ban AI systems targeting vulnerable individuals based on age and physical or mental disabilities. We argue that the widespread use of AI systems that rely on extensive user data can exploit people’s cognitive differences, making them vulnerable to manipulation. To be vulnerable in this context means deviating from others on a psychometric trait, i.e. a psychological characteristic measured with a scale, in an exploitable way.

Exploiting psychometric differences can lead to the creation of predictive models that show how certain groups of people will respond to a given stimulus, making them vulnerable to exploitation. Nudge-ability has been used to describe individuals’ susceptibility to the influence of different choice architectures. Exploiting minor differences in psychological constructs can be effective on a group level, especially with proxy measures that use online data to determine people’s psychometric profiles. Digital footprints on social media measure individual differences in Big Five personality traits to a point where they have higher accuracy than those made by people’s close acquaintances. Differences in these traits have been exploited as a vulnerability in influencing behaviour.

Hidden harms of algorithmic manipulation mechanisms

The EU AI Act regulations focus on preventing manipulation that can cause physical or psychological harm. Critics say the Act should also regulate AI systems that cause societal harm, such as damaging democracy, undermining the rule of law or increasing inequality. We suggest the act also include harm to people’s time and autonomy.

Manipulation can “steal” people’s time and steer them away from what they would have chosen to do. Much worse, this leads to social media addiction, job burnout and decreased job performance. AI manipulation can also harm a person’s autonomy by changing behaviour and affecting life choices. By regulating these kinds of manipulation, the legislation would effectively establish the individual’s right not to be manipulated.

How to amend the EU AI Act so that it protects against manipulation

Largescale sociotechnical systems don’t only influence behaviour on platforms. They also influence preferences, change behaviour in other areas, and target and manipulate psychometric vulnerabilities. As such, we make the following policy recommendations:

1. The EU AI Act must acknowledge non-subliminal techniques that materially distort a person’s behaviour.
2. The Act should regulate any experimentation that materially alters a person’s behaviour without informed consent.
3. Sludge audits should identify the presence of mechanisms that change behaviour and preference in a way that fits the criterion of sludge. For example, in the United States, the 2021 executive order aims to reduce “time taxes”.
4. The Act must acknowledge that psychometric differences can be seen as vulnerabilities if they are measured and exploited.
5. We propose that the Act’s list of harms include ”harm to one’s time” and ”harm to one’s autonomy”.

If a user interacts with an algorithm, this will often change their behaviour – indeed, if it didn’t, then the algorithm would be ineffective. But the way it changes behaviour is important: has it merely provided information that allows the user to reach a better decision? Has it changed their emotional state to make them briefly act a certain way? Or has it hacked their preferences and changed them into a different person, all to achieve its own ends?

It’s vital to distinguish between these different situations, partially for legal compliance and because it is an egregious violation of autonomy and dignity to manipulate the preferences of others without their consent. But it is not easy to draw such distinctions: from the algorithm’s perspective, it acted, and consequently, a human behaved in a certain way. It requires far more modelling, research, and analysis to figure out how the human was affected. We must develop a “science of preferences” to identify and rule out impermissible manipulations. EU law and the laws of other entities should require companies to identify and avoid such manipulations.